**Contents** [[!toc levels=3]] # Meta Note that there is also a [Xen HOWTO](../../ports/xen/howto/). Arguably this content could be folded in there. # Requirements Xen3 is supported from NetBSD-4.0 onward. If you plan on using NetBSD-CURRENT, please read the article [[How to build NetBSD-current]] to do so. Guest operating systems can run from their own partitions, or from image files in the main (DOM0) install. This tutorial describes how to: * Install and configure NetBSD as a DOM0 * Install and run a NetBSD as a DOMU * Install and run a Windows XP system as as DOMU * Install and run a Debian system as a DOMU # Installing Xen tools and kernels ## Xen tools To run and administer xen domains, we need the xentools3 or xentools33 packages which are available in pkgsrc. Xen 3.1 packages are under [sysutils/xentools3](http://pkgsrc.se/sysutils/xentools3) for traditional xentools, and [sysutils/xentools3-hvm](http://pkgsrc.se/sysutils/xentools3-hvm) for the additional **HVM** support to run un- modified OSes such as Windows XP. Xen 3.3 packages are under [sysutils/xentools33](http://pkgsrc.se/sysutils/xentools33). Unlike Xen 3.1, no extra package is required for **HVM** support. Note, it is not possible to install Xen 3.1 and Xen 3.3 packages at the same time. They conflict with each other. **HVM** stands for **Hardware Virtualization Managed**. The benefit of hardware virtualization is that you can run OSes that don't know they are being virutalized like Windows XP, for example. However, you must have a CPU which supports this. Intel CPUs must have the 'VT' instruction. AMD CPUs will have the 'SVM' instruction. You can find out if your CPU supports HVM by taking a look at this page: In NetBSD 5.0 there's a new cpuctl command. This is an example output of an AMD CPU: # cpuctl identify 0 cpu0: AMD Unknown K8 (Athlon) (686-class), 2210.22 MHz, id 0x60f82 cpu0: features 0x178bfbff cpu0: features 0x178bfbff cpu0: features 0x178bfbff cpu0: features2 0x2001 cpu0: features3 0xebd3fbff cpu0: features4 0x11f cpu0: "AMD Turion(tm) 64 X2 Mobile Technology TL-64" cpu0: I-cache 64KB 64B/line 2-way, D-cache 64KB 64B/line 2-waycpu0: L2 cache 1MB 64B/line 16-way cpu0: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative cpu0: DTLB 32 4KB entries fully associative, 8 4MB entries fully associativecpu0: Initial APIC ID 0 cpu0: AMD Power Management features: 0x7f cpu0: family 0f model 08 extfamily 00 extmodel 06 Note the **SVM** feature flag in the **features4** line indicating that **HVM** support is present on this CPU. However, this feature may be disabled in the BIOS. In this case since NetBSD 5.0 there will show up this dmesg line on AMD CPUs only: cpu0: SVM disabled by the BIOS Xen 3.1 (the xentools3-hvm package will automatically bring in the base xentools3): # cd /usr/pkgsrc/sysutils/xentools3-hvm # make install Xen 3.3: # cd /usr/pkgsrc/sysutils/xentools33 # make install ## Xen kernel Next, we will get the xen hypervisor kernel itself. For NetBSD 4.x and 5.x the i386 port does not support 'PAE' kernels and must run the Xen 3.1 package. This restriction has been removed in -current and is not relevant for the amd64 port. For Xen 3.1, in pkgsrc this is [sysutils/xenkernel3](http://pkgsrc.se/sysutils/xenkernel3), for Xen 3.3, this is [sysutils/xenkernel33](http://pkgsrc.se/sysutils/xenkernel33): # cd /usr/pkgsrc/sysutils/xenkernel3 # make install And copy it into / directory, like this: # cp /usr/pkg/xen3-kernel/xen.gz / ## Xen DOM0 kernel Lastly, we need a XEN-enabled kernel for our DOM0 domain. There are two possibilities: downloading the binary version, or building it from source. ## Downloading the binary version From NetBSD-4.0 onward, NetBSD supports Xen, and provides some XEN-enabled kernel directly from [[1]][36], in each `binary/kernel` directory associated with a particular release. For example, with NetBSD-4.0, You can grab one from (/netbsd-XEN3_DOM0.gz): # ftp -a ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-4.0/i386/binary/kernel/netbsd-XEN3_DOM0.gz The `netbsd-XEN3_DOM0.gz` file contains a gzipped version of the kernel. Just copy and move it into the root directory like this: # cp netbsd-XEN3_DOM0.gz / ## Building it from source Building a kernel from source is out of the scope of this section. Please consult [How to build a kernel](http://www.netbsd.org/docs/kernel/#how_to_build_a_kernel) from NetBSD's FAQ for more details. Once building is done, you can find them in `/usr/src/obj/_releasedir_/i386/binary/kernel/`. Identically to binary version, copy or move netbsd-XEN3_DOM0 in the root directory `/`. # Selecting a bootloader In NetBSD 5.0 the native boot loader, /boot, can load Xen directly. The NetBSD 5.0 bootloader can be easily dropped into a NetBSD 4.x system by coping them into /boot and running installboot(8) properly. ## Updating /boot For full details refer to installboot(8), but for a standard configuration with VGA console and an IDE or SATA drive with an FFSv1 root file system use the following: # cp /usr/mdec/boot /boot # installboot -v -o timeout=5 /dev/rwd0a /usr/mdec/bootxx_ffsv1 ## Updating /boot.cfg NetBSD 5.0 or later will already have a /boot.cfg file with a basic configuration. Enabling Xen support only requires one additional line in this case. If you're upgrading from an earlier version or do not have an existing /boot.cfg use the following example: banner=Welcome to NetBSD banner================== banner= banner=Please choose an option from the following menu: menu=Boot normally:boot netbsd menu=Boot single-user:boot netbsd -s menu=Boot backup kernel:boot onetbsd menu=Drop to boot prompt:prompt menu=Boot Xen with 256MB for dom0:load /netbsd-XEN3_DOM0 console=pc;multiboot /usr/pkg/xen3-kernel/xen.gz dom0_mem=256M menu=Boot Xen with 256MB for dom0 (serial):load /netbsd-XEN3_DOM0 console=com0;multiboot /usr/pkg/xen3-kernel/xen.gz dom0_mem=256M console=com1 com1=115200,8n1 menu=Boot Xen with dom0 in single-user mode:load /netbsd-XEN3_DOM0 -s;multiboot /usr/pkg/xen3-kernel/xen.gz dom0_mem=256M timeout=5 default=1 Make sure you update the "load /netbsd-XEN3_DOM0" and the "dom0_mem=256M" arguments to match your setup. On next boot select the 'Boot Xen with 256MB for dom0' option and make sure you see Xen kernel messages and the normal NetBSD kernel messages. Once you're satisfied it is working you can change the "default=1" line to "default=5" to automatically boot Xen on reboot. # Setting up DOM0 ## Creating xen devices To create all xen devices, change to /dev and run ./MAKEDEV xen cd /dev ./MAKEDEV xen This should create the devices **xencons**, **xenevt** and **xsd_kva**. If any of these are missing you may not have updated to _CURRENT_ using the latest sources and you will have to create the missing device files. ## Configuring the bridge interface The [[!template id=man name="bridge" section="4"]] interface is used to provide network access to DOMUs. To use one, edit (or create) the file `/etc/ifconfig.bridge0` and insert following lines to: create !brconfig $int add bge0 stp bge0 up Where 'bge0' should be changed to the name of the interface you want to use with your guest operating systems. use [[!template id=man name="ifconfig" section="8"]] to get more details about your actual interfaces. ## Rebooting into DOM0 Time to reboot: # shutdown -r now If all has gone well, you should have booted into the XEN3_DOM0 kernel. Check this with [[!template id=man name="uname" section="1"]]: # uname -v NetBSD 4.0 (XEN3_DOM0) #0: Sun Dec 16 01:20:31 PST 2007 builds@wb34:/home/builds/ab/netbsd-4-0-RELEASE/i386/200712160005Z-obj/home/builds/ab/netbsd-4-0-RELEASE/src/sys/arch/i386/compile/XEN3_DOM0 You should have **XEN3 DOM0** mentioned somewhere. ## Configuring rc scripts Copy or symlink xend, xenbackendd and xendomains from `/usr/pkg/share/examples/rc.d` to `/etc/rc.d`. # cp /usr/pkg/share/examples/rc.d/xend /etc/rc.d/ # cp /usr/pkg/share/examples/rc.d/xenbackendd /etc/rc.d/ # cp /usr/pkg/share/examples/rc.d/xendomains /etc/rc.d/ Edit `/etc/rc.conf` and add the following lines: xend=YES xenbackendd=YES xendomains="dom1" Later on, when you have created a configuration file for 'dom1', the xendomains variable specified above will trigger 'dom1' to be started when the system is booted. At this point no configuration exists for dom1, therefore it does nothing at this point. If you choose to name your configuration file something else, adapt the name accordingly. To avoid rebooting a second, start all three services: # /etc/rc.d/xend start # /etc/rc.d/xenbackendd start # /etc/rc.d/xendomains start Run `ifconfig -a` to ensure the bridge interface is present and issue a `ps ax | grep xen` to ensure you have a similar output. 12 ? DK 0:00.00 [xenwatch] 13 ? DK 0:00.00 [xenbus] 411 ? I 0:00.24 xenstored --pid-file=/var/run/xenstore.pid 594 ? IWa 0:00.26 xenconsoled 629 ? IW 0:00.00 /usr/pkg/bin/python2.3 /usr/pkg/sbin/xend start 631 ? IWsa 0:00.02 /usr/pkg/sbin/xenbackendd 639 ? IWa 0:00.52 /usr/pkg/bin/python2.3 /usr/pkg/sbin/xend start The DOM0 configuration is now done. We will proceed to configuring DOMU domains. # Configuring DOMU ## Configuring and installing a NetBSD DOMU Create (or modify) `/usr/pkg/etc/xen/dom1` and include this: kernel = "/usr/src/obj/releasedir/i386/binary/kernel/netbsd-INSTALL_XEN3_DOMU.gz" #kernel = "/netbsd-XEN3_DOMU" memory = 64 name = "dom1" #vcpus = 1 disk = [ 'phy:/dev/wd0g,0x03,w','file:/usr/src/obj/releasedir/i386/installation/cdrom/netbsd-i386.iso,0x04,r' ] vif = [ 'bridge=bridge0' ] root = "/dev/wd0d" This configuration boots into the NetBSD sysinst program and allows you to install a NetBSD DOMU using the normal sysinst method. This configuration uses a DOMU_INSTALL kernel and an ISO image provided by a successful 'build release' and 'build iso-image'. You may be able to locate a valid Xen3 DOMU_INSTALL kernel from but if not, building a release is your best bet. In this configuration file, `/dev/wd0g` is the reserved partition for the guest operating system. This should be changed to the partition you reserved prior to following the instructions within this document. If you would like to use a physical CDROM instead of an ISO image, change the disk line to: disk = [ 'phy:/dev/wd0g,0x03,w','phy:/dev/cd0a,0x04,r' ] Now boot into sysinst using the command: xm create dom1 -c The reserved partition will appear as `/dev/xbd0`. Proceed as you would with a normal NetBSD installation using xbd0 as the target drive and xbd1 as the CDROM. When you have finished, run `shutdown -hp now` to dom1. Now edit `/usr/pkg/etc/xen/dom1`. Comment the INSTALL kernel and uncomment the DOMU kernel. You should now have a working NetBSD DOMU (dom1). Boot into dom1 again with the command: xm create dom1 -c and ensure the file, `/etc/ttys` contains only this line or has all other lines commented: console "/usr/libexec/getty Pc" vt100 on secure and the file `/etc/wscons.conf` is completely empty or has all lines commented out. These last two steps ensure no errors should be present on boot. Setting wscons=NO in `/etc/rc.conf` may effectively do the same thing. From here, configure `/etc/rc.conf` and all runtime configuration files as you would normally. The network interface name should be _**xennet0**_. Use this name when configuring an IP address. More information can be obtained by referencing the [Xen user guide](http://www.cl.cam.ac.uk/Research/SRG/netos/xen/readmes/user/) and [the official NetBSD Xen Howto](http://www.netbsd.org/Ports/xen/howto.html). Questions can be addressed to the port- xen@NetBSD.org mailling list. ## Configuring and installing a Windows XP DOMU This requires an HVM capable processor and xentools (see sections above). This assumes you have a copy of the Windows install CD in /home/xen/winxp.iso, and wish to create a file /home/xen/winxp.img to hold the install. First create a blank file to hold the install. This assumes a size of 4GB (4096M). If you want a different size adjust the numbers to match: # dd if=/dev/zero of=/home/xen/winxp.img bs=1m count=4096 Create `/usr/pkg/etc/xen/win01`: kernel = '/usr/pkg/lib/xen/boot/hvmloader' builder = 'hvm' memory = '400' device_model='/usr/pkg/libexec/qemu-dm' disk = [ 'file:/home/xen/winxp.img,ioemu:hda,w', 'file:/home/xen/winxp.iso,ioemu:hdb:cdrom,r', ] # Hostname name = "win01" vif = [ 'type=ioemu, bridge=bridge0' ] boot= 'd' vnc = 1 usbdevice = 'tablet' # Helps with mouse pointer positioning You may want to modify the amount of memory and pathnames. Ensure you have a vncviewer installed, such as [net/tightvncviewer](http://pkgsrc.se/net/tightvncviewer) or [net/vncviewer](http://pkgsrc.se/net/vncviewer) from pkgsrc. Then start the XENU and connect to it via VNC. # xm create /usr/pkg/etc/xen/win01 # vncviewer :0 This will boot the Windows ISO image and let you install Windows as normal. As Windows reboots during install you may need to restart vncviewer. After install change the **boot d** to **boot c** to have the system boot directly from the disk image. ## Configuring and installing a GNU/Linux DOMU We will do this in two steps: * install a GNU/Linux system, from a livecd or any installation media * configure the DOM0 so that it can create and start the Linux DOMU. ### Installing a Linux distribution (soon-to-be DOMU) Before proceeding with DOMU configuration, we will install our favorite GNU/Linux distribution on the computer. In order to do it, we need at least two partitions (only one, if you do not consider the swap). These partitions must reside outside of the NetBSD slice, and may be either of primary or extended type. Of course, you can use more than two, but adapt your labels and partitions accordingly. We do not cover the partition/slices manipulations through [[!template id=man name="fdisk" section="8"]] and [[!template id=man name="disklabel" section="8"]], as it depends strongly on how you manage your hard drive's space. For this tutorial, we will use this partitioning: # fdisk /dev/wd0d fdisk: removing corrupt bootsel information fdisk: Cannot determine the number of heads Disk: /dev/wd0d NetBSD disklabel disk geometry: cylinders: 486344, heads: 16, sectors/track: 63 (1008 sectors/cylinder) total sectors: 490234752 BIOS disk geometry: cylinders: 1023, heads: 255, sectors/track: 63 (16065 sectors/cylinder) total sectors: 490234752 Partition table: 0: Linux native (sysid 131) start 63, size 20482812 (10001 MB, Cyls 0-1274) PBR is not bootable: All bytes are identical (0x00) 1: Linux swap or Prime or Solaris (sysid 130) start 20482875, size 1959930 (957 MB, Cyls 1275-1396) PBR is not bootable: All bytes are identical (0x00) 2: NetBSD (sysid 169) start 61464690, size 428770062 (209360 MB, Cyls 3826-30515/178/63), Active 3: Drive serial number: -286527765 (0xeeebeeeb) Here, you notice that we decide to use two primary partitions for our future Linux DOMU: * partition 0 (for the **root directory /**) * partition 1 (for the **swap**) Labels: 16 partitions: # size offset fstype [fsize bsize cpg/sgs] a: 30720816 61464690 4.2BSD 2048 16384 0 # (Cyl. 60976*- 91453*) b: 1049328 92185506 swap # (Cyl. 91453*- 92494*) c: 428770062 61464690 unused 0 0 # (Cyl. 60976*- 486343) d: 490234752 0 unused 0 0 # (Cyl. 0 - 486343) e: 20480000 93234834 4.2BSD 0 0 0 # (Cyl. 92494*- 112812*) f: 20480000 113714834 4.2BSD 0 0 0 # (Cyl. 112812*- 133129*) g: 20480000 134194834 4.2BSD 0 0 0 # (Cyl. 133129*- 153447*) h: 335559918 154674834 4.2BSD 0 0 0 # (Cyl. 153447*- 486343) i: 20482812 63 Linux Ext2 0 0 # (Cyl. 0*- 20320*) j: 1959930 20482875 swap # (Cyl. 20320*- 22264*) Bear in mind that we added two labels here, namely **i** and **j**, which maps respectively to partition 0 and partition 1 of the disk. We will use these labels later for DOMU configuration. Now that we have partitioned the disk, proceed with installing your Linux distribution. We will not cover that part in this tutorial. You can either install it from an installation media (a cdrom from example), or copy files from an already installed distribution on your computer. Tip: to manipulate ext2/3 filesystems (the traditional fs under Linux) from NetBSD, you can use [sysutils/e2fsprogs](http://pkgsrc.se/sysutils/e2fsprogs) from pkgsrc: # cd /usr/pkgsrc/sysutils/e2fsprogs # make install And then use e2fsck, mke2fs and [[!template id=man name="mount_ext2fs" section="8"]] directly from NetBSD. ### Getting XEN aware Linux kernels Once installation is done, reboot your computer and return to our Xen-NetBSD system. To boot our Linux DOMU, we will need a Linux kernel supporting the XENU virtualisation. Depending on your Linux distribution, you can grab one from its repository (it is up to you to find it through aptitude, yum or whatever package manager you use), or get one from the Xen binary distribution. To get a XENU Linux kernel from Xen binary distribution, get it directly from [Xen website download page](http://www.xen.org/download/dl_31tarballs.html). Download the tarball and extract the **vmlinuz-*-xen** from it. In our case, with a 2.6.18 Linux kernel: # ftp -a http://bits.xensource.com/oss-xen/release/3.1.0/bin.tgz/xen-3.1.0-install-x86_32.tgz # cd /tmp # tar -xzf xen-3.1.0-install-x86_32.tgz dist/install/boot/vmlinuz-2.6.18-xen **vmlinuz-2.6.18-xen** is the kernel that Xen will use to start the DOMU. Move it to any directory you like (just remember it when configuring the _kernel_ entry in the DOMU configuration file): # mv dist/install/boot/vmlinuz-2.6.18-xen /vmlinuz-XEN3-DOMU ### Configuring DOMU Configuring the Linux DOMU is a bit different than a NetBSD one; some options tend to differ. Edit (or create) the configuration file **domu-linux**, in `/usr/pkg/etc/xen/`: # vi /usr/pkg/etc/xen/domu-linux Here's a typical config file for a Linux DOMU: #---------------------------------------------------------------------------- # Kernel image file. This kernel will be loaded in the new domain. kernel = "/vmlinuz-XEN3-DOMU" # Memory allocation (in megabytes) for the new domain. memory = 256 # A handy name for your new domain. This will appear in 'xm list', # and you can use this as parameters for xm in place of the domain # number. All domains must have different names. # name = "domu-linux" # Which CPU to start domain on (only relevant for SMP hardware). CPUs # numbered starting from ``0''. # cpu = "^1" # leave to Xen to pick #---------------------------------------------------------------------------- # Define network interfaces for the new domain. # Number of network interfaces (must be at least 1). Default is 1. vif = [ '' ] # Define MAC and/or bridge for the network interfaces. # # The MAC address specified in ``mac'' is the one used for the interface # in the new domain. The interface in domain0 will use this address XOR'd # with 00:00:00:01:00:00 (i.e. aa:00:00:51:02:f0 in our example). Random # MACs are assigned if not given. # # ``bridge'' is a required parameter, which will be passed to the # vif-script called by xend(8) when a new domain is created to configure # the new xvif interface in domain0. # # In this example, the xvif is added to bridge0, which should have been # set up prior to the new domain being created -- either in the # ``network'' script or using a /etc/ifconfig.bridge0 file. # vif = [ 'mac=aa:00:00:50:02:f0, bridge=bridge0' ] #---------------------------------------------------------------------------- # Define the disk devices you want the domain to have access to, and # what you want them accessible as. # # Each disk entry is of the form: # # phy:DEV,VDEV,MODE # # where DEV is the device, VDEV is the device name the domain will see, # and MODE is r for read-only, w for read-write. You can also create # file-backed domains using disk entries of the form: # # file:PATH,VDEV,MODE # # where PATH is the path to the file used as the virtual disk, and VDEV # and MODE have the same meaning as for ``phy'' devices. # # /dev/wd0i will be seen as "hda1" under DOMU (the root partition) # /dev/wd0j will be seen as "hda2" under DOMU (the swap) # disk = [ 'phy:/dev/wd0i,hda1,w','phy:/dev/wd0j,hda2,w' ] #---------------------------------------------------------------------------- # Set the kernel command line for the new domain. # Set root device. root = "/dev/hda1" Now, you should be able to start your first Linux DOMU! # xm create -c /usr/pkg/etc/xen/domu-linux # Possible caveats If you intend to have more than one box configured with the above configuration on the same network, you will most likely have to specify a unique MAC address per guest OS, otherwise it is likely you will have a conflict. I'm not sure if the MAC assignment is random, incremental or if Xen is able to check for the existence of the proposed MAC address, so specifying the MAC address is recommended. Here is a method to assign a MAC address to a newly created Xen Domu. First as described before, use the following vif parameter in your config file : vif = [ 'bridge=bridge0' ] Then, run the Xen DomU and, once logged run the following command : # ifconfig xennet0 Output Sample : xennet0: flags=8863 mtu 1500 capabilities=2800 enabled=0 address: 00:16:3e:2e:32:5f inet 192.168.4.81 netmask 0xffffff00 broadcast 192.168.4.255 inet6 fe80::216:3eff:fe2e:325f%xennet0 prefixlen 64 scopeid 0x2 A MAC address is automaticaly generated, use it in your config file with the following syntax : vif = [ 'mac=00:16:3e:2e:32:5f, bridge=bridge0' ] And now you can restart the Xen DomU. Please, note that the MAC Address MUST start with "00:16:3e". # See also * [The official NetBSD Xen Howto](http://www.netbsd.org/Ports/xen/howto.html) * [The official Xen Wiki](http://wiki.xensource.com/xenwiki/)