Contents

Meta

Note that there is also a Xen HOWTO. Arguably this content could be folded in there.

Requirements

Xen3 is supported from NetBSD-4.0 onward. If you plan on using NetBSD-CURRENT, please read the article ?How to build NetBSD-current to do so. Guest operating systems can run from their own partitions, or from image files in the main (DOM0) install.

This tutorial describes how to:

• Install and configure NetBSD as a DOM0
• Install and run a NetBSD as a DOMU
• Install and run a Windows XP system as as DOMU
• Install and run a Debian system as a DOMU

Installing Xen tools and kernels

Xen tools

To run and administer xen domains, we need the xentools3 or xentools33 packages which are available in pkgsrc.

Xen 3.1 packages are under sysutils/xentools3 for traditional xentools, and sysutils/xentools3-hvm for the additional HVM support to run un- modified OSes such as Windows XP.

Xen 3.3 packages are under sysutils/xentools33. Unlike Xen 3.1, no extra package is required for HVM support. Note, it is not possible to install Xen 3.1 and Xen 3.3 packages at the same time. They conflict with each other.

HVM stands for Hardware Virtualization Managed. The benefit of hardware virtualization is that you can run OSes that don't know they are being virutalized like Windows XP, for example. However, you must have a CPU which supports this. Intel CPUs must have the 'VT' instruction. AMD CPUs will have the 'SVM' instruction. You can find out if your CPU supports HVM by taking a look at this page:

http://wiki.xensource.com/xenwiki/HVM_Compatible_Processors

In NetBSD 5.0 there's a new cpuctl command. This is an example output of an AMD CPU:

 # cpuctl identify 0
 cpu0: AMD Unknown K8 (Athlon) (686-class), 2210.22 MHz, id 0x60f82
 cpu0: features 0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
 cpu0: features 0x178bfbff<PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX>
 cpu0: features 0x178bfbff<FXSR,SSE,SSE2,HTT>
 cpu0: features2 0x2001<SSE3,CX16>
 cpu0: features3 0xebd3fbff<SCALL/RET,NOX,MXX,FFXSR,RDTSCP,LONG,3DNOW2,3DNOW>
 cpu0: features4 0x11f<LAHF,CMPLEGACY,SVM,EAPIC,ALTMOVCR0,3DNOWPREFETCH>
 cpu0: "AMD Turion(tm) 64 X2 Mobile Technology TL-64"
 cpu0: I-cache 64KB 64B/line 2-way, D-cache 64KB 64B/line 2-waycpu0: L2 cache 1MB 64B/line 16-way
 cpu0: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
 cpu0: DTLB 32 4KB entries fully associative, 8 4MB entries fully associativecpu0: Initial APIC ID 0
 cpu0: AMD Power Management features: 0x7f<TS,FID,VID,TTP,HTC,STC,100>
 cpu0: family 0f model 08 extfamily 00 extmodel 06

Note the SVM feature flag in the features4 line indicating that HVM support is present on this CPU. However, this feature may be disabled in the BIOS. In this case since NetBSD 5.0 there will show up this dmesg line on AMD CPUs only:

 cpu0: SVM disabled by the BIOS

Xen 3.1 (the xentools3-hvm package will automatically bring in the base xentools3):

# cd /usr/pkgsrc/sysutils/xentools3-hvm
# make install

Xen 3.3:

# cd /usr/pkgsrc/sysutils/xentools33
# make install

Xen kernel

Next, we will get the xen hypervisor kernel itself. For NetBSD 4.x and 5.x the i386 port does not support 'PAE' kernels and must run the Xen 3.1 package. This restriction has been removed in -current and is not relevant for the amd64 port.

For Xen 3.1, in pkgsrc this is sysutils/xenkernel3, for Xen 3.3, this is sysutils/xenkernel33:

# cd /usr/pkgsrc/sysutils/xenkernel3
# make install

And copy it into / directory, like this:

# cp /usr/pkg/xen3-kernel/xen.gz /

Xen DOM0 kernel

Lastly, we need a XEN-enabled kernel for our DOM0 domain. There are two possibilities: downloading the binary version, or building it from source.

Downloading the binary version

From NetBSD-4.0 onward, NetBSD supports Xen, and provides some XEN-enabled kernel directly from ?1[36], in each binary/kernel directory associated with a particular release.

For example, with NetBSD-4.0, You can grab one from (<ftp.NetBSD.org/pub/NetBSD/NetBSD-4.0/i386/binary/kernel>/netbsd-XEN3_DOM0.gz):

# ftp -a ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-4.0/i386/binary/kernel/netbsd-XEN3_DOM0.gz

The netbsd-XEN3_DOM0.gz file contains a gzipped version of the kernel. Just copy and move it into the root directory like this:

# cp netbsd-XEN3_DOM0.gz /

Building it from source

Building a kernel from source is out of the scope of this section. Please consult How to build a kernel from NetBSD's FAQ for more details.

Once building is done, you can find them in /usr/src/obj/_releasedir_/i386/binary/kernel/. Identically to binary version, copy or move netbsd-XEN3_DOM0 in the root directory /.

Selecting a bootloader

In NetBSD 5.0 the native boot loader, /boot, can load Xen directly. The NetBSD 5.0 bootloader can be easily dropped into a NetBSD 4.x system by coping them into /boot and running installboot(8) properly.

Updating /boot

For full details refer to installboot(8), but for a standard configuration with VGA console and an IDE or SATA drive with an FFSv1 root file system use the following:

# cp /usr/mdec/boot /boot
# installboot -v -o timeout=5 /dev/rwd0a /usr/mdec/bootxx_ffsv1

Updating /boot.cfg

NetBSD 5.0 or later will already have a /boot.cfg file with a basic configuration. Enabling Xen support only requires one additional line in this case. If you're upgrading from an earlier version or do not have an existing /boot.cfg use the following example:

banner=Welcome to NetBSD
banner==================
banner=
banner=Please choose an option from the following menu:
menu=Boot normally:boot netbsd
menu=Boot single-user:boot netbsd -s
menu=Boot backup kernel:boot onetbsd
menu=Drop to boot prompt:prompt
menu=Boot Xen with 256MB for dom0:load /netbsd-XEN3_DOM0 console=pc;multiboot /usr/pkg/xen3-kernel/xen.gz dom0_mem=256M
menu=Boot Xen with 256MB for dom0 (serial):load /netbsd-XEN3_DOM0 console=com0;multiboot /usr/pkg/xen3-kernel/xen.gz dom0_mem=256M console=com1 com1=115200,8n1
menu=Boot Xen with dom0 in single-user mode:load /netbsd-XEN3_DOM0 -s;multiboot /usr/pkg/xen3-kernel/xen.gz dom0_mem=256M
timeout=5
default=1

Make sure you update the "load /netbsd-XEN3_DOM0" and the "dom0_mem=256M" arguments to match your setup. On next boot select the 'Boot Xen with 256MB for dom0' option and make sure you see Xen kernel messages and the normal NetBSD kernel messages. Once you're satisfied it is working you can change the "default=1" line to "default=5" to automatically boot Xen on reboot.

Setting up DOM0

Creating xen devices

To create all xen devices, change to /dev and run ./MAKEDEV xen

cd /dev
./MAKEDEV xen

This should create the devices xencons, xenevt and xsd_kva. If any of these are missing you may not have updated to CURRENT using the latest sources and you will have to create the missing device files.

Configuring the bridge interface

The bridge(4) interface is used to provide network access to DOMUs.

To use one, edit (or create) the file /etc/ifconfig.bridge0 and insert following lines to:

create
!brconfig $int add bge0 stp bge0 up

Where 'bge0' should be changed to the name of the interface you want to use with your guest operating systems. use ifconfig(8) to get more details about your actual interfaces.

Rebooting into DOM0

Time to reboot:

# shutdown -r now

If all has gone well, you should have booted into the XEN3_DOM0 kernel. Check this with uname(1):

# uname -v
NetBSD 4.0 (XEN3_DOM0) #0: Sun Dec 16 01:20:31 PST 2007
builds@wb34:/home/builds/ab/netbsd-4-0-RELEASE/i386/200712160005Z-obj/home/builds/ab/netbsd-4-0-RELEASE/src/sys/arch/i386/compile/XEN3_DOM0

You should have XEN3 DOM0 mentioned somewhere.

Configuring rc scripts

Copy or symlink xend, xenbackendd and xendomains from /usr/pkg/share/examples/rc.d to /etc/rc.d.

# cp /usr/pkg/share/examples/rc.d/xend /etc/rc.d/
# cp /usr/pkg/share/examples/rc.d/xenbackendd /etc/rc.d/
# cp /usr/pkg/share/examples/rc.d/xendomains /etc/rc.d/

Edit /etc/rc.conf and add the following lines:

xend=YES
xenbackendd=YES
xendomains="dom1"

Later on, when you have created a configuration file for 'dom1', the xendomains variable specified above will trigger 'dom1' to be started when the system is booted. At this point no configuration exists for dom1, therefore it does nothing at this point. If you choose to name your configuration file something else, adapt the name accordingly.

To avoid rebooting a second, start all three services:

# /etc/rc.d/xend start
# /etc/rc.d/xenbackendd start
# /etc/rc.d/xendomains start

Run ifconfig -a to ensure the bridge interface is present and issue a ps ax | grep xen to ensure you have a similar output.

12 ?     DK   0:00.00 [xenwatch]
13 ?     DK   0:00.00 [xenbus]
411 ?     I    0:00.24 xenstored --pid-file=/var/run/xenstore.pid
594 ?     IWa  0:00.26 xenconsoled
629 ?     IW   0:00.00 /usr/pkg/bin/python2.3 /usr/pkg/sbin/xend start
631 ?     IWsa 0:00.02 /usr/pkg/sbin/xenbackendd
639 ?     IWa  0:00.52 /usr/pkg/bin/python2.3 /usr/pkg/sbin/xend start

The DOM0 configuration is now done. We will proceed to configuring DOMU domains.

Configuring DOMU

Configuring and installing a NetBSD DOMU

Create (or modify) /usr/pkg/etc/xen/dom1 and include this:

kernel = "/usr/src/obj/releasedir/i386/binary/kernel/netbsd-INSTALL_XEN3_DOMU.gz"
#kernel = "/netbsd-XEN3_DOMU"
memory = 64
name = "dom1"
#vcpus = 1
disk = [ 'phy:/dev/wd0g,0x03,w','file:/usr/src/obj/releasedir/i386/installation/cdrom/netbsd-i386.iso,0x04,r' ]
vif = [ 'bridge=bridge0' ]
root = "/dev/wd0d"

This configuration boots into the NetBSD sysinst program and allows you to install a NetBSD DOMU using the normal sysinst method. This configuration uses a DOMU_INSTALL kernel and an ISO image provided by a successful 'build release' and 'build iso-image'. You may be able to locate a valid Xen3 DOMU_INSTALL kernel from ftp://ftp.netbsd.org/pub/NetBSD-daily/ but if not, building a release is your best bet.

In this configuration file, /dev/wd0g is the reserved partition for the guest operating system. This should be changed to the partition you reserved prior to following the instructions within this document.

If you would like to use a physical CDROM instead of an ISO image, change the disk line to:

disk = [ 'phy:/dev/wd0g,0x03,w','phy:/dev/cd0a,0x04,r' ]

Now boot into sysinst using the command:

xm create dom1 -c

The reserved partition will appear as /dev/xbd0. Proceed as you would with a normal NetBSD installation using xbd0 as the target drive and xbd1 as the CDROM.

When you have finished, run shutdown -hp now to dom1.

Now edit /usr/pkg/etc/xen/dom1. Comment the INSTALL kernel and uncomment the DOMU kernel.

You should now have a working NetBSD DOMU (dom1). Boot into dom1 again with the command:

xm create dom1 -c

and ensure the file, /etc/ttys contains only this line or has all other lines commented:

console "/usr/libexec/getty Pc"         vt100   on secure

and the file /etc/wscons.conf is completely empty or has all lines commented out. These last two steps ensure no errors should be present on boot. Setting wscons=NO in /etc/rc.conf may effectively do the same thing.

From here, configure /etc/rc.conf and all runtime configuration files as you would normally. The network interface name should be xennet0. Use this name when configuring an IP address.

More information can be obtained by referencing the Xen user guide and the official NetBSD Xen Howto. Questions can be addressed to the port- xen@NetBSD.org mailling list.

Configuring and installing a Windows XP DOMU

This requires an HVM capable processor and xentools (see sections above).

This assumes you have a copy of the Windows install CD in /home/xen/winxp.iso, and wish to create a file /home/xen/winxp.img to hold the install. First create a blank file to hold the install. This assumes a size of 4GB (4096M). If you want a different size adjust the numbers to match:

# dd if=/dev/zero of=/home/xen/winxp.img bs=1m count=4096

Create /usr/pkg/etc/xen/win01:

kernel = '/usr/pkg/lib/xen/boot/hvmloader'
builder = 'hvm'
memory = '400'
device_model='/usr/pkg/libexec/qemu-dm'

disk = [ 'file:/home/xen/winxp.img,ioemu:hda,w',
          'file:/home/xen/winxp.iso,ioemu:hdb:cdrom,r', ]

# Hostname
name = "win01"

vif = [ 'type=ioemu, bridge=bridge0' ]

boot= 'd'
vnc = 1
usbdevice = 'tablet' # Helps with mouse pointer positioning

You may want to modify the amount of memory and pathnames.

Ensure you have a vncviewer installed, such as net/tightvncviewer or net/vncviewer from pkgsrc.

Then start the XENU and connect to it via VNC.

# xm create /usr/pkg/etc/xen/win01
# vncviewer :0

This will boot the Windows ISO image and let you install Windows as normal. As Windows reboots during install you may need to restart vncviewer.

After install change the boot d to boot c to have the system boot directly from the disk image.

Configuring and installing a GNU/Linux DOMU

We will do this in two steps:

• install a GNU/Linux system, from a livecd or any installation media
• configure the DOM0 so that it can create and start the Linux DOMU.

Installing a Linux distribution (soon-to-be DOMU)

Before proceeding with DOMU configuration, we will install our favorite GNU/Linux distribution on the computer.

In order to do it, we need at least two partitions (only one, if you do not consider the swap). These partitions must reside outside of the NetBSD slice, and may be either of primary or extended type. Of course, you can use more than two, but adapt your labels and partitions accordingly.

We do not cover the partition/slices manipulations through fdisk(8) and disklabel(8), as it depends strongly on how you manage your hard drive's space.

For this tutorial, we will use this partitioning:

# fdisk /dev/wd0d                                                      
fdisk: removing corrupt bootsel information
fdisk: Cannot determine the number of heads
Disk: /dev/wd0d
NetBSD disklabel disk geometry:
cylinders: 486344, heads: 16, sectors/track: 63 (1008 sectors/cylinder)
total sectors: 490234752

BIOS disk geometry:
cylinders: 1023, heads: 255, sectors/track: 63 (16065 sectors/cylinder)
total sectors: 490234752

Partition table:
0: Linux native (sysid 131)
    start 63, size 20482812 (10001 MB, Cyls 0-1274)
        PBR is not bootable: All bytes are identical (0x00)
1: Linux swap or Prime or Solaris (sysid 130)
    start 20482875, size 1959930 (957 MB, Cyls 1275-1396)
        PBR is not bootable: All bytes are identical (0x00)
2: NetBSD (sysid 169)
    start 61464690, size 428770062 (209360 MB, Cyls 3826-30515/178/63), Active
3: <UNUSED>
Drive serial number: -286527765 (0xeeebeeeb)

Here, you notice that we decide to use two primary partitions for our future Linux DOMU:

• partition 0 (for the root directory /)
• partition 1 (for the swap)

Labels:

16 partitions:
#        size    offset     fstype [fsize bsize cpg/sgs]
 a:  30720816  61464690     4.2BSD   2048 16384     0  # (Cyl.  60976*-  91453*)
 b:   1049328  92185506       swap                     # (Cyl.  91453*-  92494*)
 c: 428770062  61464690     unused      0     0        # (Cyl.  60976*- 486343)
 d: 490234752         0     unused      0     0        # (Cyl.      0 - 486343)
 e:  20480000  93234834     4.2BSD      0     0     0  # (Cyl.  92494*- 112812*)
 f:  20480000 113714834     4.2BSD      0     0     0  # (Cyl. 112812*- 133129*)
 g:  20480000 134194834     4.2BSD      0     0     0  # (Cyl. 133129*- 153447*)
 h: 335559918 154674834     4.2BSD      0     0     0  # (Cyl. 153447*- 486343)
 i:  20482812        63 Linux Ext2      0     0        # (Cyl.      0*-  20320*)
 j:   1959930  20482875       swap                     # (Cyl.  20320*-  22264*)

Bear in mind that we added two labels here, namely i and j, which maps respectively to partition 0 and partition 1 of the disk. We will use these labels later for DOMU configuration.

Now that we have partitioned the disk, proceed with installing your Linux distribution. We will not cover that part in this tutorial. You can either install it from an installation media (a cdrom from example), or copy files from an already installed distribution on your computer.

Tip: to manipulate ext2/3 filesystems (the traditional fs under Linux) from NetBSD, you can use sysutils/e2fsprogs from pkgsrc:

# cd /usr/pkgsrc/sysutils/e2fsprogs
# make install

And then use e2fsck, mke2fs and mount_ext2fs(8) directly from NetBSD.

Getting XEN aware Linux kernels

Once installation is done, reboot your computer and return to our Xen-NetBSD system.

To boot our Linux DOMU, we will need a Linux kernel supporting the XENU virtualisation. Depending on your Linux distribution, you can grab one from its repository (it is up to you to find it through aptitude, yum or whatever package manager you use), or get one from the Xen binary distribution.

To get a XENU Linux kernel from Xen binary distribution, get it directly from Xen website download page. Download the tarball and extract the vmlinuz-*-xen from it. In our case, with a 2.6.18 Linux kernel:

# ftp -a http://bits.xensource.com/oss-xen/release/3.1.0/bin.tgz/xen-3.1.0-install-x86_32.tgz
# cd /tmp
# tar -xzf xen-3.1.0-install-x86_32.tgz dist/install/boot/vmlinuz-2.6.18-xen

vmlinuz-2.6.18-xen is the kernel that Xen will use to start the DOMU. Move it to any directory you like (just remember it when configuring the kernel entry in the DOMU configuration file):

# mv dist/install/boot/vmlinuz-2.6.18-xen /vmlinuz-XEN3-DOMU

Configuring DOMU

Configuring the Linux DOMU is a bit different than a NetBSD one; some options tend to differ.

Edit (or create) the configuration file domu-linux, in /usr/pkg/etc/xen/:

# vi /usr/pkg/etc/xen/domu-linux

Here's a typical config file for a Linux DOMU:

#----------------------------------------------------------------------------
# Kernel image file. This kernel will be loaded in the new domain.
kernel = "/vmlinuz-XEN3-DOMU"

# Memory allocation (in megabytes) for the new domain.
memory = 256

# A handy name for your new domain. This will appear in 'xm list',
# and you can use this as parameters for xm in place of the domain
# number. All domains must have different names.
#
name = "domu-linux"

# Which CPU to start domain on (only relevant for SMP hardware).  CPUs
# numbered starting from ``0''.
#
cpu = "^1"   # leave to Xen to pick

#----------------------------------------------------------------------------
# Define network interfaces for the new domain.

# Number of network interfaces (must be at least 1). Default is 1.
vif = [ '' ]

# Define MAC and/or bridge for the network interfaces.
#
# The MAC address specified in ``mac'' is the one used for the interface
# in the new domain. The interface in domain0 will use this address XOR'd
# with 00:00:00:01:00:00 (i.e. aa:00:00:51:02:f0 in our example). Random
# MACs are assigned if not given.
#
# ``bridge'' is a required parameter, which will be passed to the
# vif-script called by xend(8) when a new domain is created to configure
# the new xvif interface in domain0.
#
# In this example, the xvif is added to bridge0, which should have been
# set up prior to the new domain being created -- either in the
# ``network'' script or using a /etc/ifconfig.bridge0 file.
#
vif = [ 'mac=aa:00:00:50:02:f0, bridge=bridge0' ]

#----------------------------------------------------------------------------
# Define the disk devices you want the domain to have access to, and
# what you want them accessible as.
#
# Each disk entry is of the form:
#
#       phy:DEV,VDEV,MODE
#
# where DEV is the device, VDEV is the device name the domain will see,
# and MODE is r for read-only, w for read-write.  You can also create
# file-backed domains using disk entries of the form:
#
#       file:PATH,VDEV,MODE
#
# where PATH is the path to the file used as the virtual disk, and VDEV
# and MODE have the same meaning as for ``phy'' devices.
#
# /dev/wd0i will be seen as "hda1" under DOMU (the root partition)
# /dev/wd0j will be seen as "hda2" under DOMU (the swap)
#
disk = [ 'phy:/dev/wd0i,hda1,w','phy:/dev/wd0j,hda2,w' ]

#----------------------------------------------------------------------------
# Set the kernel command line for the new domain.

# Set root device.
root = "/dev/hda1"

Now, you should be able to start your first Linux DOMU!

# xm create -c /usr/pkg/etc/xen/domu-linux

Possible caveats

If you intend to have more than one box configured with the above configuration on the same network, you will most likely have to specify a unique MAC address per guest OS, otherwise it is likely you will have a conflict. I'm not sure if the MAC assignment is random, incremental or if Xen is able to check for the existence of the proposed MAC address, so specifying the MAC address is recommended.

Here is a method to assign a MAC address to a newly created Xen Domu. First as described before, use the following vif parameter in your config file :

vif = [ 'bridge=bridge0' ]

Then, run the Xen DomU and, once logged run the following command :

# ifconfig xennet0

Output Sample : 

xennet0: flags=8863<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
        enabled=0
        address: 00:16:3e:2e:32:5f
        inet 192.168.4.81 netmask 0xffffff00 broadcast 192.168.4.255
        inet6 fe80::216:3eff:fe2e:325f%xennet0 prefixlen 64 scopeid 0x2

A MAC address is automaticaly generated, use it in your config file with the following syntax :

vif = [ 'mac=00:16:3e:2e:32:5f, bridge=bridge0' ]

And now you can restart the Xen DomU.

Please, note that the MAC Address MUST start with "00:16:3e".

See also

• The official NetBSD Xen Howto
• The official Xen Wiki