File:  [NetBSD Developer Wiki] / wikisrc / tutorials / how_to_set_up_a_guest_os_using_xen3.mdwn
Revision 1.6: download - view: text, annotated - select for diffs
Wed Sep 9 14:28:56 2020 UTC (4 months, 1 week ago) by kim
Branches: MAIN
CVS tags: HEAD
Use man template


[[!toc levels=3]]

# Meta

Note that there is also a [Xen HOWTO](../../ports/xen/howto/).
Arguably this content could be folded in there.

#  Requirements

Xen3 is supported from NetBSD-4.0 onward. If you plan on using NetBSD-CURRENT,
please read the article [[How to build NetBSD-current]] to do so. Guest
operating systems can run from their own partitions, or from image files in
the main (DOM0) install.

This tutorial describes how to:

  * Install and configure NetBSD as a DOM0 
  * Install and run a NetBSD as a DOMU 
  * Install and run a Windows XP system as as DOMU 
  * Install and run a Debian system as a DOMU 

#  Installing Xen tools and kernels

##  Xen tools

To run and administer xen domains, we need the xentools3 or xentools33
packages which are available in pkgsrc.

Xen 3.1 packages are under [sysutils/xentools3]( for traditional xentools,
and [sysutils/xentools3-hvm]( for the additional **HVM** support to run un-
modified OSes such as Windows XP.

Xen 3.3 packages are under [sysutils/xentools33]( Unlike Xen 3.1, no extra
package is required for **HVM** support. Note, it is not possible to install
Xen 3.1 and Xen 3.3 packages at the same time. They conflict with each other.

**HVM** stands for **Hardware Virtualization Managed**. The benefit of hardware virtualization is that you can run OSes that don't know they are being virutalized like Windows XP, for example. However, you must have a CPU which supports this. Intel CPUs must have the 'VT' instruction. AMD CPUs will have the 'SVM' instruction. You can find out if your CPU supports HVM by taking a look at this page: 


In NetBSD 5.0 there's a new cpuctl command. This is an example output of an

     # cpuctl identify 0
     cpu0: AMD Unknown K8 (Athlon) (686-class), 2210.22 MHz, id 0x60f82
     cpu0: features 0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
     cpu0: features 0x178bfbff<PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX>
     cpu0: features 0x178bfbff<FXSR,SSE,SSE2,HTT>
     cpu0: features2 0x2001<SSE3,CX16>
     cpu0: features3 0xebd3fbff<SCALL/RET,NOX,MXX,FFXSR,RDTSCP,LONG,3DNOW2,3DNOW>
     cpu0: "AMD Turion(tm) 64 X2 Mobile Technology TL-64"
     cpu0: I-cache 64KB 64B/line 2-way, D-cache 64KB 64B/line 2-waycpu0: L2 cache 1MB 64B/line 16-way
     cpu0: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
     cpu0: DTLB 32 4KB entries fully associative, 8 4MB entries fully associativecpu0: Initial APIC ID 0
     cpu0: AMD Power Management features: 0x7f<TS,FID,VID,TTP,HTC,STC,100>
     cpu0: family 0f model 08 extfamily 00 extmodel 06

Note the **SVM** feature flag in the **features4** line indicating that
**HVM** support is present on this CPU. However, this feature may be disabled
in the BIOS. In this case since NetBSD 5.0 there will show up this dmesg line
on AMD CPUs only:

     cpu0: SVM disabled by the BIOS

Xen 3.1 (the xentools3-hvm package will automatically bring in the base

    # cd /usr/pkgsrc/sysutils/xentools3-hvm
    # make install

Xen 3.3:

    # cd /usr/pkgsrc/sysutils/xentools33
    # make install

##  Xen kernel

Next, we will get the xen hypervisor kernel itself. For NetBSD 4.x and 5.x the
i386 port does not support 'PAE' kernels and must run the Xen 3.1 package.
This restriction has been removed in -current and is not relevant for the
amd64 port.

For Xen 3.1, in pkgsrc this is [sysutils/xenkernel3](, for Xen 3.3, this is

    # cd /usr/pkgsrc/sysutils/xenkernel3
    # make install

And copy it into / directory, like this:

    # cp /usr/pkg/xen3-kernel/xen.gz /

##  Xen DOM0 kernel

Lastly, we need a XEN-enabled kernel for our DOM0 domain. There are two
possibilities: downloading the binary version, or building it from source.

##  Downloading the binary version

From NetBSD-4.0 onward, NetBSD supports Xen, and provides some XEN-enabled
kernel directly from [[1]][36], in each `binary/kernel` directory associated
with a particular release.

For example, with NetBSD-4.0, You can grab one from

    # ftp -a

The `netbsd-XEN3_DOM0.gz` file contains a gzipped version of the kernel. Just
copy and move it into the root directory like this:

    # cp netbsd-XEN3_DOM0.gz /

##  Building it from source

Building a kernel from source is out of the scope of this section. Please
consult [How to build a kernel]( from NetBSD's FAQ for more details.

Once building is done, you can find them in
`/usr/src/obj/_releasedir_/i386/binary/kernel/`. Identically to binary
version, copy or move netbsd-XEN3_DOM0 in the root directory `/`.

#  Selecting a bootloader

In NetBSD 5.0 the native boot loader, /boot, can load Xen directly.
The NetBSD 5.0 bootloader can be easily dropped into a NetBSD 4.x system by coping them into
/boot and running installboot(8) properly.

##  Updating /boot

For full details refer to installboot(8), but for a standard configuration
with VGA console and an IDE or SATA drive with an FFSv1 root file system use
the following:

    # cp /usr/mdec/boot /boot
    # installboot -v -o timeout=5 /dev/rwd0a /usr/mdec/bootxx_ffsv1

##  Updating /boot.cfg

NetBSD 5.0 or later will already have a /boot.cfg file with a basic
configuration. Enabling Xen support only requires one additional line in this
case. If you're upgrading from an earlier version or do not have an existing
/boot.cfg use the following example:

    banner=Welcome to NetBSD
    banner=Please choose an option from the following menu:
    menu=Boot normally:boot netbsd
    menu=Boot single-user:boot netbsd -s
    menu=Boot backup kernel:boot onetbsd
    menu=Drop to boot prompt:prompt
    menu=Boot Xen with 256MB for dom0:load /netbsd-XEN3_DOM0 console=pc;multiboot /usr/pkg/xen3-kernel/xen.gz dom0_mem=256M
    menu=Boot Xen with 256MB for dom0 (serial):load /netbsd-XEN3_DOM0 console=com0;multiboot /usr/pkg/xen3-kernel/xen.gz dom0_mem=256M console=com1 com1=115200,8n1
    menu=Boot Xen with dom0 in single-user mode:load /netbsd-XEN3_DOM0 -s;multiboot /usr/pkg/xen3-kernel/xen.gz dom0_mem=256M

Make sure you update the "load /netbsd-XEN3_DOM0" and the "dom0_mem=256M"
arguments to match your setup. On next boot select the 'Boot Xen with 256MB
for dom0' option and make sure you see Xen kernel messages and the normal
NetBSD kernel messages. Once you're satisfied it is working you can change the
"default=1" line to "default=5" to automatically boot Xen on reboot.

#  Setting up DOM0

##  Creating xen devices

To create all xen devices, change to /dev and run ./MAKEDEV xen

    cd /dev
    ./MAKEDEV xen

This should create the devices **xencons**, **xenevt** and **xsd_kva**. If any
of these are missing you may not have updated to _CURRENT_ using the latest
sources and you will have to create the missing device files.

##  Configuring the bridge interface

The [[!template id=man name="bridge" section="4"]] interface is used to provide network access to DOMUs.

To use one, edit (or create) the file `/etc/ifconfig.bridge0` and insert
following lines to:

    !brconfig $int add bge0 stp bge0 up

Where 'bge0' should be changed to the name of the interface you want to use
with your guest operating systems. use [[!template id=man name="ifconfig" section="8"]] to get more details
about your actual interfaces.

##  Rebooting into DOM0

Time to reboot:

    # shutdown -r now

If all has gone well, you should have booted into the XEN3_DOM0 kernel. Check
this with [[!template id=man name="uname" section="1"]]:

    # uname -v
    NetBSD 4.0 (XEN3_DOM0) #0: Sun Dec 16 01:20:31 PST 2007

You should have **XEN3 DOM0** mentioned somewhere.

##  Configuring rc scripts

Copy or symlink xend, xenbackendd and xendomains from
`/usr/pkg/share/examples/rc.d` to `/etc/rc.d`.

    # cp /usr/pkg/share/examples/rc.d/xend /etc/rc.d/
    # cp /usr/pkg/share/examples/rc.d/xenbackendd /etc/rc.d/
    # cp /usr/pkg/share/examples/rc.d/xendomains /etc/rc.d/

Edit `/etc/rc.conf` and add the following lines:


Later on, when you have created a configuration file for 'dom1', the
xendomains variable specified above will trigger 'dom1' to be started when the
system is booted. At this point no configuration exists for dom1, therefore it
does nothing at this point. If you choose to name your configuration file
something else, adapt the name accordingly.

To avoid rebooting a second, start all three services:

    # /etc/rc.d/xend start
    # /etc/rc.d/xenbackendd start
    # /etc/rc.d/xendomains start

Run `ifconfig -a` to ensure the bridge interface is present and issue a `ps ax
| grep xen` to ensure you have a similar output.

    12 ?     DK   0:00.00 [xenwatch]
    13 ?     DK   0:00.00 [xenbus]
    411 ?     I    0:00.24 xenstored --pid-file=/var/run/
    594 ?     IWa  0:00.26 xenconsoled
    629 ?     IW   0:00.00 /usr/pkg/bin/python2.3 /usr/pkg/sbin/xend start
    631 ?     IWsa 0:00.02 /usr/pkg/sbin/xenbackendd
    639 ?     IWa  0:00.52 /usr/pkg/bin/python2.3 /usr/pkg/sbin/xend start

The DOM0 configuration is now done. We will proceed to configuring DOMU

#  Configuring DOMU

##  Configuring and installing a NetBSD DOMU

Create (or modify) `/usr/pkg/etc/xen/dom1` and include this:

    kernel = "/usr/src/obj/releasedir/i386/binary/kernel/netbsd-INSTALL_XEN3_DOMU.gz"
    #kernel = "/netbsd-XEN3_DOMU"
    memory = 64
    name = "dom1"
    #vcpus = 1
    disk = [ 'phy:/dev/wd0g,0x03,w','file:/usr/src/obj/releasedir/i386/installation/cdrom/netbsd-i386.iso,0x04,r' ]
    vif = [ 'bridge=bridge0' ]
    root = "/dev/wd0d"

This configuration boots into the NetBSD sysinst program and allows you to
install a NetBSD DOMU using the normal sysinst method. This configuration uses
a DOMU_INSTALL kernel and an ISO image provided by a successful 'build
release' and 'build iso-image'. You may be able to locate a valid Xen3
DOMU_INSTALL kernel from <> but if
not, building a release is your best bet.

In this configuration file, `/dev/wd0g` is the reserved partition for the
guest operating system. This should be changed to the partition you reserved
prior to following the instructions within this document.

If you would like to use a physical CDROM instead of an ISO image, change the
disk line to:

    disk = [ 'phy:/dev/wd0g,0x03,w','phy:/dev/cd0a,0x04,r' ]

Now boot into sysinst using the command:

    xm create dom1 -c

The reserved partition will appear as `/dev/xbd0`. Proceed as you would with a
normal NetBSD installation using xbd0 as the target drive and xbd1 as the

When you have finished, run `shutdown -hp now` to dom1.

Now edit `/usr/pkg/etc/xen/dom1`. Comment the INSTALL kernel and uncomment the
DOMU kernel.

You should now have a working NetBSD DOMU (dom1). Boot into dom1 again with
the command:

    xm create dom1 -c

and ensure the file, `/etc/ttys` contains only this line or has all other
lines commented:

    console "/usr/libexec/getty Pc"         vt100   on secure

and the file `/etc/wscons.conf` is completely empty or has all lines commented
out. These last two steps ensure no errors should be present on boot. Setting
wscons=NO in `/etc/rc.conf` may effectively do the same thing.

From here, configure `/etc/rc.conf` and all runtime configuration files as you
would normally. The network interface name should be _**xennet0**_. Use this
name when configuring an IP address.

More information can be obtained by referencing the [Xen user guide]( and
[the official NetBSD Xen Howto]( Questions can be addressed to the port- mailling list.

##  Configuring and installing a Windows XP DOMU

This requires an HVM capable processor and xentools (see sections above).

This assumes you have a copy of the Windows install CD in /home/xen/winxp.iso,
and wish to create a file /home/xen/winxp.img to hold the install. First
create a blank file to hold the install. This assumes a size of 4GB (4096M).
If you want a different size adjust the numbers to match:

    # dd if=/dev/zero of=/home/xen/winxp.img bs=1m count=4096

Create `/usr/pkg/etc/xen/win01`:

    kernel = '/usr/pkg/lib/xen/boot/hvmloader'
    builder = 'hvm'
    memory = '400'
    disk = [ 'file:/home/xen/winxp.img,ioemu:hda,w',
              'file:/home/xen/winxp.iso,ioemu:hdb:cdrom,r', ]
    # Hostname
    name = "win01"
    vif = [ 'type=ioemu, bridge=bridge0' ]
    boot= 'd'
    vnc = 1
    usbdevice = 'tablet' # Helps with mouse pointer positioning

You may want to modify the amount of memory and pathnames.

Ensure you have a vncviewer installed, such as [net/tightvncviewer]( or
[net/vncviewer]( from pkgsrc.

Then start the XENU and connect to it via VNC.

    # xm create /usr/pkg/etc/xen/win01
    # vncviewer :0

This will boot the Windows ISO image and let you install Windows as normal. As
Windows reboots during install you may need to restart vncviewer.

After install change the **boot d** to **boot c** to have the system boot
directly from the disk image.

##  Configuring and installing a GNU/Linux DOMU

We will do this in two steps:

  * install a GNU/Linux system, from a livecd or any installation media 
  * configure the DOM0 so that it can create and start the Linux DOMU. 

###  Installing a Linux distribution (soon-to-be DOMU)

Before proceeding with DOMU configuration, we will install our favorite
GNU/Linux distribution on the computer.

In order to do it, we need at least two partitions (only one, if you do not
consider the swap). These partitions must reside outside of the NetBSD slice,
and may be either of primary or extended type. Of course, you can use more
than two, but adapt your labels and partitions accordingly.

We do not cover the partition/slices manipulations through [[!template id=man name="fdisk" section="8"]] and
[[!template id=man name="disklabel" section="8"]], as it depends strongly on how you manage your hard drive's

For this tutorial, we will use this partitioning:

    # fdisk /dev/wd0d                                                      
    fdisk: removing corrupt bootsel information
    fdisk: Cannot determine the number of heads
    Disk: /dev/wd0d
    NetBSD disklabel disk geometry:
    cylinders: 486344, heads: 16, sectors/track: 63 (1008 sectors/cylinder)
    total sectors: 490234752
    BIOS disk geometry:
    cylinders: 1023, heads: 255, sectors/track: 63 (16065 sectors/cylinder)
    total sectors: 490234752
    Partition table:
    0: Linux native (sysid 131)
        start 63, size 20482812 (10001 MB, Cyls 0-1274)
            PBR is not bootable: All bytes are identical (0x00)
    1: Linux swap or Prime or Solaris (sysid 130)
        start 20482875, size 1959930 (957 MB, Cyls 1275-1396)
            PBR is not bootable: All bytes are identical (0x00)
    2: NetBSD (sysid 169)
        start 61464690, size 428770062 (209360 MB, Cyls 3826-30515/178/63), Active
    3: <UNUSED>
    Drive serial number: -286527765 (0xeeebeeeb)

Here, you notice that we decide to use two primary partitions for our future
Linux DOMU:

  * partition 0 (for the **root directory /**) 
  * partition 1 (for the **swap**) 


    16 partitions:
    #        size    offset     fstype [fsize bsize cpg/sgs]
     a:  30720816  61464690     4.2BSD   2048 16384     0  # (Cyl.  60976*-  91453*)
     b:   1049328  92185506       swap                     # (Cyl.  91453*-  92494*)
     c: 428770062  61464690     unused      0     0        # (Cyl.  60976*- 486343)
     d: 490234752         0     unused      0     0        # (Cyl.      0 - 486343)
     e:  20480000  93234834     4.2BSD      0     0     0  # (Cyl.  92494*- 112812*)
     f:  20480000 113714834     4.2BSD      0     0     0  # (Cyl. 112812*- 133129*)
     g:  20480000 134194834     4.2BSD      0     0     0  # (Cyl. 133129*- 153447*)
     h: 335559918 154674834     4.2BSD      0     0     0  # (Cyl. 153447*- 486343)
     i:  20482812        63 Linux Ext2      0     0        # (Cyl.      0*-  20320*)
     j:   1959930  20482875       swap                     # (Cyl.  20320*-  22264*)

Bear in mind that we added two labels here, namely **i** and **j**, which maps
respectively to partition 0 and partition 1 of the disk. We will use these
labels later for DOMU configuration.

Now that we have partitioned the disk, proceed with installing your Linux
distribution. We will not cover that part in this tutorial. You can either
install it from an installation media (a cdrom from example), or copy files
from an already installed distribution on your computer.

Tip: to manipulate ext2/3 filesystems (the traditional fs under Linux) from
NetBSD, you can use [sysutils/e2fsprogs]( from pkgsrc:

    # cd /usr/pkgsrc/sysutils/e2fsprogs
    # make install

And then use e2fsck, mke2fs and [[!template id=man name="mount_ext2fs" section="8"]] directly from NetBSD.

###  Getting XEN aware Linux kernels

Once installation is done, reboot your computer and return to our Xen-NetBSD

To boot our Linux DOMU, we will need a Linux kernel supporting the XENU
virtualisation. Depending on your Linux distribution, you can grab one from
its repository (it is up to you to find it through aptitude, yum or whatever
package manager you use), or get one from the Xen binary distribution.

To get a XENU Linux kernel from Xen binary distribution, get it directly from
[Xen website download page]( Download the tarball and extract the
**vmlinuz-*-xen** from it. In our case, with a 2.6.18 Linux kernel:

    # ftp -a
    # cd /tmp
    # tar -xzf xen-3.1.0-install-x86_32.tgz dist/install/boot/vmlinuz-2.6.18-xen

**vmlinuz-2.6.18-xen** is the kernel that Xen will use to start the DOMU. Move it to any directory you like (just remember it when configuring the _kernel_ entry in the DOMU configuration file): 
    # mv dist/install/boot/vmlinuz-2.6.18-xen /vmlinuz-XEN3-DOMU

###  Configuring DOMU

Configuring the Linux DOMU is a bit different than a NetBSD one; some options
tend to differ.

Edit (or create) the configuration file **domu-linux**, in

    # vi /usr/pkg/etc/xen/domu-linux

Here's a typical config file for a Linux DOMU:

    # Kernel image file. This kernel will be loaded in the new domain.
    kernel = "/vmlinuz-XEN3-DOMU"
    # Memory allocation (in megabytes) for the new domain.
    memory = 256
    # A handy name for your new domain. This will appear in 'xm list',
    # and you can use this as parameters for xm in place of the domain
    # number. All domains must have different names.
    name = "domu-linux"
    # Which CPU to start domain on (only relevant for SMP hardware).  CPUs
    # numbered starting from ``0''.
    cpu = "^1"   # leave to Xen to pick
    # Define network interfaces for the new domain.
    # Number of network interfaces (must be at least 1). Default is 1.
    vif = [ '' ]
    # Define MAC and/or bridge for the network interfaces.
    # The MAC address specified in ``mac'' is the one used for the interface
    # in the new domain. The interface in domain0 will use this address XOR'd
    # with 00:00:00:01:00:00 (i.e. aa:00:00:51:02:f0 in our example). Random
    # MACs are assigned if not given.
    # ``bridge'' is a required parameter, which will be passed to the
    # vif-script called by xend(8) when a new domain is created to configure
    # the new xvif interface in domain0.
    # In this example, the xvif is added to bridge0, which should have been
    # set up prior to the new domain being created -- either in the
    # ``network'' script or using a /etc/ifconfig.bridge0 file.
    vif = [ 'mac=aa:00:00:50:02:f0, bridge=bridge0' ]
    # Define the disk devices you want the domain to have access to, and
    # what you want them accessible as.
    # Each disk entry is of the form:
    #       phy:DEV,VDEV,MODE
    # where DEV is the device, VDEV is the device name the domain will see,
    # and MODE is r for read-only, w for read-write.  You can also create
    # file-backed domains using disk entries of the form:
    #       file:PATH,VDEV,MODE
    # where PATH is the path to the file used as the virtual disk, and VDEV
    # and MODE have the same meaning as for ``phy'' devices.
    # /dev/wd0i will be seen as "hda1" under DOMU (the root partition)
    # /dev/wd0j will be seen as "hda2" under DOMU (the swap)
    disk = [ 'phy:/dev/wd0i,hda1,w','phy:/dev/wd0j,hda2,w' ]
    # Set the kernel command line for the new domain.
    # Set root device.
    root = "/dev/hda1"

Now, you should be able to start your first Linux DOMU!

    # xm create -c /usr/pkg/etc/xen/domu-linux

#  Possible caveats

If you intend to have more than one box configured with the above
configuration on the same network, you will most likely have to specify a
unique MAC address per guest OS, otherwise it is likely you will have a
conflict. I'm not sure if the MAC assignment is random, incremental or if Xen
is able to check for the existence of the proposed MAC address, so specifying
the MAC address is recommended.

Here is a method to assign a MAC address to a newly created Xen Domu. First as
described before, use the following vif parameter in your config file :

    vif = [ 'bridge=bridge0' ]

Then, run the Xen DomU and, once logged run the following command :

    # ifconfig xennet0
    Output Sample : 
            address: 00:16:3e:2e:32:5f
            inet netmask 0xffffff00 broadcast
            inet6 fe80::216:3eff:fe2e:325f%xennet0 prefixlen 64 scopeid 0x2

A MAC address is automaticaly generated, use it in your config file with the
following syntax :

    vif = [ 'mac=00:16:3e:2e:32:5f, bridge=bridge0' ]

And now you can restart the Xen DomU.

Please, note that the MAC Address MUST start with "00:16:3e".


#  See also

  * [The official NetBSD Xen Howto](
  * [The official Xen Wiki](

CVSweb for NetBSD wikisrc <> software: FreeBSD-CVSweb