--- wikisrc/ports/xen/howto.mdwn 2014/12/24 00:04:47 1.15 +++ wikisrc/ports/xen/howto.mdwn 2014/12/24 01:34:47 1.23 @@ -27,6 +27,13 @@ code for Xen and need not be aware that Attempts to access hardware registers are trapped and emulated. This style is less efficient but can run unmodified guests. +Generally any amd64 machine will work with Xen and PV guests. For HVM +guests, the VT or VMX cpu feature (Intel) or SVM/HVM/VT (amd64) is +needed; "cpuctl identify 0" will show this. TODO: Clean up and check +the above features. TODO: Explain if i386 (non-amd64) machines can +still be used --- I think that the requirement to use PAE kernels is +about the hypervisor being amd64 only. + At boot, the dom0 kernel is loaded as module with Xen as the kernel. The dom0 can start one or more domUs. (Booting is explained in detail in the dom0 section.) @@ -36,6 +43,11 @@ and that Xen kernels and tools are avail attempts to address both the case of running a NetBSD dom0 on hardware and running NetBSD as a domU in a VPS. +Some versions of Xen support "PCI passthrough", which means that +specific PCI devices can be made available to a specific domU instead +of the dom0. This can be useful to let a domU run X11, or access some +network interface or other peripheral. + Prerequisites ------------- @@ -46,12 +58,18 @@ things must be done, guiding the reader path when there are no known good reasons to stray. This HOWTO presumes a basic familiarity with the Xen system -architecture. - -This HOWTO presumes familiarity with installing NetBSD on i386/amd64 -hardware and installing software from pkgsrc. - -For more details on Xen, see [](http://www.xen.org/). +architecture. This HOWTO presumes familiarity with installing NetBSD +on i386/amd64 hardware and installing software from pkgsrc. +See also the [Xen website](http://www.xen.org/). + +History +------- + +NetBSD used to support Xen2; this has been removed. + +Before NetBSD's native bootloader could support Xen, the use of +grub was recommended. If necessary, see the +[old grub information](/xen/howto-grub/). Versions of Xen and NetBSD ========================== @@ -70,7 +88,8 @@ but note that both packages must be inst matching versions. xenkernel3 and xenkernel33 provide Xen 3.1 and 3.3. These no longer -receive security patches and should not be used. +receive security patches and should not be used. Xen 3.1 supports PCI +passthrough. xenkernel41 provides Xen 4.1. This is no longer maintained by Xen, but as of 2014-12 receives backported security patches. It is a @@ -81,6 +100,16 @@ of 2014-12. Ideally newer versions of Xen will be added to pkgsrc. +Note that NetBSD support is called XEN3; it works with 3.1 through +4.2, because the hypercall interface has been stable. + +Xen command program +------------------- + +Early Xen used a program called "xm" to manipulate the system from the +dom0. Starting in 4.1, a replacement program with similar behavior +called "xl" is provided. In 4.2, "xm" is no longer available. + NetBSD ------ @@ -94,17 +123,34 @@ no SMP support for NetBSD as dom0. (The need SMP; the lack of support is really a problem when using a dom0 as a normal computer.) +Architecture +------------ + +Xen is basically amd64 only at this point. One can either run i386 +domains or amd64 domains. If running i386, PAE versions are required, +for both dom0 and domU. These versions are built by default in NetBSD +releases. While i386 dom0 works fine, amd64 is recommended as more +normal. (Note that emacs (at least) fails if run on i386 with PAE when +built without, and vice versa, presumably due to bugs in the undump +code.) + Recommendation -------------- -Therefore, this HOWTO recommends running xenkernel42 (and xentools42) -and NetBSD 6 stable branch. +Therefore, this HOWTO recommends running xenkernel42 (and xentools42), +xl, the NetBSD 6 stable branch, and to use amd64 as the dom0. Either +the i386 or amd64 of NetBSD may be used as domUs. NetBSD as a dom0 ================ NetBSD can be used as a dom0 and works very well. The following sections address installation, updating NetBSD, and updating Xen. +Note that it doesn't make sense to talk about installing a dom0 OS +without also installing Xen itself. We first address installing +NetBSD, which is not yet a dom0, and then adding Xen, pivoting the +NetBSD install to a dom0 install by just changing the kernel and boot +configuration. Styles of dom0 operation ------------------------ @@ -127,165 +173,96 @@ Xen daemons when not running Xen. Note that NetBSD as dom0 does not support multiple CPUs. This will limit the performance of the Xen/dom0 workstation approach. -Installation of NetBSD and Xen ------------------------------- +Installation of NetBSD +---------------------- -Note that it doesn't make sense to talk about installing a dom0 OS -without also installing Xen itself. - -First do a NetBSD/i386 or NetBSD/amd64 -[installation](../../docs/guide/en/chap-inst.html) of the 5.1 release -(or newer) as you usually do on x86 hardware. The binary releases are -available from [](ftp://ftp.NetBSD.org/pub/NetBSD/). Binary snapshots -for current and the stable branches are available on daily autobuilds. -If you plan to use the `grub` boot loader, when partitioning the disk -you have to make the root partition smaller than 512Mb, and formatted as -FFSv1 with 8k block/1k fragments. If the partition is larger than this, -uses FFSv2 or has different block/fragment sizes, grub may fail to load -some files. Also keep in mind that you'll probably want to provide -virtual disks to other domains, so reserve some partitions for these -virtual disks. Alternatively, you can create large files in the file -system, map them to vnd(4) devices and export theses vnd devices to -other domains. - -Next step is to install the Xen packages via pkgsrc or from binary -packages. See [the pkgsrc -documentation](http://www.NetBSD.org/docs/pkgsrc/) if you are unfamiliar -with pkgsrc and/or handling of binary packages. Xen 3.1, 3.3, 4.1 and -4.2 are available. 3.1 supports PCI pass-through while other versions do -not. You'll need either `sysutils/xentools3` and `sysutils/xenkernel3` -for Xen 3.1, `sysutils/xentools33` and `sysutils/xenkernel33` for Xen -3.3, `sysutils/xentools41` and `sysutils/xenkernel41` for Xen 4.1. or -`sysutils/xentools42` and `sysutils/xenkernel42` for Xen 4.2. You'll -also need `sysutils/grub` if you plan do use the grub boot loader. If -using Xen 3.1, you may also want to install `sysutils/xentools3-hvm` -which contains the utilities to run unmodified guests OSes using the -*HVM* support (for later versions this is included in -`sysutils/xentools`). Note that your CPU needs to support this. Intel -CPUs must have the 'VT' instruction, AMD CPUs the 'SVM' instruction. You -can easily find out if your CPU support HVM by using NetBSD's cpuctl -command: - - # cpuctl identify 0 - cpu0: Intel Core 2 (Merom) (686-class), id 0x6f6 - cpu0: features 0xbfebfbff - cpu0: features 0xbfebfbff - cpu0: features 0xbfebfbff - cpu0: features2 0x4e33d - cpu0: features3 0x20100800 - cpu0: "Intel(R) Xeon(R) CPU 5130 @ 2.00GHz" - cpu0: I-cache 32KB 64B/line 8-way, D-cache 32KB 64B/line 8-way - cpu0: L2 cache 4MB 64B/line 16-way - cpu0: ITLB 128 4KB entries 4-way - cpu0: DTLB 256 4KB entries 4-way, 32 4MB entries 4-way - cpu0: Initial APIC ID 0 - cpu0: Cluster/Package ID 0 - cpu0: Core ID 0 - cpu0: family 06 model 0f extfamily 00 extmodel 00 - -Depending on your CPU, the feature you are looking for is called HVM, -SVM or VMX. - -Next you need to copy the selected Xen kernel itself. pkgsrc installed -them under `/usr/pkg/xen*-kernel/`. The file you're looking for is -`xen.gz`. Copy it to your root file system. `xen-debug.gz` is a kernel -with more consistency checks and more details printed on the serial -console. It is useful for debugging crashing guests if you use a serial -console. It is not useful with a VGA console. - -You'll then need a NetBSD/Xen kernel for *domain0* on your root file -system. The XEN3PAE\_DOM0 kernel or XEN3\_DOM0 provided as part of the -i386 or amd64 binaries is suitable for this, but you may want to -customize it. Keep your native kernel around, as it can be useful for -recovery. *Note:* the *domain0* kernel must support KERNFS and `/kern` -must be mounted because *xend* needs access to `/kern/xen/privcmd`. - -Next you need to get a bootloader to load the `xen.gz` kernel, and the -NetBSD *domain0* kernel as a module. This can be `grub` or NetBSD's boot -loader. Below is a detailled example for grub, see the boot.cfg(5) -manual page for an example using the latter. - -This is also where you'll specify the memory allocated to *domain0*, the -console to use, etc ... - -Here is a commented `/grub/menu.lst` file: - - #Grub config file for NetBSD/xen. Copy as /grub/menu.lst and run - # grub-install /dev/rwd0d (assuming your boot device is wd0). - # - # The default entry to load will be the first one - default=0 - - # boot the default entry after 10s if the user didn't hit keyboard - timeout=10 - - # Configure serial port to use as console. Ignore if you'll use VGA only - serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1 - - # Let the user select which console to use (serial or VGA), default - # to serial after 10s - terminal --timeout=10 serial console - - # An entry for NetBSD/xen, using /netbsd as the domain0 kernel, and serial - # console. Domain0 will have 64MB RAM allocated. - # Assume NetBSD is installed in the first MBR partition. - title Xen 3 / NetBSD (hda0, serial) - root(hd0,0) - kernel (hd0,a)/xen.gz dom0_mem=65536 com1=115200,8n1 - module (hd0,a)/netbsd bootdev=wd0a ro console=ttyS0 - - # Same as above, but using VGA console - # We can use console=tty0 (Linux syntax) or console=pc (NetBSD syntax) - title Xen 3 / NetBSD (hda0, vga) - root(hd0,0) - kernel (hd0,a)/xen.gz dom0_mem=65536 - module (hd0,a)/netbsd bootdev=wd0a ro console=tty0 - - # NetBSD/xen using a backup domain0 kernel (in case you installed a - # nonworking kernel as /netbsd - title Xen 3 / NetBSD (hda0, backup, serial) - root(hd0,0) - kernel (hd0,a)/xen.gz dom0_mem=65536 com1=115200,8n1 - module (hd0,a)/netbsd.backup bootdev=wd0a ro console=ttyS0 - title Xen 3 / NetBSD (hda0, backup, VGA) - root(hd0,0) - kernel (hd0,a)/xen.gz dom0_mem=65536 - module (hd0,a)/netbsd.backup bootdev=wd0a ro console=tty0 - - #Load a regular NetBSD/i386 kernel. Can be useful if you end up with a - #nonworking /xen.gz - title NetBSD 5.1 - root (hd0,a) - kernel --type=netbsd /netbsd-GENERIC - - #Load the NetBSD bootloader, letting it load the NetBSD/i386 kernel. - #May be better than the above, as grub can't pass all required infos - #to the NetBSD/i386 kernel (e.g. console, root device, ...) - title NetBSD chain - root (hd0,0) - chainloader +1 - - ## end of grub config file. - - -Install grub with the following command: - - # grub --no-floppy - - grub> root (hd0,a) - Filesystem type is ffs, partition type 0xa9 - - grub> setup (hd0) - Checking if "/boot/grub/stage1" exists... no - Checking if "/grub/stage1" exists... yes - Checking if "/grub/stage2" exists... yes - Checking if "/grub/ffs_stage1_5" exists... yes - Running "embed /grub/ffs_stage1_5 (hd0)"... 14 sectors are embedded. - succeeded - Running "install /grub/stage1 (hd0) (hd0)1+14 p (hd0,0,a)/grub/stage2 /grub/menu.lst"... - succeeded - Done. - +First, +[install NetBSD/amd64](../../docs/guide/en/chap-inst.html) +just as you would if you were not using Xen. +However, the partitioning approach is very important. + +If you want to use RAIDframe for the dom0, there are no special issues +for Xen. Typically one provides RAID storage for the dom0, and the +domU systems are unaware of RAID. The 2nd-stage loader bootxx_* skips +over a RAID1 header to find /boot from a filesystem within a RAID +partition; this is no different when booting Xen. + +There are 4 styles of providing backing storage for the virtual disks +used by domUs: raw partitions, LVM, file-backed vnd(4), and SAN, + +With raw partitions, one has a disklabel (or gpt) partition sized for +each virtual disk to be used by the domU. (If you are able to predict +how domU usage will evolve, please add an explanation to the HOWTO. +Seriously, needs tend to change over time.) + +One can use lvm(8) to create logical devices to use for domU disks. +This is almost as efficient sa raw disk partitions and more flexible. +Hence raw disk partitions should typically not be used. + +One can use files in the dom0 filesystem, typically created by dd'ing +/dev/zero to create a specific size. This is somewhat less efficient, +but very convenient, as one can cp the files for backup, or move them +between dom0 hosts. + +Finally, in theory one can place the files backing the domU disks in a +SAN. (This is an invitation for someone who has done this to add a +HOWTO page.) + +Installation of Xen +------------------- + +In the dom0, install sysutils/xenkernel42 and sysutils/xentools42 from +pkgsrc (or another matching pair). +See [the pkgsrc +documentation](http://www.NetBSD.org/docs/pkgsrc/) for help with pkgsrc. + +For Xen 3.1, support for HVM guests is in sysutils/xentool3-hvm. More +recent versions have HVM support integrated in the main xentools +package. It is entirely reasonable to run only PV guests. + +Next you need to install the selected Xen kernel itself, which is +installed by pkgsrc as "/usr/pkg/xen*-kernel/xen.gz". Copy it to /. +For debugging, one may copy xen-debug.gz; this is conceptually similar +to DIAGNOSTIC and DEBUG in NetBSD. xen-debug.gz is basically only +useful with a serial console. Then, place a NetBSD XEN3_DOM0 kernel +in /, copied from releasedir/amd64/binary/kernel/netbsd-XEN3_DOM0.gz +of a NetBSD build. Both xen and NetBSD may be left compressed. (If +using i386, use releasedir/i386/binary/kernel/netbsd-XEN3PAE_DOM0.gz.) + +In a dom0 kernel, kernfs is mandatory for xend to comunicate with the +kernel, so ensure that /kern is in fstab. + +Because you already installed NetBSD, you have a working boot setup +with an MBR bootblock, either bootxx_ffsv1 or bootxx_ffsv2 at the +beginning of your root filesystem, /boot present, and likely +/boot.cfg. (If not, fix before continuing!) + +See boot.cfg(5) for an example. The basic line is + +"menu=Xen:load /netbsd-XEN3_DOM0.gz console=pc;multiboot /xen.gz dom0_mem=256M" + +which specifies that the dom0 should have 256M, leaving the rest to be +allocated for domUs. + +As with non-Xen systems, you should have a line to boot /netbsd (a +kernel that works without Xen) and fallback versions of the non-Xen +kernel, Xen, and the dom0 kernel. + +Configuring Xen +--------------- + +Now, you have a system that will boot Xen and the dom0 kernel, and +just run the dom0 kernel. There will be no domUs, and none can be +started because you still have to configure the dom0 tools. + +For 3.3 (and probably 3.1), add to rc.conf (but note that you should +have installed 4.2): + xend=YES + xenbackendd=YES + +For 4.1 and 4.2, add to rc.conf: + xend=YES + xencommons=YES Updating NetBSD in a dom0 ------------------------- @@ -298,10 +275,22 @@ and adjusts /etc. Note that one must update both the non-Xen kernel typically used for rescue purposes and the DOM0 kernel used with Xen. +To convert from grub to /boot, install an mbr bootblock with fdisk, +bootxx_ with installboot, /boot and /boot.cfg. This really should be +no different than completely reinstalling boot blocks on a non-Xen +system. + Updating Xen versions --------------------- -TODO: write +Updating Xen is conceptually not difficult, but can run into all the +issues found when installing Xen. Assuming migration from 4.1 to 4.2, +remove the xenkernel41 and xentools41 packages and install the +xenkernel42 and xentools42 packages. Copy the 4.2 xen.gz to /. + +Ensure that the contents of /etc/rc.d/xen* are correct. Enable the +correct set of daemons. Ensure that the domU config files are valid +for the new version. Creating unprivileged domains (domU) ==================================== @@ -452,7 +441,7 @@ working vif-bridge is also provided with #!/bin/sh #============================================================================ - # $NetBSD: howto.mdwn,v 1.14 2014/12/23 23:43:27 gdt Exp $ + # $NetBSD: howto.mdwn,v 1.22 2014/12/24 01:27:36 gdt Exp $ # # /usr/pkg/etc/xen/vif-bridge #