--- wikisrc/ports/xen/howto.mdwn 2014/12/24 01:38:26 1.26 +++ wikisrc/ports/xen/howto.mdwn 2014/12/26 20:00:44 1.48 @@ -27,14 +27,14 @@ code for Xen and need not be aware that Attempts to access hardware registers are trapped and emulated. This style is less efficient but can run unmodified guests. -Generally any amd64 machine will work with Xen and PV guests. For HVM -guests, the VT or VMX cpu feature (Intel) or SVM/HVM/VT (amd64) is -needed; "cpuctl identify 0" will show this. TODO: Clean up and check -the above features. TODO: Explain if i386 (non-amd64) machines can -still be used --- I think that the requirement to use PAE kernels is -about the hypervisor being amd64 only. +Generally any amd64 machine will work with Xen and PV guests. In +theory i386 computers without amd64 support can be used for Xen <= +4.2, but we have no recent reports of this working (this is a hint). +For HVM guests, the VT or VMX cpu feature (Intel) or SVM/HVM/VT +(amd64) is needed; "cpuctl identify 0" will show this. TODO: Clean up +and check the above features. -At boot, the dom0 kernel is loaded as module with Xen as the kernel. +At boot, the dom0 kernel is loaded as a module with Xen as the kernel. The dom0 can start one or more domUs. (Booting is explained in detail in the dom0 section.) @@ -61,7 +61,7 @@ path when there are no known good reason This HOWTO presumes a basic familiarity with the Xen system architecture. This HOWTO presumes familiarity with installing NetBSD on i386/amd64 hardware and installing software from pkgsrc. -See also the [Xen website](http://www.xen.org/). +See also the [Xen website](http://www.xenproject.org/). History ------- @@ -70,15 +70,15 @@ NetBSD used to support Xen2; this has be Before NetBSD's native bootloader could support Xen, the use of grub was recommended. If necessary, see the -[old grub information](/xen/howto-grub/). +[old grub information](/ports/xen/howto-grub/). Versions of Xen and NetBSD ========================== -Most of the installation concepts and instructions are independent of -Xen version. This section gives advice on which version to choose. -Versions not in pkgsrc and older unsupported versions of NetBSD are -intentionally ignored. +Most of the installation concepts and instructions are independent +of Xen version and NetBSD version. This section gives advice on +which version to choose. Versions not in pkgsrc and older unsupported +versions of NetBSD are intentionally ignored. Xen --- @@ -90,7 +90,7 @@ matching versions. xenkernel3 and xenkernel33 provide Xen 3.1 and 3.3. These no longer receive security patches and should not be used. Xen 3.1 supports PCI -passthrough. +passthrough. Xen 3.1 supports non-PAE on i386. xenkernel41 provides Xen 4.1. This is no longer maintained by Xen, but as of 2014-12 receives backported security patches. It is a @@ -109,7 +109,8 @@ Xen command program Early Xen used a program called "xm" to manipulate the system from the dom0. Starting in 4.1, a replacement program with similar behavior -called "xl" is provided. In 4.2, "xm" is no longer available. +called "xl" is provided. In 4.2 and later, "xl" is preferred. 4.4 is +the last version that has "xm". NetBSD ------ @@ -117,7 +118,9 @@ NetBSD The netbsd-5, netbsd-6, netbsd-7, and -current branches are all reasonable choices, with more or less the same considerations for non-Xen use. Therefore, netbsd-6 is recommended as the stable version -of the most recent release. +of the most recent release for production use. For those wanting to +learn Xen or without production stability concerns, netbsd-7 is likely +most appropriate. As of NetBSD 6, a NetBSD domU will support multiple vcpus. There is no SMP support for NetBSD as dom0. (The dom0 itself doesn't really @@ -127,20 +130,45 @@ a normal computer.) Architecture ------------ -Xen is basically amd64 only at this point. One can either run i386 -domains or amd64 domains. If running i386, PAE versions are required, -for both dom0 and domU. These versions are built by default in NetBSD -releases. While i386 dom0 works fine, amd64 is recommended as more -normal. (Note that emacs (at least) fails if run on i386 with PAE when -built without, and vice versa, presumably due to bugs in the undump -code.) +Xen itself can run on i386 or amd64 machines. (Practically, almost +any computer where one would want to run Xen supports amd64.) If +using an i386 NetBSD kernel for the dom0, PAE is required (PAE +versions are built by default). While i386 dom0 works fine, amd64 is +recommended as more normal. + +Xen 4.2 is the last version to support i386 as a host. TODO: Clarify +if this is about the CPU having to be amd64, or about the dom0 kernel +having to be amd64. + +One can then run i386 domUs and amd64 domUs, in any combination. If +running an i386 NetBSD kernel as a domU, the PAE version is required. +(Note that emacs (at least) fails if run on i386 with PAE when built +without, and vice versa, presumably due to bugs in the undump code.) Recommendation -------------- Therefore, this HOWTO recommends running xenkernel42 (and xentools42), -xl, the NetBSD 6 stable branch, and to use amd64 as the dom0. Either -the i386 or amd64 of NetBSD may be used as domUs. +xl, the NetBSD 6 stable branch, and to use an amd64 kernel as the +dom0. Either the i386 or amd64 of NetBSD may be used as domUs. + +Build problems +-------------- + +Ideally, all versions of Xen in pkgsrc would build on all versions of +NetBSD on both i386 and amd64. However, that isn't the case. Besides +aging code and aging compilers, qemu (included in xentools for HVM +support) is difficult to build. The following are known to fail: + + xenkernel3 netbsd-6 i386 + xentools42 netbsd-6 i386 + +The following are known to work: + + xenkernel41 netbsd-5 amd64 + xentools41 netbsd-5 amd64 + xenkernel41 netbsd-6 i386 + xentools41 netbsd-6 i386 NetBSD as a dom0 ================ @@ -153,6 +181,10 @@ NetBSD, which is not yet a dom0, and the NetBSD install to a dom0 install by just changing the kernel and boot configuration. +For experimenting with Xen, a machine with as little as 1G of RAM and +100G of disk can work. For running many domUs in productions, far +more will be needed. + Styles of dom0 operation ------------------------ @@ -178,7 +210,7 @@ Installation of NetBSD ---------------------- First, -[install NetBSD/amd64](../../docs/guide/en/chap-inst.html) +[install NetBSD/amd64](/guide/inst/) just as you would if you were not using Xen. However, the partitioning approach is very important. @@ -196,9 +228,10 @@ each virtual disk to be used by the domU how domU usage will evolve, please add an explanation to the HOWTO. Seriously, needs tend to change over time.) -One can use lvm(8) to create logical devices to use for domU disks. -This is almost as efficient sa raw disk partitions and more flexible. -Hence raw disk partitions should typically not be used. +One can use [lvm(8)](/guide/lvm/) to create logical devices to use +for domU disks. This is almost as efficient as raw disk partitions +and more flexible. Hence raw disk partitions should typically not +be used. One can use files in the dom0 filesystem, typically created by dd'ing /dev/zero to create a specific size. This is somewhat less efficient, @@ -240,30 +273,112 @@ beginning of your root filesystem, /boot See boot.cfg(5) for an example. The basic line is -"menu=Xen:load /netbsd-XEN3_DOM0.gz console=pc;multiboot /xen.gz dom0_mem=256M" + menu=Xen:load /netbsd-XEN3_DOM0.gz console=pc;multiboot /xen.gz dom0_mem=256M which specifies that the dom0 should have 256M, leaving the rest to be -allocated for domUs. +allocated for domUs. In an attempt to add performance, one can also +add + + dom0_max_vcpus=1 dom0_vcpus_pin + +to force only one vcpu to be provided (since NetBSD dom0 can't use +more) and to pin that vcpu to a physical cpu. TODO: benchmark this. As with non-Xen systems, you should have a line to boot /netbsd (a kernel that works without Xen) and fallback versions of the non-Xen kernel, Xen, and the dom0 kernel. +The [HowTo on Installing into +RAID-1](http://mail-index.NetBSD.org/port-xen/2006/03/01/0010.html) +explains how to set up booting a dom0 with Xen using grub with +NetBSD's RAIDframe. (This is obsolete with the use of NetBSD's native +boot.) + Configuring Xen --------------- Now, you have a system that will boot Xen and the dom0 kernel, and just run the dom0 kernel. There will be no domUs, and none can be -started because you still have to configure the dom0 tools. +started because you still have to configure the dom0 tools. The +daemons which should be run vary with Xen version and with whether one +is using xm or xl. Note that xend is for supporting "xm", and should +only be used if you plan on using "xm". Do NOT enable xend if you +plan on using "xl" as it will cause problems. + +The installation of NetBSD should already have created devices for xen +(xencons, xenevt), but if they are not present, create them: + + cd /dev && sh MAKEDEV xen + +TODO: Give 3.1 advice (or remove it from pkgsrc). + +For 3.3 (and thus xm), add to rc.conf (but note that you should have +installed 4.1 or 4.2): + + xend=YES + xenbackendd=YES + +For 4.1 (and thus xm; xl is believed not to work well), add to rc.conf: + + xend=YES + xencommons=YES + +TODO: Explain why if xm is preferred on 4.1, rc.d/xendomains has xl. +Or fix the package. + +For 4.2 with xm, add to rc.conf + + xend=YES + xencommons=YES + +For 4.2 with xl (preferred), add to rc.conf: + + TODO: explain if there is a xend replacement + xencommons=YES + +TODO: Recommend for/against xen-watchdog. + +After you have configured the daemons and either started them or +rebooted, run the following (or use xl) to inspect Xen's boot +messages, available resources, and running domains: + + # xm dmesg + [xen's boot info] + # xm info + [available memory, etc.] + # xm list + Name Id Mem(MB) CPU State Time(s) Console + Domain-0 0 64 0 r---- 58.1 + +anita (for testing NetBSD) +-------------------------- + +With the setup so far, one should be able to run anita (see +pkgsrc/sysutils/py-anita) to test NetBSD releases, by doing (as root, +because anita must create a domU): + + anita --vmm=xm test file:///usr/obj/i386/ + +Alternatively, one can use --vmm=xl to use xl-based domU creation instead. +TODO: check this. + +Xen-specific NetBSD issues +-------------------------- + +There are (at least) two additional things different about NetBSD as a +dom0 kernel compared to hardware. + +One is that modules are not usable in DOM0 kernels, so one must +compile in what's needed. It's not really that modules cannot work, +but that modules must be built for XEN3_DOM0 because some of the +defines change and the normal module builds don't do this. Basically, +enabling Xen changes the kernel ABI, and the module build system +doesn't cope with this. -For 3.3 (and probably 3.1), add to rc.conf (but note that you should -have installed 4.2): - xend=YES - xenbackendd=YES - -For 4.1 and 4.2, add to rc.conf: - xend=YES - xencommons=YES +The other difference is that XEN3_DOM0 does not have exactly the same +options as GENERIC. While it is debatable whether or not this is a +bug, users should be aware of this and can simply add missing config +items if desired. Updating NetBSD in a dom0 ------------------------- @@ -293,39 +408,217 @@ Ensure that the contents of /etc/rc.d/xe correct set of daemons. Ensure that the domU config files are valid for the new version. -Creating unprivileged domains (domU) -==================================== + +Unprivileged domains (domU) +=========================== + +This section describes general concepts about domUs. It does not +address specific domU operating systems or how to install them. The +config files for domUs are typically in /usr/pkg/etc/xen, and are +typically named so that the file anme, domU name and the domU's host +name match. + +The domU is provided with cpu and memory by Xen, configured by the +dom0. The domU is provided with disk and network by the dom0, +mediated by Xen, and configured in the dom0. + +Entropy in domUs can be an issue; physical disks and network are on +the dom0. NetBSD's /dev/random system works, but is often challenged. + +Config files +------------ + +There is no good order to present config files and the concepts +surrounding what is being configured. We first show an example config +file, and then in the various sections give details. + +See (at least in xentools41) /usr/pkg/share/examples/xen/xmexample*, +for a large number of well-commented examples, mostly for running +GNU/Linux. + +The following is an example minimal domain configuration file +"/usr/pkg/etc/xen/foo". It is (with only a name change) an actual +known working config file on Xen 4.1 (NetBSD 5 amd64 dom0 and NetBSD 5 +i386 domU). The domU serves as a network file server. + + # -*- mode: python; -*- + + kernel = "/netbsd-XEN3PAE_DOMU-i386-foo.gz" + memory = 1024 + vif = [ 'mac=aa:00:00:d1:00:09,bridge=bridge0' ] + disk = [ 'file:/n0/xen/foo-wd0,0x0,w', + 'file:/n0/xen/foo-wd1,0x1,w' ] + +The domain will have the same name as the file. The kernel has the +host/domU name in it, so that on the dom0 one can update the various +domUs independently. The vif line causes an interface to be provided, +with a specific mac address (do not reuse MAC addresses!), in bridge +mode. Two disks are provided, and they are both writable; the bits +are stored in files and Xen attaches them to a vnd(4) device in the +dom0 on domain creation. The system treates xbd0 as the boot device +without needing explicit configuration. + +By default xm looks for domain config files in /usr/pkg/etc/xen. Note +that "xm create" takes the name of a config file, while other commands +take the name of a domain. To create the domain, connect to the +console, create the domain while attaching the console, shutdown the +domain, and see if it has finished stopping, do (or xl with Xen >= +4.2): + + xm create foo + xm console foo + xm create -c foo + xm shutdown foo + xm list + +Typing ^] will exit the console session. Shutting down a domain is +equivalent to pushing the power button; a NetBSD domU will receive a +power-press event and do a clean shutdown. Shutting down the dom0 +will trigger controlled shutdowns of all configured domUs. + +domU kernels +------------ + +On a physical computer, the BIOS reads sector 0, and a chain of boot +loaders finds and loads a kernel. Normally this comes from the root +filesystem. With Xen domUs, the process is totally different. The +normal path is for the domU kernel to be a file in the dom0's +filesystem. At the request of the dom0, Xen loads that kernel into a +new domU instance and starts execution. While domU kernels can be +anyplace, reasonable places to store domU kernels on the dom0 are in / +(so they are near the dom0 kernel), in /usr/pkg/etc/xen (near the +config files), or in /u0/xen (where the vdisks are). + +See the VPS section near the end for discussion of alternate ways to +obtain domU kernels. + +CPU and memory +-------------- + +A domain is provided with some number of vcpus, less than the number +of cpus seen by the hypervisor. (For a dom0, this is controlled by +the boot argument "dom0_max_vcpus=1".) For a domU, it is controlled +from the config file by the "vcpus = N" directive. + +A domain is provided with memory; this is controlled in the config +file by "memory = N" (in megabytes). In the straightforward case, the +sum of the the memory allocated to the dom0 and all domUs must be less +than the available memory. + +Xen also provides a "balloon" driver, which can be used to let domains +use more memory temporarily. TODO: Explain better, and explain how +well it works with NetBSD. + +Virtual disks +------------- + +With the file/vnd style, typically one creates a directory, +e.g. /u0/xen, on a disk large enough to hold virtual disks for all +domUs. Then, for each domU disk, one writes zeros to a file that then +serves to hold the virtual disk's bits; a suggested name is foo-xbd0 +for the first virtual disk for the domU called foo. Writing zeros to +the file serves two purposes. One is that preallocating the contents +improves performance. The other is that vnd on sparse files has +failed to work. TODO: give working/notworking NetBSD versions for +sparse vnd. Note that the use of file/vnd for Xen is not really +different than creating a file-backed virtual disk for some other +purpose, except that xentools handles the vnconfig commands. To +create an empty 4G virtual disk, simply do + + dd if=/dev/zero of=foo-xbd0 bs=1m count=4096 + +With the lvm style, one creates logical devices. They are then used +similarly to vnds. TODO: Add an example with lvm. + +In domU config files, the disks are defined as a sequence of 3-tuples. +The first element is "method:/path/to/disk". Common methods are +"file:" for file-backed vnd. and "phy:" for something that is already +a (TODO: character or block) device. + +The second element is an artifact of how virtual disks are passed to +Linux, and a source of confusion with NetBSD Xen usage. Linux domUs +are given a device name to associate with the disk, and values like +"hda1" or "sda1" are common. In a NetBSD domU, the first disk appears +as xbd0, the second as xbd1, and so on. However, xm/xl demand a +second argument. The name given is converted to a major/minor by +consulting /dev and this is passed to the domU (TODO: check this). In +the general case, the dom0 and domU can be different operating +systems, and it is an unwarranted assumption that they have consistent +numbering in /dev, or even that the dom0 OS has a /dev. With NetBSD +as both dom0 and domU, using values of 0x0 for the first disk and 0x1 +for the second works fine and avoids this issue. + +The third element is "w" for writable disks, and "r" for read-only +disks. + +Virtual Networking +------------------ + +Xen provides virtual ethernets, each of which connects the dom0 and a +domU. For each virtual network, there is an interface "xvifN.M" in +the dom0, and in domU index N, a matching interface xennetM (NetBSD +name). The interfaces behave as if there is an Ethernet with two +adaptors connected. From this primitive, one can construct various +configurations. We focus on two common and useful cases for which +there are existing scripts: bridging and NAT. + +With bridging (in the example above), the domU perceives itself to be +on the same network as the dom0. For server virtualization, this is +usually best. Bridging is accomplished by creating a bridge(4) device +and adding the dom0's physical interface and the various xvifN.0 +interfaces to the bridge. One specifies "bridge=bridge0" in the domU +config file. The bridge must be set up already in the dom0; an +example /etc/ifconfig.bridge0 is: + + create + up + !brconfig bridge0 add wm0 + +With NAT, the domU perceives itself to be behind a NAT running on the +dom0. This is often appropriate when running Xen on a workstation. +TODO: NAT appears to be configured by "vif = [ '' ]". + +Sizing domains +-------------- + +Modern x86 hardware has vast amounts of resources. However, many +virtual servers can function just fine on far less. A system with +256M of RAM and a 4G disk can be a reasonable choice. Note that it is +far easier to adjust virtual resources than physical ones. For +memory, it's just a config file edit and a reboot. For disk, one can +create a new file and vnconfig it (or lvm), and then dump/restore, +just like updating physical disks, but without having to be there and +without those pesky connectors. + +Starting domains automatically +------------------------------ + +To start domains foo at bar at boot and shut them down cleanly on dom0 +shutdown, in rc.conf add: + + xendomains="foo bar" + +TODO: Explain why 4.1 rc.d/xendomains has xl, when one should use xm +on 4.1. Or fix the xentools41 package to have xm + +Creating specific unprivileged domains (domU) +============================================= Creating domUs is almost entirely independent of operating system. We first explain NetBSD, and then differences for Linux and Solaris. +Note that you must have already completed the dom0 setup so that "xm +list" (or "xl list") works. Creating an unprivileged NetBSD domain (domU) --------------------------------------------- -Once you have *domain0* running, you need to start the xen tool daemon -(`/usr/pkg/share/examples/rc.d/xend start`) and the xen backend daemon -(`/usr/pkg/share/examples/rc.d/xenbackendd start` for Xen3\*, -`/usr/pkg/share/examples/rc.d/xencommons start` for Xen4.\*). Make sure -that `/dev/xencons` and `/dev/xenevt` exist before starting `xend`. You -can create them with this command: - - # cd /dev && sh MAKEDEV xen - -xend will write logs to `/var/log/xend.log` and -`/var/log/xend-debug.log`. You can then control xen with the xm tool. -'xm list' will show something like: - - # xm list - Name Id Mem(MB) CPU State Time(s) Console - Domain-0 0 64 0 r---- 58.1 - 'xm create' allows you to create a new domain. It uses a config file in PKG\_SYSCONFDIR for its parameters. By default, this file will be in `/usr/pkg/etc/xen/`. On creation, a kernel has to be specified, which will be executed in the new domain (this kernel is in the *domain0* file system, not on the new domain virtual disk; but please note, you should install the same kernel into *domainU* as `/netbsd` in order to make -your system tools, like MAN.SAVECORE.8, work). A suitable kernel is +your system tools, like savecore(8), work). A suitable kernel is provided as part of the i386 and amd64 binary sets: XEN3\_DOMU. Here is an /usr/pkg/etc/xen/nbsd example config file: @@ -435,14 +728,14 @@ like this: !brconfig $int add ex0 up (replace `ex0` with the name of your physical interface). Then bridge0 -will be created on boot. See the MAN.BRIDGE.4 man page for details. +will be created on boot. See the bridge(4) man page for details. So, here is a suitable `/usr/pkg/etc/xen/vif-bridge` for xvif?.? (a working vif-bridge is also provided with xentools20) configuring: #!/bin/sh #============================================================================ - # $NetBSD: howto.mdwn,v 1.25 2014/12/24 01:37:30 gdt Exp $ + # $NetBSD: howto.mdwn,v 1.47 2014/12/26 18:35:45 gdt Exp $ # # /usr/pkg/etc/xen/vif-bridge # @@ -819,13 +1112,25 @@ to use PCI devices in a domU. Here's a k sd* at scsibus? target ? lun ? # SCSI disk drives cd* at scsibus? target ? lun ? # SCSI CD-ROM drives -Links and further information -============================= -- The [HowTo on Installing into RAID-1](http://mail-index.NetBSD.org/port-xen/2006/03/01/0010.html) - explains how to set up booting a dom0 with Xen using grub - with NetBSD's RAIDframe. (This is obsolete with the use of - NetBSD's native boot.) -- An example of how to use NetBSD's native bootloader to load - NetBSD/Xen instead of Grub can be found in the i386/amd64 boot(8) - and boot.cfg(5) manpages. +NetBSD as a domU in a VPS +========================= + +The bulk of the HOWTO is about using NetBSD as a dom0 on your own +hardware. This section explains how to deal with Xen in a domU as a +virtual private server where you do not control or have access to the +dom0. + +TODO: Perhaps reference panix, prmgr, amazon as interesting examples. + +TODO: Somewhere, discuss pvgrub and py-grub to load the domU kernel +from the domU filesystem. + +Using npf +--------- + +In standard kernels, npf is a module, and thus cannot be loadeed in a +DOMU kernel. + +TODO: explain how to compile npf into a custom kernel, answering: +http://mail-index.netbsd.org/netbsd-users/2014/12/26/msg015576.html