--- wikisrc/ports/xen/howto.mdwn 2015/01/17 13:04:01 1.80 +++ wikisrc/ports/xen/howto.mdwn 2015/03/05 13:27:37 1.93 @@ -92,10 +92,13 @@ reasonable although trailing-edge choice xenkernel42 provides Xen 4.2. This is maintained by Xen, but old as of 2014-12. +xenkernel45 provides Xen 4.5. This is new to pkgsrc as of 2015-01 and +not yet recommended for other than experimental/testing use. + Ideally newer versions of Xen will be added to pkgsrc. -Note that NetBSD support is called XEN3. It works with 3.1 through -4.2 because the hypercall interface has been stable. +Note that NetBSD support is called XEN3. It works with Xen 3 and Xen +4 because the hypercall interface has been stable. Xen command program ------------------- @@ -139,6 +142,18 @@ running an i386 NetBSD kernel as a domU, (Note that emacs (at least) fails if run on i386 with PAE when built without, and vice versa, presumably due to bugs in the undump code.) +Stability +--------- + +Mostly, NetBSD as a dom0 or domU is quite stable. +However, there are some open PRs indicating problems. + + - [PR 48125](http://gnats.netbsd.org/48125) + - [PR 47720](http://gnats.netbsd.org/47720) + +Note also that there are issues with sparse vnd(4) instances, but +these are not about Xen. + Recommendation -------------- @@ -211,7 +226,7 @@ dom0 is what the computer would have bee desktop or laptop. Then, one can run domUs at will. Purists will deride this as less secure than the previous approach, and for a computer whose purpose is to run domUs, they are right. But Xen and a -dom0 (without domUs) is not meaingfully less secure than the same +dom0 (without domUs) is not meaningfully less secure than the same things running without Xen. One can boot Xen or boot regular NetBSD alternately with little problems, simply refraining from starting the Xen daemons when not running Xen. @@ -236,7 +251,7 @@ over a RAID1 header to find /boot from a partition; this is no different when booting Xen. There are 4 styles of providing backing storage for the virtual disks -used by domUs: raw partitions, LVM, file-backed vnd(4), and SAN, +used by domUs: raw partitions, LVM, file-backed vnd(4), and SAN. With raw partitions, one has a disklabel (or gpt) partition sized for each virtual disk to be used by the domU. (If you are able to predict @@ -309,6 +324,11 @@ starting at 0). In an attempt to add pe to force only one vcpu to be provided (since NetBSD dom0 can't use more) and to pin that vcpu to a physical cpu. TODO: benchmark this. +Xen has [many boot +options](http://xenbits.xenproject.org/docs/4.5-testing/misc/xen-command-line.html), +and other tham dom0 memory and max_vcpus, they are generally not +necessary. + As with non-Xen systems, you should have a line to boot /netbsd (a kernel that works without Xen) and fallback versions of the non-Xen kernel, Xen, and the dom0 kernel. @@ -341,7 +361,8 @@ to configure the dom0 tools. The daemon with Xen version and with whether one is using xm or xl. Note that xend is for supporting "xm", and should only be used if you plan on using "xm". Do NOT enable xend if you plan on using "xl" as it will -cause problems. +cause problems. Running xl without xencommons=YES (and starting it) +will result in a hang (so don't do that; follow the HOWTO!). The installation of NetBSD should already have created devices for xen (xencons, xenevt), but if they are not present, create them: @@ -374,6 +395,11 @@ For 4.2 with xl, add to rc.conf: xencommons=YES TODO: explain if there is a xend replacement +For 4.5 (and thus with xl), add to rc.conf: + + xencommons=YES + TODO: explain if there is a xend replacement + TODO: Recommend for/against xen-watchdog. After you have configured the daemons and either started them (in the @@ -389,21 +415,33 @@ available resources, and running domains Domain-0 0 64 0 r---- 58.1 With xl, the commands are the same, and the output may be slightly -different. TODO: add example output for xl, after confirming on 4.2 -and resolving the TODO about rc.conf. +different. TODO: add example output for xl before the xm example, +after confirming on 4.2 and resolving the TODO about rc.conf. -anita (for testing NetBSD) --------------------------- +### Issues with xencommons -With the setup so far, one should be able to run anita (see -pkgsrc/misc/py-anita) to test NetBSD releases, by doing (as root, -because anita must create a domU): +xencommons starts xenstored, which stores data on behalf of dom0 and +domUs. It does not currently work to stop and start xenstored. +Certainly all domUs should be shutdown first, following the sort order +of the rc.d scripts. However, the dom0 sets up state with xenstored, +and is not notified when xenstored exits, leading to not recreating +the state when the new xenstored starts. Until there's a mechanism to +make this work, one should not expect to be able to restart xenstored +(and thus xencommons). There is currently no reason to expect that +this will get fixed any time soon. - anita --vmm=xm test file:///usr/obj/i386/ - -Alternatively, one can use --vmm=xl to use xl-based domU creation instead. -TODO: check this, and make the example use xl when confirmed. +anita (for testing NetBSD) +-------------------------- +With the setup so far (assuming 4.2/xl), one should be able to run +anita (see pkgsrc/misc/py-anita) to test NetBSD releases, by doing (as +root, because anita must create a domU): + + anita --vmm=xl test file:///usr/obj/i386/ + +Alternatively, one can use --vmm=xm to use xm-based domU creation +instead (and must, on Xen <= 4.1). TODO: confirm that anita xl really works. + Xen-specific NetBSD issues -------------------------- @@ -477,6 +515,25 @@ correct set of daemons. Ensure that the for the new version. +Running Xen under qemu +---------------------- + +The astute reader will note that this section is somewhat twisted. +However, it can be useful to run Xen under qemu either because the +version of NetBSD as a dom0 does not run on the hardware in use, or to +generate automated test cases involving Xen. + +In 2015-01, the following combination was reported to mostly work: + + host OS: NetBSD/amd64 6.1.4 + qemu: 2.2.0 from pkgsrc + Xen kernel: xenkernel42-4.2.5nb1 from pkgsrc + dom0 kernel: NetBSD/amd64 6.1.5 + Xen tools: xentools42-4.2.5 from pkgsrc + +See [PR 47720](http://gnats.netbsd.org/47720) for a problem with dom0 +shutdown. + Unprivileged domains (domU) =========================== @@ -537,7 +594,7 @@ domain, and see if it has finished stopp xm console foo xm create -c foo xm shutdown foo - xm list + xm list Typing ^] will exit the console session. Shutting down a domain is equivalent to pushing the power button; a NetBSD domU will receive a @@ -597,6 +654,10 @@ create an empty 4G virtual disk, simply dd if=/dev/zero of=foo-xbd0 bs=1m count=4096 +Do not use qemu-img-xen, because this will create sparse file. There +have been recent (2015) reports of sparse vnd(4) devices causing +lockups, but there is apparently no PR. + With the lvm style, one creates logical devices. They are then used similarly to vnds. TODO: Add an example with lvm. @@ -674,8 +735,8 @@ shutdown, in rc.conf add: xendomains="foo bar" -TODO: Explain why 4.1 rc.d/xendomains has xl, when one should use xm -on 4.1. Or fix the xentools41 package to have xm +Note that earlier versions of the xentools41 xendomains rc.d scripth +usd xl, when one should use xm with 4.1. Creating specific unprivileged domains (domU) ============================================= @@ -753,6 +814,14 @@ It is also desirable to add in rc.conf. This way, the domain will be properly shut down if `xm shutdown -R` or `xm shutdown -H` is used on the dom0. +It is not strictly necessary to have a kernel (as /netbsd) in the domU +filesystem. However, various programs (e.g. netstat) will use that +kernel to look up symbols to read from kernel virtual memory. If +/netbsd is not the running kernel, those lookups will fail. (This is +not really a Xen-specific issue, but because the domU kernel is +obtained from the dom0, it is far more likely to be out of sync or +missing with Xen.) + Your domain should be now ready to work, enjoy. Creating an unprivileged Linux domain (domU) @@ -946,7 +1015,7 @@ TODO: add link to NetBSD amazon howto. Using npf --------- -In standard kernels, npf is a module, and thus cannot be loadeed in a +In standard kernels, npf is a module, and thus cannot be loaded in a DOMU kernel. TODO: explain how to compile npf into a custom kernel, answering (but @@ -956,8 +1025,10 @@ http://mail-index.netbsd.org/netbsd-user TODO items for improving NetBSD/xen =================================== -* Package Xen 4.4. -* Get PCI passthrough working on Xen 4.2 (or 4.4). +* Make the NetBSD dom0 kernel work with SMP. +* Test the Xen 4.5 packages adequately to be able to recommend them as + the standard approach. +* Get PCI passthrough working on Xen 4.5 * Get pvgrub into pkgsrc, either via xentools or separately. * grub * Check/add support to pkgsrc grub2 for UFS2 and arbitrary @@ -969,3 +1040,14 @@ TODO items for improving NetBSD/xen * Get UFS2 patches into pvgrub. * Add support for PV ops to a version of /boot, and make it usable as a kernel in Xen, similar to pvgrub. +* Solve somehow the issue with modules for GENERIC not being loadable + in a Xen dom0 or domU kernel. + +Random pointers +=============== + +TODO: This section contains links from elsewhere not yet integrated +into the HOWTO. + +* http://www.lumbercartel.ca/library/xen/ +* http://pbraun.nethence.com/doc/sysutils/xen_netbsd_dom0.html