[NetBSD Developer Wiki]
- view: text
- select for diffs
Thu Jul 26 10:36:26 2018 UTC
(22 months, 1 week ago) by maxv
CVS tags: HEAD
Reduce the noise:
- Sum up the "Xen" section in a single, readable table.
- Remove the "Status" section, it is pointless and probably
- Remove the "Using npf" section. We do have support for kernel
modules now, and basically, that kind of stuff shouldn't be in
a Xen howto.
- Remove the "missing kernel modules support" from the TODO list,
we do have that now. By the way, we should probably remove this
TODO section. This page is supposed to be a HowTo, not a page
where we put some random things.
1: [[!meta title="Xen HowTo"]]
9: Xen is a hypervisor (or virtual machine monitor) for x86 hardware
10: (i686-class or higher), which supports running multiple guest
11: operating systems on a single physical machine. Xen is a Type 1 or
12: bare-metal hypervisor; one uses the Xen kernel to control the CPU,
13: memory and console, a dom0 operating system which mediates access to
14: other hardware (e.g., disks, network, USB), and one or more domU
15: operating systems which operate in an unprivileged virtualized
16: environment. IO requests from the domU systems are forwarded by the
17: hypervisor (Xen) to the dom0 to be fulfilled.
19: Xen supports two styles of guests. The original is Para-Virtualized
20: (PV) which means that the guest OS does not attempt to access hardware
21: directly, but instead makes hypercalls to the hypervisor. This is
22: analogous to a user-space program making system calls. (The dom0
23: operating system uses PV calls for some functions, such as updating
24: memory mapping page tables, but has direct hardware access for disk
25: and network.) PV guests must be specifically coded for Xen.
27: The more recent style is HVM, which means that the guest does not have
28: code for Xen and need not be aware that it is running under Xen.
29: Attempts to access hardware registers are trapped and emulated. This
30: style is less efficient but can run unmodified guests.
32: Generally any machine that runs NetBSD/amd64 will work with Xen and PV
33: guests. In theory i386 computers (without x86_64/amd64 support) can
34: be used for Xen <= 4.2, but we have no recent reports of this working
35: (this is a hint). For HVM guests, hardware support is needed, but it
36: is common on recent machines. For Intel CPUs, one needs the VT-x
37: extension, shown in "cpuctl identify 0" as VMX. For AMD CPUs, one
38: needs the AMD-V extensions, shown in "cpuctl identify 0" as SVM.
39: There are further features for IOMMU virtualization, Intel's VT-d and
40: AMD's AMD-Vi. TODO: Explain whether Xen on NetBSD makes use of these
41: features. TODO: Review by someone who really understands this.
43: Note that a FreeBSD dom0 requires VT-x and VT-d (or equivalent); this
44: is because the FreeBSD dom0 does not run in PV mode.
46: At boot, the dom0 kernel is loaded as a module with Xen as the kernel.
47: The dom0 can start one or more domUs. (Booting is explained in detail
48: in the dom0 section.)
50: NetBSD supports Xen in that it can serve as dom0, be used as a domU,
51: and that Xen kernels and tools are available in pkgsrc. This HOWTO
52: attempts to address both the case of running a NetBSD dom0 on hardware
53: and running domUs under it (NetBSD and other), and also running NetBSD
54: as a domU in a VPS.
56: Xen 3.1 in pkgsrc used to support "PCI passthrough", which means that
57: specific PCI devices can be made available to a specific domU instead
58: of the dom0. This can be useful to let a domU run X11, or access some
59: network interface or other peripheral.
61: NetBSD 6 and earlier supported Xen 2; support was removed from NetBSD
62: 7. Xen 2 has been removed from pkgsrc.
67: Installing NetBSD/Xen is not extremely difficult, but it is more
68: complex than a normal installation of NetBSD.
69: In general, this HOWTO is occasionally overly restrictive about how
70: things must be done, guiding the reader to stay on the established
71: path when there are no known good reasons to stray.
73: This HOWTO presumes a basic familiarity with the Xen system
74: architecture, with installing NetBSD on i386/amd64 hardware, and with
75: installing software from pkgsrc. See also the [Xen
78: Versions of Xen and NetBSD
81: Most of the installation concepts and instructions are independent
82: of Xen version and NetBSD version. This section gives advice on
83: which version to choose. Versions not in pkgsrc and older unsupported
84: versions of NetBSD are intentionally ignored.
86: The term "amd64" is used to refer to both the NetBSD port and to the
87: hardware architecture on which it runs. (Such hardware is made by
88: both Intel and AMD, and in 2016 a normal PC has this CPU
94: In NetBSD, Xen is provided in pkgsrc, via matching pairs of packages
95: xenkernel and xentools. We will refer only to the kernel versions,
96: but note that both packages must be installed together and must have
97: matching versions.
99: Versions available in pkgsrc:
101: [[!table data="""
102: Xen Version |Package Name |Xen CPU Support |EOL'ed By Upstream
103: 4.2 |xenkernel42 |32bit, 64bit |Yes
104: 4.5 |xenkernel45 |64bit |Yes
105: 4.6 |xenkernel46 |64bit |Partially
106: 4.8 |xenkernel48 |64bit |No
107: 4.11 |xenkernel411 |64bit |No
110: See also the [Xen Security Advisory page](http://xenbits.xen.org/xsa/).
112: Note: Xen 4.2 was the last version to support 32bit CPUs.
114: Note that NetBSD support is called XEN3. It works with Xen 3 and Xen
115: 4 because the hypercall interface has been stable.
117: Xen command program
120: Early Xen used a program called xm to manipulate the system from the
121: dom0. Starting in 4.1, a replacement program with similar behavior
122: called xl is provided, but it does not work well in 4.1. In 4.2, both
123: xm and xl work fine. 4.4 is the last version that has xm.
125: You must make a global choice to use xm or xl, because it affects not
126: only which command you use, but the command used by rc.d scripts
127: (specifically xendomains) and which daemons should be run. The
128: xentools packages provide xm for 3.1, 3.3 and 4.1 and xl for 4.2 and up.
130: In 4.2, you can choose to use xm by simply changing the ctl_command
131: variable and setting xend=YES in rc.conf.
133: With xl, virtual devices are configured in parallel, which can cause
134: problems if they are written assuming serial operation (e.g., updating
135: firewall rules without explicit locking). There is now locking for
136: the provided scripts, which works for normal casses (e.g, file-backed
137: xbd, where a vnd must be allocated). But, as of 201612, it has not
138: been adequately tested for a complex custom setup with a large number
139: of interfaces.
144: The netbsd-7, netbsd-8, and -current branches are all reasonable
145: choices, with more or less the same considerations for non-Xen use.
146: Therefore, netbsd-7 is recommended as the stable version of the most
147: recent release for production use. (Note that netbsd-7 (and therefore
148: 8/current) have a important scheduler fix (in November of 2015)
149: affecting contention between dom0 and domUs; see
150: https://releng.netbsd.org/cgi-bin/req-7.cgi?show=1040 for a
151: description.) For production, netbsd-7 is appropriate. For learning,
152: netbsd-8 is appropriate. For developing Xen, netbsd-current may be
155: As of NetBSD 6, a NetBSD domU will support multiple vcpus. There is
156: no SMP support for NetBSD as dom0. (The dom0 itself doesn't really
157: need SMP for dom0 functions; the lack of support is really a problem
158: when using a dom0 as a normal computer.)
163: Xen itself can run on i386 (Xen < 4.2) or amd64 hardware (all Xen
164: versions). (Practically, almost any computer where one would want to
165: run Xen today supports amd64.)
167: Xen, the dom0 system, and each domU system can be either i386 or
168: amd64. When building a xenkernel package, one obtains an i386 Xen
169: kernel on an i386 host, and an amd64 Xen kernel on an amd64 host. If
170: the Xen kernel is i386, then the dom0 kernel and all domU kernels must
171: be i386. With an amd64 Xen kernel, an amd64 dom0 kernel is known to
172: work, and an i386 dom0 kernel should in theory work. An amd64
173: Xen/dom0 is known to support both i386 and amd64 domUs.
175: i386 dom0 and domU kernels must be PAE (except for an i386 Xen 3.1
176: kernel, where one can use non-PAE for dom0 and all domUs); PAE kernels
177: are included in the NetBSD default build. (Note that emacs (at least)
178: fails if run on i386 with PAE when built without, and vice versa,
179: presumably due to bugs in the undump code.)
181: Because of the above, the standard approach is to use an amd64 Xen
182: kernel and NetBSD/amd64 for the dom0. For domUs, NetBSD/i386 (with
183: the PAE kernel) and NetBSD/amd64 are in widespread use, and there is
184: little to no Xen-specific reason to prefer one over the other.
186: Note that to use an i386 dom0 with Xen 4.5 or higher, one must build
187: (or obtain from pre-built packages) an amd64 Xen kernel and install
188: that on the system. (One must also use a PAE i386 kernel, but this is
189: also required with an i386 Xen kernel.). Almost no one in the
190: NetBSD/Xen community does this, and the standard, well-tested,
191: approach is to use an amd64 dom0.
193: A [posting on
195: explained that PV system call overhead was higher on amd64, and thus
196: there is some notion that i386 guests are faster. It goes on to
197: caution that the total situation is complex and not entirely
198: understood. On top of that caution, the post is about Linux, not
199: NetBSD. TODO: Include link to benchmarks, if someone posts them.
204: Mostly, NetBSD as a dom0 or domU is quite stable. However, just like every
205: other architecture, there are some open PRs indicating problems.
207: Note also that there are issues with sparse vnd(4) instances, but
208: these are not about Xen -- they just are noticed with sparse vnd(4)
209: instances in support of virtual disks in a dom0.
214: Therefore, this HOWTO recommends running xenkernel46, xl, the NetBSD 7
215: stable branch, and therefore to use an amd64 kernel as the dom0.
216: Either the i386PAE or amd64 version of NetBSD may be used as domUs.
218: A tentative replacement recommendation is xenkernel48, xl, and NetBSD
221: Because bugs are fixed quite often, and because of Xen security
222: advisories, it is good to stay up to date with NetBSD (tracking a
223: stable branch), with the Xen kernel (tracking a Xen version via
224: pkgsrc), and with the Xen tools. Specifically, NetBSD (-7 and
225: -current) got an important fix affecting dom0/domU timesharing in
226: November, 2015, and xentools46 got a fix to enable Ubuntu guests to
227: boot in December, 2016.
229: NetBSD as a dom0
232: NetBSD can be used as a dom0 and works very well. The following
233: sections address installation, updating NetBSD, and updating Xen.
234: Note that it doesn't make sense to talk about installing a dom0 OS
235: without also installing Xen itself. We first address installing
236: NetBSD, which is not yet a dom0, and then adding Xen, pivoting the
237: NetBSD install to a dom0 install by just changing the kernel and boot
240: For experimenting with Xen, a machine with as little as 1G of RAM and
241: 100G of disk can work. For running many domUs in productions, far
242: more will be needed; e.g. 4-8G and 1T of disk is reasonable for a
243: half-dozen domUs of 512M and 32G each. Basically, the RAM and disk
244: have to be bigger than the sum of the RAM/disk needs of the dom0 and
245: all the domUs.
247: In 2018-05, trouble booting a dom0 was reported with 256M of RAM: with
248: 512M it worked reliably. This does not make sense, but if you see
249: "not ELF" after Xen boots, try increasing dom0 RAM.
251: Styles of dom0 operation
254: There are two basic ways to use Xen. The traditional method is for
255: the dom0 to do absolutely nothing other than providing support to some
256: number of domUs. Such a system was probably installed for the sole
257: purpose of hosting domUs, and sits in a server room on a UPS.
259: The other way is to put Xen under a normal-usage computer, so that the
260: dom0 is what the computer would have been without Xen, perhaps a
261: desktop or laptop. Then, one can run domUs at will. Purists will
262: deride this as less secure than the previous approach, and for a
263: computer whose purpose is to run domUs, they are right. But Xen and a
264: dom0 (without domUs) is not meaningfully less secure than the same
265: things running without Xen. One can boot Xen or boot regular NetBSD
266: alternately with little problems, simply refraining from starting the
267: Xen daemons when not running Xen.
269: Note that NetBSD as dom0 does not support multiple CPUs. This will
270: limit the performance of the Xen/dom0 workstation approach. In theory
271: the only issue is that the "backend drivers" are not yet MPSAFE:
274: Installation of NetBSD
278: [install NetBSD/amd64](/guide/inst/)
279: just as you would if you were not using Xen.
280: However, the partitioning approach is very important.
282: If you want to use RAIDframe for the dom0, there are no special issues
283: for Xen. Typically one provides RAID storage for the dom0, and the
284: domU systems are unaware of RAID. The 2nd-stage loader bootxx_* skips
285: over a RAID1 header to find /boot from a file system within a RAID
286: partition; this is no different when booting Xen.
288: There are 4 styles of providing backing storage for the virtual disks
289: used by domUs: raw partitions, LVM, file-backed vnd(4), and SAN.
291: With raw partitions, one has a disklabel (or gpt) partition sized for
292: each virtual disk to be used by the domU. (If you are able to predict
293: how domU usage will evolve, please add an explanation to the HOWTO.
294: Seriously, needs tend to change over time.)
296: One can use [lvm(8)](/guide/lvm/) to create logical devices to use
297: for domU disks. This is almost as efficient as raw disk partitions
298: and more flexible. Hence raw disk partitions should typically not
299: be used.
301: One can use files in the dom0 file system, typically created by dd'ing
302: /dev/zero to create a specific size. This is somewhat less efficient,
303: but very convenient, as one can cp the files for backup, or move them
304: between dom0 hosts.
306: Finally, in theory one can place the files backing the domU disks in a
307: SAN. (This is an invitation for someone who has done this to add a
308: HOWTO page.)
310: Installation of Xen
313: In the dom0, install sysutils/xenkernel42 and sysutils/xentools42 from
314: pkgsrc (or another matching pair). See [the pkgsrc
315: documentation](https://www.NetBSD.org/docs/pkgsrc/) for help with
316: pkgsrc. Ensure that your packages are recent; the HOWTO does not
317: contemplate old builds.
320: For Xen 3.1, support for HVM guests is in sysutils/xentool3-hvm. More
321: recent versions have HVM support integrated in the main xentools
322: package. It is entirely reasonable to run only PV guests.
324: Next you need to install the selected Xen kernel itself, which is
325: installed by pkgsrc as "/usr/pkg/xen*-kernel/xen.gz". Copy it to /.
326: For debugging, one may copy xen-debug.gz; this is conceptually similar
327: to DIAGNOSTIC and DEBUG in NetBSD. xen-debug.gz is basically only
328: useful with a serial console. Then, place a NetBSD XEN3_DOM0 kernel
329: in /, copied from releasedir/amd64/binary/kernel/netbsd-XEN3_DOM0.gz
330: of a NetBSD build. If using i386, use
331: releasedir/i386/binary/kernel/netbsd-XEN3PAE_DOM0.gz. (If using Xen
332: 3.1 and i386, you may use XEN3_DOM0 with the non-PAE Xen. But you
333: should not use Xen 3.1.) Both xen and the NetBSD kernel may be (and
334: typically are) left compressed.
336: In a dom0, kernfs is mandatory for xend to communicate with the
337: kernel, so ensure that /kern is in fstab. (A standard NetBSD install
338: should already mount /kern.)
340: Because you already installed NetBSD, you have a working boot setup
341: with an MBR bootblock, either bootxx_ffsv1 or bootxx_ffsv2 at the
342: beginning of your root file system, have /boot, and likely also
343: /boot.cfg. (If not, fix before continuing!)
345: Add a line to to /boot.cfg to boot Xen. See boot.cfg(5) for an
346: example. The basic line is
348: menu=Xen:load /netbsd-XEN3_DOM0.gz console=pc;multiboot /xen.gz dom0_mem=512M
350: which specifies that the dom0 should have 512M, leaving the rest to be
351: allocated for domUs. To use a serial console, use
353: menu=Xen:load /netbsd-XEN3_DOM0.gz;multiboot /xen.gz dom0_mem=512M console=com1 com1=9600,8n1
355: which will use the first serial port for Xen (which counts starting
356: from 1, unlike NetBSD which counts starting from 0), forcing
357: speed/parity. Because the NetBSD command line lacks a
358: "console=pc" argument, it will use the default "xencons" console device,
359: which directs the console I/O through Xen to the same console device Xen
360: itself uses (in this case, the serial port).
362: In an attempt to add performance, one can also add
364: dom0_max_vcpus=1 dom0_vcpus_pin
366: to force only one vcpu to be provided (since NetBSD dom0 can't use
367: more) and to pin that vcpu to a physical CPU. TODO: benchmark this.
369: Xen has [many boot
371: and other than dom0 memory and max_vcpus, they are generally not
374: As with non-Xen systems, you should have a line to boot /netbsd (a
375: kernel that works without Xen). Consider a line to boot /netbsd.ok (a
376: fallback version of the non-Xen kernel, updated manually when you are
377: sure /netbsd is ok). Consider also a line to boot fallback versions
378: of Xen and the dom0 kernel, but note that non-Xen NetBSD can be used
379: to resolve Xen booting issues.
381: Probably you want a default=N line to choose Xen in the absence of
384: Now, reboot so that you are running a DOM0 kernel under Xen, rather
385: than GENERIC without Xen.
387: Using grub (historic)
390: Before NetBSD's native bootloader could support Xen, the use of
391: grub was recommended. If necessary, see the
392: [old grub information](/ports/xen/howto-grub).
394: The [HowTo on Installing into
396: explains how to set up booting a dom0 with Xen using grub with
397: NetBSD's RAIDframe. (This is obsolete with the use of NetBSD's native
398: boot. Now, just create a system with RAID-1, and alter /boot.cfg as
399: described above.)
401: Configuring Xen
404: Xen logs will be in /var/log/xen.
406: Now, you have a system that will boot Xen and the dom0 kernel, but not
407: do anything else special. Make sure that you have rebooted into Xen.
408: There will be no domUs, and none can be started because you still have
409: to configure the dom0 daemons.
411: The daemons which should be run vary with Xen version and with whether
412: one is using xm or xl. The Xen 3.1, 3.3 and 4.1 packages use xm. Xen
413: 4.2 and up packages use xl. To use xm with 4.2, edit xendomains to
414: use xm instead.
416: For 3.1 and 3.3, you should enable xend and xenbackendd:
421: For 4.1 and up, you should enable xencommons. Not enabling xencommons
422: will result in a hang; it is necessary to hit ^C on the console to let
423: the machine finish booting. If you are using xm (default in 4.1, or
424: if you changed xendomains in 4.2), you should also enable xend:
426: xend=YES # only if using xm, and only installed <= 4.2
429: TODO: Recommend for/against xen-watchdog.
431: After you have configured the daemons and either started them (in the
432: order given) or rebooted, use xm or xl to inspect Xen's boot messages,
433: available resources, and running domains. An example with xl follows:
435: # xl dmesg
436: [xen's boot info]
437: # xl info
438: [available memory, etc.]
439: # xl list
440: Name Id Mem(MB) CPU State Time(s) Console
441: Domain-0 0 64 0 r---- 58.1
443: ### Issues with xencommons
445: xencommons starts xenstored, which stores data on behalf of dom0 and
446: domUs. It does not currently work to stop and start xenstored.
447: Certainly all domUs should be shutdown first, following the sort order
448: of the rc.d scripts. However, the dom0 sets up state with xenstored,
449: and is not notified when xenstored exits, leading to not recreating
450: the state when the new xenstored starts. Until there's a mechanism to
451: make this work, one should not expect to be able to restart xenstored
452: (and thus xencommons). There is currently no reason to expect that
453: this will get fixed any time soon.
455: ### No-longer needed advice about devices
457: The installation of NetBSD should already have created devices for xen
458: (xencons, xenevt, xsd_kva), but if they are not present, create them:
460: cd /dev && sh MAKEDEV xen
462: anita (for testing NetBSD)
465: With the setup so far (assuming 4.2/xl), one should be able to run
466: anita (see pkgsrc/misc/py-anita) to test NetBSD releases, by doing (as
467: root, because anita must create a domU):
469: anita --vmm=xl test file:///usr/obj/i386/
471: Alternatively, one can use --vmm=xm to use xm-based domU creation
472: instead (and must, on Xen <= 4.1). TODO: confirm that anita xl really works.
474: Xen-specific NetBSD issues
477: There are (at least) two additional things different about NetBSD as a
478: dom0 kernel compared to hardware.
480: One is that the module ABI is different because some of the #defines
481: change, so one must build modules for Xen. As of netbsd-7, the build
482: system does this automatically. TODO: check this. (Before building
483: Xen modules was added, it was awkward to use modules to the point
484: where it was considered that it did not work.)
486: The other difference is that XEN3_DOM0 does not have exactly the same
487: options as GENERIC. While it is debatable whether or not this is a
488: bug, users should be aware of this and can simply add missing config
489: items if desired.
491: Updating NetBSD in a dom0
494: This is just like updating NetBSD on bare hardware, assuming the new
495: version supports the version of Xen you are running. Generally, one
496: replaces the kernel and reboots, and then overlays userland binaries
497: and adjusts /etc.
499: Note that one must update both the non-Xen kernel typically used for
500: rescue purposes and the DOM0 kernel used with Xen.
502: Converting from grub to /boot
505: These instructions were [TODO: will be] used to convert a system from
506: grub to /boot. The system was originally installed in February of
507: 2006 with a RAID1 setup and grub to boot Xen 2, and has been updated
508: over time. Before these commands, it was running NetBSD 6 i386, Xen
509: 4.1 and grub, much like the message linked earlier in the grub
512: # Install MBR bootblocks on both disks.
513: fdisk -i /dev/rwd0d
514: fdisk -i /dev/rwd1d
515: # Install NetBSD primary boot loader (/ is FFSv1) into RAID1 components.
516: installboot -v /dev/rwd0d /usr/mdec/bootxx_ffsv1
517: installboot -v /dev/rwd1d /usr/mdec/bootxx_ffsv1
518: # Install secondary boot loader
519: cp -p /usr/mdec/boot /
520: # Create boot.cfg following earlier guidance:
521: menu=Xen:load /netbsd-XEN3PAE_DOM0.gz console=pc;multiboot /xen.gz dom0_mem=512M
522: menu=Xen.ok:load /netbsd-XEN3PAE_DOM0.ok.gz console=pc;multiboot /xen.ok.gz dom0_mem=512M
524: menu=GENERIC single-user:boot -s
525: menu=GENERIC.ok:boot netbsd.ok
526: menu=GENERIC.ok single-user:boot netbsd.ok -s
527: menu=Drop to boot prompt:prompt
531: TODO: actually do this and fix it if necessary.
533: Upgrading Xen versions
536: Minor version upgrades are trivial. Just rebuild/replace the
537: xenkernel version and copy the new xen.gz to / (where /boot.cfg
538: references it), and reboot.
540: Major version upgrades are conceptually not difficult, but can run
541: into all the issues found when installing Xen. Assuming migration
542: from 4.1 to 4.2, remove the xenkernel41 and xentools41 packages and
543: install the xenkernel42 and xentools42 packages. Copy the 4.2 xen.gz
544: to /.
546: Ensure that the contents of /etc/rc.d/xen* are correct. Specifically,
547: they must match the package you just installed and not be left over
548: from some previous installation.
550: Enable the correct set of daemons; see the configuring section above.
551: (Upgrading from 3.x to 4.x without doing this will result in a hang.)
553: Ensure that the domU config files are valid for the new version.
554: Specifically, for 4.x remove autorestart=True, and ensure that disks
555: are specified with numbers as the second argument, as the examples
556: above show, and not NetBSD device names.
558: Hardware known to work
561: Arguably, this section is misplaced, and there should be a page of
562: hardware that runs NetBSD/amd64 well, with the mostly-well-founded
563: assumption that NetBSD/xen runs fine on any modern hardware that
564: NetBSD/amd64 runs well on. Until then, we give motherboard/CPU (and
565: sometimes RAM) pairs/triples to aid those choosing a motherboard.
566: Note that Xen systems usually do not run X, so a listing here does not
567: imply that X works at all.
569: Supermicro X9SRL-F, Xeon E5-1650 v2, 96 GiB ECC
570: Supermicro ??, Atom C2758 (8 core), 32 GiB ECC
571: ASUS M5A78L-M/USB3 AM3+ microATX, AMD Piledriver X8 4000MHz, 16 GiB ECC
573: Older hardware:
575: Intel D915GEV, Pentium4 CPU 3.40GHz, 4GB 533MHz Synchronous DDR2
576: INTEL DG33FB, "Intel(R) Core(TM)2 Duo CPU E6850 @ 3.00GHz"
577: INTEL DG33FB, "Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz"
579: Running Xen under qemu
582: The astute reader will note that this section is somewhat twisted.
583: However, it can be useful to run Xen under qemu either because the
584: version of NetBSD as a dom0 does not run on the hardware in use, or to
585: generate automated test cases involving Xen.
587: In 2015-01, the following combination was reported to mostly work:
589: host OS: NetBSD/amd64 6.1.4
590: qemu: 2.2.0 from pkgsrc
591: Xen kernel: xenkernel42-4.2.5nb1 from pkgsrc
592: dom0 kernel: NetBSD/amd64 6.1.5
593: Xen tools: xentools42-4.2.5 from pkgsrc
595: See [PR 47720](https://gnats.netbsd.org/47720) for a problem with dom0
598: Unprivileged domains (domU)
601: This section describes general concepts about domUs. It does not
602: address specific domU operating systems or how to install them. The
603: config files for domUs are typically in /usr/pkg/etc/xen, and are
604: typically named so that the file name, domU name and the domU's host
605: name match.
607: The domU is provided with CPU and memory by Xen, configured by the
608: dom0. The domU is provided with disk and network by the dom0,
609: mediated by Xen, and configured in the dom0.
611: Entropy in domUs can be an issue; physical disks and network are on
612: the dom0. NetBSD's /dev/random system works, but is often challenged.
614: Config files
617: There is no good order to present config files and the concepts
618: surrounding what is being configured. We first show an example config
619: file, and then in the various sections give details.
621: See (at least in xentools41) /usr/pkg/share/examples/xen/xmexample*,
622: for a large number of well-commented examples, mostly for running
625: The following is an example minimal domain configuration file
626: "/usr/pkg/etc/xen/foo". It is (with only a name change) an actual
627: known working config file on Xen 4.1 (NetBSD 5 amd64 dom0 and NetBSD 5
628: i386 domU). The domU serves as a network file server.
630: # -*- mode: python; -*-
632: kernel = "/netbsd-XEN3PAE_DOMU-i386-foo.gz"
633: memory = 1024
634: vif = [ 'mac=aa:00:00:d1:00:09,bridge=bridge0' ]
635: disk = [ 'file:/n0/xen/foo-wd0,0x0,w',
636: 'file:/n0/xen/foo-wd1,0x1,w' ]
638: The domain will have the same name as the file. The kernel has the
639: host/domU name in it, so that on the dom0 one can update the various
640: domUs independently. The vif line causes an interface to be provided,
641: with a specific mac address (do not reuse MAC addresses!), in bridge
642: mode. Two disks are provided, and they are both writable; the bits
643: are stored in files and Xen attaches them to a vnd(4) device in the
644: dom0 on domain creation. The system treats xbd0 as the boot device
645: without needing explicit configuration.
647: By default xm looks for domain config files in /usr/pkg/etc/xen. Note
648: that "xm create" takes the name of a config file, while other commands
649: take the name of a domain. To create the domain, connect to the
650: console, create the domain while attaching the console, shutdown the
651: domain, and see if it has finished stopping, do (or xl with Xen >=
654: xm create foo
655: xm console foo
656: xm create -c foo
657: xm shutdown foo
658: xm list
660: Typing ^] will exit the console session. Shutting down a domain is
661: equivalent to pushing the power button; a NetBSD domU will receive a
662: power-press event and do a clean shutdown. Shutting down the dom0
663: will trigger controlled shutdowns of all configured domUs.
665: domU kernels
668: On a physical computer, the BIOS reads sector 0, and a chain of boot
669: loaders finds and loads a kernel. Normally this comes from the root
670: file system. With Xen domUs, the process is totally different. The
671: normal path is for the domU kernel to be a file in the dom0's
672: file system. At the request of the dom0, Xen loads that kernel into a
673: new domU instance and starts execution. While domU kernels can be
674: anyplace, reasonable places to store domU kernels on the dom0 are in /
675: (so they are near the dom0 kernel), in /usr/pkg/etc/xen (near the
676: config files), or in /u0/xen (where the vdisks are).
678: Note that loading the domU kernel from the dom0 implies that boot
679: blocks, /boot, /boot.cfg, and so on are all ignored in the domU.
680: See the VPS section near the end for discussion of alternate ways to
681: obtain domU kernels.
683: CPU and memory
686: A domain is provided with some number of vcpus, less than the number
687: of CPUs seen by the hypervisor. (For a dom0, this is controlled by
688: the boot argument "dom0_max_vcpus=1".) For a domU, it is controlled
689: from the config file by the "vcpus = N" directive.
691: A domain is provided with memory; this is controlled in the config
692: file by "memory = N" (in megabytes). In the straightforward case, the
693: sum of the the memory allocated to the dom0 and all domUs must be less
694: than the available memory.
696: Xen also provides a "balloon" driver, which can be used to let domains
697: use more memory temporarily. TODO: Explain better, and explain how
698: well it works with NetBSD.
700: Virtual disks
703: With the file/vnd style, typically one creates a directory,
704: e.g. /u0/xen, on a disk large enough to hold virtual disks for all
705: domUs. Then, for each domU disk, one writes zeros to a file that then
706: serves to hold the virtual disk's bits; a suggested name is foo-xbd0
707: for the first virtual disk for the domU called foo. Writing zeros to
708: the file serves two purposes. One is that preallocating the contents
709: improves performance. The other is that vnd on sparse files has
710: failed to work. TODO: give working/notworking NetBSD versions for
711: sparse vnd and gnats reference. Note that the use of file/vnd for Xen
712: is not really different than creating a file-backed virtual disk for
713: some other purpose, except that xentools handles the vnconfig
714: commands. To create an empty 4G virtual disk, simply do
716: dd if=/dev/zero of=foo-xbd0 bs=1m count=4096
718: Do not use qemu-img-xen, because this will create sparse file. There
719: have been recent (2015) reports of sparse vnd(4) devices causing
720: lockups, but there is apparently no PR.
722: With the lvm style, one creates logical devices. They are then used
723: similarly to vnds. TODO: Add an example with lvm.
725: In domU config files, the disks are defined as a sequence of 3-tuples.
726: The first element is "method:/path/to/disk". Common methods are
727: "file:" for file-backed vnd. and "phy:" for something that is already
728: a (TODO: character or block) device.
730: The second element is an artifact of how virtual disks are passed to
731: Linux, and a source of confusion with NetBSD Xen usage. Linux domUs
732: are given a device name to associate with the disk, and values like
733: "hda1" or "sda1" are common. In a NetBSD domU, the first disk appears
734: as xbd0, the second as xbd1, and so on. However, xm/xl demand a
735: second argument. The name given is converted to a major/minor by
736: calling stat(2) on the name in /dev and this is passed to the domU.
737: In the general case, the dom0 and domU can be different operating
738: systems, and it is an unwarranted assumption that they have consistent
739: numbering in /dev, or even that the dom0 OS has a /dev. With NetBSD
740: as both dom0 and domU, using values of 0x0 for the first disk and 0x1
741: for the second works fine and avoids this issue. For a GNU/Linux
742: guest, one can create /dev/hda1 in /dev, or to pass 0x301 for
745: The third element is "w" for writable disks, and "r" for read-only
748: Note that NetBSD by default creates only vnd. If you need more
749: than 4 total virtual disks at a time, run e.g. "./MAKEDEV vnd4" in the
752: Note that NetBSD by default creates only xbd. If you need more
753: virtual disks in a domU, run e.g. "./MAKEDEV xbd4" in the domU.
755: Virtual Networking
758: Xen provides virtual Ethernets, each of which connects the dom0 and a
759: domU. For each virtual network, there is an interface "xvifN.M" in
760: the dom0, and in domU index N, a matching interface xennetM (NetBSD
761: name). The interfaces behave as if there is an Ethernet with two
762: adapters connected. From this primitive, one can construct various
763: configurations. We focus on two common and useful cases for which
764: there are existing scripts: bridging and NAT.
766: With bridging (in the example above), the domU perceives itself to be
767: on the same network as the dom0. For server virtualization, this is
768: usually best. Bridging is accomplished by creating a bridge(4) device
769: and adding the dom0's physical interface and the various xvifN.0
770: interfaces to the bridge. One specifies "bridge=bridge0" in the domU
771: config file. The bridge must be set up already in the dom0; an
772: example /etc/ifconfig.bridge0 is:
776: !brconfig bridge0 add wm0
778: With NAT, the domU perceives itself to be behind a NAT running on the
779: dom0. This is often appropriate when running Xen on a workstation.
780: TODO: NAT appears to be configured by "vif = [ '' ]".
782: The MAC address specified is the one used for the interface in the new
783: domain. The interface in dom0 will use this address XOR'd with
784: 00:00:00:01:00:00. Random MAC addresses are assigned if not given.
786: Sizing domains
789: Modern x86 hardware has vast amounts of resources. However, many
790: virtual servers can function just fine on far less. A system with
791: 512M of RAM and a 4G disk can be a reasonable choice. Note that it is
792: far easier to adjust virtual resources than physical ones. For
793: memory, it's just a config file edit and a reboot. For disk, one can
794: create a new file and vnconfig it (or lvm), and then dump/restore,
795: just like updating physical disks, but without having to be there and
796: without those pesky connectors.
798: Starting domains automatically
801: To start domains foo at bar at boot and shut them down cleanly on dom0
802: shutdown, in rc.conf add:
804: xendomains="foo bar"
806: Note that earlier versions of the xentools41 xendomains rc.d script
807: used xl, when one should use xm with 4.1.
809: Creating specific unprivileged domains (domU)
812: Creating domUs is almost entirely independent of operating system. We
813: have already presented the basics of config files. Note that you must
814: have already completed the dom0 setup so that "xl list" (or "xm list")
817: Creating an unprivileged NetBSD domain (domU)
820: See the earlier config file, and adjust memory. Decide on how much
821: storage you will provide, and prepare it (file or lvm).
823: While the kernel will be obtained from the dom0 file system, the same
824: file should be present in the domU as /netbsd so that tools like
825: savecore(8) can work. (This is helpful but not necessary.)
827: The kernel must be specifically for Xen and for use as a domU. The
828: i386 and amd64 provide the following kernels:
830: i386 XEN3_DOMU
831: i386 XEN3PAE_DOMU
832: amd64 XEN3_DOMU
834: Unless using Xen 3.1 (and you shouldn't) with i386-mode Xen, you must
835: use the PAE version of the i386 kernel.
837: This will boot NetBSD, but this is not that useful if the disk is
838: empty. One approach is to unpack sets onto the disk outside of xen
839: (by mounting it, just as you would prepare a physical disk for a
840: system you can't run the installer on).
842: A second approach is to run an INSTALL kernel, which has a miniroot
843: and can load sets from the network. To do this, copy the INSTALL
844: kernel to / and change the kernel line in the config file to:
846: kernel = "/home/bouyer/netbsd-INSTALL_XEN3_DOMU"
848: Then, start the domain as "xl create -c configname".
850: Alternatively, if you want to install NetBSD/Xen with a CDROM image, the following
851: line should be used in the config file.
853: disk = [ 'phy:/dev/wd0e,0x1,w', 'phy:/dev/cd0a,0x2,r' ]
855: After booting the domain, the option to install via CDROM may be
856: selected. The CDROM device should be changed to `xbd1d`.
858: Once done installing, "halt -p" the new domain (don't reboot or halt,
859: it would reload the INSTALL_XEN3_DOMU kernel even if you changed the
860: config file), switch the config file back to the XEN3_DOMU kernel,
861: and start the new domain again. Now it should be able to use "root on
862: xbd0a" and you should have a, functional NetBSD domU.
864: TODO: check if this is still accurate.
865: When the new domain is booting you'll see some warnings about *wscons*
866: and the pseudo-terminals. These can be fixed by editing the files
867: `/etc/ttys` and `/etc/wscons.conf`. You must disable all terminals in
868: `/etc/ttys`, except *console*, like this:
870: console "/usr/libexec/getty Pc" vt100 on secure
871: ttyE0 "/usr/libexec/getty Pc" vt220 off secure
872: ttyE1 "/usr/libexec/getty Pc" vt220 off secure
873: ttyE2 "/usr/libexec/getty Pc" vt220 off secure
874: ttyE3 "/usr/libexec/getty Pc" vt220 off secure
876: Finally, all screens must be commented out from `/etc/wscons.conf`.
878: It is also desirable to add
882: in rc.conf. This way, the domain will be properly shut down if
883: `xm shutdown -R` or `xm shutdown -H` is used on the dom0.
885: It is not strictly necessary to have a kernel (as /netbsd) in the domU
886: file system. However, various programs (e.g. netstat) will use that
887: kernel to look up symbols to read from kernel virtual memory. If
888: /netbsd is not the running kernel, those lookups will fail. (This is
889: not really a Xen-specific issue, but because the domU kernel is
890: obtained from the dom0, it is far more likely to be out of sync or
891: missing with Xen.)
893: Creating an unprivileged Linux domain (domU)
896: Creating unprivileged Linux domains isn't much different from
897: unprivileged NetBSD domains, but there are some details to know.
899: First, the second parameter passed to the disk declaration (the '0x1' in
900: the example below)
902: disk = [ 'phy:/dev/wd0e,0x1,w' ]
904: does matter to Linux. It wants a Linux device number here (e.g. 0x300
905: for hda). Linux builds device numbers as: (major \<\< 8 + minor).
906: So, hda1 which has major 3 and minor 1 on a Linux system will have
907: device number 0x301. Alternatively, devices names can be used (hda,
908: hdb, ...) as xentools has a table to map these names to devices
909: numbers. To export a partition to a Linux guest we can use:
911: disk = [ 'phy:/dev/wd0e,0x300,w' ]
912: root = "/dev/hda1 ro"
914: and it will appear as /dev/hda on the Linux system, and be used as root
917: To install the Linux system on the partition to be exported to the
918: guest domain, the following method can be used: install
919: sysutils/e2fsprogs from pkgsrc. Use mke2fs to format the partition
920: that will be the root partition of your Linux domain, and mount it.
921: Then copy the files from a working Linux system, make adjustments in
922: `/etc` (fstab, network config). It should also be possible to extract
923: binary packages such as .rpm or .deb directly to the mounted partition
924: using the appropriate tool, possibly running under NetBSD's Linux
925: emulation. Once the file system has been populated, umount it. If
926: desirable, the file system can be converted to ext3 using tune2fs -j.
927: It should now be possible to boot the Linux guest domain, using one of
928: the vmlinuz-\*-xenU kernels available in the Xen binary distribution.
930: To get the Linux console right, you need to add:
932: extra = "xencons=tty1"
934: to your configuration since not all Linux distributions auto-attach a
935: tty to the xen console.
937: Creating an unprivileged Solaris domain (domU)
940: See possibly outdated
941: [Solaris domU instructions](/ports/xen/howto-solaris/).
944: PCI passthrough: Using PCI devices in guest domains
947: The dom0 can give other domains access to selected PCI
948: devices. This can allow, for example, a non-privileged domain to have
949: access to a physical network interface or disk controller. However,
950: keep in mind that giving a domain access to a PCI device most likely
951: will give the domain read/write access to the whole physical memory,
952: as PCs don't have an IOMMU to restrict memory access to DMA-capable
953: device. Also, it's not possible to export ISA devices to non-dom0
954: domains, which means that the primary VGA adapter can't be exported.
955: A guest domain trying to access the VGA registers will panic.
957: If the dom0 is NetBSD, it has to be running Xen 3.1, as support has
958: not been ported to later versions at this time.
960: For a PCI device to be exported to a domU, is has to be attached to
961: the "pciback" driver in dom0. Devices passed to the dom0 via the
962: pciback.hide boot parameter will attach to "pciback" instead of the
963: usual driver. The list of devices is specified as "(bus:dev.func)",
964: where bus and dev are 2-digit hexadecimal numbers, and func a
965: single-digit number:
969: pciback devices should show up in the dom0's boot messages, and the
970: devices should be listed in the `/kern/xen/pci` directory.
972: PCI devices to be exported to a domU are listed in the "pci" array of
973: the domU's config file, with the format "0000:bus:dev.func".
975: pci = [ '0000:00:06.0', '0000:00:0a.0' ]
977: In the domU an "xpci" device will show up, to which one or more pci
978: buses will attach. Then the PCI drivers will attach to PCI buses as
979: usual. Note that the default NetBSD DOMU kernels do not have "xpci"
980: or any PCI drivers built in by default; you have to build your own
981: kernel to use PCI devices in a domU. Here's a kernel config example;
982: note that only the "xpci" lines are unusual.
984: include "arch/i386/conf/XEN3_DOMU"
986: # Add support for PCI buses to the XEN3_DOMU kernel
987: xpci* at xenbus ?
988: pci* at xpci ?
990: # PCI USB controllers
991: uhci* at pci? dev ? function ? # Universal Host Controller (Intel)
993: # USB bus support
994: usb* at uhci?
996: # USB Hubs
997: uhub* at usb?
998: uhub* at uhub? port ? configuration ? interface ?
1000: # USB Mass Storage
1001: umass* at uhub? port ? configuration ? interface ?
1002: wd* at umass?
1003: # SCSI controllers
1004: ahc* at pci? dev ? function ? # Adaptec 94x, aic78x0 SCSI
1006: # SCSI bus support (for both ahc and umass)
1007: scsibus* at scsi?
1009: # SCSI devices
1010: sd* at scsibus? target ? lun ? # SCSI disk drives
1011: cd* at scsibus? target ? lun ? # SCSI CD-ROM drives
1014: NetBSD as a domU in a VPS
1017: The bulk of the HOWTO is about using NetBSD as a dom0 on your own
1018: hardware. This section explains how to deal with Xen in a domU as a
1019: virtual private server where you do not control or have access to the
1020: dom0. This is not intended to be an exhaustive list of VPS providers;
1021: only a few are mentioned that specifically support NetBSD.
1023: VPS operators provide varying degrees of access and mechanisms for
1024: configuration. The big issue is usually how one controls which kernel
1025: is booted, because the kernel is nominally in the dom0 file system (to
1026: which VPS users do not normally have access). A second issue is how
1027: to install NetBSD.
1028: A VPS user may want to compile a kernel for security updates, to run
1029: npf, run IPsec, or any other reason why someone would want to change
1030: their kernel.
1032: One approach is to have an administrative interface to upload a kernel,
1033: or to select from a prepopulated list. Other approaches are pygrub
1034: (deprecated) and pvgrub, which are ways to have a bootloader obtain a
1035: kernel from the domU file system. This is closer to a regular physical
1036: computer, where someone who controls a machine can replace the kernel.
1038: A second issue is multiple CPUs. With NetBSD 6, domUs support
1039: multiple vcpus, and it is typical for VPS providers to enable multiple
1040: CPUs for NetBSD domUs.
1045: pygrub runs in the dom0 and looks into the domU file system. This
1046: implies that the domU must have a kernel in a file system in a format
1047: known to pygrub. As of 2014, pygrub seems to be of mostly historical
1053: pvgrub is a version of grub that uses PV operations instead of BIOS
1054: calls. It is booted from the dom0 as the domU kernel, and then reads
1055: /grub/menu.lst and loads a kernel from the domU file system.
1057: [Panix](http://www.panix.com/) lets users use pvgrub. Panix reports
1058: that pvgrub works with FFsv2 with 16K/2K and 32K/4K block/frag sizes
1059: (and hence with defaults from "newfs -O 2"). See [Panix's pvgrub
1060: page](http://www.panix.com/v-colo/grub.html), which describes only
1061: Linux but should be updated to cover NetBSD :-).
1063: [prgmr.com](http://prgmr.com/) also lets users with pvgrub to boot
1064: their own kernel. See then [prgmr.com NetBSD
1066: (which is in need of updating).
1068: It appears that [grub's FFS
1070: does not support all aspects of modern FFS, but there are also reports
1071: that FFSv2 works fine. At prgmr, typically one has an ext2 or FAT
1072: partition for the kernel with the intent that grub can understand it,
1073: which leads to /netbsd not being the actual kernel. One must remember
1074: to update the special boot partition.
1079: See the [Amazon EC2 page](/amazon_ec2/).
1081: TODO items for improving NetBSD/xen
1084: * Make the NetBSD dom0 kernel work with SMP.
1085: * Test the Xen 4.5 packages adequately to be able to recommend them as
1086: the standard approach.
1087: * Get PCI passthrough working on Xen 4.5
1088: * Get pvgrub into pkgsrc, either via xentools or separately.
1089: * grub
1090: * Check/add support to pkgsrc grub2 for UFS2 and arbitrary
1091: fragsize/blocksize (UFS2 support may be present; the point is to
1092: make it so that with any UFS1/UFS2 file system setup that works
1093: with NetBSD grub will also work).
1094: See [pkg/40258](https://gnats.netbsd.org/40258).
1095: * Push patches upstream.
1096: * Get UFS2 patches into pvgrub.
1097: * Add support for PV ops to a version of /boot, and make it usable as
1098: a kernel in Xen, similar to pvgrub.
1100: Random pointers
1103: This section contains links from elsewhere not yet integrated into the
1104: HOWTO, and other guides.
1106: * http://www.lumbercartel.ca/library/xen/
1107: * http://pbraun.nethence.com/doc/sysutils/xen_netbsd_dom0.html
1108: * https://gmplib.org/~tege/xen.html
CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb