--- wikisrc/ports/xen/howto.mdwn 2021/03/15 00:25:54 1.204 +++ wikisrc/ports/xen/howto.mdwn 2021/03/15 23:01:56 1.205 @@ -40,42 +40,50 @@ approach and limits discussion of altern ## Guest Styles -Xen supports different styles of guests. +Xen supports different styles of guests. See +https://wiki.xenproject.org/wiki/Virtualization_Spectrum for a +discussion. + +This table shows the styles, and if a NetBSD dom0 can run in that +style, if a NetBSD dom0 can sypport that style of guest in a domU, and +if NetBSD as a domU can support that style. [[!table data=""" -Style of guest |description |NetBSD dom0? |NetBSD domU? -PV |Paravirtualization (hypercalls) |yes |yes -HVM |unmodified guest - domU perceives HW |can run guests |yes -PVHVM |Like HVM but also PV drivers |can run guests |current only -PVH |Lightweight HVM (no qemu) and PV |not yet |current only +Style of guest |dom0 can be? |dom0 can support? |domU can be? +PV |yes |yes |yes +HVM |N/A |yes |yes +PVHVM |N/A |yes |current only +PVH |not yet |current only |current only """]] -In Para-Virtualized (PV) mode, the guest OS does not attempt to access +In PV (paravirtualized) mode, the guest OS does not attempt to access hardware directly, but instead makes hypercalls to the hypervisor; PV -guests must be specifically coded for Xen. -See [PV](https://wiki.xen.org/wiki/Paravirtualization_(PV\)). +guests must be specifically coded for Xen. See +[PV](https://wiki.xen.org/wiki/Paravirtualization_(PV\)). -In HVM mode, no guest modification is required; however, hardware -support is required, such as VT-x on Intel CPUs and SVM on AMD CPUs. -The dom0 runs qemu to emulate hardware. It is therefore non-sensical -to have an HVM dom0. +In HVM (Hardware Virtual Machine) mode, no guest modification is +required. However, hardware support is required, such as VT-x on +Intel CPUs and SVM on AMD CPUs to assist with the processor emulation. +The dom0 runs qemu to emulate hardware other than the processor. It +is therefore non-sensical to have an HVM dom0, because there is no +underlying system to provide emulation. -In PVHVM mode, the guest runs as HVM, but additionally can use PV +In PVHVM mode, the guest runs as HVM, but additionally uses PV drivers for efficiency. Therefore it is non-sensical for to have a PVHVM dom0. See [PV on HVM](https://wiki.xen.org/wiki/PV_on_HVM). There have been two PVH modes: original PVH and PVHv2. Original PVH was based on PV mode and is no longer relevant at all. Therefore -PVHv2 is abreviated PVH. PVHv2 is basically lightweight HVM with PV -drivers. A critical feature of it is that qemu is not needed; the -hypervisor can do the emulation that is required. Thus, a dom0 can be -PVHv2. The source code uses PVH and config files use pvh, but NB that -this refers to PVHv2. See +PVHv2 is written as PVH, here and elsewhere. PVH is basically +lightweight HVM with PV drivers. A critical feature of it is that +qemu is not needed; the hypervisor can do the emulation that is +required. Thus, a dom0 can be PVH. The source code uses PVH and +config files use pvh, but NB that this refers to PVHv2. See [PVH(v2)](https://wiki.xenproject.org/wiki/PVH_(v2\)_Domu). -At system boot, the dom0 kernel is loaded as a module with Xen as the kernel. -The dom0 can start one or more domUs. (Booting is explained in detail -in the dom0 section.) +At system boot, the dom0 kernel is loaded as a module with Xen as the +kernel. The dom0 can start one or more domUs. (Booting is explained +in detail in the dom0 section.) ## CPU Architecture @@ -116,7 +124,7 @@ Initially Xen was PV only. NetBSD Xen has always supported PV, in both dom0 and domU; for a long time this was the only way. NetBSD >=8 as a dom0 supports HVM mode in -domUs (HVM as a dom0 does not make sense). +domUs. Support for PVHVM and PVH is available only in NetBSD-current; this is currently somewhat experimental, although PVHVM appears reasonably @@ -154,15 +162,16 @@ In 2018-05, trouble booting a dom0 was r using Xen. Therefore, use the most recent release, or a build from the most recent stable branch. Alternatively, use -current, being mindful of all the usual caveats of lower stability of current, and -likely a bit more so. +likely a bit more so. Think about how you will provide storage for +disk images. ## Installation of Xen ### Building Xen -Use the most recent version of Xen in pkgsrc, unless the DESCR says that it is not suitable. -Therefore, choose 4.13. -In the dom0, install xenkernel413 and xentools413 from pkgsrc. +Use the most recent version of Xen in pkgsrc, unless the DESCR says +that it is not suitable. Therefore, choose 4.13. In the dom0, +install xenkernel413 and xentools413 from pkgsrc. Once this is done, copy the Xen kernel from where pkgsrc puts it to where the boot process will be able to find it: @@ -208,7 +217,7 @@ wd0a causes boot to fail. Beware that userconf statements must be attached to the dom0 load, and may not be at top-level, because then they would try to configure the -hypervisor, if there is a way to pass them via multiboot . It appears +hypervisor, if there is a way to pass them via multiboot. It appears that adding `userconf=pckbc` to `/boot.cfg` causes Xen to crash very early with a heap overflow. @@ -264,12 +273,14 @@ messages from the hypervisor's console; ### Tuning -In an attempt to add performance, one can also add `dom0_max_vcpus=1 dom0_vcpus_pin`, -to force only one vcpu to be provided (since NetBSD dom0 can't use -more) and to pin that vcpu to a physical CPU. Xen has -[many boot options](http://xenbits.xenproject.org/docs/4.13-testing/misc/xen-command-line.html), +In an attempt to add performance, one can also add `dom0_max_vcpus=1 +dom0_vcpus_pin`, to force only one vcpu to be provided (since NetBSD +dom0 can't use more) and to pin that vcpu to a physical CPU. Xen has +[many boot +options](http://xenbits.xenproject.org/docs/4.13-testing/misc/xen-command-line.html), and other than dom0 memory and max_vcpus, they are generally not necessary. + \todo Revisit this advice with current. \todo Explain if anyone has ever actually measured that this helps. @@ -328,15 +339,18 @@ dom0 kernel compared to hardware. One is that through NetBSD 9 the module ABI is different because some of the #defines change, so there are separate sets of modules in -/stand. In NetBSD-current, there is only one set of modules. +/stand. (Further, zfs in Xen is troubled because of differing +MAXPHYS; see the zfs howto for more.) In NetBSD-current, there is +only one set of modules. The other difference is that XEN3_DOM0 does not have exactly the same -options as GENERIC. While it is debatable whether or not this is a -bug, users should be aware of this and can simply add missing config +options as GENERIC. While this is roughly agreed to be in large part +a bug, users should be aware of this and can simply add missing config items if desired. Finally, there have been occasional reports of trouble with X11 -servers in NetBSD as a dom0. +servers in NetBSD as a dom0. Some hardware support is intentionally +disabled in XEN3_DOM0. ## Updating Xen in a dom0 @@ -357,9 +371,9 @@ rescue purposes, as well as the DOM0 ker ## anita (for testing NetBSD) -With a NetBSD dom0, even without any domUs, one should be able to run -anita (see pkgsrc/misc/py-anita) to test NetBSD releases, by doing (as -root, because anita must create a domU): +With a NetBSD dom0, even without any domUs, one can run anita (see +pkgsrc/misc/py-anita) to test NetBSD releases, by doing (as root, +because anita must create a domU): [[!template id=programlisting text=""" anita --vmm=xl test file:///usr/obj/i386/ @@ -385,8 +399,8 @@ the dom0. NetBSD's /dev/random system w See /usr/pkg/share/examples/xen/xlexample* for a very small number of examples for running GNU/Linux. -The following is an example minimal domain configuration file. The domU -serves as a network file server. +The following is an example minimal domain configuration file. The +domU serves as a network file server. [[!template id=filecontent name="/usr/pkg/etc/xen/foo" text=""" name = "domU-id" @@ -406,9 +420,9 @@ are stored in files and Xen attaches the dom0 on domain creation. The system treats xbd0 as the boot device without needing explicit configuration. -There is not type line; that implicitly defines a pv domU. Otherwise, -one sets type to the lower-case version of the domU type in the table -above, e.g. `type = "hvm"`. +There is not a type line; that implicitly defines a pv domU. +Otherwise, one sets type to the lower-case version of the domU type in +the table above; see later sections. By convention, domain config files are kept in `/usr/pkg/etc/xen`. Note that "xl create" takes the name of a config file, while other commands @@ -431,9 +445,11 @@ will trigger controlled shutdowns of all ## CPU and memory -A domain is provided with some number of vcpus, up to the number -of CPUs seen by the hypervisor. For a domU, it is controlled -from the config file by the "vcpus = N" directive. +A domain is provided with some number of vcpus; any domain can have up +to the number of CPUs seen by the hypervisor. For a domU, it is +controlled from the config file by the "vcpus = N" directive. It is +normal to overcommit vcpus; a 4-core machine machine might well provide 4 +vcpus to each domU. One might also configure fewer vcpus for a domU. A domain is provided with memory; this is controlled in the config file by "memory = N" (in megabytes). In the straightforward case, the @@ -745,14 +761,20 @@ note that only the "xpci" lines are unus ## Nesting under Linux KVM -It is possible to run a Xen and a NetBSD dom0 under Linux KVM. One +It is possible to run Xen and a NetBSD dom0 under Linux KVM. One can enable virtio in the dom0 for greater speed. +## Nesting under qemu + +It is possible to run Xen and a NetBSD dom0 under qemu on NetBSD, and +also with nvmm. +\todo Check this. + ## Other nesting In theory, any full emulation should be able to run Xen and a NetBSD dom0. The HOWTO does not currently have information about Xen XVM -mode, nvmm, qemu, Virtualbox, etc. +mode, Virtualbox, etc. ## NetBSD 5 as domU @@ -793,7 +815,7 @@ different choices. ### stub domains -Some (Linux only?) dom0 systems use something called "stub domains" to +Some (Linux) dom0 systems use something called "stub domains" to isolate qemu from the dom0 system, as a security and reliabilty mechanism when running HVM domUs. Somehow, NetBSD's GENERIC kernel ends up using PIO for disks rather than DMA. Of course, all of this @@ -803,8 +825,8 @@ currently understood. ### Grant tables There are multiple versions of using grant tables, and some security -advisories have suggested disabling some versions. Some versions of -NetBSD apparently only use specific versions and this can lead to +advisories have suggested disabling some versions. NetBSD through 9 +uses version 1 and NetBSD-current uses version 2. This can lead to "NetBSD current doesn't run on hosting provider X" situations. \todo Explain better. @@ -817,41 +839,49 @@ pvgrub is a version of grub that uses PV calls. It is booted from the dom0 as the domU kernel, and then reads /grub/menu.lst and loads a kernel from the domU file system. -[Panix](http://www.panix.com/) lets users use pvgrub. Panix reports -that pvgrub works with FFsv2 with 16K/2K and 32K/4K block/frag sizes -(and hence with defaults from "newfs -O 2"). See [Panix's pvgrub -page](http://www.panix.com/v-colo/grub.html), which describes only -Linux but should be updated to cover NetBSD :-). - -[prgmr.com](http://prgmr.com/) also lets users with pvgrub to boot -their own kernel. See then [prgmr.com NetBSD -HOWTO](http://wiki.prgmr.com/mediawiki/index.php/NetBSD_as_a_DomU) -(which is in need of updating). - It appears that [grub's FFS code](http://xenbits.xensource.com/hg/xen-unstable.hg/file/bca284f67702/tools/libfsimage/ufs/fsys_ufs.c) does not support all aspects of modern FFS, but there are also reports -that FFSv2 works fine. At prgmr, typically one has an ext2 or FAT -partition for the kernel with the intent that grub can understand it, -which leads to /netbsd not being the actual kernel. One must remember -to update the special boot partition. +that FFSv2 works fine. ### pygrub +As of 2014, pygrub seems to be of mostly historical interest. As of +2021, the section should perhaps be outright deleted. + pygrub runs in the dom0 and looks into the domU file system. This implies that the domU must have a kernel in a file system in a format known to pygrub. pygrub doesn't seem to work to load Linux images under NetBSD dom0, -and is inherently less secure than pvgrub due to running inside dom0. For both these -reasons, pygrub should not be used, and is only still present so that -historical DomU images using it still work. - -As of 2014, pygrub seems to be of mostly historical -interest. New DomUs should use pvgrub. +and is inherently less secure than pvgrub due to running inside +dom0. For both these reasons, pygrub should not be used, and is only +still present so that historical DomU images using it still work. ## Specific Providers +The intent is to list providers only if they document support for +running NetBSD, and to point to their resources briefly. + +### panix.com + +[Panix](http://www.panix.com/) provides NetBSD as an OS option. See +https://www.panix.com/v-colo/nupgrade.html for some information. +Users can use pvgrub. Panix reports that pvgrub works with FFsv2 with +16K/2K and 32K/4K block/frag sizes (and hence with defaults from +"newfs -O 2"). See [Panix's pvgrub +page](http://www.panix.com/v-colo/grub.html) which describes how to +boot NetBSD. + +### prgmr.com + +[prgmr.com](http://prgmr.com/) provides released versions of +NetBSD/amd64 as installation options. Users can use pvgrub to boot +their own kernel, and a small FAT32 /boot is encouraged. See the +[prgmr.com NetBSD +HOWTO](http://wiki.prgmr.com/mediawiki/index.php/NetBSD_as_a_DomU) +(which is in need of updating). + ### Amazon See the [Amazon EC2 page](/amazon_ec2/).