version 1.204, 2021/03/15 00:25:54
|
version 1.205, 2021/03/15 23:01:56
|
Line 40 approach and limits discussion of altern
|
Line 40 approach and limits discussion of altern
|
|
|
## Guest Styles |
## Guest Styles |
|
|
Xen supports different styles of guests. |
Xen supports different styles of guests. See |
|
https://wiki.xenproject.org/wiki/Virtualization_Spectrum for a |
|
discussion. |
|
|
|
This table shows the styles, and if a NetBSD dom0 can run in that |
|
style, if a NetBSD dom0 can sypport that style of guest in a domU, and |
|
if NetBSD as a domU can support that style. |
|
|
[[!table data=""" |
[[!table data=""" |
Style of guest |description |NetBSD dom0? |NetBSD domU? |
Style of guest |dom0 can be? |dom0 can support? |domU can be? |
PV |Paravirtualization (hypercalls) |yes |yes |
PV |yes |yes |yes |
HVM |unmodified guest - domU perceives HW |can run guests |yes |
HVM |N/A |yes |yes |
PVHVM |Like HVM but also PV drivers |can run guests |current only |
PVHVM |N/A |yes |current only |
PVH |Lightweight HVM (no qemu) and PV |not yet |current only |
PVH |not yet |current only |current only |
"""]] |
"""]] |
|
|
In Para-Virtualized (PV) mode, the guest OS does not attempt to access |
In PV (paravirtualized) mode, the guest OS does not attempt to access |
hardware directly, but instead makes hypercalls to the hypervisor; PV |
hardware directly, but instead makes hypercalls to the hypervisor; PV |
guests must be specifically coded for Xen. |
guests must be specifically coded for Xen. See |
See [PV](https://wiki.xen.org/wiki/Paravirtualization_(PV\)). |
[PV](https://wiki.xen.org/wiki/Paravirtualization_(PV\)). |
|
|
In HVM mode, no guest modification is required; however, hardware |
In HVM (Hardware Virtual Machine) mode, no guest modification is |
support is required, such as VT-x on Intel CPUs and SVM on AMD CPUs. |
required. However, hardware support is required, such as VT-x on |
The dom0 runs qemu to emulate hardware. It is therefore non-sensical |
Intel CPUs and SVM on AMD CPUs to assist with the processor emulation. |
to have an HVM dom0. |
The dom0 runs qemu to emulate hardware other than the processor. It |
|
is therefore non-sensical to have an HVM dom0, because there is no |
|
underlying system to provide emulation. |
|
|
In PVHVM mode, the guest runs as HVM, but additionally can use PV |
In PVHVM mode, the guest runs as HVM, but additionally uses PV |
drivers for efficiency. Therefore it is non-sensical for to have a |
drivers for efficiency. Therefore it is non-sensical for to have a |
PVHVM dom0. See [PV on HVM](https://wiki.xen.org/wiki/PV_on_HVM). |
PVHVM dom0. See [PV on HVM](https://wiki.xen.org/wiki/PV_on_HVM). |
|
|
There have been two PVH modes: original PVH and PVHv2. Original PVH |
There have been two PVH modes: original PVH and PVHv2. Original PVH |
was based on PV mode and is no longer relevant at all. Therefore |
was based on PV mode and is no longer relevant at all. Therefore |
PVHv2 is abreviated PVH. PVHv2 is basically lightweight HVM with PV |
PVHv2 is written as PVH, here and elsewhere. PVH is basically |
drivers. A critical feature of it is that qemu is not needed; the |
lightweight HVM with PV drivers. A critical feature of it is that |
hypervisor can do the emulation that is required. Thus, a dom0 can be |
qemu is not needed; the hypervisor can do the emulation that is |
PVHv2. The source code uses PVH and config files use pvh, but NB that |
required. Thus, a dom0 can be PVH. The source code uses PVH and |
this refers to PVHv2. See |
config files use pvh, but NB that this refers to PVHv2. See |
[PVH(v2)](https://wiki.xenproject.org/wiki/PVH_(v2\)_Domu). |
[PVH(v2)](https://wiki.xenproject.org/wiki/PVH_(v2\)_Domu). |
|
|
At system boot, the dom0 kernel is loaded as a module with Xen as the kernel. |
At system boot, the dom0 kernel is loaded as a module with Xen as the |
The dom0 can start one or more domUs. (Booting is explained in detail |
kernel. The dom0 can start one or more domUs. (Booting is explained |
in the dom0 section.) |
in detail in the dom0 section.) |
|
|
## CPU Architecture |
## CPU Architecture |
|
|
Line 116 Initially Xen was PV only.
|
Line 124 Initially Xen was PV only.
|
|
|
NetBSD Xen has always supported PV, in both dom0 and domU; for a long |
NetBSD Xen has always supported PV, in both dom0 and domU; for a long |
time this was the only way. NetBSD >=8 as a dom0 supports HVM mode in |
time this was the only way. NetBSD >=8 as a dom0 supports HVM mode in |
domUs (HVM as a dom0 does not make sense). |
domUs. |
|
|
Support for PVHVM and PVH is available only in NetBSD-current; this is |
Support for PVHVM and PVH is available only in NetBSD-current; this is |
currently somewhat experimental, although PVHVM appears reasonably |
currently somewhat experimental, although PVHVM appears reasonably |
Line 154 In 2018-05, trouble booting a dom0 was r
|
Line 162 In 2018-05, trouble booting a dom0 was r
|
using Xen. Therefore, use the most recent release, or a build from |
using Xen. Therefore, use the most recent release, or a build from |
the most recent stable branch. Alternatively, use -current, being |
the most recent stable branch. Alternatively, use -current, being |
mindful of all the usual caveats of lower stability of current, and |
mindful of all the usual caveats of lower stability of current, and |
likely a bit more so. |
likely a bit more so. Think about how you will provide storage for |
|
disk images. |
|
|
## Installation of Xen |
## Installation of Xen |
|
|
### Building Xen |
### Building Xen |
|
|
Use the most recent version of Xen in pkgsrc, unless the DESCR says that it is not suitable. |
Use the most recent version of Xen in pkgsrc, unless the DESCR says |
Therefore, choose 4.13. |
that it is not suitable. Therefore, choose 4.13. In the dom0, |
In the dom0, install xenkernel413 and xentools413 from pkgsrc. |
install xenkernel413 and xentools413 from pkgsrc. |
|
|
Once this is done, copy the Xen kernel from where pkgsrc puts it to |
Once this is done, copy the Xen kernel from where pkgsrc puts it to |
where the boot process will be able to find it: |
where the boot process will be able to find it: |
Line 208 wd0a causes boot to fail.
|
Line 217 wd0a causes boot to fail.
|
|
|
Beware that userconf statements must be attached to the dom0 load, and |
Beware that userconf statements must be attached to the dom0 load, and |
may not be at top-level, because then they would try to configure the |
may not be at top-level, because then they would try to configure the |
hypervisor, if there is a way to pass them via multiboot . It appears |
hypervisor, if there is a way to pass them via multiboot. It appears |
that adding `userconf=pckbc` to `/boot.cfg` causes Xen to crash very |
that adding `userconf=pckbc` to `/boot.cfg` causes Xen to crash very |
early with a heap overflow. |
early with a heap overflow. |
|
|
Line 264 messages from the hypervisor's console;
|
Line 273 messages from the hypervisor's console;
|
|
|
### Tuning |
### Tuning |
|
|
In an attempt to add performance, one can also add `dom0_max_vcpus=1 dom0_vcpus_pin`, |
In an attempt to add performance, one can also add `dom0_max_vcpus=1 |
to force only one vcpu to be provided (since NetBSD dom0 can't use |
dom0_vcpus_pin`, to force only one vcpu to be provided (since NetBSD |
more) and to pin that vcpu to a physical CPU. Xen has |
dom0 can't use more) and to pin that vcpu to a physical CPU. Xen has |
[many boot options](http://xenbits.xenproject.org/docs/4.13-testing/misc/xen-command-line.html), |
[many boot |
|
options](http://xenbits.xenproject.org/docs/4.13-testing/misc/xen-command-line.html), |
and other than dom0 memory and max_vcpus, they are generally not |
and other than dom0 memory and max_vcpus, they are generally not |
necessary. |
necessary. |
|
|
\todo Revisit this advice with current. |
\todo Revisit this advice with current. |
\todo Explain if anyone has ever actually measured that this helps. |
\todo Explain if anyone has ever actually measured that this helps. |
|
|
Line 328 dom0 kernel compared to hardware.
|
Line 339 dom0 kernel compared to hardware.
|
|
|
One is that through NetBSD 9 the module ABI is different because some |
One is that through NetBSD 9 the module ABI is different because some |
of the #defines change, so there are separate sets of modules in |
of the #defines change, so there are separate sets of modules in |
/stand. In NetBSD-current, there is only one set of modules. |
/stand. (Further, zfs in Xen is troubled because of differing |
|
MAXPHYS; see the zfs howto for more.) In NetBSD-current, there is |
|
only one set of modules. |
|
|
The other difference is that XEN3_DOM0 does not have exactly the same |
The other difference is that XEN3_DOM0 does not have exactly the same |
options as GENERIC. While it is debatable whether or not this is a |
options as GENERIC. While this is roughly agreed to be in large part |
bug, users should be aware of this and can simply add missing config |
a bug, users should be aware of this and can simply add missing config |
items if desired. |
items if desired. |
|
|
Finally, there have been occasional reports of trouble with X11 |
Finally, there have been occasional reports of trouble with X11 |
servers in NetBSD as a dom0. |
servers in NetBSD as a dom0. Some hardware support is intentionally |
|
disabled in XEN3_DOM0. |
|
|
## Updating Xen in a dom0 |
## Updating Xen in a dom0 |
|
|
Line 357 rescue purposes, as well as the DOM0 ker
|
Line 371 rescue purposes, as well as the DOM0 ker
|
|
|
## anita (for testing NetBSD) |
## anita (for testing NetBSD) |
|
|
With a NetBSD dom0, even without any domUs, one should be able to run |
With a NetBSD dom0, even without any domUs, one can run anita (see |
anita (see pkgsrc/misc/py-anita) to test NetBSD releases, by doing (as |
pkgsrc/misc/py-anita) to test NetBSD releases, by doing (as root, |
root, because anita must create a domU): |
because anita must create a domU): |
|
|
[[!template id=programlisting text=""" |
[[!template id=programlisting text=""" |
anita --vmm=xl test file:///usr/obj/i386/ |
anita --vmm=xl test file:///usr/obj/i386/ |
Line 385 the dom0. NetBSD's /dev/random system w
|
Line 399 the dom0. NetBSD's /dev/random system w
|
See /usr/pkg/share/examples/xen/xlexample* for a very small number of |
See /usr/pkg/share/examples/xen/xlexample* for a very small number of |
examples for running GNU/Linux. |
examples for running GNU/Linux. |
|
|
The following is an example minimal domain configuration file. The domU |
The following is an example minimal domain configuration file. The |
serves as a network file server. |
domU serves as a network file server. |
|
|
[[!template id=filecontent name="/usr/pkg/etc/xen/foo" text=""" |
[[!template id=filecontent name="/usr/pkg/etc/xen/foo" text=""" |
name = "domU-id" |
name = "domU-id" |
Line 406 are stored in files and Xen attaches the
|
Line 420 are stored in files and Xen attaches the
|
dom0 on domain creation. The system treats xbd0 as the boot device |
dom0 on domain creation. The system treats xbd0 as the boot device |
without needing explicit configuration. |
without needing explicit configuration. |
|
|
There is not type line; that implicitly defines a pv domU. Otherwise, |
There is not a type line; that implicitly defines a pv domU. |
one sets type to the lower-case version of the domU type in the table |
Otherwise, one sets type to the lower-case version of the domU type in |
above, e.g. `type = "hvm"`. |
the table above; see later sections. |
|
|
By convention, domain config files are kept in `/usr/pkg/etc/xen`. Note |
By convention, domain config files are kept in `/usr/pkg/etc/xen`. Note |
that "xl create" takes the name of a config file, while other commands |
that "xl create" takes the name of a config file, while other commands |
Line 431 will trigger controlled shutdowns of all
|
Line 445 will trigger controlled shutdowns of all
|
|
|
## CPU and memory |
## CPU and memory |
|
|
A domain is provided with some number of vcpus, up to the number |
A domain is provided with some number of vcpus; any domain can have up |
of CPUs seen by the hypervisor. For a domU, it is controlled |
to the number of CPUs seen by the hypervisor. For a domU, it is |
from the config file by the "vcpus = N" directive. |
controlled from the config file by the "vcpus = N" directive. It is |
|
normal to overcommit vcpus; a 4-core machine machine might well provide 4 |
|
vcpus to each domU. One might also configure fewer vcpus for a domU. |
|
|
A domain is provided with memory; this is controlled in the config |
A domain is provided with memory; this is controlled in the config |
file by "memory = N" (in megabytes). In the straightforward case, the |
file by "memory = N" (in megabytes). In the straightforward case, the |
Line 745 note that only the "xpci" lines are unus
|
Line 761 note that only the "xpci" lines are unus
|
|
|
## Nesting under Linux KVM |
## Nesting under Linux KVM |
|
|
It is possible to run a Xen and a NetBSD dom0 under Linux KVM. One |
It is possible to run Xen and a NetBSD dom0 under Linux KVM. One |
can enable virtio in the dom0 for greater speed. |
can enable virtio in the dom0 for greater speed. |
|
|
|
## Nesting under qemu |
|
|
|
It is possible to run Xen and a NetBSD dom0 under qemu on NetBSD, and |
|
also with nvmm. |
|
\todo Check this. |
|
|
## Other nesting |
## Other nesting |
|
|
In theory, any full emulation should be able to run Xen and a NetBSD |
In theory, any full emulation should be able to run Xen and a NetBSD |
dom0. The HOWTO does not currently have information about Xen XVM |
dom0. The HOWTO does not currently have information about Xen XVM |
mode, nvmm, qemu, Virtualbox, etc. |
mode, Virtualbox, etc. |
|
|
## NetBSD 5 as domU |
## NetBSD 5 as domU |
|
|
Line 793 different choices.
|
Line 815 different choices.
|
|
|
### stub domains |
### stub domains |
|
|
Some (Linux only?) dom0 systems use something called "stub domains" to |
Some (Linux) dom0 systems use something called "stub domains" to |
isolate qemu from the dom0 system, as a security and reliabilty |
isolate qemu from the dom0 system, as a security and reliabilty |
mechanism when running HVM domUs. Somehow, NetBSD's GENERIC kernel |
mechanism when running HVM domUs. Somehow, NetBSD's GENERIC kernel |
ends up using PIO for disks rather than DMA. Of course, all of this |
ends up using PIO for disks rather than DMA. Of course, all of this |
Line 803 currently understood.
|
Line 825 currently understood.
|
### Grant tables |
### Grant tables |
|
|
There are multiple versions of using grant tables, and some security |
There are multiple versions of using grant tables, and some security |
advisories have suggested disabling some versions. Some versions of |
advisories have suggested disabling some versions. NetBSD through 9 |
NetBSD apparently only use specific versions and this can lead to |
uses version 1 and NetBSD-current uses version 2. This can lead to |
"NetBSD current doesn't run on hosting provider X" situations. |
"NetBSD current doesn't run on hosting provider X" situations. |
|
|
\todo Explain better. |
\todo Explain better. |
Line 817 pvgrub is a version of grub that uses PV
|
Line 839 pvgrub is a version of grub that uses PV
|
calls. It is booted from the dom0 as the domU kernel, and then reads |
calls. It is booted from the dom0 as the domU kernel, and then reads |
/grub/menu.lst and loads a kernel from the domU file system. |
/grub/menu.lst and loads a kernel from the domU file system. |
|
|
[Panix](http://www.panix.com/) lets users use pvgrub. Panix reports |
|
that pvgrub works with FFsv2 with 16K/2K and 32K/4K block/frag sizes |
|
(and hence with defaults from "newfs -O 2"). See [Panix's pvgrub |
|
page](http://www.panix.com/v-colo/grub.html), which describes only |
|
Linux but should be updated to cover NetBSD :-). |
|
|
|
[prgmr.com](http://prgmr.com/) also lets users with pvgrub to boot |
|
their own kernel. See then [prgmr.com NetBSD |
|
HOWTO](http://wiki.prgmr.com/mediawiki/index.php/NetBSD_as_a_DomU) |
|
(which is in need of updating). |
|
|
|
It appears that [grub's FFS |
It appears that [grub's FFS |
code](http://xenbits.xensource.com/hg/xen-unstable.hg/file/bca284f67702/tools/libfsimage/ufs/fsys_ufs.c) |
code](http://xenbits.xensource.com/hg/xen-unstable.hg/file/bca284f67702/tools/libfsimage/ufs/fsys_ufs.c) |
does not support all aspects of modern FFS, but there are also reports |
does not support all aspects of modern FFS, but there are also reports |
that FFSv2 works fine. At prgmr, typically one has an ext2 or FAT |
that FFSv2 works fine. |
partition for the kernel with the intent that grub can understand it, |
|
which leads to /netbsd not being the actual kernel. One must remember |
|
to update the special boot partition. |
|
|
|
### pygrub |
### pygrub |
|
|
|
As of 2014, pygrub seems to be of mostly historical interest. As of |
|
2021, the section should perhaps be outright deleted. |
|
|
pygrub runs in the dom0 and looks into the domU file system. This |
pygrub runs in the dom0 and looks into the domU file system. This |
implies that the domU must have a kernel in a file system in a format |
implies that the domU must have a kernel in a file system in a format |
known to pygrub. |
known to pygrub. |
|
|
pygrub doesn't seem to work to load Linux images under NetBSD dom0, |
pygrub doesn't seem to work to load Linux images under NetBSD dom0, |
and is inherently less secure than pvgrub due to running inside dom0. For both these |
and is inherently less secure than pvgrub due to running inside |
reasons, pygrub should not be used, and is only still present so that |
dom0. For both these reasons, pygrub should not be used, and is only |
historical DomU images using it still work. |
still present so that historical DomU images using it still work. |
|
|
As of 2014, pygrub seems to be of mostly historical |
|
interest. New DomUs should use pvgrub. |
|
|
|
## Specific Providers |
## Specific Providers |
|
|
|
The intent is to list providers only if they document support for |
|
running NetBSD, and to point to their resources briefly. |
|
|
|
### panix.com |
|
|
|
[Panix](http://www.panix.com/) provides NetBSD as an OS option. See |
|
https://www.panix.com/v-colo/nupgrade.html for some information. |
|
Users can use pvgrub. Panix reports that pvgrub works with FFsv2 with |
|
16K/2K and 32K/4K block/frag sizes (and hence with defaults from |
|
"newfs -O 2"). See [Panix's pvgrub |
|
page](http://www.panix.com/v-colo/grub.html) which describes how to |
|
boot NetBSD. |
|
|
|
### prgmr.com |
|
|
|
[prgmr.com](http://prgmr.com/) provides released versions of |
|
NetBSD/amd64 as installation options. Users can use pvgrub to boot |
|
their own kernel, and a small FAT32 /boot is encouraged. See the |
|
[prgmr.com NetBSD |
|
HOWTO](http://wiki.prgmr.com/mediawiki/index.php/NetBSD_as_a_DomU) |
|
(which is in need of updating). |
|
|
### Amazon |
### Amazon |
|
|
See the [Amazon EC2 page](/amazon_ec2/). |
See the [Amazon EC2 page](/amazon_ec2/). |