|
version 1.159, 2018/09/05 09:25:10
|
version 1.180, 2020/11/15 14:31:58
|
|
Line 8 systems which operate in an unprivileged
|
Line 8 systems which operate in an unprivileged
|
| from the domU systems are forwarded by the Xen hypervisor to the dom0 to be |
from the domU systems are forwarded by the Xen hypervisor to the dom0 to be |
| fulfilled. |
fulfilled. |
| |
|
| Xen supports different styles of guest: |
Xen supports different styles of guests; see [PV on HVM](https://wiki.xen.org/wiki/PV_on_HVM) and [PVH(v2)](https://wiki.xenproject.org/wiki/PVH_(v2\)_Domu) for upstream documentation. |
| |
|
| [[!table data=""" |
[[!table data=""" |
| Style of guest |Supported by NetBSD |
Style of guest |Supported by NetBSD |
| PV |Yes (dom0, domU) |
PV |Yes (dom0, domU) |
| HVM |Yes (domU) |
HVM |Yes (domU) |
| PVHVM |No |
PVHVM |current-only (domU) |
| PVH |No |
PVH |current-only (domU, dom0 not yet) |
| """]] |
"""]] |
| |
|
| In Para-Virtualized (PV) mode, the guest OS does not attempt to access |
In Para-Virtualized (PV) mode, the guest OS does not attempt to access |
| hardware directly, but instead makes hypercalls to the hypervisor; PV |
hardware directly, but instead makes hypercalls to the hypervisor; PV |
| guests must be specifically coded for Xen. In HVM mode, no guest |
guests must be specifically coded for Xen. |
| modification is required; however, hardware support is required, such |
|
| as VT-x on Intel CPUs and SVM on AMD CPUs. |
In HVM mode, no guest modification is required; however, hardware |
| |
support is required, such as VT-x on Intel CPUs and SVM on AMD CPUs. |
| |
The dom0 runs qemu to emulate hardware. |
| |
|
| |
In PVHVM mode, the guest runs as HVM, but additionally can use PV |
| |
drivers for efficiency. |
| |
|
| |
There have been two PVH modes: original PVH and PVHv2. Original PVH |
| |
was based on PV mode and is no longer relevant at all. PVHv2 is |
| |
basically lightweight HVM with PV drivers. A critical feature of it |
| |
is that qemu is not needed; the hypervisor can do the emulation that |
| |
is required. Thus, a dom0 can be PVHv2. |
| |
|
| |
The source code uses PVH and config files use pvh; this refers to PVHv2. |
| |
|
| At boot, the dom0 kernel is loaded as a module with Xen as the kernel. |
At boot, the dom0 kernel is loaded as a module with Xen as the kernel. |
| The dom0 can start one or more domUs. (Booting is explained in detail |
The dom0 can start one or more domUs. (Booting is explained in detail |
|
Line 33 architecture, with installing NetBSD on
|
Line 46 architecture, with installing NetBSD on
|
| installing software from pkgsrc. See also the [Xen |
installing software from pkgsrc. See also the [Xen |
| website](http://www.xenproject.org/). |
website](http://www.xenproject.org/). |
| |
|
| This HOWTO attempts to address both the case of running a NetBSD dom0 |
[[!toc]] |
| on hardware and running domUs under it (NetBSD and other), and also |
|
| running NetBSD as a domU in a VPS. |
|
| |
|
| Versions and Support |
# Versions and Support |
| ==================== |
|
| |
|
| In NetBSD, Xen is provided in pkgsrc, via matching pairs of packages |
In NetBSD, Xen is provided in pkgsrc, via matching pairs of packages |
| xenkernel and xentools. We will refer only to the kernel versions, |
xenkernel and xentools. We will refer only to the kernel versions, |
|
Line 48 matching versions.
|
Line 58 matching versions.
|
| Versions available in pkgsrc: |
Versions available in pkgsrc: |
| |
|
| [[!table data=""" |
[[!table data=""" |
| Xen Version |Package Name |Xen CPU Support |EOL'ed By Upstream |
Xen Version |Package Name |Xen CPU Support |xm? |EOL'ed By Upstream |
| 4.2 |xenkernel42 |32bit, 64bit |Yes |
4.11 |xenkernel411 |x86_64 | |No |
| 4.5 |xenkernel45 |64bit |Yes |
4.13 |xenkernel413 |x86_64 | |No |
| 4.6 |xenkernel46 |64bit |Partially |
|
| 4.8 |xenkernel48 |64bit |No |
|
| 4.11 |xenkernel411 |64bit |No |
|
| """]] |
"""]] |
| |
|
| See also the [Xen Security Advisory page](http://xenbits.xen.org/xsa/). |
See also the [Xen Security Advisory page](http://xenbits.xen.org/xsa/). |
|
Line 66 dom0 |No
|
Line 73 dom0 |No
|
| domU |Yes |
domU |Yes |
| """]] |
"""]] |
| |
|
| Note: NetBSD support is called XEN3. However, it does support Xen 4, |
Note: NetBSD support is called XEN3. However, it does support Xen 4, |
| because the hypercall interface has remained identical. |
because the hypercall interface has remained identical. |
| |
|
| |
Older Xen had a python-based management tool called xm, now replaced |
| |
by xl. |
| |
|
| Architecture |
Architecture |
| ------------ |
------------ |
| |
|
| Xen itself runs on x86_64 hardware. |
Xen 4.5 and later runs on x86_64 hardware (the NetBSD amd64 port). |
| |
There is a concept of Xen running on ARM, but there are no reports of this working with NetBSD. |
| The dom0 system, plus each domU, can be either i386PAE or amd64. |
|
| i386 without PAE is not supported. |
|
| |
|
| The standard approach is to use NetBSD/amd64 for the dom0. |
|
| |
|
| To use an i386PAE dom0, one must build or obtain a 64bit Xen kernel and |
The dom0 system should be amd64. (Instructions for i386PAE dom0 have been removed from the HOWTO.) |
| install it on the system. |
|
| |
|
| For domUs, i386PAE is considered as |
The domU can be i386PAE or amd64. |
| [faster](https://lists.xen.org/archives/html/xen-devel/2012-07/msg00085.html) |
i386PAE at one point was considered as [faster](https://lists.xen.org/archives/html/xen-devel/2012-07/msg00085.html) than amd64. |
| than amd64. |
|
| |
|
| NetBSD as a dom0 |
# Creating a dom0 |
| ================ |
|
| |
|
| In order to install a NetBSD as a dom0, one must first install a normal |
In order to install a NetBSD as a dom0, one must first install a normal |
| NetBSD system, and then pivot the install to a dom0 install by changing |
NetBSD system, and then pivot the install to a dom0 install by changing |
|
Line 106 just as you would if you were not using
|
Line 109 just as you would if you were not using
|
| Installation of Xen |
Installation of Xen |
| ------------------- |
------------------- |
| |
|
| We will consider that you chose to use Xen 4.8, with NetBSD/amd64 as |
We will consider that you chose to use Xen 4.13, with NetBSD/amd64 as |
| dom0. In the dom0, install xenkernel48 and xentools48 from pkgsrc. |
dom0. In the dom0, install xenkernel48 and xentools48 from pkgsrc. |
| |
|
| Once this is done, install the Xen kernel itself: |
Once this is done, install the Xen kernel itself: |
|
Line 146 itself uses (in this case, the serial po
|
Line 149 itself uses (in this case, the serial po
|
| In an attempt to add performance, one can also add `dom0_max_vcpus=1 dom0_vcpus_pin`, |
In an attempt to add performance, one can also add `dom0_max_vcpus=1 dom0_vcpus_pin`, |
| to force only one vcpu to be provided (since NetBSD dom0 can't use |
to force only one vcpu to be provided (since NetBSD dom0 can't use |
| more) and to pin that vcpu to a physical CPU. Xen has |
more) and to pin that vcpu to a physical CPU. Xen has |
| [many boot options](http://xenbits.xenproject.org/docs/4.8-testing/misc/xen-command-line.html), |
[many boot options](http://xenbits.xenproject.org/docs/4.13-testing/misc/xen-command-line.html), |
| and other than dom0 memory and max_vcpus, they are generally not |
and other than dom0 memory and max_vcpus, they are generally not |
| necessary. |
necessary. |
| |
|
|
Line 197 this will get fixed any time soon.
|
Line 200 this will get fixed any time soon.
|
| anita (for testing NetBSD) |
anita (for testing NetBSD) |
| -------------------------- |
-------------------------- |
| |
|
| With the setup so far (assuming 4.8/xl), one should be able to run |
With the setup so far, one should be able to run |
| anita (see pkgsrc/misc/py-anita) to test NetBSD releases, by doing (as |
anita (see pkgsrc/misc/py-anita) to test NetBSD releases, by doing (as |
| root, because anita must create a domU): |
root, because anita must create a domU): |
| |
|
|
Line 269 Minor version upgrades are trivial. Jus
|
Line 272 Minor version upgrades are trivial. Jus
|
| xenkernel version and copy the new xen.gz to `/` (where `/boot.cfg` |
xenkernel version and copy the new xen.gz to `/` (where `/boot.cfg` |
| references it), and reboot. |
references it), and reboot. |
| |
|
| Unprivileged domains (domU) |
#Unprivileged domains (domU) |
| =========================== |
|
| |
|
| This section describes general concepts about domUs. It does not |
This section describes general concepts about domUs. It does not |
| address specific domU operating systems or how to install them. The |
address specific domU operating systems or how to install them. The |
|
Line 288 the dom0. NetBSD's /dev/random system w
|
Line 290 the dom0. NetBSD's /dev/random system w
|
| Config files |
Config files |
| ------------ |
------------ |
| |
|
| See /usr/pkg/share/examples/xen/xlexample*, |
See /usr/pkg/share/examples/xen/xlexample* |
| for a large number of well-commented examples, mostly for running |
for a small number of well-commented examples, mostly for running |
| GNU/Linux. |
GNU/Linux. |
| |
|
| The following is an example minimal domain configuration file. The domU |
The following is an example minimal domain configuration file. The domU |
|
Line 313 are stored in files and Xen attaches the
|
Line 315 are stored in files and Xen attaches the
|
| dom0 on domain creation. The system treats xbd0 as the boot device |
dom0 on domain creation. The system treats xbd0 as the boot device |
| without needing explicit configuration. |
without needing explicit configuration. |
| |
|
| By default, `xl` looks for domain config files in `/usr/pkg/etc/xen`. Note |
By convention, domain config files are kept in `/usr/pkg/etc/xen`. Note |
| that "xl create" takes the name of a config file, while other commands |
that "xl create" takes the name of a config file, while other commands |
| take the name of a domain. |
take the name of a domain. |
| |
|
| Examples of commands: |
Examples of commands: |
| |
|
| [[!template id=programlisting text=""" |
[[!template id=programlisting text=""" |
| xl create foo |
xl create /usr/pkg/etc/xen/foo |
| xl console foo |
xl console domU-id |
| xl create -c foo |
xl create -c /usr/pkg/etc/xen/foo |
| xl shutdown foo |
xl shutdown domU-id |
| xl list |
xl list |
| """]] |
"""]] |
| |
|
|
Line 335 will trigger controlled shutdowns of all
|
Line 337 will trigger controlled shutdowns of all
|
| CPU and memory |
CPU and memory |
| -------------- |
-------------- |
| |
|
| A domain is provided with some number of vcpus, less than the number |
A domain is provided with some number of vcpus, up to the number |
| of CPUs seen by the hypervisor. For a domU, it is controlled |
of CPUs seen by the hypervisor. For a domU, it is controlled |
| from the config file by the "vcpus = N" directive. |
from the config file by the "vcpus = N" directive. |
| |
|
|
Line 353 Virtual disks
|
Line 355 Virtual disks
|
| In domU config files, the disks are defined as a sequence of 3-tuples: |
In domU config files, the disks are defined as a sequence of 3-tuples: |
| |
|
| * The first element is "method:/path/to/disk". Common methods are |
* The first element is "method:/path/to/disk". Common methods are |
| "file:" for file-backed vnd, and "phy:" for something that is already |
"file:" for a file-backed vnd, and "phy:" for something that is already |
| a device. |
a device, such as an LVM logical volume. |
| |
|
| * The second element is an artifact of how virtual disks are passed to |
* The second element is an artifact of how virtual disks are passed to |
| Linux, and a source of confusion with NetBSD Xen usage. Linux domUs |
Linux, and a source of confusion with NetBSD Xen usage. Linux domUs |
|
Line 429 down cleanly on dom0 shutdown, add the f
|
Line 431 down cleanly on dom0 shutdown, add the f
|
| xendomains="domU-netbsd domU-linux" |
xendomains="domU-netbsd domU-linux" |
| """]] |
"""]] |
| |
|
| Creating a domU |
# Creating a domU |
| =============== |
|
| |
|
| Creating domUs is almost entirely independent of operating system. We |
Creating domUs is almost entirely independent of operating system. We |
| have already presented the basics of config files. Note that you must |
have already presented the basics of config files. Note that you must |
| have already completed the dom0 setup so that "xl list" works. |
have already completed the dom0 setup so that "xl list" works. |
| |
|
| Creating a NetBSD domU |
Creating a NetBSD PV domU |
| ---------------------- |
-------------------------- |
| |
|
| See the earlier config file, and adjust memory. Decide on how much |
See the earlier config file, and adjust memory. Decide on how much |
| storage you will provide, and prepare it (file or lvm). |
storage you will provide, and prepare it (file or LVM). |
| |
|
| While the kernel will be obtained from the dom0 file system, the same |
While the kernel will be obtained from the dom0 file system, the same |
| file should be present in the domU as /netbsd so that tools like |
file should be present in the domU as /netbsd so that tools like |
|
Line 463 kernel to / and change the kernel line i
|
Line 464 kernel to / and change the kernel line i
|
| |
|
| kernel = "/home/bouyer/netbsd-INSTALL_XEN3_DOMU" |
kernel = "/home/bouyer/netbsd-INSTALL_XEN3_DOMU" |
| |
|
| Then, start the domain as "xl create -c configname". |
Then, start the domain as "xl create -c configfile". |
| |
|
| Alternatively, if you want to install NetBSD/Xen with a CDROM image, the following |
Alternatively, if you want to install NetBSD/Xen with a CDROM image, the following |
| line should be used in the config file. |
line should be used in the config file. |
|
Line 552 To get the Linux console right, you need
|
Line 553 To get the Linux console right, you need
|
| to your configuration since not all Linux distributions auto-attach a |
to your configuration since not all Linux distributions auto-attach a |
| tty to the xen console. |
tty to the xen console. |
| |
|
| |
## Creating a NetBSD HVM domU |
| |
|
| |
Use type='hmv', probably. Use a GENERIC kernel within the disk image. |
| |
|
| |
## Creating a NetBSD PVH domU |
| |
|
| |
Use type='pvh'. |
| |
|
| |
\todo Explain where the kernel comes from. |
| |
|
| |
|
| Creating a Solaris domU |
Creating a Solaris domU |
| ----------------------- |
----------------------- |
| |
|
|
Line 562 See possibly outdated
|
Line 574 See possibly outdated
|
| PCI passthrough: Using PCI devices in guest domains |
PCI passthrough: Using PCI devices in guest domains |
| --------------------------------------------------- |
--------------------------------------------------- |
| |
|
| |
NB: PCI passthrough only works on some Xen versions and as of 2020 it |
| |
is not clear that it works on any version in pkgsrc. Reports |
| |
confirming or denying this notion should be sent to port-xen@. |
| |
|
| The dom0 can give other domains access to selected PCI |
The dom0 can give other domains access to selected PCI |
| devices. This can allow, for example, a non-privileged domain to have |
devices. This can allow, for example, a non-privileged domain to have |
| access to a physical network interface or disk controller. However, |
access to a physical network interface or disk controller. However, |
|
Line 629 note that only the "xpci" lines are unus
|
Line 645 note that only the "xpci" lines are unus
|
| cd* at scsibus? target ? lun ? # SCSI CD-ROM drives |
cd* at scsibus? target ? lun ? # SCSI CD-ROM drives |
| |
|
| |
|
| NetBSD as a domU in a VPS |
# Specific Issues |
| ========================= |
|
| |
## domU |
| |
|
| |
[NetBSD 5 is known to panic.](http://mail-index.netbsd.org/port-xen/2018/04/17/msg009181.html) |
| |
(However, NetBSD 5 systems should be updated to a supported version.) |
| |
|
| |
# NetBSD as a domU in a VPS |
| |
|
| The bulk of the HOWTO is about using NetBSD as a dom0 on your own |
The bulk of the HOWTO is about using NetBSD as a dom0 on your own |
| hardware. This section explains how to deal with Xen in a domU as a |
hardware. This section explains how to deal with Xen in a domU as a |
|
Line 657 A second issue is multiple CPUs. With N
|
Line 679 A second issue is multiple CPUs. With N
|
| multiple vcpus, and it is typical for VPS providers to enable multiple |
multiple vcpus, and it is typical for VPS providers to enable multiple |
| CPUs for NetBSD domUs. |
CPUs for NetBSD domUs. |
| |
|
| pygrub |
## Complexities due to Xen changes |
| ------- |
|
| |
|
| pygrub runs in the dom0 and looks into the domU file system. This |
Xen has many security advisories and people running Xen systems make |
| implies that the domU must have a kernel in a file system in a format |
different choices. |
| known to pygrub. As of 2014, pygrub seems to be of mostly historical |
|
| interest. |
### stub domains |
| |
|
| |
Some (Linux only?) dom0 systems use something called "stub domains" to |
| |
isolate qemu from the dom0 system, as a security and reliabilty |
| |
mechanism when running HVM domUs. Somehow, NetBSD's GENERIC kernel |
| |
ends up using PIO for disks rather than DMA. Of course, all of this |
| |
is emulated, but emulated PIO is unusably slow. This problem is not |
| |
currently understood. |
| |
|
| |
### Grant tables |
| |
|
| |
There are multiple versions of using grant tables, and some security |
| |
advisories have suggested disabling some versions. Some versions of |
| |
NetBSD apparently only use specific versions and this can lead to |
| |
"NetBSD current doesn't run on hosting provider X" situations. |
| |
|
| |
\todo Explain better. |
| |
|
| pvgrub |
pvgrub |
| ------ |
------ |
|
Line 691 partition for the kernel with the intent
|
Line 728 partition for the kernel with the intent
|
| which leads to /netbsd not being the actual kernel. One must remember |
which leads to /netbsd not being the actual kernel. One must remember |
| to update the special boot partition. |
to update the special boot partition. |
| |
|
| Amazon |
pygrub |
| ------ |
------- |
| |
|
| See the [Amazon EC2 page](/amazon_ec2/). |
pygrub runs in the dom0 and looks into the domU file system. This |
| |
implies that the domU must have a kernel in a file system in a format |
| |
known to pygrub. |
| |
|
| Random pointers |
pygrub doesn't seem to work to load Linux images under NetBSD dom0, |
| =============== |
and is inherently less secure than pvgrub due to running inside dom0. For both these |
| |
reasons, pygrub should not be used, and is only still present so that |
| |
historical DomU images using it still work. |
| |
|
| This section contains links from elsewhere not yet integrated into the |
As of 2014, pygrub seems to be of mostly historical |
| HOWTO, and other guides. |
interest. New DomUs should use pvgrub. |
| |
|
| * http://www.lumbercartel.ca/library/xen/ |
Amazon |
| * http://pbraun.nethence.com/doc/sysutils/xen_netbsd_dom0.html |
------ |
| * https://gmplib.org/~tege/xen.html |
|
| |
See the [Amazon EC2 page](/amazon_ec2/). |
![[BACK]](/icons/back.gif){kind=icon}
Return to howto.mdwn CVS log ![[TXT]](/icons/text.gif){kind=icon}
![[DIR]](/icons/dir.gif){kind=icon}
Up to [NetBSD Developer Wiki] / wikisrc / ports / xen