|
version 1.178, 2020/11/14 01:00:40
|
version 1.182, 2020/11/15 14:47:55
|
|
Line 8 systems which operate in an unprivileged
|
Line 8 systems which operate in an unprivileged
|
| from the domU systems are forwarded by the Xen hypervisor to the dom0 to be |
from the domU systems are forwarded by the Xen hypervisor to the dom0 to be |
| fulfilled. |
fulfilled. |
| |
|
| Xen supports different styles of guests; see [PV on HVM](https://wiki.xen.org/wiki/PV_on_HVM) and [PVH(v2)](https://wiki.xenproject.org/wiki/PVH_(v2)_Domu) for upstream documentation. |
This HOWTO presumes a basic familiarity with the Xen system |
| |
architecture, with installing NetBSD on amd64 hardware, and with |
| |
installing software from pkgsrc. See also the [Xen |
| |
website](http://www.xenproject.org/). |
| |
|
| |
[[!toc]] |
| |
|
| |
# Overview: Versions of Xen, Styles, NetBSD |
| |
|
| |
The basic concept of Xen is that the hypervisor (xenkernel) runs on |
| |
the hardware, and runs a privileged domain ("dom0") that can access |
| |
disks/networking/etc. One then runs additonal unprivileged domains |
| |
(each a "domU"), presumably to do something useful. |
| |
|
| |
This HOWTO addresses how to run a NetBSD dom0 (and hence also build |
| |
xen itself). It also addresses how to run domUs in that environment, |
| |
and how to deal with having a domU in a Xen environment run by someone |
| |
else and/or not running NetBSD. |
| |
|
| |
## Guest Styles |
| |
|
| |
Xen supports different styles of guests. |
| |
|
| [[!table data=""" |
[[!table data=""" |
| Style of guest |Supported by NetBSD |
Style of guest |Supported by NetBSD |
| PV |Yes (dom0, domU) |
PV |Yes (dom0, domU) |
| HVM |Yes (domU) |
HVM |Yes (domU) |
| PVHVM |current-only (domU) |
PVHVM |current-only (domU) |
| PVHv2 |current-only (domU, dom0 not yet) |
PVH |current-only (domU, dom0 not yet) |
| """]] |
"""]] |
| |
|
| In Para-Virtualized (PV) mode, the guest OS does not attempt to access |
In Para-Virtualized (PV) mode, the guest OS does not attempt to access |
| hardware directly, but instead makes hypercalls to the hypervisor; PV |
hardware directly, but instead makes hypercalls to the hypervisor; PV |
| guests must be specifically coded for Xen. |
guests must be specifically coded for Xen. |
| |
See [PV](https://wiki.xen.org/wiki/Paravirtualization_(PV\)). |
| |
|
| In HVM mode, no guest modification is required; however, hardware |
In HVM mode, no guest modification is required; however, hardware |
| support is required, such as VT-x on Intel CPUs and SVM on AMD CPUs. |
support is required, such as VT-x on Intel CPUs and SVM on AMD CPUs. |
|
Line 28 The dom0 runs qemu to emulate hardware.
|
Line 50 The dom0 runs qemu to emulate hardware.
|
| |
|
| In PVHVM mode, the guest runs as HVM, but additionally can use PV |
In PVHVM mode, the guest runs as HVM, but additionally can use PV |
| drivers for efficiency. |
drivers for efficiency. |
| |
See [PV on HVM](https://wiki.xen.org/wiki/PV_on_HVM). |
| |
|
| In PVHv2H mode, operation is similar to PVHVM, except that qemu is not |
There have been two PVH modes: original PVH and PVHv2. Original PVH |
| run and thus the PV interfaces for console, disks, networking are the |
was based on PV mode and is no longer relevant at all. PVHv2 is |
| only way to access these resources. |
basically lightweight HVM with PV drivers. A critical feature of it |
| |
is that qemu is not needed; the hypervisor can do the emulation that |
| |
is required. Thus, a dom0 can be PVHv2. |
| |
The source code uses PVH and config files use pvh; this refers to PVHv2. |
| |
See [PVH(v2)](https://wiki.xenproject.org/wiki/PVH_(v2\)_Domu). |
| |
|
| At boot, the dom0 kernel is loaded as a module with Xen as the kernel. |
|
| |
At system boot, the dom0 kernel is loaded as a module with Xen as the kernel. |
| The dom0 can start one or more domUs. (Booting is explained in detail |
The dom0 can start one or more domUs. (Booting is explained in detail |
| in the dom0 section.) |
in the dom0 section.) |
| |
|
| This HOWTO presumes a basic familiarity with the Xen system |
## CPU Architecture |
| architecture, with installing NetBSD on i386/amd64 hardware, and with |
|
| installing software from pkgsrc. See also the [Xen |
|
| website](http://www.xenproject.org/). |
|
| |
|
| [[!toc]] |
Xen runs on x86_64 hardware (the NetBSD amd64 port). |
| |
|
| |
There is a concept of Xen running on ARM, but there are no reports of this working with NetBSD. |
| |
|
| |
The dom0 system should be amd64. (Instructions for i386PAE dom0 have been removed from the HOWTO.) |
| |
|
| # Versions and Support |
The domU can be i386PAE or amd64. |
| |
i386PAE at one point was considered as [faster](https://lists.xen.org/archives/html/xen-devel/2012-07/msg00085.html) than amd64. |
| |
|
| |
## Xen Versions |
| |
|
| In NetBSD, Xen is provided in pkgsrc, via matching pairs of packages |
In NetBSD, Xen is provided in pkgsrc, via matching pairs of packages |
| xenkernel and xentools. We will refer only to the kernel versions, |
xenkernel and xentools. We will refer only to the kernel versions, |
|
Line 75 because the hypercall interface has rema
|
Line 107 because the hypercall interface has rema
|
| Older Xen had a python-based management tool called xm, now replaced |
Older Xen had a python-based management tool called xm, now replaced |
| by xl. |
by xl. |
| |
|
| Architecture |
|
| ------------ |
|
| |
|
| Xen 4.5 and later runs on x86_64 hardware (the NetBSD amd64 port). |
|
| There is a concept of Xen running on ARM, but there are no reports of this working with NetBSD. |
|
| |
|
| The dom0 system should be amd64. (Instructions for i386PAE dom0 have been removed from the HOWTO.) |
|
| |
|
| The domU can be i386PAE or amd64. |
|
| i386PAE at one point was considered as [faster](https://lists.xen.org/archives/html/xen-devel/2012-07/msg00085.html) than amd64. |
|
| |
|
| # Creating a dom0 |
# Creating a dom0 |
| |
|
| In order to install a NetBSD as a dom0, one must first install a normal |
In order to install a NetBSD as a dom0, one must first install a normal |
|
Line 96 In 2018-05, trouble booting a dom0 was r
|
Line 117 In 2018-05, trouble booting a dom0 was r
|
| 512M it worked reliably. This does not make sense, but if you see |
512M it worked reliably. This does not make sense, but if you see |
| "not ELF" after Xen boots, try increasing dom0 RAM. |
"not ELF" after Xen boots, try increasing dom0 RAM. |
| |
|
| Installation of NetBSD |
## Installation of NetBSD |
| ---------------------- |
|
| |
|
| [Install NetBSD/amd64](/guide/inst/) |
[Install NetBSD/amd64](/guide/inst/) |
| just as you would if you were not using Xen. |
just as you would if you were not using Xen. |
| |
|
| Installation of Xen |
## Installation of Xen |
| ------------------- |
|
| |
|
| We will consider that you chose to use Xen 4.13, with NetBSD/amd64 as |
We will consider that you chose to use Xen 4.13, with NetBSD/amd64 as |
| dom0. In the dom0, install xenkernel48 and xentools48 from pkgsrc. |
dom0. In the dom0, install xenkernel48 and xentools48 from pkgsrc. |
|
Line 193 make this work, one should not expect to
|
Line 212 make this work, one should not expect to
|
| (and thus xencommons). There is currently no reason to expect that |
(and thus xencommons). There is currently no reason to expect that |
| this will get fixed any time soon. |
this will get fixed any time soon. |
| |
|
| anita (for testing NetBSD) |
## anita (for testing NetBSD) |
| -------------------------- |
|
| |
|
| With the setup so far, one should be able to run |
With the setup so far, one should be able to run |
| anita (see pkgsrc/misc/py-anita) to test NetBSD releases, by doing (as |
anita (see pkgsrc/misc/py-anita) to test NetBSD releases, by doing (as |
|
Line 204 root, because anita must create a domU):
|
Line 222 root, because anita must create a domU):
|
| anita --vmm=xl test file:///usr/obj/i386/ |
anita --vmm=xl test file:///usr/obj/i386/ |
| """]] |
"""]] |
| |
|
| Xen-specific NetBSD issues |
## Xen-specific NetBSD issues |
| -------------------------- |
|
| |
|
| There are (at least) two additional things different about NetBSD as a |
There are (at least) two additional things different about NetBSD as a |
| dom0 kernel compared to hardware. |
dom0 kernel compared to hardware. |
|
Line 219 options as GENERIC. While it is debatab
|
Line 236 options as GENERIC. While it is debatab
|
| bug, users should be aware of this and can simply add missing config |
bug, users should be aware of this and can simply add missing config |
| items if desired. |
items if desired. |
| |
|
| Updating NetBSD in a dom0 |
## Updating NetBSD in a dom0 |
| ------------------------- |
|
| |
|
| This is just like updating NetBSD on bare hardware, assuming the new |
This is just like updating NetBSD on bare hardware, assuming the new |
| version supports the version of Xen you are running. Generally, one |
version supports the version of Xen you are running. Generally, one |
|
Line 230 and adjusts `/etc`.
|
Line 246 and adjusts `/etc`.
|
| Note that one must update both the non-Xen kernel typically used for |
Note that one must update both the non-Xen kernel typically used for |
| rescue purposes and the DOM0 kernel used with Xen. |
rescue purposes and the DOM0 kernel used with Xen. |
| |
|
| Converting from grub to /boot |
## Converting from grub to /boot |
| ----------------------------- |
|
| |
|
| These instructions were used to convert a system from |
These instructions were used to convert a system from |
| grub to /boot. The system was originally installed in February of |
grub to /boot. The system was originally installed in February of |
|
Line 261 default=1
|
Line 276 default=1
|
| timeout=30 |
timeout=30 |
| """]] |
"""]] |
| |
|
| Upgrading Xen versions |
## Upgrading Xen versions |
| --------------------- |
|
| |
|
| Minor version upgrades are trivial. Just rebuild/replace the |
Minor version upgrades are trivial. Just rebuild/replace the |
| xenkernel version and copy the new xen.gz to `/` (where `/boot.cfg` |
xenkernel version and copy the new xen.gz to `/` (where `/boot.cfg` |
|
Line 283 mediated by Xen, and configured in the d
|
Line 297 mediated by Xen, and configured in the d
|
| Entropy in domUs can be an issue; physical disks and network are on |
Entropy in domUs can be an issue; physical disks and network are on |
| the dom0. NetBSD's /dev/random system works, but is often challenged. |
the dom0. NetBSD's /dev/random system works, but is often challenged. |
| |
|
| Config files |
## Config files |
| ------------ |
|
| |
|
| See /usr/pkg/share/examples/xen/xlexample* |
See /usr/pkg/share/examples/xen/xlexample* |
| for a small number of well-commented examples, mostly for running |
for a small number of well-commented examples, mostly for running |
|
Line 330 equivalent to pushing the power button;
|
Line 343 equivalent to pushing the power button;
|
| power-press event and do a clean shutdown. Shutting down the dom0 |
power-press event and do a clean shutdown. Shutting down the dom0 |
| will trigger controlled shutdowns of all configured domUs. |
will trigger controlled shutdowns of all configured domUs. |
| |
|
| CPU and memory |
## CPU and memory |
| -------------- |
|
| |
|
| A domain is provided with some number of vcpus, up to the number |
A domain is provided with some number of vcpus, up to the number |
| of CPUs seen by the hypervisor. For a domU, it is controlled |
of CPUs seen by the hypervisor. For a domU, it is controlled |
|
Line 345 than the available memory.
|
Line 357 than the available memory.
|
| Xen also provides a "balloon" driver, which can be used to let domains |
Xen also provides a "balloon" driver, which can be used to let domains |
| use more memory temporarily. |
use more memory temporarily. |
| |
|
| Virtual disks |
## Virtual disks |
| ------------- |
|
| |
|
| In domU config files, the disks are defined as a sequence of 3-tuples: |
In domU config files, the disks are defined as a sequence of 3-tuples: |
| |
|
|
Line 427 down cleanly on dom0 shutdown, add the f
|
Line 438 down cleanly on dom0 shutdown, add the f
|
| xendomains="domU-netbsd domU-linux" |
xendomains="domU-netbsd domU-linux" |
| """]] |
"""]] |
| |
|
| #Creating a domU |
# Creating a domU |
| |
|
| Creating domUs is almost entirely independent of operating system. We |
Creating domUs is almost entirely independent of operating system. We |
| have already presented the basics of config files. Note that you must |
have already presented the basics of config files. Note that you must |
| have already completed the dom0 setup so that "xl list" works. |
have already completed the dom0 setup so that "xl list" works. |
| |
|
| Creating a NetBSD domU |
Creating a NetBSD PV domU |
| ---------------------- |
-------------------------- |
| |
|
| See the earlier config file, and adjust memory. Decide on how much |
See the earlier config file, and adjust memory. Decide on how much |
| storage you will provide, and prepare it (file or LVM). |
storage you will provide, and prepare it (file or LVM). |
|
Line 549 To get the Linux console right, you need
|
Line 560 To get the Linux console right, you need
|
| to your configuration since not all Linux distributions auto-attach a |
to your configuration since not all Linux distributions auto-attach a |
| tty to the xen console. |
tty to the xen console. |
| |
|
| |
## Creating a NetBSD HVM domU |
| |
|
| |
Use type='hmv', probably. Use a GENERIC kernel within the disk image. |
| |
|
| |
## Creating a NetBSD PVH domU |
| |
|
| |
Use type='pvh'. |
| |
|
| |
\todo Explain where the kernel comes from. |
| |
|
| |
|
| Creating a Solaris domU |
Creating a Solaris domU |
| ----------------------- |
----------------------- |
| |
|
|
Line 559 See possibly outdated
|
Line 581 See possibly outdated
|
| PCI passthrough: Using PCI devices in guest domains |
PCI passthrough: Using PCI devices in guest domains |
| --------------------------------------------------- |
--------------------------------------------------- |
| |
|
| |
NB: PCI passthrough only works on some Xen versions and as of 2020 it |
| |
is not clear that it works on any version in pkgsrc. Reports |
| |
confirming or denying this notion should be sent to port-xen@. |
| |
|
| The dom0 can give other domains access to selected PCI |
The dom0 can give other domains access to selected PCI |
| devices. This can allow, for example, a non-privileged domain to have |
devices. This can allow, for example, a non-privileged domain to have |
| access to a physical network interface or disk controller. However, |
access to a physical network interface or disk controller. However, |
|
Line 660 A second issue is multiple CPUs. With N
|
Line 686 A second issue is multiple CPUs. With N
|
| multiple vcpus, and it is typical for VPS providers to enable multiple |
multiple vcpus, and it is typical for VPS providers to enable multiple |
| CPUs for NetBSD domUs. |
CPUs for NetBSD domUs. |
| |
|
| pvgrub |
## Complexities due to Xen changes |
| ------ |
|
| |
Xen has many security advisories and people running Xen systems make |
| |
different choices. |
| |
|
| |
### stub domains |
| |
|
| |
Some (Linux only?) dom0 systems use something called "stub domains" to |
| |
isolate qemu from the dom0 system, as a security and reliabilty |
| |
mechanism when running HVM domUs. Somehow, NetBSD's GENERIC kernel |
| |
ends up using PIO for disks rather than DMA. Of course, all of this |
| |
is emulated, but emulated PIO is unusably slow. This problem is not |
| |
currently understood. |
| |
|
| |
### Grant tables |
| |
|
| |
There are multiple versions of using grant tables, and some security |
| |
advisories have suggested disabling some versions. Some versions of |
| |
NetBSD apparently only use specific versions and this can lead to |
| |
"NetBSD current doesn't run on hosting provider X" situations. |
| |
|
| |
\todo Explain better. |
| |
|
| |
## Boot methods |
| |
|
| |
### pvgrub |
| |
|
| pvgrub is a version of grub that uses PV operations instead of BIOS |
pvgrub is a version of grub that uses PV operations instead of BIOS |
| calls. It is booted from the dom0 as the domU kernel, and then reads |
calls. It is booted from the dom0 as the domU kernel, and then reads |
|
Line 686 partition for the kernel with the intent
|
Line 736 partition for the kernel with the intent
|
| which leads to /netbsd not being the actual kernel. One must remember |
which leads to /netbsd not being the actual kernel. One must remember |
| to update the special boot partition. |
to update the special boot partition. |
| |
|
| pygrub |
### pygrub |
| ------- |
|
| |
|
| pygrub runs in the dom0 and looks into the domU file system. This |
pygrub runs in the dom0 and looks into the domU file system. This |
| implies that the domU must have a kernel in a file system in a format |
implies that the domU must have a kernel in a file system in a format |
|
Line 701 historical DomU images using it still wo
|
Line 750 historical DomU images using it still wo
|
| As of 2014, pygrub seems to be of mostly historical |
As of 2014, pygrub seems to be of mostly historical |
| interest. New DomUs should use pvgrub. |
interest. New DomUs should use pvgrub. |
| |
|
| Amazon |
## Specific Providers |
| ------ |
|
| |
### Amazon |
| |
|
| See the [Amazon EC2 page](/amazon_ec2/). |
See the [Amazon EC2 page](/amazon_ec2/). |
![[BACK]](/icons/back.gif){kind=icon}
Return to howto.mdwn CVS log ![[TXT]](/icons/text.gif){kind=icon}
![[DIR]](/icons/dir.gif){kind=icon}
Up to [NetBSD Developer Wiki] / wikisrc / ports / xen