version 1.157, 2018/08/27 16:54:42
|
version 1.189, 2020/11/19 13:55:58
|
Line 8 systems which operate in an unprivileged
|
Line 8 systems which operate in an unprivileged
|
from the domU systems are forwarded by the Xen hypervisor to the dom0 to be |
from the domU systems are forwarded by the Xen hypervisor to the dom0 to be |
fulfilled. |
fulfilled. |
|
|
Xen supports different styles of guest: |
This HOWTO presumes a basic familiarity with the Xen system |
|
architecture, with installing NetBSD on amd64 hardware, and with |
|
installing software from pkgsrc. See also the [Xen |
|
website](http://www.xenproject.org/). |
|
|
|
[[!toc]] |
|
|
|
# Overview |
|
|
|
The basic concept of Xen is that the hypervisor (xenkernel) runs on |
|
the hardware, and runs a privileged domain ("dom0") that can access |
|
disks/networking/etc. One then runs additonal unprivileged domains |
|
(each a "domU"), presumably to do something useful. |
|
|
|
This HOWTO addresses how to run a NetBSD dom0 (and hence also build |
|
xen itself). It also addresses how to run domUs in that environment, |
|
and how to deal with having a domU in a Xen environment run by someone |
|
else and/or not running NetBSD. |
|
|
|
There are many choices one can make; the HOWTO recommends the standard |
|
approach and limits discussion of alternatives in many cases. |
|
|
|
## Guest Styles |
|
|
|
Xen supports different styles of guests. |
|
|
[[!table data=""" |
[[!table data=""" |
Style of guest |Supported by NetBSD |
Style of guest |Supported by NetBSD |
PV |Yes (dom0, domU) |
PV |Yes (dom0, domU) |
HVM |Yes (domU) |
HVM |Yes (domU) |
PVHVM |No |
PVHVM |current-only (domU) |
PVH |No |
PVH |current-only (domU, dom0 not yet) |
"""]] |
"""]] |
|
|
In Para-Virtualized (PV) mode, the guest OS does not attempt to access |
In Para-Virtualized (PV) mode, the guest OS does not attempt to access |
hardware directly, but instead makes hypercalls to the hypervisor; PV |
hardware directly, but instead makes hypercalls to the hypervisor; PV |
guests must be specifically coded for Xen. In HVM mode, no guest |
guests must be specifically coded for Xen. |
modification is required; however, hardware support is required, such |
See [PV](https://wiki.xen.org/wiki/Paravirtualization_(PV\)). |
as VT-x on Intel CPUs and SVM on AMD CPUs. |
|
|
In HVM mode, no guest modification is required; however, hardware |
|
support is required, such as VT-x on Intel CPUs and SVM on AMD CPUs. |
|
The dom0 runs qemu to emulate hardware. |
|
|
|
In PVHVM mode, the guest runs as HVM, but additionally can use PV |
|
drivers for efficiency. |
|
See [PV on HVM](https://wiki.xen.org/wiki/PV_on_HVM). |
|
|
|
There have been two PVH modes: original PVH and PVHv2. Original PVH |
|
was based on PV mode and is no longer relevant at all. PVHv2 is |
|
basically lightweight HVM with PV drivers. A critical feature of it |
|
is that qemu is not needed; the hypervisor can do the emulation that |
|
is required. Thus, a dom0 can be PVHv2. |
|
The source code uses PVH and config files use pvh; this refers to PVHv2. |
|
See [PVH(v2)](https://wiki.xenproject.org/wiki/PVH_(v2\)_Domu). |
|
|
At boot, the dom0 kernel is loaded as a module with Xen as the kernel. |
At system boot, the dom0 kernel is loaded as a module with Xen as the kernel. |
The dom0 can start one or more domUs. (Booting is explained in detail |
The dom0 can start one or more domUs. (Booting is explained in detail |
in the dom0 section.) |
in the dom0 section.) |
|
|
This HOWTO presumes a basic familiarity with the Xen system |
## CPU Architecture |
architecture, with installing NetBSD on i386/amd64 hardware, and with |
|
installing software from pkgsrc. See also the [Xen |
|
website](http://www.xenproject.org/). |
|
|
|
This HOWTO attempts to address both the case of running a NetBSD dom0 |
Xen runs on x86_64 hardware (the NetBSD amd64 port). |
on hardware and running domUs under it (NetBSD and other), and also |
|
running NetBSD as a domU in a VPS. |
|
|
|
Versions and Support |
There is a concept of Xen running on ARM, but there are no reports of this working with NetBSD. |
==================== |
|
|
The dom0 system should be amd64. (Instructions for i386PAE dom0 have been removed from the HOWTO.) |
|
|
|
The domU can be i386PAE or amd64. |
|
i386PAE at one point was considered as [faster](https://lists.xen.org/archives/html/xen-devel/2012-07/msg00085.html) than amd64. |
|
|
|
## Xen Versions |
|
|
In NetBSD, Xen is provided in pkgsrc, via matching pairs of packages |
In NetBSD, Xen is provided in pkgsrc, via matching pairs of packages |
xenkernel and xentools. We will refer only to the kernel versions, |
xenkernel and xentools. We will refer only to the kernel versions, |
Line 49 Versions available in pkgsrc:
|
Line 89 Versions available in pkgsrc:
|
|
|
[[!table data=""" |
[[!table data=""" |
Xen Version |Package Name |Xen CPU Support |EOL'ed By Upstream |
Xen Version |Package Name |Xen CPU Support |EOL'ed By Upstream |
4.2 |xenkernel42 |32bit, 64bit |Yes |
4.11 |xenkernel411 |x86_64 |No |
4.5 |xenkernel45 |64bit |Yes |
4.13 |xenkernel413 |x86_64 |No |
4.6 |xenkernel46 |64bit |Partially |
|
4.8 |xenkernel48 |64bit |No |
|
4.11 |xenkernel411 |64bit |No |
|
"""]] |
"""]] |
|
|
See also the [Xen Security Advisory page](http://xenbits.xen.org/xsa/). |
See also the [Xen Security Advisory page](http://xenbits.xen.org/xsa/). |
|
|
Multiprocessor (SMP) support in NetBSD differs depending on the domain: |
Older Xen had a python-based management tool called xm, now replaced |
|
by xl. |
|
|
[[!table data=""" |
## NetBSD versions |
Domain |Supports SMP |
|
dom0 |No |
|
domU |Yes |
|
"""]] |
|
|
|
Note: NetBSD support is called XEN3. However, it does support Xen 4, |
|
because the hypercall interface has remained identical. |
|
|
|
Architecture |
Xen has been supported in NetBSD for a long time, at least since 2005. |
------------ |
Initially Xen was PV only. |
|
|
Xen itself runs on x86_64 hardware. |
NetBSD 8 and up support PV and HVM modes. |
|
|
The dom0 system, plus each domU, can be either i386PAE or amd64. |
Support for PVHVM and PVH is available only in NetBSD-current. |
i386 without PAE is not supported. |
|
|
|
The standard approach is to use NetBSD/amd64 for the dom0. |
NetBSD as a dom0 does not run SMP, because some drivers are not yet |
|
safe for this. \todo Link to more information about what needs work. |
|
|
To use an i386PAE dom0, one must build or obtain a 64bit Xen kernel and |
NetBSD, when run as a domU, can and does typically run SMP. |
install it on the system. |
|
|
|
For domUs, i386PAE is considered as |
Note: NetBSD support is called XEN3. However, it does support Xen 4, |
[faster](https://lists.xen.org/archives/html/xen-devel/2012-07/msg00085.html) |
because the hypercall interface has remained identical. |
than amd64. |
|
|
|
NetBSD as a dom0 |
|
================ |
|
|
|
NetBSD can be used as a dom0 and works very well. The following |
# Creating a NetBSD dom0 |
sections address installation, updating NetBSD, and updating Xen. |
|
|
|
Note that it doesn't make sense to talk about installing a dom0 OS |
In order to install a NetBSD as a dom0, one must first install a normal |
without also installing Xen itself. We first address installing |
NetBSD system, and then pivot the install to a dom0 install by changing |
NetBSD, which is not yet a dom0, and then adding Xen, pivoting the |
the kernel and boot configuration. |
NetBSD install to a dom0 install by just changing the kernel and boot |
|
configuration. |
|
|
|
In 2018-05, trouble booting a dom0 was reported with 256M of RAM: with |
In 2018-05, trouble booting a dom0 was reported with 256M of RAM: with |
512M it worked reliably. This does not make sense, but if you see |
512M it worked reliably. This does not make sense, but if you see |
"not ELF" after Xen boots, try increasing dom0 RAM. |
"not ELF" after Xen boots, try increasing dom0 RAM. |
|
|
Styles of dom0 operation |
## Installation of NetBSD |
------------------------ |
|
|
|
There are two basic ways to use Xen. The traditional method is for |
|
the dom0 to do absolutely nothing other than providing support to some |
|
number of domUs. Such a system was probably installed for the sole |
|
purpose of hosting domUs, and sits in a server room on a UPS. |
|
|
|
The other way is to put Xen under a normal-usage computer, so that the |
|
dom0 is what the computer would have been without Xen, perhaps a |
|
desktop or laptop. Then, one can run domUs at will. |
|
|
|
Installation of NetBSD |
|
---------------------- |
|
|
|
[Install NetBSD/amd64](/guide/inst/) |
[Install NetBSD/amd64](/guide/inst/) |
just as you would if you were not using Xen. |
just as you would if you were not using Xen. |
|
Therefore, use the most recent release, or a build from the most recent stable branch. |
|
|
Installation of Xen |
## Installation of Xen |
------------------- |
|
|
|
We will consider that you chose to use Xen 4.8, with NetBSD/amd64 as |
Use the most recent version of Xen in pkgsrc, unless the DESCR says that it is not suitable. |
dom0. In the dom0, install xenkernel48 and xentools48 from pkgsrc. |
Therefore, choose 4.13. |
Ensure that your packages are recent. |
In the dom0, install xenkernel413 and xentools413 from pkgsrc. |
|
|
Once this is done, install the Xen kernel itself: |
Once this is done, copy the Xen kernel from where pkgsrc puts it to |
|
where the boot process will be able to find it: |
|
|
[[!template id=programlisting text=""" |
[[!template id=programlisting text=""" |
# cp /usr/pkg/xen48-kernel/xen.gz / |
# cp -p /usr/pkg/xen413-kernel/xen.gz / |
"""]] |
"""]] |
|
|
Then, place a NetBSD XEN3_DOM0 kernel in the `/` directory, copied from |
Then, place a NetBSD XEN3_DOM0 kernel in the `/` directory. Such kernel |
`releasedir/amd64/binary/kernel/netbsd-XEN3_DOM0.gz` of a NetBSD build. |
can either be compiled manually, or downloaded from the NetBSD FTP, for |
|
example at: |
|
|
|
[[!template id=programlisting text=""" |
|
ftp.netbsd.org/pub/NetBSD/NetBSD-9.1/amd64/binary/kernel/netbsd-XEN3_DOM0.gz |
|
"""]] |
|
|
Add a line to /boot.cfg to boot Xen: |
Add a line to /boot.cfg to boot Xen: |
|
|
Line 142 Add a line to /boot.cfg to boot Xen:
|
Line 158 Add a line to /boot.cfg to boot Xen:
|
menu=Xen:load /netbsd-XEN3_DOM0.gz console=pc;multiboot /xen.gz dom0_mem=512M |
menu=Xen:load /netbsd-XEN3_DOM0.gz console=pc;multiboot /xen.gz dom0_mem=512M |
"""]] |
"""]] |
|
|
which specifies that the dom0 should have 512MB of ram, leaving the rest |
This specifies that the dom0 should have 512MB of ram, leaving the rest |
to be allocated for domUs. To use a serial console, use |
to be allocated for domUs. |
|
|
|
NB: This says add, not replace, so that you will be able to boot a |
|
NetBSD kernel without Xen. Once Xen boots ok, you may want to set it |
|
as default. |
|
|
|
To use a serial console, add settings as follows: |
|
|
[[!template id=filecontent name="/boot.cfg" text=""" |
[[!template id=filecontent name="/boot.cfg" text=""" |
menu=Xen:load /netbsd-XEN3_DOM0.gz;multiboot /xen.gz dom0_mem=512M console=com1 com1=9600,8n1 |
menu=Xen:load /netbsd-XEN3_DOM0.gz;multiboot /xen.gz dom0_mem=512M console=com1 com1=9600,8n1 |
Line 158 itself uses (in this case, the serial po
|
Line 180 itself uses (in this case, the serial po
|
|
|
In an attempt to add performance, one can also add `dom0_max_vcpus=1 dom0_vcpus_pin`, |
In an attempt to add performance, one can also add `dom0_max_vcpus=1 dom0_vcpus_pin`, |
to force only one vcpu to be provided (since NetBSD dom0 can't use |
to force only one vcpu to be provided (since NetBSD dom0 can't use |
more) and to pin that vcpu to a physical CPU. |
more) and to pin that vcpu to a physical CPU. Xen has |
|
[many boot options](http://xenbits.xenproject.org/docs/4.13-testing/misc/xen-command-line.html), |
Xen has [many boot |
|
options](http://xenbits.xenproject.org/docs/4.5-testing/misc/xen-command-line.html), |
|
and other than dom0 memory and max_vcpus, they are generally not |
and other than dom0 memory and max_vcpus, they are generally not |
necessary. |
necessary. |
|
|
Now, reboot so that you are running a DOM0 kernel under Xen, rather |
Ensure that the boot scripts installed in |
than GENERIC without Xen. |
`/usr/pkg/share/examples/rc.d` are in `/etc/rc.d`, either because you |
|
have `PKG_RCD_SCRIPTS=yes`, or manually. (This is not special to Xen, |
Configuring Xen |
but a normal part of pkgsrc usage.) |
--------------- |
|
|
|
Now, you have a system that will boot Xen and the dom0 kernel, but not |
Set `xencommons=YES` in rc.conf: |
do anything else special. Make sure that you have rebooted into Xen. |
|
There will be no domUs, and none can be started because you still have |
|
to configure the dom0 daemons. |
|
|
|
Since Xen 4.2, the tool which should be used is `xl`. |
|
|
|
For 4.1 and up, you should enable `xencommons`: |
|
|
|
[[!template id=filecontent name="/etc/rc.conf" text=""" |
[[!template id=filecontent name="/etc/rc.conf" text=""" |
xencommons=YES |
xencommons=YES |
"""]] |
"""]] |
|
|
Not enabling xencommons will result in a hang; it is necessary to hit `^C` on |
\todo Recommend for/against xen-watchdog. |
the console to let the machine finish booting: |
|
|
|
TODO: Recommend for/against xen-watchdog. |
Now, reboot so that you are running a DOM0 kernel under Xen, rather |
|
than GENERIC without Xen. |
|
|
After you have configured the daemons and either started them (in the |
Once the reboot is done, use `xl` to inspect Xen's boot messages, |
order given) or rebooted, use `xl` to inspect Xen's boot messages, |
|
available resources, and running domains. For example: |
available resources, and running domains. For example: |
|
|
[[!template id=programlisting text=""" |
[[!template id=programlisting text=""" |
Line 216 the state when the new xenstored starts.
|
Line 227 the state when the new xenstored starts.
|
make this work, one should not expect to be able to restart xenstored |
make this work, one should not expect to be able to restart xenstored |
(and thus xencommons). There is currently no reason to expect that |
(and thus xencommons). There is currently no reason to expect that |
this will get fixed any time soon. |
this will get fixed any time soon. |
|
\todo Confirm if this is still true in 2020. |
|
|
anita (for testing NetBSD) |
## Xen-specific NetBSD issues |
-------------------------- |
|
|
|
With the setup so far (assuming 4.8/xl), one should be able to run |
|
anita (see pkgsrc/misc/py-anita) to test NetBSD releases, by doing (as |
|
root, because anita must create a domU): |
|
|
|
[[!template id=programlisting text=""" |
|
anita --vmm=xl test file:///usr/obj/i386/ |
|
"""]] |
|
|
|
Xen-specific NetBSD issues |
|
-------------------------- |
|
|
|
There are (at least) two additional things different about NetBSD as a |
There are (at least) two additional things different about NetBSD as a |
dom0 kernel compared to hardware. |
dom0 kernel compared to hardware. |
Line 243 options as GENERIC. While it is debatab
|
Line 243 options as GENERIC. While it is debatab
|
bug, users should be aware of this and can simply add missing config |
bug, users should be aware of this and can simply add missing config |
items if desired. |
items if desired. |
|
|
Updating NetBSD in a dom0 |
Finally, there have been occasional reports of trouble with X11 |
------------------------- |
servers in NetBSD as a dom0. |
|
|
|
## Updating Xen in a dom0 |
|
|
|
Basically, update the xenkernel and xentools packages and copy the new |
|
Xen kernel into place, and reboot. This procedure should be usable to |
|
update to a new Xen release, but the reader is reminded that having a |
|
non-Xen boot methods was recommended earlier. |
|
|
|
## Updating NetBSD in a dom0 |
|
|
This is just like updating NetBSD on bare hardware, assuming the new |
This is just like updating NetBSD on bare hardware, assuming the new |
version supports the version of Xen you are running. Generally, one |
version supports the version of Xen you are running. Generally, one |
Line 254 and adjusts `/etc`.
|
Line 263 and adjusts `/etc`.
|
Note that one must update both the non-Xen kernel typically used for |
Note that one must update both the non-Xen kernel typically used for |
rescue purposes and the DOM0 kernel used with Xen. |
rescue purposes and the DOM0 kernel used with Xen. |
|
|
Converting from grub to /boot |
## anita (for testing NetBSD) |
----------------------------- |
|
|
With a NetBSD dom0, even without any domUs, one should be able to run |
|
anita (see pkgsrc/misc/py-anita) to test NetBSD releases, by doing (as |
|
root, because anita must create a domU): |
|
|
|
[[!template id=programlisting text=""" |
|
anita --vmm=xl test file:///usr/obj/i386/ |
|
"""]] |
|
|
|
## Converting from grub to /boot (historical note) |
|
|
|
These instructions are provided only to help people using grub, which |
|
used to be the normal approach. |
|
|
These instructions were used to convert a system from |
These instructions were used to convert a system from |
grub to /boot. The system was originally installed in February of |
grub to /boot. The system was originally installed in February of |
Line 285 default=1
|
Line 306 default=1
|
timeout=30 |
timeout=30 |
"""]] |
"""]] |
|
|
Upgrading Xen versions |
# Unprivileged domains (domU) |
--------------------- |
|
|
|
Minor version upgrades are trivial. Just rebuild/replace the |
|
xenkernel version and copy the new xen.gz to `/` (where `/boot.cfg` |
|
references it), and reboot. |
|
|
|
Unprivileged domains (domU) |
|
=========================== |
|
|
|
This section describes general concepts about domUs. It does not |
This section describes general concepts about domUs. It does not |
address specific domU operating systems or how to install them. The |
address specific domU operating systems or how to install them. The |
Line 308 mediated by Xen, and configured in the d
|
Line 321 mediated by Xen, and configured in the d
|
Entropy in domUs can be an issue; physical disks and network are on |
Entropy in domUs can be an issue; physical disks and network are on |
the dom0. NetBSD's /dev/random system works, but is often challenged. |
the dom0. NetBSD's /dev/random system works, but is often challenged. |
|
|
Config files |
## Config files |
------------ |
|
|
|
There is no good order to present config files and the concepts |
See /usr/pkg/share/examples/xen/xlexample* |
surrounding what is being configured. We first show an example config |
for a small number of well-commented examples, mostly for running |
file, and then in the various sections give details. |
|
|
|
See /usr/pkg/share/examples/xen/xmexample*, |
|
for a large number of well-commented examples, mostly for running |
|
GNU/Linux. |
GNU/Linux. |
|
|
The following is an example minimal domain configuration file. The domU |
The following is an example minimal domain configuration file. The domU |
Line 340 are stored in files and Xen attaches the
|
Line 348 are stored in files and Xen attaches the
|
dom0 on domain creation. The system treats xbd0 as the boot device |
dom0 on domain creation. The system treats xbd0 as the boot device |
without needing explicit configuration. |
without needing explicit configuration. |
|
|
By default, `xl` looks for domain config files in `/usr/pkg/etc/xen`. Note |
There is not type line; that implicitly defines a pv domU. |
|
|
|
By convention, domain config files are kept in `/usr/pkg/etc/xen`. Note |
that "xl create" takes the name of a config file, while other commands |
that "xl create" takes the name of a config file, while other commands |
take the name of a domain. |
take the name of a domain. |
|
|
Examples of commands: |
Examples of commands: |
|
|
[[!template id=programlisting text=""" |
[[!template id=programlisting text=""" |
xl create foo |
xl create /usr/pkg/etc/xen/foo |
xl console foo |
xl console domU-id |
xl create -c foo |
xl create -c /usr/pkg/etc/xen/foo |
xl shutdown foo |
xl shutdown domU-id |
xl list |
xl list |
"""]] |
"""]] |
|
|
Line 359 equivalent to pushing the power button;
|
Line 369 equivalent to pushing the power button;
|
power-press event and do a clean shutdown. Shutting down the dom0 |
power-press event and do a clean shutdown. Shutting down the dom0 |
will trigger controlled shutdowns of all configured domUs. |
will trigger controlled shutdowns of all configured domUs. |
|
|
domU kernels |
## CPU and memory |
------------ |
|
|
|
On a physical computer, the BIOS reads sector 0, and a chain of boot |
|
loaders finds and loads a kernel. Normally this comes from the root |
|
file system. With Xen domUs, the process is totally different. The |
|
normal path is for the domU kernel to be a file in the dom0's |
|
file system. At the request of the dom0, Xen loads that kernel into a |
|
new domU instance and starts execution. domU kernels can be anyplace. |
|
|
|
Note that loading the domU kernel from the dom0 implies that boot |
|
blocks, /boot, /boot.cfg, and so on are all ignored in the domU. |
|
See the VPS section near the end for discussion of alternate ways to |
|
obtain domU kernels. |
|
|
|
CPU and memory |
|
-------------- |
|
|
|
A domain is provided with some number of vcpus, less than the number |
A domain is provided with some number of vcpus, up to the number |
of CPUs seen by the hypervisor. For a domU, it is controlled |
of CPUs seen by the hypervisor. For a domU, it is controlled |
from the config file by the "vcpus = N" directive. |
from the config file by the "vcpus = N" directive. |
|
|
Line 389 than the available memory.
|
Line 383 than the available memory.
|
Xen also provides a "balloon" driver, which can be used to let domains |
Xen also provides a "balloon" driver, which can be used to let domains |
use more memory temporarily. |
use more memory temporarily. |
|
|
Virtual disks |
## Virtual disks |
------------- |
|
|
|
With the file/vnd style, typically one creates a directory, |
In domU config files, the disks are defined as a sequence of 3-tuples: |
e.g. /u0/xen, on a disk large enough to hold virtual disks for all |
|
domUs. Then, for each domU disk, one writes zeros to a file that then |
|
serves to hold the virtual disk's bits; a suggested name is foo-xbd0 |
|
for the first virtual disk for the domU called foo. Writing zeros to |
|
the file serves two purposes. One is that preallocating the contents |
|
improves performance. The other is that vnd on sparse files has |
|
failed to work. TODO: give working/notworking NetBSD versions for |
|
sparse vnd and gnats reference. Note that the use of file/vnd for Xen |
|
is not really different than creating a file-backed virtual disk for |
|
some other purpose, except that xentools handles the vnconfig |
|
commands. To create an empty 4G virtual disk, simply do |
|
|
|
dd if=/dev/zero of=foo-xbd0 bs=1m count=4096 |
|
|
|
Do not use qemu-img-xen, because this will create sparse file. There |
|
have been recent (2015) reports of sparse vnd(4) devices causing |
|
lockups, but there is apparently no PR. |
|
|
|
With the lvm style, one creates logical devices. They are then used |
|
similarly to vnds. TODO: Add an example with lvm. |
|
|
|
In domU config files, the disks are defined as a sequence of 3-tuples. |
|
The first element is "method:/path/to/disk". Common methods are |
|
"file:" for file-backed vnd. and "phy:" for something that is already |
|
a (TODO: character or block) device. |
|
|
|
The second element is an artifact of how virtual disks are passed to |
|
Linux, and a source of confusion with NetBSD Xen usage. Linux domUs |
|
are given a device name to associate with the disk, and values like |
|
"hda1" or "sda1" are common. In a NetBSD domU, the first disk appears |
|
as xbd0, the second as xbd1, and so on. However, xm/xl demand a |
|
second argument. The name given is converted to a major/minor by |
|
calling stat(2) on the name in /dev and this is passed to the domU. |
|
In the general case, the dom0 and domU can be different operating |
|
systems, and it is an unwarranted assumption that they have consistent |
|
numbering in /dev, or even that the dom0 OS has a /dev. With NetBSD |
|
as both dom0 and domU, using values of 0x0 for the first disk and 0x1 |
|
for the second works fine and avoids this issue. For a GNU/Linux |
|
guest, one can create /dev/hda1 in /dev, or to pass 0x301 for |
|
/dev/hda1. |
|
|
|
The third element is "w" for writable disks, and "r" for read-only |
* The first element is "method:/path/to/disk". Common methods are |
disks. |
"file:" for a file-backed vnd, and "phy:" for something that is already |
|
a device, such as an LVM logical volume. |
|
|
|
* The second element is an artifact of how virtual disks are passed to |
|
Linux, and a source of confusion with NetBSD Xen usage. Linux domUs |
|
are given a device name to associate with the disk, and values like |
|
"hda1" or "sda1" are common. In a NetBSD domU, the first disk appears |
|
as xbd0, the second as xbd1, and so on. However, xl demands a |
|
second argument. The name given is converted to a major/minor by |
|
calling stat(2) on the name in /dev and this is passed to the domU. |
|
In the general case, the dom0 and domU can be different operating |
|
systems, and it is an unwarranted assumption that they have consistent |
|
numbering in /dev, or even that the dom0 OS has a /dev. With NetBSD |
|
as both dom0 and domU, using values of 0x0 for the first disk and 0x1 |
|
for the second works fine and avoids this issue. For a GNU/Linux |
|
guest, one can create /dev/hda1 in /dev, or to pass 0x301 for |
|
/dev/hda1. |
|
|
|
* The third element is "w" for writable disks, and "r" for read-only |
|
disks. |
|
|
|
Example: |
|
[[!template id=filecontent name="/usr/pkg/etc/xen/foo" text=""" |
|
disk = [ 'file:/n0/xen/foo-wd0,0x0,w' ] |
|
"""]] |
|
|
Note that NetBSD by default creates only vnd[0123]. If you need more |
Note that NetBSD by default creates only vnd[0123]. If you need more |
than 4 total virtual disks at a time, run e.g. "./MAKEDEV vnd4" in the |
than 4 total virtual disks at a time, run e.g. "./MAKEDEV vnd4" in the |
dom0. |
dom0. |
|
|
Note that NetBSD by default creates only xbd[0123]. If you need more |
## Virtual Networking |
virtual disks in a domU, run e.g. "./MAKEDEV xbd4" in the domU. |
|
|
|
Virtual Networking |
|
------------------ |
|
|
|
Xen provides virtual Ethernets, each of which connects the dom0 and a |
Xen provides virtual Ethernets, each of which connects the dom0 and a |
domU. For each virtual network, there is an interface "xvifN.M" in |
domU. For each virtual network, there is an interface "xvifN.M" in |
the dom0, and in domU index N, a matching interface xennetM (NetBSD |
the dom0, and a matching interface xennetM (NetBSD name) in domU index N. |
name). The interfaces behave as if there is an Ethernet with two |
The interfaces behave as if there is an Ethernet with two |
adapters connected. From this primitive, one can construct various |
adapters connected. From this primitive, one can construct various |
configurations. We focus on two common and useful cases for which |
configurations. We focus on two common and useful cases for which |
there are existing scripts: bridging and NAT. |
there are existing scripts: bridging and NAT. |
Line 477 The MAC address specified is the one use
|
Line 450 The MAC address specified is the one use
|
domain. The interface in dom0 will use this address XOR'd with |
domain. The interface in dom0 will use this address XOR'd with |
00:00:00:01:00:00. Random MAC addresses are assigned if not given. |
00:00:00:01:00:00. Random MAC addresses are assigned if not given. |
|
|
Sizing domains |
## Starting domains automatically |
-------------- |
|
|
|
Modern x86 hardware has vast amounts of resources. However, many |
To start domains `domU-netbsd` and `domU-linux` at boot and shut them |
virtual servers can function just fine on far less. A system with |
down cleanly on dom0 shutdown, add the following in rc.conf: |
512M of RAM and a 4G disk can be a reasonable choice. Note that it is |
|
far easier to adjust virtual resources than physical ones. For |
|
memory, it's just a config file edit and a reboot. For disk, one can |
|
create a new file and vnconfig it (or lvm), and then dump/restore, |
|
just like updating physical disks, but without having to be there and |
|
without those pesky connectors. |
|
|
|
Starting domains automatically |
|
------------------------------ |
|
|
|
To start domains foo at bar at boot and shut them down cleanly on dom0 |
|
shutdown, in rc.conf add: |
|
|
|
xendomains="foo bar" |
[[!template id=filecontent name="/etc/rc.conf" text=""" |
|
xendomains="domU-netbsd domU-linux" |
Note that earlier versions of the xentools41 xendomains rc.d script |
"""]] |
used xl, when one should use xm with 4.1. |
|
|
|
Creating specific unprivileged domains (domU) |
# domU setup for specific systems |
============================================= |
|
|
|
Creating domUs is almost entirely independent of operating system. We |
Creating domUs is almost entirely independent of operating system. We |
have already presented the basics of config files. Note that you must |
have already presented the basics of config files in the previous system. |
have already completed the dom0 setup so that "xl list" (or "xm list") |
|
works. |
|
|
|
Creating an unprivileged NetBSD domain (domU) |
Of course, this section presumes that you have a working dom0. |
--------------------------------------------- |
|
|
## Creating a NetBSD PV domU |
|
|
See the earlier config file, and adjust memory. Decide on how much |
See the earlier config file, and adjust memory. Decide on how much |
storage you will provide, and prepare it (file or lvm). |
storage you will provide, and prepare it (file or LVM). |
|
|
While the kernel will be obtained from the dom0 file system, the same |
While the kernel will be obtained from the dom0 file system, the same |
file should be present in the domU as /netbsd so that tools like |
file should be present in the domU as /netbsd so that tools like |
savecore(8) can work. (This is helpful but not necessary.) |
savecore(8) can work. (This is helpful but not necessary.) |
|
|
The kernel must be specifically for Xen and for use as a domU. The |
The kernel must be specifically built for Xen, to use PV interfacesas |
i386 and amd64 provide the following kernels: |
a domU. NetBSD release builds provide the following kernels: |
|
|
i386 XEN3PAE_DOMU |
i386 XEN3PAE_DOMU |
amd64 XEN3_DOMU |
amd64 XEN3_DOMU |
|
|
This will boot NetBSD, but this is not that useful if the disk is |
This will boot NetBSD, but this is not that useful if the disk is |
empty. One approach is to unpack sets onto the disk outside of xen |
empty. One approach is to unpack sets onto the disk outside of Xen |
(by mounting it, just as you would prepare a physical disk for a |
(by mounting it, just as you would prepare a physical disk for a |
system you can't run the installer on). |
system you can't run the installer on). |
|
|
Line 535 kernel to / and change the kernel line i
|
Line 492 kernel to / and change the kernel line i
|
|
|
kernel = "/home/bouyer/netbsd-INSTALL_XEN3_DOMU" |
kernel = "/home/bouyer/netbsd-INSTALL_XEN3_DOMU" |
|
|
Then, start the domain as "xl create -c configname". |
Then, start the domain as "xl create -c configfile". |
|
|
Alternatively, if you want to install NetBSD/Xen with a CDROM image, the following |
Alternatively, if you want to install NetBSD/Xen with a CDROM image, the following |
line should be used in the config file. |
line should be used in the config file. |
Line 545 line should be used in the config file.
|
Line 502 line should be used in the config file.
|
After booting the domain, the option to install via CDROM may be |
After booting the domain, the option to install via CDROM may be |
selected. The CDROM device should be changed to `xbd1d`. |
selected. The CDROM device should be changed to `xbd1d`. |
|
|
Once done installing, "halt -p" the new domain (don't reboot or halt, |
Once done installing, "halt -p" the new domain (don't reboot or halt: |
it would reload the INSTALL_XEN3_DOMU kernel even if you changed the |
it would reload the INSTALL_XEN3_DOMU kernel even if you changed the |
config file), switch the config file back to the XEN3_DOMU kernel, |
config file), switch the config file back to the XEN3_DOMU kernel, |
and start the new domain again. Now it should be able to use "root on |
and start the new domain again. Now it should be able to use "root on |
xbd0a" and you should have a, functional NetBSD domU. |
xbd0a" and you should have a functional NetBSD domU. |
|
|
TODO: check if this is still accurate. |
TODO: check if this is still accurate. |
When the new domain is booting you'll see some warnings about *wscons* |
When the new domain is booting you'll see some warnings about *wscons* |
Line 570 It is also desirable to add
|
Line 527 It is also desirable to add
|
powerd=YES |
powerd=YES |
|
|
in rc.conf. This way, the domain will be properly shut down if |
in rc.conf. This way, the domain will be properly shut down if |
`xm shutdown -R` or `xm shutdown -H` is used on the dom0. |
`xl shutdown -R` or `xl shutdown -H` is used on the dom0. |
|
\todo Check the translation to xl. |
|
|
It is not strictly necessary to have a kernel (as /netbsd) in the domU |
It is not strictly necessary to have a kernel (as /netbsd) in the domU |
file system. However, various programs (e.g. netstat) will use that |
file system. However, various programs (e.g. netstat) will use that |
Line 580 not really a Xen-specific issue, but bec
|
Line 538 not really a Xen-specific issue, but bec
|
obtained from the dom0, it is far more likely to be out of sync or |
obtained from the dom0, it is far more likely to be out of sync or |
missing with Xen.) |
missing with Xen.) |
|
|
Creating an unprivileged Linux domain (domU) |
Note that NetBSD by default creates only xbd[0123]. If you need more |
-------------------------------------------- |
virtual disks in a domU, run e.g. "./MAKEDEV xbd4" in the domU. |
|
|
|
## Creating a Linux domU |
|
|
Creating unprivileged Linux domains isn't much different from |
Creating unprivileged Linux domains isn't much different from |
unprivileged NetBSD domains, but there are some details to know. |
unprivileged NetBSD domains, but there are some details to know. |
Line 624 To get the Linux console right, you need
|
Line 584 To get the Linux console right, you need
|
to your configuration since not all Linux distributions auto-attach a |
to your configuration since not all Linux distributions auto-attach a |
tty to the xen console. |
tty to the xen console. |
|
|
Creating an unprivileged Solaris domain (domU) |
## Creating a NetBSD HVM domU |
---------------------------------------------- |
|
|
Use type='hmv', probably. Use a GENERIC kernel within the disk image. |
|
|
|
## Creating a NetBSD PVH domU |
|
|
|
Use type='pvh'. |
|
|
|
\todo Explain where the kernel comes from. |
|
|
|
|
|
## Creating a Solaris domU |
|
|
See possibly outdated |
See possibly outdated |
[Solaris domU instructions](/ports/xen/howto-solaris/). |
[Solaris domU instructions](/ports/xen/howto-solaris/). |
|
|
|
|
PCI passthrough: Using PCI devices in guest domains |
## PCI passthrough: Using PCI devices in guest domains |
--------------------------------------------------- |
|
|
NB: PCI passthrough only works on some Xen versions and as of 2020 it |
|
is not clear that it works on any version in pkgsrc. Reports |
|
confirming or denying this notion should be sent to port-xen@. |
|
|
The dom0 can give other domains access to selected PCI |
The dom0 can give other domains access to selected PCI |
devices. This can allow, for example, a non-privileged domain to have |
devices. This can allow, for example, a non-privileged domain to have |
Line 701 note that only the "xpci" lines are unus
|
Line 674 note that only the "xpci" lines are unus
|
cd* at scsibus? target ? lun ? # SCSI CD-ROM drives |
cd* at scsibus? target ? lun ? # SCSI CD-ROM drives |
|
|
|
|
NetBSD as a domU in a VPS |
# Miscellaneous Information |
========================= |
|
|
## Nesting under Linux KVM |
|
|
|
It is possible to run a Xen and a NetBSD dom0 under Linux KVM. One |
|
can enable virtio in the dom0 for greater speed. |
|
|
|
## Other nesting |
|
|
|
In theory, any full emulation should be able to run Xen and a NetBSD |
|
dom0. The HOWTO does not currently have information about Xen XVM |
|
mode, nvmm, qemu, Virtualbox, etc. |
|
|
|
## NetBSD 5 as domU |
|
|
|
[NetBSD 5 is known to panic.](http://mail-index.netbsd.org/port-xen/2018/04/17/msg009181.html) |
|
(However, NetBSD 5 systems should be updated to a supported version.) |
|
|
|
# NetBSD as a domU in a VPS |
|
|
The bulk of the HOWTO is about using NetBSD as a dom0 on your own |
The bulk of the HOWTO is about using NetBSD as a dom0 on your own |
hardware. This section explains how to deal with Xen in a domU as a |
hardware. This section explains how to deal with Xen in a domU as a |
Line 729 A second issue is multiple CPUs. With N
|
Line 719 A second issue is multiple CPUs. With N
|
multiple vcpus, and it is typical for VPS providers to enable multiple |
multiple vcpus, and it is typical for VPS providers to enable multiple |
CPUs for NetBSD domUs. |
CPUs for NetBSD domUs. |
|
|
pygrub |
## Complexities due to Xen changes |
------- |
|
|
|
pygrub runs in the dom0 and looks into the domU file system. This |
Xen has many security advisories and people running Xen systems make |
implies that the domU must have a kernel in a file system in a format |
different choices. |
known to pygrub. As of 2014, pygrub seems to be of mostly historical |
|
interest. |
### stub domains |
|
|
|
Some (Linux only?) dom0 systems use something called "stub domains" to |
|
isolate qemu from the dom0 system, as a security and reliabilty |
|
mechanism when running HVM domUs. Somehow, NetBSD's GENERIC kernel |
|
ends up using PIO for disks rather than DMA. Of course, all of this |
|
is emulated, but emulated PIO is unusably slow. This problem is not |
|
currently understood. |
|
|
|
### Grant tables |
|
|
|
There are multiple versions of using grant tables, and some security |
|
advisories have suggested disabling some versions. Some versions of |
|
NetBSD apparently only use specific versions and this can lead to |
|
"NetBSD current doesn't run on hosting provider X" situations. |
|
|
|
\todo Explain better. |
|
|
|
## Boot methods |
|
|
pvgrub |
### pvgrub |
------ |
|
|
|
pvgrub is a version of grub that uses PV operations instead of BIOS |
pvgrub is a version of grub that uses PV operations instead of BIOS |
calls. It is booted from the dom0 as the domU kernel, and then reads |
calls. It is booted from the dom0 as the domU kernel, and then reads |
Line 763 partition for the kernel with the intent
|
Line 769 partition for the kernel with the intent
|
which leads to /netbsd not being the actual kernel. One must remember |
which leads to /netbsd not being the actual kernel. One must remember |
to update the special boot partition. |
to update the special boot partition. |
|
|
Amazon |
### pygrub |
------ |
|
|
|
See the [Amazon EC2 page](/amazon_ec2/). |
pygrub runs in the dom0 and looks into the domU file system. This |
|
implies that the domU must have a kernel in a file system in a format |
|
known to pygrub. |
|
|
Random pointers |
pygrub doesn't seem to work to load Linux images under NetBSD dom0, |
=============== |
and is inherently less secure than pvgrub due to running inside dom0. For both these |
|
reasons, pygrub should not be used, and is only still present so that |
|
historical DomU images using it still work. |
|
|
This section contains links from elsewhere not yet integrated into the |
As of 2014, pygrub seems to be of mostly historical |
HOWTO, and other guides. |
interest. New DomUs should use pvgrub. |
|
|
* http://www.lumbercartel.ca/library/xen/ |
## Specific Providers |
* http://pbraun.nethence.com/doc/sysutils/xen_netbsd_dom0.html |
|
* https://gmplib.org/~tege/xen.html |
### Amazon |
|
|
|
See the [Amazon EC2 page](/amazon_ec2/). |