version 1.127, 2016/12/20 19:25:35
|
version 1.141, 2017/12/15 16:40:59
|
Line 2 Introduction
|
Line 2 Introduction
|
============ |
============ |
|
|
[![[Xen |
[![[Xen |
screenshot]](http://www.netbsd.org/gallery/in-Action/hubertf-xens.png)](http://www.netbsd.org/gallery/in-Action/hubertf-xen.png) |
screenshot]](https://www.netbsd.org/gallery/in-Action/hubertf-xens.png)](https://www.netbsd.org/gallery/in-Action/hubertf-xen.png) |
|
|
Xen is a hypervisor (or virtual machine monitor) for x86 hardware |
Xen is a hypervisor (or virtual machine monitor) for x86 hardware |
(i686-class or higher), which supports running multiple guest |
(i686-class or higher), which supports running multiple guest |
Line 81 of Xen version and NetBSD version. This
|
Line 81 of Xen version and NetBSD version. This
|
which version to choose. Versions not in pkgsrc and older unsupported |
which version to choose. Versions not in pkgsrc and older unsupported |
versions of NetBSD are intentionally ignored. |
versions of NetBSD are intentionally ignored. |
|
|
|
The term "amd64" is used to refer to both the NetBSD port and to the |
|
hardware architecture on which it runs. (Such hardware is made by |
|
both Intel and AMD, and in 2016 a normal PC has this CPU |
|
architecture.) |
|
|
Xen |
Xen |
--- |
--- |
|
|
Line 89 xenkernel and xentools. We will refer o
|
Line 94 xenkernel and xentools. We will refer o
|
but note that both packages must be installed together and must have |
but note that both packages must be installed together and must have |
matching versions. |
matching versions. |
|
|
xenkernel3 provides Xen 3.1. It is no longer maintained by Xen, and the last applied security patch was in |
xenkernel3 provides Xen 3.1. It is no longer maintained by Xen, and |
2011. Thus, it should not be used. It supports PCI passthrough, |
the last applied security patch was in 2011. Thus, it should not be |
which is why people use it anyway. Xen 3.1 supports i386, both PAE and |
used. It supports PCI passthrough, which is why people use it anyway. |
non-PAE. |
Xen 3.1 runs on i386 (both non-PAE and PAE) and amd64 hardware. |
|
|
xenkernel33 provides Xen 3.3. It is no longer maintained by Xen, and |
xenkernel33 provides Xen 3.3. It is no longer maintained by Xen, and |
the last applied security patch was in 2012. Thus, it should not be |
the last applied security patch was in 2012. Thus, it should not be |
used. Xen 3.3 supports i386, but only in PAE mode. There are no good |
used. Xen 3.3 runs on i386 PAE and amd64 hardware. There are no good |
reasons to run this version. |
reasons to run this version. |
|
|
xenkernel41 provides Xen 4.1. It is no longer maintained by Xen, but |
xenkernel41 provides Xen 4.1. It is no longer maintained by Xen, but |
as of 2016-12 received backported security patches. Xen 4.1 supports |
as of 2016-12 received backported security patches. Xen 4.1 runs on |
i386, but only in PAE mode. There are no good reasons to run this |
i386 PAE and amd64 hardware. There are no good reasons to run this |
version. |
version. |
|
|
|
Note that 3.1, 3.3 and 4.1 have been removed from pkgsrc-current, but |
|
are in 2016Q4. They will be removed from this HOWTO sometime after |
|
2017Q1. |
|
|
xenkernel42 provides Xen 4.2. It is no longer maintained by Xen, but |
xenkernel42 provides Xen 4.2. It is no longer maintained by Xen, but |
as of 2016-12 received backported security patches. Xen 4.2 supports |
as of 2016-12 received backported security patches. Xen 4.2 runs on |
i386, but only in PAE mode. The only reason to run this is if you |
i386 PAE and amd64 hardware. The only reason to run this is if you |
need to use xm instead of xl, or if you need to run an i386 dom0 |
need to use xm instead of xl, or if you need to run on hardware that |
(because your hardware is i386 only). |
supports i386 but not amd64. (This might also be useful if you need |
|
an i386 dom0, if it turns out that an amd64 Xen kernel and an i386 |
|
dom0 is problematic.) |
|
|
xenkernel45 provides Xen 4.5. As of 2016-12, security patches were |
xenkernel45 provides Xen 4.5. As of 2016-12, security patches were |
released by Xen and applied to pkgsrc. Xen 4.5 requires using a dom0 |
released by Xen and applied to pkgsrc. Xen 4.5 runs on amd64 hardware |
running NetBSD's amd64 port (Intel or AMD hardware is fine), but domUs |
only. While slightly old, 4.5 has been tested and run by others, so |
can be amd64 or i386 PAE. TODO: It is either a conservative choice or |
it is the conservative choice. |
somewhat old. |
|
|
|
xenkernel46 provides Xen 4.6. It is new to pkgsrc as of 2016-05. As |
xenkernel46 provides Xen 4.6. It is new to pkgsrc as of 2016-05. As |
of 2016-12, security patches were released by Xen and applied to |
of 2016-12, security patches were released by Xen and applied to |
pkgsrc. Xen 4.6 similarly requires a NetBSD/amd64 dom0, but domUs can |
pkgsrc. Xen 4.6 runs on amd64 hardware only For new installations, |
be amd64 or i386 PAE. TODO: It is either a somewhat aggressive choice |
4.6 is probably the appropriate choice and it will likely soon be the |
or the standard choice. |
standard approach. (If using Ubuntu guests, be sure to have the |
|
xentools46 from December, 2016). |
|
|
Xen 4.7 (released 2016-06) and 4.8 (released 2016-12) are not yet in |
Xen 4.7 (released 2016-06) and 4.8 (released 2016-12) are not yet in |
pkgsrc. |
pkgsrc. |
Line 144 only which command you use, but the comm
|
Line 155 only which command you use, but the comm
|
xentools packages provide xm for 3.1, 3.3 and 4.1 and xl for 4.2 and up. |
xentools packages provide xm for 3.1, 3.3 and 4.1 and xl for 4.2 and up. |
|
|
In 4.2, you can choose to use xm by simply changing the ctl_command |
In 4.2, you can choose to use xm by simply changing the ctl_command |
variable. |
variable and setting xend=YES in rc.conf. |
|
|
With xl, virtual devices are configured in parallel, which can cause |
With xl, virtual devices are configured in parallel, which can cause |
problems if they are written assuming serial operation (e.g., updating |
problems if they are written assuming serial operation (e.g., updating |
firewall rules without explicit locking). |
firewall rules without explicit locking). There is now locking for |
|
the provided scripts, which works for normal casses (e.g, file-backed |
There is now locking for xl startup, but as of 201612 it has not been |
xbd, where a vnd must be allocated). But, as of 201612, it has not |
adequately tested. |
been adequately tested for a complex custom setup with a large number |
|
of interfaces. |
|
|
NetBSD |
NetBSD |
------ |
------ |
Line 177 when using a dom0 as a normal computer.)
|
Line 189 when using a dom0 as a normal computer.)
|
Architecture |
Architecture |
------------ |
------------ |
|
|
Xen itself can run on i386 (Xen < 3.1) or amd64 machines (all Xen |
Xen itself can run on i386 (Xen < 4.2) or amd64 hardware (all Xen |
versions). (Practically, almost any computer where one would want to |
versions). (Practically, almost any computer where one would want to |
run Xen today supports amd64.) |
run Xen today supports amd64.) |
|
|
Xen, the dom0 kernel, and each domU kernel can be either i386 or |
Xen, the dom0 system, and each domU system can be either i386 or |
amd64. When building a xenkernel package, one obtains i386 on an i386 |
amd64. When building a xenkernel package, one obtains an i386 Xen |
host, and amd64 on an amd64 host. If the Xen kernel is i386, then the |
kernel on an i386 host, and an amd64 Xen kernel on an amd64 host. If |
dom0 kernel and all domU kernels must be i386. With an amd64 Xen |
the Xen kernel is i386, then the dom0 kernel and all domU kernels must |
kernel, an amd64 dom0 kernel is known to work, and an i386PAE dom0 |
be i386. With an amd64 Xen kernel, an amd64 dom0 kernel is known to |
kernel should in theory work. An amd64 Xen/dom0 is known to support |
work, and an i386 dom0 kernel should in theory work. An amd64 |
both i386PAE and amd64 domUs. |
Xen/dom0 is known to support both i386 and amd64 domUs. |
|
|
i386 dom0 and domU kernels must be PAE (except for Xen 3.1); these are |
i386 dom0 and domU kernels must be PAE (except for an i386 Xen 3.1 |
built by default. (Note that emacs (at least) fails if run on i386 |
kernel, where one can use non-PAE for dom0 and all domUs); PAE kernels |
with PAE when built without, and vice versa, presumably due to bugs in |
are included in the NetBSD default build. (Note that emacs (at least) |
the undump code.) |
fails if run on i386 with PAE when built without, and vice versa, |
|
presumably due to bugs in the undump code.) |
Because of the above, the standard approach is to use amd64 for the |
|
dom0. |
Because of the above, the standard approach is to use an amd64 Xen |
|
kernel and NetBSD/amd64 for the dom0. For domUs, NetBSD/i386 (with |
Xen 4.2 is the last version to support i386 as a host. TODO: Clarify |
the PAE kernel) and NetBSD/amd64 are in widespread use, and there is |
if this is about the CPU, the Xen kernel, or the dom0 kernel having to |
little to no Xen-specific reason to prefer one over the other. |
be amd64. |
|
|
Note that to use an i386 dom0 with Xen 4.5 or higher, one must build |
|
(or obtain from pre-built packages) an amd64 Xen kernel and install |
|
that on the system. (One must also use a PAE i386 kernel, but this is |
|
also required with an i386 Xen kernel.). Almost no one in the |
|
NetBSD/Xen community does this, and the standard, well-tested, |
|
approach is to use an amd64 dom0. |
|
|
|
A [posting on |
|
xen-devel](https://lists.xen.org/archives/html/xen-devel/2012-07/msg00085.html) |
|
explained that PV system call overhead was higher on amd64, and thus |
|
there is some notion that i386 guests are faster. It goes on to |
|
caution that the total situation is complex and not entirely |
|
understood. On top of that caution, the post is about Linux, not |
|
NetBSD. TODO: Include link to benchmarks, if someone posts them. |
|
|
Stability |
Stability |
--------- |
--------- |
Line 209 Mostly, NetBSD as a dom0 or domU is quit
|
Line 234 Mostly, NetBSD as a dom0 or domU is quit
|
However, there are some open PRs indicating problems. |
However, there are some open PRs indicating problems. |
|
|
- [PR 48125](http://gnats.netbsd.org/48125) |
- [PR 48125](http://gnats.netbsd.org/48125) |
- [PR 47720](http://gnats.netbsd.org/47720) |
|
|
|
Note also that there are issues with sparse vnd(4) instances, but |
Note also that there are issues with sparse vnd(4) instances, but |
these are not about Xen -- they just are noticed with sparse vnd(4) |
these are not about Xen -- they just are noticed with sparse vnd(4) |
Line 223 xl, the NetBSD 7 stable branch, and to u
|
Line 247 xl, the NetBSD 7 stable branch, and to u
|
dom0. Either the i386PAE or amd64 version of NetBSD may be used as |
dom0. Either the i386PAE or amd64 version of NetBSD may be used as |
domUs. |
domUs. |
|
|
|
Because bugs are fixed quite often, and because of Xen security |
|
advisories, it is good to stay up to date with NetBSD (tracking a |
|
stable branch), with the Xen kernel (tracking a Xen version via |
|
pkgsrc), and with the Xen tools. Specifically, NetBSD (-7 and |
|
-current) got an important fix affecting dom0/domU timesharing in |
|
November, 2015, and xentools46 got a fix to enable Ubuntu guests to |
|
boot in December, 2016. |
|
|
Status |
Status |
------ |
------ |
|
|
Line 238 The following table gives status, with t
|
Line 270 The following table gives status, with t
|
(generally on the most recent quarterly branch). The first code is |
(generally on the most recent quarterly branch). The first code is |
"builds" if it builds ok, and "FAIL" for a failure to build. The |
"builds" if it builds ok, and "FAIL" for a failure to build. The |
second code/date only appears for xenkernel* and is "works" if it runs |
second code/date only appears for xenkernel* and is "works" if it runs |
ok as a dom0, and "FAIL" if it won't boot or run a domU. |
ok as a dom0 and can support a domU, and "FAIL" if it won't boot or |
|
run a domU. |
|
|
xenkernel3 netbsd-6 i386 FAIL 201612 |
xenkernel3 netbsd-6 i386 FAIL 201612 |
xenkernel33 netbsd-6 i386 FAIL 201612 |
xenkernel33 netbsd-6 i386 FAIL 201612 |
xenkernel41 netbsd-6 i386 builds 201612 |
xenkernel41 netbsd-6 i386 builds 201612 |
xenkernel42 netbsd-6 i386 builds 201612 |
xenkernel42 netbsd-6 i386 builds 201612 |
xentools3 netbsd-6 i386 FAIL 201612 |
xentools3 netbsd-6 i386 FAIL 201612 |
xentools3-hvm netbsd-6 i386 FAIL 201412 |
xentools33 netbsd-6 i386 FAIL 201612 |
xentools33 netbsd-6 i386 builds 201412 |
|
xentools41 netbsd-6 i386 builds 201612 |
xentools41 netbsd-6 i386 builds 201612 |
xentools42 netbsd-6 i386 FAIL 201612 |
xentools42 netbsd-6 i386 FAIL 201612 |
|
|
Line 295 configuration.
|
Line 327 configuration.
|
|
|
For experimenting with Xen, a machine with as little as 1G of RAM and |
For experimenting with Xen, a machine with as little as 1G of RAM and |
100G of disk can work. For running many domUs in productions, far |
100G of disk can work. For running many domUs in productions, far |
more will be needed. |
more will be needed; e.g. 4-8G and 1T of disk is reasonable for a |
|
half-dozen domUs of 512M and 32G each. Basically, the RAM and disk |
|
have to be bigger than the sum of the RAM/disk needs of the dom0 and |
|
all the domUs. |
|
|
Styles of dom0 operation |
Styles of dom0 operation |
------------------------ |
------------------------ |
Line 318 Xen daemons when not running Xen.
|
Line 353 Xen daemons when not running Xen.
|
Note that NetBSD as dom0 does not support multiple CPUs. This will |
Note that NetBSD as dom0 does not support multiple CPUs. This will |
limit the performance of the Xen/dom0 workstation approach. In theory |
limit the performance of the Xen/dom0 workstation approach. In theory |
the only issue is that the "backend drivers" are not yet MPSAFE: |
the only issue is that the "backend drivers" are not yet MPSAFE: |
http://mail-index.netbsd.org/netbsd-users/2014/08/29/msg015195.html |
https://mail-index.netbsd.org/netbsd-users/2014/08/29/msg015195.html |
|
|
Installation of NetBSD |
Installation of NetBSD |
---------------------- |
---------------------- |
Line 361 Installation of Xen
|
Line 396 Installation of Xen
|
|
|
In the dom0, install sysutils/xenkernel42 and sysutils/xentools42 from |
In the dom0, install sysutils/xenkernel42 and sysutils/xentools42 from |
pkgsrc (or another matching pair). See [the pkgsrc |
pkgsrc (or another matching pair). See [the pkgsrc |
documentation](http://www.NetBSD.org/docs/pkgsrc/) for help with |
documentation](https://www.NetBSD.org/docs/pkgsrc/) for help with |
pkgsrc. Ensure that your packages are recent; the HOWTO does not |
pkgsrc. Ensure that your packages are recent; the HOWTO does not |
contemplate old builds. |
contemplate old builds. |
|
|
Line 382 releasedir/i386/binary/kernel/netbsd-XEN
|
Line 417 releasedir/i386/binary/kernel/netbsd-XEN
|
should not use Xen 3.1.) Both xen and the NetBSD kernel may be (and |
should not use Xen 3.1.) Both xen and the NetBSD kernel may be (and |
typically are) left compressed. |
typically are) left compressed. |
|
|
In a dom0 kernel, kernfs is mandatory for xend to communicate with the |
In a dom0, kernfs is mandatory for xend to communicate with the |
kernel, so ensure that /kern is in fstab. TODO: Say this is default, |
kernel, so ensure that /kern is in fstab. (A standard NetBSD install |
or file a PR and give a reference. |
should already mount /kern.) |
|
|
Because you already installed NetBSD, you have a working boot setup |
Because you already installed NetBSD, you have a working boot setup |
with an MBR bootblock, either bootxx_ffsv1 or bootxx_ffsv2 at the |
with an MBR bootblock, either bootxx_ffsv1 or bootxx_ffsv2 at the |
beginning of your root file system, /boot present, and likely |
beginning of your root file system, have /boot, and likely also |
/boot.cfg. (If not, fix before continuing!) |
/boot.cfg. (If not, fix before continuing!) |
|
|
Add a line to to /boot.cfg to boot Xen. See boot.cfg(5) for an |
Add a line to to /boot.cfg to boot Xen. See boot.cfg(5) for an |
Line 399 example. The basic line is
|
Line 434 example. The basic line is
|
which specifies that the dom0 should have 256M, leaving the rest to be |
which specifies that the dom0 should have 256M, leaving the rest to be |
allocated for domUs. To use a serial console, use |
allocated for domUs. To use a serial console, use |
|
|
menu=Xen:load /netbsd-XEN3_DOM0.gz console=com0;multiboot /xen.gz dom0_mem=256M console=com1 com1=9600,8n1 |
menu=Xen:load /netbsd-XEN3_DOM0.gz;multiboot /xen.gz dom0_mem=256M console=com1 com1=9600,8n1 |
|
|
which will use the first serial port for Xen (which counts starting |
which will use the first serial port for Xen (which counts starting |
from 1), forcing speed/parity, and also for NetBSD (which counts |
from 1, unlike NetBSD which counts starting from 0), forcing |
starting at 0). In an attempt to add performance, one can also add |
speed/parity. Because the NetBSD command line lacks a |
|
"console=pc" argument, it will use the default "xencons" console device, |
|
which directs the console I/O through Xen to the same console device Xen |
|
itself uses (in this case, the serial port). |
|
|
|
In an attempt to add performance, one can also add |
|
|
dom0_max_vcpus=1 dom0_vcpus_pin |
dom0_max_vcpus=1 dom0_vcpus_pin |
|
|
Line 433 Using grub (historic)
|
Line 473 Using grub (historic)
|
|
|
Before NetBSD's native bootloader could support Xen, the use of |
Before NetBSD's native bootloader could support Xen, the use of |
grub was recommended. If necessary, see the |
grub was recommended. If necessary, see the |
[old grub information](/ports/xen/howto-grub/). |
[old grub information](/ports/xen/howto-grub). |
|
|
The [HowTo on Installing into |
The [HowTo on Installing into |
RAID-1](http://mail-index.NetBSD.org/port-xen/2006/03/01/0010.html) |
RAID-1](https://mail-index.NetBSD.org/port-xen/2006/03/01/0010.html) |
explains how to set up booting a dom0 with Xen using grub with |
explains how to set up booting a dom0 with Xen using grub with |
NetBSD's RAIDframe. (This is obsolete with the use of NetBSD's native |
NetBSD's RAIDframe. (This is obsolete with the use of NetBSD's native |
boot.) |
boot. Now, just create a system with RAID-1, and alter /boot.cfg as |
|
described above.) |
|
|
Configuring Xen |
Configuring Xen |
--------------- |
--------------- |
Line 456 one is using xm or xl. The Xen 3.1, 3.3
|
Line 497 one is using xm or xl. The Xen 3.1, 3.3
|
4.2 and up packages use xl. To use xm with 4.2, edit xendomains to |
4.2 and up packages use xl. To use xm with 4.2, edit xendomains to |
use xm instead. |
use xm instead. |
|
|
|
For 3.1 and 3.3, you should enable xend and xenbackendd: |
For "xm", you should enable xend and xenbackendd: |
|
|
|
xend=YES |
xend=YES |
xenbackendd=YES |
xenbackendd=YES |
|
|
For "xl", you should enable xend and xencommons (xenstored). Trying |
For 4.1 and up, you should enable xencommons. Not enabling xencommons |
to use xl without xencommons=YES will result in a hang; it is |
will result in a hang; it is necessary to hit ^C on the console to let |
necessary to hit ^C on the console to let the machine finish booting. |
the machine finish booting. If you are using xm (default in 4.1, or |
TODO: Check if xend really should be used, and if not why it is |
if you changed xendomains in 4.2), you should also enable xend: |
installed. |
|
|
|
xend=YES |
xend=YES # only if using xm, and only installed <= 4.2 |
xencommons=YES |
xencommons=YES |
|
|
TODO: Recommend for/against xen-watchdog. |
TODO: Recommend for/against xen-watchdog. |
Line 606 Hardware known to work
|
Line 645 Hardware known to work
|
Arguably, this section is misplaced, and there should be a page of |
Arguably, this section is misplaced, and there should be a page of |
hardware that runs NetBSD/amd64 well, with the mostly-well-founded |
hardware that runs NetBSD/amd64 well, with the mostly-well-founded |
assumption that NetBSD/xen runs fine on any modern hardware that |
assumption that NetBSD/xen runs fine on any modern hardware that |
NetBSD/amd64 runs well on. Until then, we give motherboard/CPU/RAM |
NetBSD/amd64 runs well on. Until then, we give motherboard/CPU (and |
triples to aid those choosing a motherboard. Note that Xen systems |
sometimes RAM) pairs/triples to aid those choosing a motherboard. |
usually do not run X, so a listing here does not imply that X works at |
Note that Xen systems usually do not run X, so a listing here does not |
all. |
imply that X works at all. |
|
|
Supermicro X9SRL-F, Xeon E5-1650 v2, 96 GiB ECC |
Supermicro X9SRL-F, Xeon E5-1650 v2, 96 GiB ECC |
Supermicro ??, Atom C2758 (8 core), 32 GiB ECC |
Supermicro ??, Atom C2758 (8 core), 32 GiB ECC |
Line 618 all.
|
Line 657 all.
|
Older hardware: |
Older hardware: |
|
|
Intel D915GEV, Pentium4 CPU 3.40GHz, 4GB 533MHz Synchronous DDR2 |
Intel D915GEV, Pentium4 CPU 3.40GHz, 4GB 533MHz Synchronous DDR2 |
|
INTEL DG33FB, "Intel(R) Core(TM)2 Duo CPU E6850 @ 3.00GHz" |
|
INTEL DG33FB, "Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz" |
|
|
Running Xen under qemu |
Running Xen under qemu |
---------------------- |
---------------------- |
Line 635 In 2015-01, the following combination wa
|
Line 676 In 2015-01, the following combination wa
|
dom0 kernel: NetBSD/amd64 6.1.5 |
dom0 kernel: NetBSD/amd64 6.1.5 |
Xen tools: xentools42-4.2.5 from pkgsrc |
Xen tools: xentools42-4.2.5 from pkgsrc |
|
|
See [PR 47720](http://gnats.netbsd.org/47720) for a problem with dom0 |
See [PR 47720](https://gnats.netbsd.org/47720) for a problem with dom0 |
shutdown. |
shutdown. |
|
|
Unprivileged domains (domU) |
Unprivileged domains (domU) |
Line 1130 DOMU kernel.
|
Line 1171 DOMU kernel.
|
TODO: Explain how to compile npf into a custom kernel, answering (but |
TODO: Explain how to compile npf into a custom kernel, answering (but |
note that the problem was caused by not booting the right kernel) |
note that the problem was caused by not booting the right kernel) |
[this email to |
[this email to |
netbsd-users](http://mail-index.netbsd.org/netbsd-users/2014/12/26/msg015576.html). |
netbsd-users](https://mail-index.netbsd.org/netbsd-users/2014/12/26/msg015576.html). |
|
|
TODO items for improving NetBSD/xen |
TODO items for improving NetBSD/xen |
=================================== |
=================================== |
Line 1145 TODO items for improving NetBSD/xen
|
Line 1186 TODO items for improving NetBSD/xen
|
fragsize/blocksize (UFS2 support may be present; the point is to |
fragsize/blocksize (UFS2 support may be present; the point is to |
make it so that with any UFS1/UFS2 file system setup that works |
make it so that with any UFS1/UFS2 file system setup that works |
with NetBSD grub will also work). |
with NetBSD grub will also work). |
See [pkg/40258](http://gnats.netbsd.org/40258). |
See [pkg/40258](https://gnats.netbsd.org/40258). |
* Push patches upstream. |
* Push patches upstream. |
* Get UFS2 patches into pvgrub. |
* Get UFS2 patches into pvgrub. |
* Add support for PV ops to a version of /boot, and make it usable as |
* Add support for PV ops to a version of /boot, and make it usable as |