version 1.75, 2015/01/17 01:32:12
|
version 1.101, 2015/03/12 23:40:07
|
Line 92 reasonable although trailing-edge choice
|
Line 92 reasonable although trailing-edge choice
|
xenkernel42 provides Xen 4.2. This is maintained by Xen, but old as |
xenkernel42 provides Xen 4.2. This is maintained by Xen, but old as |
of 2014-12. |
of 2014-12. |
|
|
|
xenkernel45 provides Xen 4.5. This is new to pkgsrc as of 2015-01 and |
|
not yet recommended for other than experimental/testing use. |
|
|
|
See also the [Xen Security Advisory page](http://xenbits.xen.org/xsa/). |
|
|
Ideally newer versions of Xen will be added to pkgsrc. |
Ideally newer versions of Xen will be added to pkgsrc. |
|
|
Note that NetBSD support is called XEN3. It works with 3.1 through |
Note that NetBSD support is called XEN3. It works with Xen 3 and Xen |
4.2 because the hypercall interface has been stable. |
4 because the hypercall interface has been stable. |
|
|
Xen command program |
Xen command program |
------------------- |
------------------- |
|
|
Early Xen used a program called "xm" to manipulate the system from the |
Early Xen used a program called xm to manipulate the system from the |
dom0. Starting in 4.1, a replacement program with similar behavior |
dom0. Starting in 4.1, a replacement program with similar behavior |
called "xl" is provided. In 4.2 and later, "xl" is preferred. 4.4 is |
called xl is provided, but it does not work well in 4.1. In 4.2, both |
the last version that has "xm". |
xm and xl work fine. 4.4 is the last version that has xm. You must |
|
choose one or the other, because it affects which daemons you run. |
|
|
NetBSD |
NetBSD |
------ |
------ |
Line 124 Architecture
|
Line 130 Architecture
|
------------ |
------------ |
|
|
Xen itself can run on i386 or amd64 machines. (Practically, almost |
Xen itself can run on i386 or amd64 machines. (Practically, almost |
any computer where one would want to run Xen supports amd64.) If |
any computer where one would want to run Xen today supports amd64.) |
using an i386 NetBSD kernel for the dom0, PAE is required (PAE |
|
versions are built by default). While i386 dom0 works fine, amd64 is |
Xen, the dom0 kernel, and each domU kernel can be either i386 or |
recommended as more normal. |
amd64. When building a xenkernel package, one obtains i386 on an i386 |
|
host, and amd64 on an amd64 host. If the xen kernel is i386, then the |
|
dom0 kernel and all domU kernels must be i386. With an amd64 xen |
|
kernel, an amd64 dom0 kernel is known to work, and an i386 dom0 kernel |
|
should in theory work. An amd64 xen/dom0 is known to support both |
|
i386 and amd64 domUs. |
|
|
|
i386 dom0 and domU kernels must be PAE (except for Xen 3.1); these are |
|
built by default. (Note that emacs (at least) fails if run on i386 |
|
with PAE when built without, and vice versa, presumably due to bugs in |
|
the undump code.) |
|
|
|
Because of the above, the standard approach is to use amd64 for the |
|
dom0. |
|
|
Xen 4.2 is the last version to support i386 as a host. TODO: Clarify |
Xen 4.2 is the last version to support i386 as a host. TODO: Clarify |
if this is about the CPU having to be amd64, or about the dom0 kernel |
if this is about the CPU, the xen kernel, or the dom0 kernel having to |
having to be amd64. |
be amd64. |
|
|
|
|
|
Stability |
|
--------- |
|
|
|
Mostly, NetBSD as a dom0 or domU is quite stable. |
|
However, there are some open PRs indicating problems. |
|
|
|
- [PR 48125](http://gnats.netbsd.org/48125) |
|
- [PR 47720](http://gnats.netbsd.org/47720) |
|
|
One can then run i386 domUs and amd64 domUs, in any combination. If |
Note also that there are issues with sparse vnd(4) instances, but |
running an i386 NetBSD kernel as a domU, the PAE version is required. |
these are not about Xen. |
(Note that emacs (at least) fails if run on i386 with PAE when built |
|
without, and vice versa, presumably due to bugs in the undump code.) |
|
|
|
Recommendation |
Recommendation |
-------------- |
-------------- |
Line 210 dom0 is what the computer would have bee
|
Line 237 dom0 is what the computer would have bee
|
desktop or laptop. Then, one can run domUs at will. Purists will |
desktop or laptop. Then, one can run domUs at will. Purists will |
deride this as less secure than the previous approach, and for a |
deride this as less secure than the previous approach, and for a |
computer whose purpose is to run domUs, they are right. But Xen and a |
computer whose purpose is to run domUs, they are right. But Xen and a |
dom0 (without domUs) is not meaingfully less secure than the same |
dom0 (without domUs) is not meaningfully less secure than the same |
things running without Xen. One can boot Xen or boot regular NetBSD |
things running without Xen. One can boot Xen or boot regular NetBSD |
alternately with little problems, simply refraining from starting the |
alternately with little problems, simply refraining from starting the |
Xen daemons when not running Xen. |
Xen daemons when not running Xen. |
Line 235 over a RAID1 header to find /boot from a
|
Line 262 over a RAID1 header to find /boot from a
|
partition; this is no different when booting Xen. |
partition; this is no different when booting Xen. |
|
|
There are 4 styles of providing backing storage for the virtual disks |
There are 4 styles of providing backing storage for the virtual disks |
used by domUs: raw partitions, LVM, file-backed vnd(4), and SAN, |
used by domUs: raw partitions, LVM, file-backed vnd(4), and SAN. |
|
|
With raw partitions, one has a disklabel (or gpt) partition sized for |
With raw partitions, one has a disklabel (or gpt) partition sized for |
each virtual disk to be used by the domU. (If you are able to predict |
each virtual disk to be used by the domU. (If you are able to predict |
Line 289 with an MBR bootblock, either bootxx_ffs
|
Line 316 with an MBR bootblock, either bootxx_ffs
|
beginning of your root filesystem, /boot present, and likely |
beginning of your root filesystem, /boot present, and likely |
/boot.cfg. (If not, fix before continuing!) |
/boot.cfg. (If not, fix before continuing!) |
|
|
See boot.cfg(5) for an example. The basic line is |
Add a line to to /boot.cfg to boot Xen. See boot.cfg(5) for an |
|
example. The basic line is |
|
|
menu=Xen:load /netbsd-XEN3_DOM0.gz console=pc;multiboot /xen.gz dom0_mem=256M |
menu=Xen:load /netbsd-XEN3_DOM0.gz console=pc;multiboot /xen.gz dom0_mem=256M |
|
|
which specifies that the dom0 should have 256M, leaving the rest to be |
which specifies that the dom0 should have 256M, leaving the rest to be |
allocated for domUs. To use In an attempt to add performance, one can |
allocated for domUs. To use a serial console, use |
also add |
|
|
menu=Xen:load /netbsd-XEN3_DOM0.gz console=com0;multiboot /xen.gz dom0_mem=256M console=com1 com1=9600,8n1 |
|
|
|
which will use the first serial port for Xen (which counts starting |
|
from 1), forcing speed/parity, and also for NetBSD (which counts |
|
starting at 0). In an attempt to add performance, one can also add |
|
|
dom0_max_vcpus=1 dom0_vcpus_pin |
dom0_max_vcpus=1 dom0_vcpus_pin |
|
|
to force only one vcpu to be provided (since NetBSD dom0 can't use |
to force only one vcpu to be provided (since NetBSD dom0 can't use |
more) and to pin that vcpu to a physical cpu. TODO: benchmark this. |
more) and to pin that vcpu to a physical cpu. TODO: benchmark this. |
|
|
|
Xen has [many boot |
|
options](http://xenbits.xenproject.org/docs/4.5-testing/misc/xen-command-line.html), |
|
and other tham dom0 memory and max_vcpus, they are generally not |
|
necessary. |
|
|
As with non-Xen systems, you should have a line to boot /netbsd (a |
As with non-Xen systems, you should have a line to boot /netbsd (a |
kernel that works without Xen) and fallback versions of the non-Xen |
kernel that works without Xen) and fallback versions of the non-Xen |
kernel, Xen, and the dom0 kernel. |
kernel, Xen, and the dom0 kernel. |
|
|
|
Now, reboot so that you are running a DOM0 kernel under Xen, rather |
|
than GENERIC without Xen. |
|
|
Using grub (historic) |
Using grub (historic) |
--------------------- |
--------------------- |
|
|
Line 324 Configuring Xen
|
Line 365 Configuring Xen
|
|
|
Xen logs will be in /var/log/xen. |
Xen logs will be in /var/log/xen. |
|
|
Now, you have a system that will boot Xen and the dom0 kernel, and |
Now, you have a system that will boot Xen and the dom0 kernel, but not |
just run the dom0 kernel. There will be no domUs, and none can be |
do anything else special. Make sure that you have rebooted into Xen. |
started because you still have to configure the dom0 tools. The |
There will be no domUs, and none can be started because you still have |
daemons which should be run vary with Xen version and with whether one |
to configure the dom0 tools. The daemons which should be run vary |
is using xm or xl. Note that xend is for supporting "xm", and should |
with Xen version and with whether one is using xm or xl. Note that |
only be used if you plan on using "xm". Do NOT enable xend if you |
xend is for supporting "xm", and should only be used if you plan on |
plan on using "xl" as it will cause problems. |
using "xm". Do NOT enable xend if you plan on using "xl" as it will |
|
cause problems. Running xl without xencommons=YES (and starting it) |
|
will result in a hang (so don't do that; follow the HOWTO!). |
|
|
The installation of NetBSD should already have created devices for xen |
The installation of NetBSD should already have created devices for xen |
(xencons, xenevt), but if they are not present, create them: |
(xencons, xenevt), but if they are not present, create them: |
Line 358 For 4.2 with xm, add to rc.conf
|
Line 401 For 4.2 with xm, add to rc.conf
|
xencommons=YES |
xencommons=YES |
xend=YES |
xend=YES |
|
|
For 4.2 with xl (preferred), add to rc.conf: |
For 4.2 with xl, add to rc.conf: |
|
|
|
xencommons=YES |
|
TODO: explain if there is a xend replacement |
|
|
|
For 4.5 (and thus with xl), add to rc.conf: |
|
|
xencommons=YES |
xencommons=YES |
TODO: explain if there is a xend replacement |
TODO: explain if there is a xend replacement |
Line 366 For 4.2 with xl (preferred), add to rc.c
|
Line 414 For 4.2 with xl (preferred), add to rc.c
|
TODO: Recommend for/against xen-watchdog. |
TODO: Recommend for/against xen-watchdog. |
|
|
After you have configured the daemons and either started them (in the |
After you have configured the daemons and either started them (in the |
order given) or rebooted, run the following (or use xl) to inspect |
order given) or rebooted, use xm or xl to inspect Xen's boot messages, |
Xen's boot messages, available resources, and running domains: |
available resources, and running domains. An example with xm follows: |
|
|
# xm dmesg |
# xm dmesg |
[xen's boot info] |
[xen's boot info] |
Line 377 Xen's boot messages, available resources
|
Line 425 Xen's boot messages, available resources
|
Name Id Mem(MB) CPU State Time(s) Console |
Name Id Mem(MB) CPU State Time(s) Console |
Domain-0 0 64 0 r---- 58.1 |
Domain-0 0 64 0 r---- 58.1 |
|
|
|
With xl, the commands are the same, and the output may be slightly |
|
different. TODO: add example output for xl before the xm example, |
|
after confirming on 4.2 and resolving the TODO about rc.conf. |
|
|
|
### Issues with xencommons |
|
|
|
xencommons starts xenstored, which stores data on behalf of dom0 and |
|
domUs. It does not currently work to stop and start xenstored. |
|
Certainly all domUs should be shutdown first, following the sort order |
|
of the rc.d scripts. However, the dom0 sets up state with xenstored, |
|
and is not notified when xenstored exits, leading to not recreating |
|
the state when the new xenstored starts. Until there's a mechanism to |
|
make this work, one should not expect to be able to restart xenstored |
|
(and thus xencommons). There is currently no reason to expect that |
|
this will get fixed any time soon. |
|
|
anita (for testing NetBSD) |
anita (for testing NetBSD) |
-------------------------- |
-------------------------- |
|
|
With the setup so far, one should be able to run anita (see |
With the setup so far (assuming 4.2/xl), one should be able to run |
pkgsrc/sysutils/py-anita) to test NetBSD releases, by doing (as root, |
anita (see pkgsrc/misc/py-anita) to test NetBSD releases, by doing (as |
because anita must create a domU): |
root, because anita must create a domU): |
|
|
anita --vmm=xm test file:///usr/obj/i386/ |
anita --vmm=xl test file:///usr/obj/i386/ |
|
|
Alternatively, one can use --vmm=xl to use xl-based domU creation instead. |
Alternatively, one can use --vmm=xm to use xm-based domU creation |
TODO: check this. |
instead (and must, on Xen <= 4.1). TODO: confirm that anita xl really works. |
|
|
Xen-specific NetBSD issues |
Xen-specific NetBSD issues |
-------------------------- |
-------------------------- |
|
|
Line 461 Ensure that the contents of /etc/rc.d/xe
|
Line 525 Ensure that the contents of /etc/rc.d/xe
|
correct set of daemons. Ensure that the domU config files are valid |
correct set of daemons. Ensure that the domU config files are valid |
for the new version. |
for the new version. |
|
|
|
Hardware known to work |
|
---------------------- |
|
|
|
Arguably, this section is misplaced, and there should be a page of |
|
hardware that runs NetBSD/amd64 well, with the mostly-well-founded |
|
assumption that NetBSD/xen runs fine on any modern hardware that |
|
NetBSD/amd64 runs well on. Until then, we give motherboard/CPU/RAM |
|
triples to aid those choosing a motherboard. Note that Xen systems |
|
usually do not run X, so a listing here does not imply that X works at |
|
all. |
|
|
|
Supermicro X9SRL-F, Xeon E5-1650 v2, 96 GiB ECC |
|
Supermicro ??, Atom C2758 (8 core), 32 GiB ECC |
|
ASUS M5A78L-M/USB3 AM3+ microATX, AMD Piledriver X8 4000MHz, 16 GiB ECC |
|
|
|
Older hardware: |
|
|
|
Intel D915GEV, Pentium4 CPU 3.40GHz, 4GB 533MHz Synchronous DDR2 |
|
|
|
Running Xen under qemu |
|
---------------------- |
|
|
|
The astute reader will note that this section is somewhat twisted. |
|
However, it can be useful to run Xen under qemu either because the |
|
version of NetBSD as a dom0 does not run on the hardware in use, or to |
|
generate automated test cases involving Xen. |
|
|
|
In 2015-01, the following combination was reported to mostly work: |
|
|
|
host OS: NetBSD/amd64 6.1.4 |
|
qemu: 2.2.0 from pkgsrc |
|
Xen kernel: xenkernel42-4.2.5nb1 from pkgsrc |
|
dom0 kernel: NetBSD/amd64 6.1.5 |
|
Xen tools: xentools42-4.2.5 from pkgsrc |
|
|
|
See [PR 47720](http://gnats.netbsd.org/47720) for a problem with dom0 |
|
shutdown. |
|
|
Unprivileged domains (domU) |
Unprivileged domains (domU) |
=========================== |
=========================== |
Line 522 domain, and see if it has finished stopp
|
Line 623 domain, and see if it has finished stopp
|
xm console foo |
xm console foo |
xm create -c foo |
xm create -c foo |
xm shutdown foo |
xm shutdown foo |
xm list |
xm list |
|
|
Typing ^] will exit the console session. Shutting down a domain is |
Typing ^] will exit the console session. Shutting down a domain is |
equivalent to pushing the power button; a NetBSD domU will receive a |
equivalent to pushing the power button; a NetBSD domU will receive a |
Line 582 create an empty 4G virtual disk, simply
|
Line 683 create an empty 4G virtual disk, simply
|
|
|
dd if=/dev/zero of=foo-xbd0 bs=1m count=4096 |
dd if=/dev/zero of=foo-xbd0 bs=1m count=4096 |
|
|
|
Do not use qemu-img-xen, because this will create sparse file. There |
|
have been recent (2015) reports of sparse vnd(4) devices causing |
|
lockups, but there is apparently no PR. |
|
|
With the lvm style, one creates logical devices. They are then used |
With the lvm style, one creates logical devices. They are then used |
similarly to vnds. TODO: Add an example with lvm. |
similarly to vnds. TODO: Add an example with lvm. |
|
|
Line 659 shutdown, in rc.conf add:
|
Line 764 shutdown, in rc.conf add:
|
|
|
xendomains="foo bar" |
xendomains="foo bar" |
|
|
TODO: Explain why 4.1 rc.d/xendomains has xl, when one should use xm |
Note that earlier versions of the xentools41 xendomains rc.d scripth |
on 4.1. Or fix the xentools41 package to have xm |
usd xl, when one should use xm with 4.1. |
|
|
Creating specific unprivileged domains (domU) |
Creating specific unprivileged domains (domU) |
============================================= |
============================================= |
Line 685 i386 and amd64 provide the following ker
|
Line 790 i386 and amd64 provide the following ker
|
|
|
i386 XEN3_DOMU |
i386 XEN3_DOMU |
i386 XEN3PAE_DOMU |
i386 XEN3PAE_DOMU |
amd64 XEN3_DOMU |
amd64 XEN3_DOMU |
|
|
Unless using Xen 3.1 (and you shouldn't) with i386-mode Xen, you must |
Unless using Xen 3.1 (and you shouldn't) with i386-mode Xen, you must |
use the PAE version of the i386 kernel. |
use the PAE version of the i386 kernel. |
Line 738 It is also desirable to add
|
Line 843 It is also desirable to add
|
in rc.conf. This way, the domain will be properly shut down if |
in rc.conf. This way, the domain will be properly shut down if |
`xm shutdown -R` or `xm shutdown -H` is used on the dom0. |
`xm shutdown -R` or `xm shutdown -H` is used on the dom0. |
|
|
Your domain should be now ready to work, enjoy. |
It is not strictly necessary to have a kernel (as /netbsd) in the domU |
|
filesystem. However, various programs (e.g. netstat) will use that |
|
kernel to look up symbols to read from kernel virtual memory. If |
|
/netbsd is not the running kernel, those lookups will fail. (This is |
|
not really a Xen-specific issue, but because the domU kernel is |
|
obtained from the dom0, it is far more likely to be out of sync or |
|
missing with Xen.) |
|
|
Creating an unprivileged Linux domain (domU) |
Creating an unprivileged Linux domain (domU) |
-------------------------------------------- |
-------------------------------------------- |
Line 926 to update the special boot partiion.
|
Line 1037 to update the special boot partiion.
|
Amazon |
Amazon |
------ |
------ |
|
|
TODO: add link to NetBSD amazon howto. |
See the [Amazon EC2 page](../amazon_ec2/). |
|
|
Using npf |
Using npf |
--------- |
--------- |
|
|
In standard kernels, npf is a module, and thus cannot be loadeed in a |
In standard kernels, npf is a module, and thus cannot be loaded in a |
DOMU kernel. |
DOMU kernel. |
|
|
TODO: explain how to compile npf into a custom kernel, answering (but |
TODO: Explain how to compile npf into a custom kernel, answering (but |
note that the problem was caused by not booting the right kernel): |
note that the problem was caused by not booting the right kernel) |
http://mail-index.netbsd.org/netbsd-users/2014/12/26/msg015576.html |
[this email to |
|
netbsd-users](http://mail-index.netbsd.org/netbsd-users/2014/12/26/msg015576.html). |
|
|
TODO items for improving NetBSD/xen |
TODO items for improving NetBSD/xen |
=================================== |
=================================== |
|
|
* Package Xen 4.4. |
* Make the NetBSD dom0 kernel work with SMP. |
* Get PCI passthrough working on Xen 4.2 (or 4.4). |
* Test the Xen 4.5 packages adequately to be able to recommend them as |
|
the standard approach. |
|
* Get PCI passthrough working on Xen 4.5 |
* Get pvgrub into pkgsrc, either via xentools or separately. |
* Get pvgrub into pkgsrc, either via xentools or separately. |
* grub |
* grub |
* Check/add support to pkgsrc grub2 for UFS2 and arbitrary |
* Check/add support to pkgsrc grub2 for UFS2 and arbitrary |
Line 954 TODO items for improving NetBSD/xen
|
Line 1068 TODO items for improving NetBSD/xen
|
* Get UFS2 patches into pvgrub. |
* Get UFS2 patches into pvgrub. |
* Add support for PV ops to a version of /boot, and make it usable as |
* Add support for PV ops to a version of /boot, and make it usable as |
a kernel in Xen, similar to pvgrub. |
a kernel in Xen, similar to pvgrub. |
|
* Solve somehow the issue with modules for GENERIC not being loadable |
|
in a Xen dom0 or domU kernel. |
|
|
|
Random pointers |
|
=============== |
|
|
|
TODO: This section contains links from elsewhere not yet integrated |
|
into the HOWTO. |
|
|
|
* http://www.lumbercartel.ca/library/xen/ |
|
* http://pbraun.nethence.com/doc/sysutils/xen_netbsd_dom0.html |