version 1.19, 2014/12/24 00:41:04
|
version 1.141, 2017/12/15 16:40:59
|
Line 2 Introduction
|
Line 2 Introduction
|
============ |
============ |
|
|
[![[Xen |
[![[Xen |
screenshot]](http://www.netbsd.org/gallery/in-Action/hubertf-xens.png)](../../gallery/in-Action/hubertf-xen.png) |
screenshot]](https://www.netbsd.org/gallery/in-Action/hubertf-xens.png)](https://www.netbsd.org/gallery/in-Action/hubertf-xen.png) |
|
|
Xen is a virtual machine monitor or hypervisor for x86 hardware |
Xen is a hypervisor (or virtual machine monitor) for x86 hardware |
(i686-class or higher), which supports running multiple guest |
(i686-class or higher), which supports running multiple guest |
operating systems on a single physical machine. With Xen, one uses |
operating systems on a single physical machine. Xen is a Type 1 or |
the Xen kernel to control the CPU, memory and console, a dom0 |
bare-metal hypervisor; one uses the Xen kernel to control the CPU, |
operating system which mediates access to other hardware (e.g., disks, |
memory and console, a dom0 operating system which mediates access to |
network, USB), and one or more domU operating systems which operate in |
other hardware (e.g., disks, network, USB), and one or more domU |
an unprivileged virtualized environment. IO requests from the domU |
operating systems which operate in an unprivileged virtualized |
systems are forwarded by the hypervisor (Xen) to the dom0 to be |
environment. IO requests from the domU systems are forwarded by the |
fulfilled. |
hypervisor (Xen) to the dom0 to be fulfilled. |
|
|
Xen supports two styles of guests. The original is Para-Virtualized |
Xen supports two styles of guests. The original is Para-Virtualized |
(PV) which means that the guest OS does not attempt to access hardware |
(PV) which means that the guest OS does not attempt to access hardware |
Line 27 code for Xen and need not be aware that
|
Line 27 code for Xen and need not be aware that
|
Attempts to access hardware registers are trapped and emulated. This |
Attempts to access hardware registers are trapped and emulated. This |
style is less efficient but can run unmodified guests. |
style is less efficient but can run unmodified guests. |
|
|
Generally any amd64 machine will work with Xen and PV guests. For HVM |
Generally any machine that runs NetBSD/amd64 will work with Xen and PV |
guests, the VMX cpu feature (Intel) or VT?? (amd64) is needed. TODO: |
guests. In theory i386 computers (without x86_64/amd64 support) can |
Explain if i386 (non-amd64) machines can still be used - I think that |
be used for Xen <= 4.2, but we have no recent reports of this working |
witthe requirement to use PAE kernels is about the hypervisor being |
(this is a hint). For HVM guests, hardware support is needed, but it |
amd64 only. |
is common on recent machines. For Intel CPUs, one needs the VT-x |
|
extension, shown in "cpuctl identify 0" as VMX. For AMD CPUs, one |
|
needs the AMD-V extensions, shown in "cpuctl identify 0" as SVM. |
|
There are further features for IOMMU virtualization, Intel's VT-d and |
|
AMD's AMD-Vi. TODO: Explain whether Xen on NetBSD makes use of these |
|
features. TODO: Review by someone who really understands this. |
|
|
At boot, the dom0 kernel is loaded as module with Xen as the kernel. |
Note that a FreeBSD dom0 requires VT-x and VT-d (or equivalent); this |
|
is because the FreeBSD dom0 does not run in PV mode. |
|
|
|
At boot, the dom0 kernel is loaded as a module with Xen as the kernel. |
The dom0 can start one or more domUs. (Booting is explained in detail |
The dom0 can start one or more domUs. (Booting is explained in detail |
in the dom0 section.) |
in the dom0 section.) |
|
|
NetBSD supports Xen in that it can serve as dom0, be used as a domU, |
NetBSD supports Xen in that it can serve as dom0, be used as a domU, |
and that Xen kernels and tools are available in pkgsrc. This HOWTO |
and that Xen kernels and tools are available in pkgsrc. This HOWTO |
attempts to address both the case of running a NetBSD dom0 on hardware |
attempts to address both the case of running a NetBSD dom0 on hardware |
and running NetBSD as a domU in a VPS. |
and running domUs under it (NetBSD and other), and also running NetBSD |
|
as a domU in a VPS. |
|
|
|
Xen 3.1 in pkgsrc supports "PCI passthrough", which means that |
|
specific PCI devices can be made available to a specific domU instead |
|
of the dom0. This can be useful to let a domU run X11, or access some |
|
network interface or other peripheral. |
|
|
|
NetBSD 6 and earlier supported Xen 2; support was removed from NetBSD |
|
7. Xen 2 has been removed from pkgsrc. |
|
|
Prerequisites |
Prerequisites |
------------- |
------------- |
Line 52 things must be done, guiding the reader
|
Line 69 things must be done, guiding the reader
|
path when there are no known good reasons to stray. |
path when there are no known good reasons to stray. |
|
|
This HOWTO presumes a basic familiarity with the Xen system |
This HOWTO presumes a basic familiarity with the Xen system |
architecture. This HOWTO presumes familiarity with installing NetBSD |
architecture, with installing NetBSD on i386/amd64 hardware, and with |
on i386/amd64 hardware and installing software from pkgsrc. |
installing software from pkgsrc. See also the [Xen |
See also the [Xen website](http://www.xen.org/). |
website](http://www.xenproject.org/). |
|
|
History |
|
------- |
|
|
|
NetBSD used to support Xen2; this has been removed. |
|
|
|
Before NetBSD's native bootloader could support Xen, the use of |
|
grub was recommended. If necessary, see the |
|
[old grub information](/xen/howto-grub/). |
|
|
|
Versions of Xen and NetBSD |
Versions of Xen and NetBSD |
========================== |
========================== |
|
|
Most of the installation concepts and instructions are independent of |
Most of the installation concepts and instructions are independent |
Xen version. This section gives advice on which version to choose. |
of Xen version and NetBSD version. This section gives advice on |
Versions not in pkgsrc and older unsupported versions of NetBSD are |
which version to choose. Versions not in pkgsrc and older unsupported |
inentionally ignored. |
versions of NetBSD are intentionally ignored. |
|
|
|
The term "amd64" is used to refer to both the NetBSD port and to the |
|
hardware architecture on which it runs. (Such hardware is made by |
|
both Intel and AMD, and in 2016 a normal PC has this CPU |
|
architecture.) |
|
|
Xen |
Xen |
--- |
--- |
|
|
In NetBSD, xen is provided in pkgsrc, via matching pairs of packages |
In NetBSD, Xen is provided in pkgsrc, via matching pairs of packages |
xenkernel and xentools. We will refer only to the kernel versions, |
xenkernel and xentools. We will refer only to the kernel versions, |
but note that both packages must be installed together and must have |
but note that both packages must be installed together and must have |
matching versions. |
matching versions. |
|
|
xenkernel3 and xenkernel33 provide Xen 3.1 and 3.3. These no longer |
xenkernel3 provides Xen 3.1. It is no longer maintained by Xen, and |
receive security patches and should not be used. |
the last applied security patch was in 2011. Thus, it should not be |
|
used. It supports PCI passthrough, which is why people use it anyway. |
|
Xen 3.1 runs on i386 (both non-PAE and PAE) and amd64 hardware. |
|
|
|
xenkernel33 provides Xen 3.3. It is no longer maintained by Xen, and |
|
the last applied security patch was in 2012. Thus, it should not be |
|
used. Xen 3.3 runs on i386 PAE and amd64 hardware. There are no good |
|
reasons to run this version. |
|
|
|
xenkernel41 provides Xen 4.1. It is no longer maintained by Xen, but |
|
as of 2016-12 received backported security patches. Xen 4.1 runs on |
|
i386 PAE and amd64 hardware. There are no good reasons to run this |
|
version. |
|
|
|
Note that 3.1, 3.3 and 4.1 have been removed from pkgsrc-current, but |
|
are in 2016Q4. They will be removed from this HOWTO sometime after |
|
2017Q1. |
|
|
|
xenkernel42 provides Xen 4.2. It is no longer maintained by Xen, but |
|
as of 2016-12 received backported security patches. Xen 4.2 runs on |
|
i386 PAE and amd64 hardware. The only reason to run this is if you |
|
need to use xm instead of xl, or if you need to run on hardware that |
|
supports i386 but not amd64. (This might also be useful if you need |
|
an i386 dom0, if it turns out that an amd64 Xen kernel and an i386 |
|
dom0 is problematic.) |
|
|
|
xenkernel45 provides Xen 4.5. As of 2016-12, security patches were |
|
released by Xen and applied to pkgsrc. Xen 4.5 runs on amd64 hardware |
|
only. While slightly old, 4.5 has been tested and run by others, so |
|
it is the conservative choice. |
|
|
|
xenkernel46 provides Xen 4.6. It is new to pkgsrc as of 2016-05. As |
|
of 2016-12, security patches were released by Xen and applied to |
|
pkgsrc. Xen 4.6 runs on amd64 hardware only For new installations, |
|
4.6 is probably the appropriate choice and it will likely soon be the |
|
standard approach. (If using Ubuntu guests, be sure to have the |
|
xentools46 from December, 2016). |
|
|
xenkernel41 provides Xen 4.1. This is no longer maintained by Xen, |
Xen 4.7 (released 2016-06) and 4.8 (released 2016-12) are not yet in |
but as of 2014-12 receives backported security patches. It is a |
pkgsrc. |
reasonable although trailing-edge choice. |
|
|
|
xenkernel42 provides Xen 4.2. This is maintained by Xen, but old as |
See also the [Xen Security Advisory page](http://xenbits.xen.org/xsa/). |
of 2014-12. |
|
|
|
Ideally newer versions of Xen will be added to pkgsrc. |
Note that NetBSD support is called XEN3. It works with Xen 3 and Xen |
|
4 because the hypercall interface has been stable. |
|
|
Xen command program |
Xen command program |
------------------- |
------------------- |
|
|
Early Xen used a program called "xm" to manipulate the system from the |
Early Xen used a program called xm to manipulate the system from the |
dom0. Starting in 4.1, a replacement program with similar behavior |
dom0. Starting in 4.1, a replacement program with similar behavior |
called "xl" is provided. In 4.2, "xm" is no longer available. |
called xl is provided, but it does not work well in 4.1. In 4.2, both |
|
xm and xl work fine. 4.4 is the last version that has xm. |
|
|
|
You must make a global choice to use xm or xl, because it affects not |
|
only which command you use, but the command used by rc.d scripts |
|
(specifically xendomains) and which daemons should be run. The |
|
xentools packages provide xm for 3.1, 3.3 and 4.1 and xl for 4.2 and up. |
|
|
|
In 4.2, you can choose to use xm by simply changing the ctl_command |
|
variable and setting xend=YES in rc.conf. |
|
|
|
With xl, virtual devices are configured in parallel, which can cause |
|
problems if they are written assuming serial operation (e.g., updating |
|
firewall rules without explicit locking). There is now locking for |
|
the provided scripts, which works for normal casses (e.g, file-backed |
|
xbd, where a vnd must be allocated). But, as of 201612, it has not |
|
been adequately tested for a complex custom setup with a large number |
|
of interfaces. |
|
|
NetBSD |
NetBSD |
------ |
------ |
|
|
The netbsd-5, netbsd-6, netbsd-7, and -current branches are all |
The netbsd-6, netbsd-7, and -current branches are all reasonable |
reasonable choices, with more or less the same considerations for |
choices, with more or less the same considerations for non-Xen use. |
non-Xen use. Therefore, netbsd-6 is recommended as the stable version |
Therefore, netbsd-7 is recommended as the stable version of the most |
of the most recent release. |
recent release for production use. In addition, netbsd-7 and -current |
|
have a important scheduler fix (in November of 2015) affecting |
|
contention between dom0 and domUs; see |
|
https://releng.netbsd.org/cgi-bin/req-7.cgi?show=1040 for a |
|
description. For those wanting to learn Xen or without production |
|
stability concerns, netbsd-7 is still likely most appropriate, but |
|
-current is also a reasonable choice. (Xen runs ok on netbsd-5, but |
|
the xentools packages are likely difficult to build, and netbsd-5 is |
|
not supported.) |
|
|
As of NetBSD 6, a NetBSD domU will support multiple vcpus. There is |
As of NetBSD 6, a NetBSD domU will support multiple vcpus. There is |
no SMP support for NetBSD as dom0. (The dom0 itself doesn't really |
no SMP support for NetBSD as dom0. (The dom0 itself doesn't really |
need SMP; the lack of support is really a problem when using a dom0 as |
need SMP for dom0 functions; the lack of support is really a problem |
a normal computer.) |
when using a dom0 as a normal computer.) |
|
|
Architecture |
Architecture |
------------ |
------------ |
|
|
Xen is basically amd64 only at this point. One can either run i386 |
Xen itself can run on i386 (Xen < 4.2) or amd64 hardware (all Xen |
domains or amd64 domains. If running i386, PAE versions are required, |
versions). (Practically, almost any computer where one would want to |
for both dom0 and domU. These versions are built by default in NetBSD |
run Xen today supports amd64.) |
releases. While i386 dom0 works fine, amd64 is recommended as more |
|
normal. (Note that emacs (at least) fails if run on i386 with PAE when |
Xen, the dom0 system, and each domU system can be either i386 or |
built without, and vice versa, presumably due to bugs in the undump |
amd64. When building a xenkernel package, one obtains an i386 Xen |
code.) |
kernel on an i386 host, and an amd64 Xen kernel on an amd64 host. If |
|
the Xen kernel is i386, then the dom0 kernel and all domU kernels must |
|
be i386. With an amd64 Xen kernel, an amd64 dom0 kernel is known to |
|
work, and an i386 dom0 kernel should in theory work. An amd64 |
|
Xen/dom0 is known to support both i386 and amd64 domUs. |
|
|
|
i386 dom0 and domU kernels must be PAE (except for an i386 Xen 3.1 |
|
kernel, where one can use non-PAE for dom0 and all domUs); PAE kernels |
|
are included in the NetBSD default build. (Note that emacs (at least) |
|
fails if run on i386 with PAE when built without, and vice versa, |
|
presumably due to bugs in the undump code.) |
|
|
|
Because of the above, the standard approach is to use an amd64 Xen |
|
kernel and NetBSD/amd64 for the dom0. For domUs, NetBSD/i386 (with |
|
the PAE kernel) and NetBSD/amd64 are in widespread use, and there is |
|
little to no Xen-specific reason to prefer one over the other. |
|
|
|
Note that to use an i386 dom0 with Xen 4.5 or higher, one must build |
|
(or obtain from pre-built packages) an amd64 Xen kernel and install |
|
that on the system. (One must also use a PAE i386 kernel, but this is |
|
also required with an i386 Xen kernel.). Almost no one in the |
|
NetBSD/Xen community does this, and the standard, well-tested, |
|
approach is to use an amd64 dom0. |
|
|
|
A [posting on |
|
xen-devel](https://lists.xen.org/archives/html/xen-devel/2012-07/msg00085.html) |
|
explained that PV system call overhead was higher on amd64, and thus |
|
there is some notion that i386 guests are faster. It goes on to |
|
caution that the total situation is complex and not entirely |
|
understood. On top of that caution, the post is about Linux, not |
|
NetBSD. TODO: Include link to benchmarks, if someone posts them. |
|
|
|
Stability |
|
--------- |
|
|
|
Mostly, NetBSD as a dom0 or domU is quite stable. |
|
However, there are some open PRs indicating problems. |
|
|
|
- [PR 48125](http://gnats.netbsd.org/48125) |
|
|
|
Note also that there are issues with sparse vnd(4) instances, but |
|
these are not about Xen -- they just are noticed with sparse vnd(4) |
|
instances in support of virtual disks in a dom0. |
|
|
Recommendation |
Recommendation |
-------------- |
-------------- |
|
|
Therefore, this HOWTO recommends running xenkernel42 (and xentools42), |
Therefore, this HOWTO recommends running xenkernel45 or xenkernel46, |
xl, the NetBSD 6 stable branch, and to use amd64 as the dom0. Either |
xl, the NetBSD 7 stable branch, and to use an amd64 kernel as the |
the i386 or amd64 of NetBSD may be used as domUs. |
dom0. Either the i386PAE or amd64 version of NetBSD may be used as |
|
domUs. |
|
|
|
Because bugs are fixed quite often, and because of Xen security |
|
advisories, it is good to stay up to date with NetBSD (tracking a |
|
stable branch), with the Xen kernel (tracking a Xen version via |
|
pkgsrc), and with the Xen tools. Specifically, NetBSD (-7 and |
|
-current) got an important fix affecting dom0/domU timesharing in |
|
November, 2015, and xentools46 got a fix to enable Ubuntu guests to |
|
boot in December, 2016. |
|
|
|
Status |
|
------ |
|
|
|
Ideally, all versions of Xen in pkgsrc would build on all supported |
|
versions of NetBSD/amd64, to the point where this section would be |
|
silly. However, that has not always been the case. Besides aging |
|
code and aging compilers, qemu (included in xentools for HVM support) |
|
is difficult to build. Note that there is intentionally no data for |
|
4.5+ up for i386, and often omits xentools info if the corresponding |
|
kernel fails. |
|
|
|
The following table gives status, with the date last checked |
|
(generally on the most recent quarterly branch). The first code is |
|
"builds" if it builds ok, and "FAIL" for a failure to build. The |
|
second code/date only appears for xenkernel* and is "works" if it runs |
|
ok as a dom0 and can support a domU, and "FAIL" if it won't boot or |
|
run a domU. |
|
|
|
xenkernel3 netbsd-6 i386 FAIL 201612 |
|
xenkernel33 netbsd-6 i386 FAIL 201612 |
|
xenkernel41 netbsd-6 i386 builds 201612 |
|
xenkernel42 netbsd-6 i386 builds 201612 |
|
xentools3 netbsd-6 i386 FAIL 201612 |
|
xentools33 netbsd-6 i386 FAIL 201612 |
|
xentools41 netbsd-6 i386 builds 201612 |
|
xentools42 netbsd-6 i386 FAIL 201612 |
|
|
|
xenkernel3 netbsd-7 i386 FAIL 201412 |
|
xenkernel33 netbsd-7 i386 FAIL 201412 |
|
xenkernel41 netbsd-7 i386 builds 201412 |
|
xenkernel42 netbsd-7 i386 builds 201412 |
|
xentools41 netbsd-7 i386 builds 201412 |
|
xentools42 netbsd-7 i386 ??FAIL 201412 |
|
|
|
xenkernel3 netbsd-6 amd64 FAIL 201612 |
|
xenkernel33 netbsd-6 amd64 FAIL 201612 |
|
xenkernel41 netbsd-6 amd64 builds 201612 works 201612 |
|
xenkernel42 netbsd-6 amd64 builds 201612 works 201612 |
|
xenkernel45 netbsd-6 amd64 builds 201612 |
|
xenkernel46 netbsd-6 amd64 builds 201612 |
|
xentools41 netbsd-6 amd64 builds 201612 |
|
xentools42 netbsd-6 amd64 builds 201612 |
|
xentools45 netbsd-6 amd64 builds 201612 |
|
xentools46 netbsd-6 amd64 FAIL 201612 |
|
|
|
xenkernel3 netbsd-7 amd64 builds 201612 |
|
xenkernel33 netbsd-7 amd64 builds 201612 |
|
xenkernel41 netbsd-7 amd64 builds 201612 |
|
xenkernel42 netbsd-7 amd64 builds 201612 |
|
xenkernel45 netbsd-7 amd64 builds 201612 |
|
xenkernel46 netbsd-7 amd64 builds 201612 |
|
xentools3 netbsd-7 amd64 builds 201612 |
|
xentools3-hvm netbsd-7 amd64 builds 201612 |
|
xentools33 netbsd-7 amd64 FAIL 201612 |
|
xentools41 netbsd-7 amd64 builds 201612 |
|
xentools42 netbsd-7 amd64 builds 201612 |
|
xentools45 netbsd-7 amd64 builds 201612 |
|
xentools46 netbsd-7 amd64 builds 201612 |
|
|
NetBSD as a dom0 |
NetBSD as a dom0 |
================ |
================ |
Line 142 NetBSD, which is not yet a dom0, and the
|
Line 325 NetBSD, which is not yet a dom0, and the
|
NetBSD install to a dom0 install by just changing the kernel and boot |
NetBSD install to a dom0 install by just changing the kernel and boot |
configuration. |
configuration. |
|
|
|
For experimenting with Xen, a machine with as little as 1G of RAM and |
|
100G of disk can work. For running many domUs in productions, far |
|
more will be needed; e.g. 4-8G and 1T of disk is reasonable for a |
|
half-dozen domUs of 512M and 32G each. Basically, the RAM and disk |
|
have to be bigger than the sum of the RAM/disk needs of the dom0 and |
|
all the domUs. |
|
|
Styles of dom0 operation |
Styles of dom0 operation |
------------------------ |
------------------------ |
|
|
Line 155 dom0 is what the computer would have bee
|
Line 345 dom0 is what the computer would have bee
|
desktop or laptop. Then, one can run domUs at will. Purists will |
desktop or laptop. Then, one can run domUs at will. Purists will |
deride this as less secure than the previous approach, and for a |
deride this as less secure than the previous approach, and for a |
computer whose purpose is to run domUs, they are right. But Xen and a |
computer whose purpose is to run domUs, they are right. But Xen and a |
dom0 (without domUs) is not meaingfully less secure than the same |
dom0 (without domUs) is not meaningfully less secure than the same |
things running without Xen. One can boot Xen or boot regular NetBSD |
things running without Xen. One can boot Xen or boot regular NetBSD |
alternately with little problems, simply refraining from starting the |
alternately with little problems, simply refraining from starting the |
Xen daemons when not running Xen. |
Xen daemons when not running Xen. |
|
|
Note that NetBSD as dom0 does not support multiple CPUs. This will |
Note that NetBSD as dom0 does not support multiple CPUs. This will |
limit the performance of the Xen/dom0 workstation approach. |
limit the performance of the Xen/dom0 workstation approach. In theory |
|
the only issue is that the "backend drivers" are not yet MPSAFE: |
|
https://mail-index.netbsd.org/netbsd-users/2014/08/29/msg015195.html |
|
|
Installation of NetBSD |
Installation of NetBSD |
---------------------- |
---------------------- |
|
|
First, |
First, |
[install NetBSD/amd64](../../docs/guide/en/chap-inst.html) |
[install NetBSD/amd64](/guide/inst/) |
just as you would if you were not using Xen. |
just as you would if you were not using Xen. |
However, the partitioning approach is very important. |
However, the partitioning approach is very important. |
|
|
If you want to use RAIDframe for the dom0, there are no special issues |
If you want to use RAIDframe for the dom0, there are no special issues |
for Xen. Typically one provides RAID storage for the dom0, and the |
for Xen. Typically one provides RAID storage for the dom0, and the |
domU systems are unaware of RAID. |
domU systems are unaware of RAID. The 2nd-stage loader bootxx_* skips |
|
over a RAID1 header to find /boot from a file system within a RAID |
|
partition; this is no different when booting Xen. |
|
|
There are 4 styles of providing backing storage for the virtual disks |
There are 4 styles of providing backing storage for the virtual disks |
used by domUs: raw partitions, LVM, file-backed vnd(4), and SAN, |
used by domUs: raw partitions, LVM, file-backed vnd(4), and SAN. |
|
|
With raw partitions, one has a disklabel (or gpt) partition sized for |
With raw partitions, one has a disklabel (or gpt) partition sized for |
each virtual disk to be used by the domU. (If you are able to predict |
each virtual disk to be used by the domU. (If you are able to predict |
how domU usage will evolve, please add an explanation to the HOWTO. |
how domU usage will evolve, please add an explanation to the HOWTO. |
Seriously, needs tend to change over time.) |
Seriously, needs tend to change over time.) |
|
|
One can use lvm(8) to create logical devices to use for domU disks. |
One can use [lvm(8)](/guide/lvm/) to create logical devices to use |
This is almost as efficient sa raw disk partitions and more flexible. |
for domU disks. This is almost as efficient as raw disk partitions |
Hence raw disk partitions should typically not be used. |
and more flexible. Hence raw disk partitions should typically not |
|
be used. |
|
|
One can use files in the dom0 filesystem, typically created by dd'ing |
One can use files in the dom0 file system, typically created by dd'ing |
/dev/zero to create a specific size. This is somewhat less efficient, |
/dev/zero to create a specific size. This is somewhat less efficient, |
but very convenient, as one can cp the files for backup, or move them |
but very convenient, as one can cp the files for backup, or move them |
between dom0 hosts. |
between dom0 hosts. |
Line 199 HOWTO page.)
|
Line 394 HOWTO page.)
|
Installation of Xen |
Installation of Xen |
------------------- |
------------------- |
|
|
Next step is to install the Xen packages via pkgsrc or from binary |
In the dom0, install sysutils/xenkernel42 and sysutils/xentools42 from |
packages. See [the pkgsrc |
pkgsrc (or another matching pair). See [the pkgsrc |
documentation](http://www.NetBSD.org/docs/pkgsrc/) if you are unfamiliar |
documentation](https://www.NetBSD.org/docs/pkgsrc/) for help with |
with pkgsrc and/or handling of binary packages. Xen 3.1, 3.3, 4.1 and |
pkgsrc. Ensure that your packages are recent; the HOWTO does not |
4.2 are available. 3.1 supports PCI pass-through while other versions do |
contemplate old builds. |
not. You'll need either `sysutils/xentools3` and `sysutils/xenkernel3` |
|
for Xen 3.1, `sysutils/xentools33` and `sysutils/xenkernel33` for Xen |
|
3.3, `sysutils/xentools41` and `sysutils/xenkernel41` for Xen 4.1. or |
For Xen 3.1, support for HVM guests is in sysutils/xentool3-hvm. More |
`sysutils/xentools42` and `sysutils/xenkernel42` for Xen 4.2. You'll |
recent versions have HVM support integrated in the main xentools |
also need `sysutils/grub` if you plan do use the grub boot loader. If |
package. It is entirely reasonable to run only PV guests. |
using Xen 3.1, you may also want to install `sysutils/xentools3-hvm` |
|
which contains the utilities to run unmodified guests OSes using the |
Next you need to install the selected Xen kernel itself, which is |
*HVM* support (for later versions this is included in |
installed by pkgsrc as "/usr/pkg/xen*-kernel/xen.gz". Copy it to /. |
`sysutils/xentools`). Note that your CPU needs to support this. Intel |
For debugging, one may copy xen-debug.gz; this is conceptually similar |
CPUs must have the 'VT' instruction, AMD CPUs the 'SVM' instruction. You |
to DIAGNOSTIC and DEBUG in NetBSD. xen-debug.gz is basically only |
can easily find out if your CPU support HVM by using NetBSD's cpuctl |
useful with a serial console. Then, place a NetBSD XEN3_DOM0 kernel |
command: |
in /, copied from releasedir/amd64/binary/kernel/netbsd-XEN3_DOM0.gz |
|
of a NetBSD build. If using i386, use |
# cpuctl identify 0 |
releasedir/i386/binary/kernel/netbsd-XEN3PAE_DOM0.gz. (If using Xen |
cpu0: Intel Core 2 (Merom) (686-class), id 0x6f6 |
3.1 and i386, you may use XEN3_DOM0 with the non-PAE Xen. But you |
cpu0: features 0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR> |
should not use Xen 3.1.) Both xen and the NetBSD kernel may be (and |
cpu0: features 0xbfebfbff<PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX> |
typically are) left compressed. |
cpu0: features 0xbfebfbff<FXSR,SSE,SSE2,SS,HTT,TM,SBF> |
|
cpu0: features2 0x4e33d<SSE3,DTES64,MONITOR,DS-CPL,,TM2,SSSE3,CX16,xTPR,PDCM,DCA> |
In a dom0, kernfs is mandatory for xend to communicate with the |
cpu0: features3 0x20100800<SYSCALL/SYSRET,XD,EM64T> |
kernel, so ensure that /kern is in fstab. (A standard NetBSD install |
cpu0: "Intel(R) Xeon(R) CPU 5130 @ 2.00GHz" |
should already mount /kern.) |
cpu0: I-cache 32KB 64B/line 8-way, D-cache 32KB 64B/line 8-way |
|
cpu0: L2 cache 4MB 64B/line 16-way |
Because you already installed NetBSD, you have a working boot setup |
cpu0: ITLB 128 4KB entries 4-way |
with an MBR bootblock, either bootxx_ffsv1 or bootxx_ffsv2 at the |
cpu0: DTLB 256 4KB entries 4-way, 32 4MB entries 4-way |
beginning of your root file system, have /boot, and likely also |
cpu0: Initial APIC ID 0 |
/boot.cfg. (If not, fix before continuing!) |
cpu0: Cluster/Package ID 0 |
|
cpu0: Core ID 0 |
Add a line to to /boot.cfg to boot Xen. See boot.cfg(5) for an |
cpu0: family 06 model 0f extfamily 00 extmodel 00 |
example. The basic line is |
|
|
Depending on your CPU, the feature you are looking for is called HVM, |
menu=Xen:load /netbsd-XEN3_DOM0.gz console=pc;multiboot /xen.gz dom0_mem=256M |
SVM or VMX. |
|
|
which specifies that the dom0 should have 256M, leaving the rest to be |
Next you need to copy the selected Xen kernel itself. pkgsrc installed |
allocated for domUs. To use a serial console, use |
them under `/usr/pkg/xen*-kernel/`. The file you're looking for is |
|
`xen.gz`. Copy it to your root file system. `xen-debug.gz` is a kernel |
menu=Xen:load /netbsd-XEN3_DOM0.gz;multiboot /xen.gz dom0_mem=256M console=com1 com1=9600,8n1 |
with more consistency checks and more details printed on the serial |
|
console. It is useful for debugging crashing guests if you use a serial |
which will use the first serial port for Xen (which counts starting |
console. It is not useful with a VGA console. |
from 1, unlike NetBSD which counts starting from 0), forcing |
|
speed/parity. Because the NetBSD command line lacks a |
You'll then need a NetBSD/Xen kernel for *domain0* on your root file |
"console=pc" argument, it will use the default "xencons" console device, |
system. The XEN3PAE\_DOM0 kernel or XEN3\_DOM0 provided as part of the |
which directs the console I/O through Xen to the same console device Xen |
i386 or amd64 binaries is suitable for this, but you may want to |
itself uses (in this case, the serial port). |
customize it. Keep your native kernel around, as it can be useful for |
|
recovery. *Note:* the *domain0* kernel must support KERNFS and `/kern` |
In an attempt to add performance, one can also add |
must be mounted because *xend* needs access to `/kern/xen/privcmd`. |
|
|
dom0_max_vcpus=1 dom0_vcpus_pin |
Next you need to get a bootloader to load the `xen.gz` kernel, and the |
|
NetBSD *domain0* kernel as a module. This can be `grub` or NetBSD's boot |
to force only one vcpu to be provided (since NetBSD dom0 can't use |
loader. Below is a detailled example for grub, see the boot.cfg(5) |
more) and to pin that vcpu to a physical CPU. TODO: benchmark this. |
manual page for an example using the latter. |
|
|
Xen has [many boot |
This is also where you'll specify the memory allocated to *domain0*, the |
options](http://xenbits.xenproject.org/docs/4.5-testing/misc/xen-command-line.html), |
console to use, etc ... |
and other than dom0 memory and max_vcpus, they are generally not |
|
necessary. |
Here is a commented `/grub/menu.lst` file: |
|
|
As with non-Xen systems, you should have a line to boot /netbsd (a |
#Grub config file for NetBSD/xen. Copy as /grub/menu.lst and run |
kernel that works without Xen). Consider a line to boot /netbsd.ok (a |
# grub-install /dev/rwd0d (assuming your boot device is wd0). |
fallback version of the non-Xen kernel, updated manually when you are |
# |
sure /netbsd is ok). Consider also a line to boot fallback versions |
# The default entry to load will be the first one |
of Xen and the dom0 kernel, but note that non-Xen NetBSD can be used |
default=0 |
to resolve Xen booting issues. |
|
|
# boot the default entry after 10s if the user didn't hit keyboard |
Probably you want a default=N line to choose Xen in the absence of |
timeout=10 |
intervention. |
|
|
# Configure serial port to use as console. Ignore if you'll use VGA only |
Now, reboot so that you are running a DOM0 kernel under Xen, rather |
serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1 |
than GENERIC without Xen. |
|
|
# Let the user select which console to use (serial or VGA), default |
Using grub (historic) |
# to serial after 10s |
--------------------- |
terminal --timeout=10 serial console |
|
|
Before NetBSD's native bootloader could support Xen, the use of |
# An entry for NetBSD/xen, using /netbsd as the domain0 kernel, and serial |
grub was recommended. If necessary, see the |
# console. Domain0 will have 64MB RAM allocated. |
[old grub information](/ports/xen/howto-grub). |
# Assume NetBSD is installed in the first MBR partition. |
|
title Xen 3 / NetBSD (hda0, serial) |
The [HowTo on Installing into |
root(hd0,0) |
RAID-1](https://mail-index.NetBSD.org/port-xen/2006/03/01/0010.html) |
kernel (hd0,a)/xen.gz dom0_mem=65536 com1=115200,8n1 |
explains how to set up booting a dom0 with Xen using grub with |
module (hd0,a)/netbsd bootdev=wd0a ro console=ttyS0 |
NetBSD's RAIDframe. (This is obsolete with the use of NetBSD's native |
|
boot. Now, just create a system with RAID-1, and alter /boot.cfg as |
# Same as above, but using VGA console |
described above.) |
# We can use console=tty0 (Linux syntax) or console=pc (NetBSD syntax) |
|
title Xen 3 / NetBSD (hda0, vga) |
Configuring Xen |
root(hd0,0) |
--------------- |
kernel (hd0,a)/xen.gz dom0_mem=65536 |
|
module (hd0,a)/netbsd bootdev=wd0a ro console=tty0 |
Xen logs will be in /var/log/xen. |
|
|
# NetBSD/xen using a backup domain0 kernel (in case you installed a |
Now, you have a system that will boot Xen and the dom0 kernel, but not |
# nonworking kernel as /netbsd |
do anything else special. Make sure that you have rebooted into Xen. |
title Xen 3 / NetBSD (hda0, backup, serial) |
There will be no domUs, and none can be started because you still have |
root(hd0,0) |
to configure the dom0 daemons. |
kernel (hd0,a)/xen.gz dom0_mem=65536 com1=115200,8n1 |
|
module (hd0,a)/netbsd.backup bootdev=wd0a ro console=ttyS0 |
The daemons which should be run vary with Xen version and with whether |
title Xen 3 / NetBSD (hda0, backup, VGA) |
one is using xm or xl. The Xen 3.1, 3.3 and 4.1 packages use xm. Xen |
root(hd0,0) |
4.2 and up packages use xl. To use xm with 4.2, edit xendomains to |
kernel (hd0,a)/xen.gz dom0_mem=65536 |
use xm instead. |
module (hd0,a)/netbsd.backup bootdev=wd0a ro console=tty0 |
|
|
For 3.1 and 3.3, you should enable xend and xenbackendd: |
#Load a regular NetBSD/i386 kernel. Can be useful if you end up with a |
|
#nonworking /xen.gz |
xend=YES |
title NetBSD 5.1 |
xenbackendd=YES |
root (hd0,a) |
|
kernel --type=netbsd /netbsd-GENERIC |
For 4.1 and up, you should enable xencommons. Not enabling xencommons |
|
will result in a hang; it is necessary to hit ^C on the console to let |
#Load the NetBSD bootloader, letting it load the NetBSD/i386 kernel. |
the machine finish booting. If you are using xm (default in 4.1, or |
#May be better than the above, as grub can't pass all required infos |
if you changed xendomains in 4.2), you should also enable xend: |
#to the NetBSD/i386 kernel (e.g. console, root device, ...) |
|
title NetBSD chain |
xend=YES # only if using xm, and only installed <= 4.2 |
root (hd0,0) |
xencommons=YES |
chainloader +1 |
|
|
TODO: Recommend for/against xen-watchdog. |
## end of grub config file. |
|
|
After you have configured the daemons and either started them (in the |
|
order given) or rebooted, use xm or xl to inspect Xen's boot messages, |
Install grub with the following command: |
available resources, and running domains. An example with xl follows: |
|
|
# grub --no-floppy |
# xl dmesg |
|
[xen's boot info] |
grub> root (hd0,a) |
# xl info |
Filesystem type is ffs, partition type 0xa9 |
[available memory, etc.] |
|
# xl list |
grub> setup (hd0) |
Name Id Mem(MB) CPU State Time(s) Console |
Checking if "/boot/grub/stage1" exists... no |
Domain-0 0 64 0 r---- 58.1 |
Checking if "/grub/stage1" exists... yes |
|
Checking if "/grub/stage2" exists... yes |
### Issues with xencommons |
Checking if "/grub/ffs_stage1_5" exists... yes |
|
Running "embed /grub/ffs_stage1_5 (hd0)"... 14 sectors are embedded. |
xencommons starts xenstored, which stores data on behalf of dom0 and |
succeeded |
domUs. It does not currently work to stop and start xenstored. |
Running "install /grub/stage1 (hd0) (hd0)1+14 p (hd0,0,a)/grub/stage2 /grub/menu.lst"... |
Certainly all domUs should be shutdown first, following the sort order |
succeeded |
of the rc.d scripts. However, the dom0 sets up state with xenstored, |
Done. |
and is not notified when xenstored exits, leading to not recreating |
|
the state when the new xenstored starts. Until there's a mechanism to |
|
make this work, one should not expect to be able to restart xenstored |
|
(and thus xencommons). There is currently no reason to expect that |
|
this will get fixed any time soon. |
|
|
|
### No-longer needed advice about devices |
|
|
|
The installation of NetBSD should already have created devices for xen |
|
(xencons, xenevt, xsd_kva), but if they are not present, create them: |
|
|
|
cd /dev && sh MAKEDEV xen |
|
|
|
anita (for testing NetBSD) |
|
-------------------------- |
|
|
|
With the setup so far (assuming 4.2/xl), one should be able to run |
|
anita (see pkgsrc/misc/py-anita) to test NetBSD releases, by doing (as |
|
root, because anita must create a domU): |
|
|
|
anita --vmm=xl test file:///usr/obj/i386/ |
|
|
|
Alternatively, one can use --vmm=xm to use xm-based domU creation |
|
instead (and must, on Xen <= 4.1). TODO: confirm that anita xl really works. |
|
|
|
Xen-specific NetBSD issues |
|
-------------------------- |
|
|
|
There are (at least) two additional things different about NetBSD as a |
|
dom0 kernel compared to hardware. |
|
|
|
One is that the module ABI is different because some of the #defines |
|
change, so one must build modules for Xen. As of netbsd-7, the build |
|
system does this automatically. TODO: check this. (Before building |
|
Xen modules was added, it was awkward to use modules to the point |
|
where it was considered that it did not work.) |
|
|
|
The other difference is that XEN3_DOM0 does not have exactly the same |
|
options as GENERIC. While it is debatable whether or not this is a |
|
bug, users should be aware of this and can simply add missing config |
|
items if desired. |
|
|
Updating NetBSD in a dom0 |
Updating NetBSD in a dom0 |
------------------------- |
------------------------- |
Line 349 and adjusts /etc.
|
Line 583 and adjusts /etc.
|
Note that one must update both the non-Xen kernel typically used for |
Note that one must update both the non-Xen kernel typically used for |
rescue purposes and the DOM0 kernel used with Xen. |
rescue purposes and the DOM0 kernel used with Xen. |
|
|
Updating Xen versions |
Converting from grub to /boot |
|
----------------------------- |
|
|
|
These instructions were [TODO: will be] used to convert a system from |
|
grub to /boot. The system was originally installed in February of |
|
2006 with a RAID1 setup and grub to boot Xen 2, and has been updated |
|
over time. Before these commands, it was running NetBSD 6 i386, Xen |
|
4.1 and grub, much like the message linked earlier in the grub |
|
section. |
|
|
|
# Install MBR bootblocks on both disks. |
|
fdisk -i /dev/rwd0d |
|
fdisk -i /dev/rwd1d |
|
# Install NetBSD primary boot loader (/ is FFSv1) into RAID1 components. |
|
installboot -v /dev/rwd0d /usr/mdec/bootxx_ffsv1 |
|
installboot -v /dev/rwd1d /usr/mdec/bootxx_ffsv1 |
|
# Install secondary boot loader |
|
cp -p /usr/mdec/boot / |
|
# Create boot.cfg following earlier guidance: |
|
menu=Xen:load /netbsd-XEN3PAE_DOM0.gz console=pc;multiboot /xen.gz dom0_mem=256M |
|
menu=Xen.ok:load /netbsd-XEN3PAE_DOM0.ok.gz console=pc;multiboot /xen.ok.gz dom0_mem=256M |
|
menu=GENERIC:boot |
|
menu=GENERIC single-user:boot -s |
|
menu=GENERIC.ok:boot netbsd.ok |
|
menu=GENERIC.ok single-user:boot netbsd.ok -s |
|
menu=Drop to boot prompt:prompt |
|
default=1 |
|
timeout=30 |
|
|
|
TODO: actually do this and fix it if necessary. |
|
|
|
Upgrading Xen versions |
--------------------- |
--------------------- |
|
|
TODO: write |
Minor version upgrades are trivial. Just rebuild/replace the |
|
xenkernel version and copy the new xen.gz to / (where /boot.cfg |
|
references it), and reboot. |
|
|
|
Major version upgrades are conceptually not difficult, but can run |
|
into all the issues found when installing Xen. Assuming migration |
|
from 4.1 to 4.2, remove the xenkernel41 and xentools41 packages and |
|
install the xenkernel42 and xentools42 packages. Copy the 4.2 xen.gz |
|
to /. |
|
|
|
Ensure that the contents of /etc/rc.d/xen* are correct. Specifically, |
|
they must match the package you just installed and not be left over |
|
from some previous installation. |
|
|
|
Enable the correct set of daemons; see the configuring section above. |
|
(Upgrading from 3.x to 4.x without doing this will result in a hang.) |
|
|
|
Ensure that the domU config files are valid for the new version. |
|
Specifically, for 4.x remove autorestart=True, and ensure that disks |
|
are specified with numbers as the second argument, as the examples |
|
above show, and not NetBSD device names. |
|
|
|
Hardware known to work |
|
---------------------- |
|
|
|
Arguably, this section is misplaced, and there should be a page of |
|
hardware that runs NetBSD/amd64 well, with the mostly-well-founded |
|
assumption that NetBSD/xen runs fine on any modern hardware that |
|
NetBSD/amd64 runs well on. Until then, we give motherboard/CPU (and |
|
sometimes RAM) pairs/triples to aid those choosing a motherboard. |
|
Note that Xen systems usually do not run X, so a listing here does not |
|
imply that X works at all. |
|
|
|
Supermicro X9SRL-F, Xeon E5-1650 v2, 96 GiB ECC |
|
Supermicro ??, Atom C2758 (8 core), 32 GiB ECC |
|
ASUS M5A78L-M/USB3 AM3+ microATX, AMD Piledriver X8 4000MHz, 16 GiB ECC |
|
|
|
Older hardware: |
|
|
|
Intel D915GEV, Pentium4 CPU 3.40GHz, 4GB 533MHz Synchronous DDR2 |
|
INTEL DG33FB, "Intel(R) Core(TM)2 Duo CPU E6850 @ 3.00GHz" |
|
INTEL DG33FB, "Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz" |
|
|
|
Running Xen under qemu |
|
---------------------- |
|
|
|
The astute reader will note that this section is somewhat twisted. |
|
However, it can be useful to run Xen under qemu either because the |
|
version of NetBSD as a dom0 does not run on the hardware in use, or to |
|
generate automated test cases involving Xen. |
|
|
|
In 2015-01, the following combination was reported to mostly work: |
|
|
|
host OS: NetBSD/amd64 6.1.4 |
|
qemu: 2.2.0 from pkgsrc |
|
Xen kernel: xenkernel42-4.2.5nb1 from pkgsrc |
|
dom0 kernel: NetBSD/amd64 6.1.5 |
|
Xen tools: xentools42-4.2.5 from pkgsrc |
|
|
|
See [PR 47720](https://gnats.netbsd.org/47720) for a problem with dom0 |
|
shutdown. |
|
|
|
Unprivileged domains (domU) |
|
=========================== |
|
|
|
This section describes general concepts about domUs. It does not |
|
address specific domU operating systems or how to install them. The |
|
config files for domUs are typically in /usr/pkg/etc/xen, and are |
|
typically named so that the file name, domU name and the domU's host |
|
name match. |
|
|
|
The domU is provided with CPU and memory by Xen, configured by the |
|
dom0. The domU is provided with disk and network by the dom0, |
|
mediated by Xen, and configured in the dom0. |
|
|
|
Entropy in domUs can be an issue; physical disks and network are on |
|
the dom0. NetBSD's /dev/random system works, but is often challenged. |
|
|
|
Config files |
|
------------ |
|
|
|
There is no good order to present config files and the concepts |
|
surrounding what is being configured. We first show an example config |
|
file, and then in the various sections give details. |
|
|
|
See (at least in xentools41) /usr/pkg/share/examples/xen/xmexample*, |
|
for a large number of well-commented examples, mostly for running |
|
GNU/Linux. |
|
|
|
The following is an example minimal domain configuration file |
|
"/usr/pkg/etc/xen/foo". It is (with only a name change) an actual |
|
known working config file on Xen 4.1 (NetBSD 5 amd64 dom0 and NetBSD 5 |
|
i386 domU). The domU serves as a network file server. |
|
|
|
# -*- mode: python; -*- |
|
|
|
kernel = "/netbsd-XEN3PAE_DOMU-i386-foo.gz" |
|
memory = 1024 |
|
vif = [ 'mac=aa:00:00:d1:00:09,bridge=bridge0' ] |
|
disk = [ 'file:/n0/xen/foo-wd0,0x0,w', |
|
'file:/n0/xen/foo-wd1,0x1,w' ] |
|
|
|
The domain will have the same name as the file. The kernel has the |
|
host/domU name in it, so that on the dom0 one can update the various |
|
domUs independently. The vif line causes an interface to be provided, |
|
with a specific mac address (do not reuse MAC addresses!), in bridge |
|
mode. Two disks are provided, and they are both writable; the bits |
|
are stored in files and Xen attaches them to a vnd(4) device in the |
|
dom0 on domain creation. The system treats xbd0 as the boot device |
|
without needing explicit configuration. |
|
|
|
By default xm looks for domain config files in /usr/pkg/etc/xen. Note |
|
that "xm create" takes the name of a config file, while other commands |
|
take the name of a domain. To create the domain, connect to the |
|
console, create the domain while attaching the console, shutdown the |
|
domain, and see if it has finished stopping, do (or xl with Xen >= |
|
4.2): |
|
|
|
xm create foo |
|
xm console foo |
|
xm create -c foo |
|
xm shutdown foo |
|
xm list |
|
|
|
Typing ^] will exit the console session. Shutting down a domain is |
|
equivalent to pushing the power button; a NetBSD domU will receive a |
|
power-press event and do a clean shutdown. Shutting down the dom0 |
|
will trigger controlled shutdowns of all configured domUs. |
|
|
Creating unprivileged domains (domU) |
domU kernels |
==================================== |
------------ |
|
|
|
On a physical computer, the BIOS reads sector 0, and a chain of boot |
|
loaders finds and loads a kernel. Normally this comes from the root |
|
file system. With Xen domUs, the process is totally different. The |
|
normal path is for the domU kernel to be a file in the dom0's |
|
file system. At the request of the dom0, Xen loads that kernel into a |
|
new domU instance and starts execution. While domU kernels can be |
|
anyplace, reasonable places to store domU kernels on the dom0 are in / |
|
(so they are near the dom0 kernel), in /usr/pkg/etc/xen (near the |
|
config files), or in /u0/xen (where the vdisks are). |
|
|
|
Note that loading the domU kernel from the dom0 implies that boot |
|
blocks, /boot, /boot.cfg, and so on are all ignored in the domU. |
|
See the VPS section near the end for discussion of alternate ways to |
|
obtain domU kernels. |
|
|
|
CPU and memory |
|
-------------- |
|
|
|
A domain is provided with some number of vcpus, less than the number |
|
of CPUs seen by the hypervisor. (For a dom0, this is controlled by |
|
the boot argument "dom0_max_vcpus=1".) For a domU, it is controlled |
|
from the config file by the "vcpus = N" directive. |
|
|
|
A domain is provided with memory; this is controlled in the config |
|
file by "memory = N" (in megabytes). In the straightforward case, the |
|
sum of the the memory allocated to the dom0 and all domUs must be less |
|
than the available memory. |
|
|
|
Xen also provides a "balloon" driver, which can be used to let domains |
|
use more memory temporarily. TODO: Explain better, and explain how |
|
well it works with NetBSD. |
|
|
|
Virtual disks |
|
------------- |
|
|
|
With the file/vnd style, typically one creates a directory, |
|
e.g. /u0/xen, on a disk large enough to hold virtual disks for all |
|
domUs. Then, for each domU disk, one writes zeros to a file that then |
|
serves to hold the virtual disk's bits; a suggested name is foo-xbd0 |
|
for the first virtual disk for the domU called foo. Writing zeros to |
|
the file serves two purposes. One is that preallocating the contents |
|
improves performance. The other is that vnd on sparse files has |
|
failed to work. TODO: give working/notworking NetBSD versions for |
|
sparse vnd and gnats reference. Note that the use of file/vnd for Xen |
|
is not really different than creating a file-backed virtual disk for |
|
some other purpose, except that xentools handles the vnconfig |
|
commands. To create an empty 4G virtual disk, simply do |
|
|
|
dd if=/dev/zero of=foo-xbd0 bs=1m count=4096 |
|
|
|
Do not use qemu-img-xen, because this will create sparse file. There |
|
have been recent (2015) reports of sparse vnd(4) devices causing |
|
lockups, but there is apparently no PR. |
|
|
|
With the lvm style, one creates logical devices. They are then used |
|
similarly to vnds. TODO: Add an example with lvm. |
|
|
|
In domU config files, the disks are defined as a sequence of 3-tuples. |
|
The first element is "method:/path/to/disk". Common methods are |
|
"file:" for file-backed vnd. and "phy:" for something that is already |
|
a (TODO: character or block) device. |
|
|
|
The second element is an artifact of how virtual disks are passed to |
|
Linux, and a source of confusion with NetBSD Xen usage. Linux domUs |
|
are given a device name to associate with the disk, and values like |
|
"hda1" or "sda1" are common. In a NetBSD domU, the first disk appears |
|
as xbd0, the second as xbd1, and so on. However, xm/xl demand a |
|
second argument. The name given is converted to a major/minor by |
|
calling stat(2) on the name in /dev and this is passed to the domU. |
|
In the general case, the dom0 and domU can be different operating |
|
systems, and it is an unwarranted assumption that they have consistent |
|
numbering in /dev, or even that the dom0 OS has a /dev. With NetBSD |
|
as both dom0 and domU, using values of 0x0 for the first disk and 0x1 |
|
for the second works fine and avoids this issue. For a GNU/Linux |
|
guest, one can create /dev/hda1 in /dev, or to pass 0x301 for |
|
/dev/hda1. |
|
|
|
The third element is "w" for writable disks, and "r" for read-only |
|
disks. |
|
|
|
Note that NetBSD by default creates only vnd[0123]. If you need more |
|
than 4 total virtual disks at a time, run e.g. "./MAKEDEV vnd4" in the |
|
dom0. |
|
|
|
Note that NetBSD by default creates only xbd[0123]. If you need more |
|
virtual disks in a domU, run e.g. "./MAKEDEV xbd4" in the domU. |
|
|
|
Virtual Networking |
|
------------------ |
|
|
|
Xen provides virtual Ethernets, each of which connects the dom0 and a |
|
domU. For each virtual network, there is an interface "xvifN.M" in |
|
the dom0, and in domU index N, a matching interface xennetM (NetBSD |
|
name). The interfaces behave as if there is an Ethernet with two |
|
adapters connected. From this primitive, one can construct various |
|
configurations. We focus on two common and useful cases for which |
|
there are existing scripts: bridging and NAT. |
|
|
|
With bridging (in the example above), the domU perceives itself to be |
|
on the same network as the dom0. For server virtualization, this is |
|
usually best. Bridging is accomplished by creating a bridge(4) device |
|
and adding the dom0's physical interface and the various xvifN.0 |
|
interfaces to the bridge. One specifies "bridge=bridge0" in the domU |
|
config file. The bridge must be set up already in the dom0; an |
|
example /etc/ifconfig.bridge0 is: |
|
|
|
create |
|
up |
|
!brconfig bridge0 add wm0 |
|
|
|
With NAT, the domU perceives itself to be behind a NAT running on the |
|
dom0. This is often appropriate when running Xen on a workstation. |
|
TODO: NAT appears to be configured by "vif = [ '' ]". |
|
|
|
The MAC address specified is the one used for the interface in the new |
|
domain. The interface in dom0 will use this address XOR'd with |
|
00:00:00:01:00:00. Random MAC addresses are assigned if not given. |
|
|
|
Sizing domains |
|
-------------- |
|
|
|
Modern x86 hardware has vast amounts of resources. However, many |
|
virtual servers can function just fine on far less. A system with |
|
256M of RAM and a 4G disk can be a reasonable choice. Note that it is |
|
far easier to adjust virtual resources than physical ones. For |
|
memory, it's just a config file edit and a reboot. For disk, one can |
|
create a new file and vnconfig it (or lvm), and then dump/restore, |
|
just like updating physical disks, but without having to be there and |
|
without those pesky connectors. |
|
|
|
Starting domains automatically |
|
------------------------------ |
|
|
|
To start domains foo at bar at boot and shut them down cleanly on dom0 |
|
shutdown, in rc.conf add: |
|
|
|
xendomains="foo bar" |
|
|
|
Note that earlier versions of the xentools41 xendomains rc.d script |
|
used xl, when one should use xm with 4.1. |
|
|
|
Creating specific unprivileged domains (domU) |
|
============================================= |
|
|
Creating domUs is almost entirely independent of operating system. We |
Creating domUs is almost entirely independent of operating system. We |
first explain NetBSD, and then differences for Linux and Solaris. |
have already presented the basics of config files. Note that you must |
|
have already completed the dom0 setup so that "xl list" (or "xm list") |
|
works. |
|
|
Creating an unprivileged NetBSD domain (domU) |
Creating an unprivileged NetBSD domain (domU) |
--------------------------------------------- |
--------------------------------------------- |
|
|
Once you have *domain0* running, you need to start the xen tool daemon |
See the earlier config file, and adjust memory. Decide on how much |
(`/usr/pkg/share/examples/rc.d/xend start`) and the xen backend daemon |
storage you will provide, and prepare it (file or lvm). |
(`/usr/pkg/share/examples/rc.d/xenbackendd start` for Xen3\*, |
|
`/usr/pkg/share/examples/rc.d/xencommons start` for Xen4.\*). Make sure |
|
that `/dev/xencons` and `/dev/xenevt` exist before starting `xend`. You |
|
can create them with this command: |
|
|
|
# cd /dev && sh MAKEDEV xen |
|
|
|
xend will write logs to `/var/log/xend.log` and |
|
`/var/log/xend-debug.log`. You can then control xen with the xm tool. |
|
'xm list' will show something like: |
|
|
|
# xm list |
|
Name Id Mem(MB) CPU State Time(s) Console |
|
Domain-0 0 64 0 r---- 58.1 |
|
|
|
'xm create' allows you to create a new domain. It uses a config file in |
|
PKG\_SYSCONFDIR for its parameters. By default, this file will be in |
|
`/usr/pkg/etc/xen/`. On creation, a kernel has to be specified, which |
|
will be executed in the new domain (this kernel is in the *domain0* file |
|
system, not on the new domain virtual disk; but please note, you should |
|
install the same kernel into *domainU* as `/netbsd` in order to make |
|
your system tools, like MAN.SAVECORE.8, work). A suitable kernel is |
|
provided as part of the i386 and amd64 binary sets: XEN3\_DOMU. |
|
|
|
Here is an /usr/pkg/etc/xen/nbsd example config file: |
|
|
|
# -*- mode: python; -*- |
|
#============================================================================ |
|
# Python defaults setup for 'xm create'. |
|
# Edit this file to reflect the configuration of your system. |
|
#============================================================================ |
|
|
|
#---------------------------------------------------------------------------- |
|
# Kernel image file. This kernel will be loaded in the new domain. |
|
kernel = "/home/bouyer/netbsd-XEN3_DOMU" |
|
#kernel = "/home/bouyer/netbsd-INSTALL_XEN3_DOMU" |
|
|
|
# Memory allocation (in megabytes) for the new domain. |
|
memory = 128 |
|
|
|
# A handy name for your new domain. This will appear in 'xm list', |
|
# and you can use this as parameters for xm in place of the domain |
|
# number. All domains must have different names. |
|
# |
|
name = "nbsd" |
|
|
|
# The number of virtual CPUs this domain has. |
|
# |
|
vcpus = 1 |
|
|
|
#---------------------------------------------------------------------------- |
|
# Define network interfaces for the new domain. |
|
|
|
# Number of network interfaces (must be at least 1). Default is 1. |
|
nics = 1 |
|
|
|
# Define MAC and/or bridge for the network interfaces. |
|
# |
|
# The MAC address specified in ``mac'' is the one used for the interface |
|
# in the new domain. The interface in domain0 will use this address XOR'd |
|
# with 00:00:00:01:00:00 (i.e. aa:00:00:51:02:f0 in our example). Random |
|
# MACs are assigned if not given. |
|
# |
|
# ``bridge'' is a required parameter, which will be passed to the |
|
# vif-script called by xend(8) when a new domain is created to configure |
|
# the new xvif interface in domain0. |
|
# |
|
# In this example, the xvif is added to bridge0, which should have been |
|
# set up prior to the new domain being created -- either in the |
|
# ``network'' script or using a /etc/ifconfig.bridge0 file. |
|
# |
|
vif = [ 'mac=aa:00:00:50:02:f0, bridge=bridge0' ] |
|
|
|
#---------------------------------------------------------------------------- |
|
# Define the disk devices you want the domain to have access to, and |
|
# what you want them accessible as. |
|
# |
|
# Each disk entry is of the form: |
|
# |
|
# phy:DEV,VDEV,MODE |
|
# |
|
# where DEV is the device, VDEV is the device name the domain will see, |
|
# and MODE is r for read-only, w for read-write. You can also create |
|
# file-backed domains using disk entries of the form: |
|
# |
|
# file:PATH,VDEV,MODE |
|
# |
|
# where PATH is the path to the file used as the virtual disk, and VDEV |
|
# and MODE have the same meaning as for ``phy'' devices. |
|
# |
|
# VDEV doesn't really matter for a NetBSD guest OS (it's just used as an index), |
|
# but it does for Linux. |
|
# Worse, the device has to exist in /dev/ of domain0, because xm will |
|
# try to stat() it. This means that in order to load a Linux guest OS |
|
# from a NetBSD domain0, you'll have to create /dev/hda1, /dev/hda2, ... |
|
# on domain0, with the major/minor from Linux :( |
|
# Alternatively it's possible to specify the device number in hex, |
|
# e.g. 0x301 for /dev/hda1, 0x302 for /dev/hda2, etc ... |
|
|
|
disk = [ 'phy:/dev/wd0e,0x1,w' ] |
While the kernel will be obtained from the dom0 file system, the same |
#disk = [ 'file:/var/xen/nbsd-disk,0x01,w' ] |
file should be present in the domU as /netbsd so that tools like |
#disk = [ 'file:/var/xen/nbsd-disk,0x301,w' ] |
savecore(8) can work. (This is helpful but not necessary.) |
|
|
|
The kernel must be specifically for Xen and for use as a domU. The |
|
i386 and amd64 provide the following kernels: |
|
|
|
i386 XEN3_DOMU |
|
i386 XEN3PAE_DOMU |
|
amd64 XEN3_DOMU |
|
|
|
Unless using Xen 3.1 (and you shouldn't) with i386-mode Xen, you must |
|
use the PAE version of the i386 kernel. |
|
|
#---------------------------------------------------------------------------- |
This will boot NetBSD, but this is not that useful if the disk is |
# Set the kernel command line for the new domain. |
empty. One approach is to unpack sets onto the disk outside of xen |
|
(by mounting it, just as you would prepare a physical disk for a |
|
system you can't run the installer on). |
|
|
# Set root device. This one does matter for NetBSD |
A second approach is to run an INSTALL kernel, which has a miniroot |
root = "xbd0" |
and can load sets from the network. To do this, copy the INSTALL |
# extra parameters passed to the kernel |
kernel to / and change the kernel line in the config file to: |
# this is where you can set boot flags like -s, -a, etc ... |
|
#extra = "" |
|
|
|
#---------------------------------------------------------------------------- |
|
# Set according to whether you want the domain restarted when it exits. |
|
# The default is False. |
|
#autorestart = True |
|
|
|
# end of nbsd config file ==================================================== |
|
|
|
When a new domain is created, xen calls the |
|
`/usr/pkg/etc/xen/vif-bridge` script for each virtual network interface |
|
created in *domain0*. This can be used to automatically configure the |
|
xvif?.? interfaces in *domain0*. In our example, these will be bridged |
|
with the bridge0 device in *domain0*, but the bridge has to exist first. |
|
To do this, create the file `/etc/ifconfig.bridge0` and make it look |
|
like this: |
|
|
|
create |
|
!brconfig $int add ex0 up |
|
|
|
(replace `ex0` with the name of your physical interface). Then bridge0 |
|
will be created on boot. See the MAN.BRIDGE.4 man page for details. |
|
|
|
So, here is a suitable `/usr/pkg/etc/xen/vif-bridge` for xvif?.? (a |
|
working vif-bridge is also provided with xentools20) configuring: |
|
|
|
#!/bin/sh |
|
#============================================================================ |
|
# $NetBSD: howto.mdwn,v 1.18 2014/12/24 00:11:15 gdt Exp $ |
|
# |
|
# /usr/pkg/etc/xen/vif-bridge |
|
# |
|
# Script for configuring a vif in bridged mode with a dom0 interface. |
|
# The xend(8) daemon calls a vif script when bringing a vif up or down. |
|
# The script name to use is defined in /usr/pkg/etc/xen/xend-config.sxp |
|
# in the ``vif-script'' field. |
|
# |
|
# Usage: vif-bridge up|down [var=value ...] |
|
# |
|
# Actions: |
|
# up Adds the vif interface to the bridge. |
|
# down Removes the vif interface from the bridge. |
|
# |
|
# Variables: |
|
# domain name of the domain the interface is on (required). |
|
# vifq vif interface name (required). |
|
# mac vif MAC address (required). |
|
# bridge bridge to add the vif to (required). |
|
# |
|
# Example invocation: |
|
# |
|
# vif-bridge up domain=VM1 vif=xvif1.0 mac="ee:14:01:d0:ec:af" bridge=bridge0 |
|
# |
|
#============================================================================ |
|
|
|
# Exit if anything goes wrong |
|
set -e |
|
|
|
echo "vif-bridge $*" |
|
|
|
# Operation name. |
|
OP=$1; shift |
|
|
|
# Pull variables in args into environment |
|
for arg ; do export "${arg}" ; done |
|
|
|
# Required parameters. Fail if not set. |
|
domain=${domain:?} |
|
vif=${vif:?} |
|
mac=${mac:?} |
|
bridge=${bridge:?} |
|
|
|
# Optional parameters. Set defaults. |
|
ip=${ip:-''} # default to null (do nothing) |
|
|
|
# Are we going up or down? |
|
case $OP in |
|
up) brcmd='add' ;; |
|
down) brcmd='delete' ;; |
|
*) |
|
echo 'Invalid command: ' $OP |
|
echo 'Valid commands are: up, down' |
|
exit 1 |
|
;; |
|
esac |
|
|
|
# Don't do anything if the bridge is "null". |
|
if [ "${bridge}" = "null" ] ; then |
|
exit |
|
fi |
|
|
|
# Don't do anything if the bridge doesn't exist. |
|
if ! ifconfig -l | grep "${bridge}" >/dev/null; then |
|
exit |
|
fi |
|
|
|
# Add/remove vif to/from bridge. |
|
ifconfig x${vif} $OP |
|
brconfig ${bridge} ${brcmd} x${vif} |
|
|
|
Now, running |
|
|
|
xm create -c /usr/pkg/etc/xen/nbsd |
|
|
|
should create a domain and load a NetBSD kernel in it. (Note: `-c` |
|
causes xm to connect to the domain's console once created.) The kernel |
|
will try to find its root file system on xbd0 (i.e., wd0e) which hasn't |
|
been created yet. wd0e will be seen as a disk device in the new domain, |
|
so it will be 'sub-partitioned'. We could attach a ccd to wd0e in |
|
*domain0* and partition it, newfs and extract the NetBSD/i386 or amd64 |
|
tarballs there, but there's an easier way: load the |
|
`netbsd-INSTALL_XEN3_DOMU` kernel provided in the NetBSD binary sets. |
|
Like other install kernels, it contains a ramdisk with sysinst, so you |
|
can install NetBSD using sysinst on your new domain. |
|
|
|
If you want to install NetBSD/Xen with a CDROM image, the following line |
kernel = "/home/bouyer/netbsd-INSTALL_XEN3_DOMU" |
should be used in the `/usr/pkg/etc/xen/nbsd` file: |
|
|
Then, start the domain as "xl create -c configname". |
|
|
|
Alternatively, if you want to install NetBSD/Xen with a CDROM image, the following |
|
line should be used in the config file. |
|
|
disk = [ 'phy:/dev/wd0e,0x1,w', 'phy:/dev/cd0a,0x2,r' ] |
disk = [ 'phy:/dev/wd0e,0x1,w', 'phy:/dev/cd0a,0x2,r' ] |
|
|
After booting the domain, the option to install via CDROM may be |
After booting the domain, the option to install via CDROM may be |
selected. The CDROM device should be changed to `xbd1d`. |
selected. The CDROM device should be changed to `xbd1d`. |
|
|
Once done installing, `halt -p` the new domain (don't reboot or halt, it |
Once done installing, "halt -p" the new domain (don't reboot or halt, |
would reload the INSTALL\_XEN3\_DOMU kernel even if you changed the |
it would reload the INSTALL_XEN3_DOMU kernel even if you changed the |
config file), switch the config file back to the XEN3\_DOMU kernel, and |
config file), switch the config file back to the XEN3_DOMU kernel, |
start the new domain again. Now it should be able to use `root on xbd0a` |
and start the new domain again. Now it should be able to use "root on |
and you should have a second, functional NetBSD system on your xen |
xbd0a" and you should have a, functional NetBSD domU. |
installation. |
|
|
|
|
TODO: check if this is still accurate. |
When the new domain is booting you'll see some warnings about *wscons* |
When the new domain is booting you'll see some warnings about *wscons* |
and the pseudo-terminals. These can be fixed by editing the files |
and the pseudo-terminals. These can be fixed by editing the files |
`/etc/ttys` and `/etc/wscons.conf`. You must disable all terminals in |
`/etc/ttys` and `/etc/wscons.conf`. You must disable all terminals in |
Line 620 Finally, all screens must be commented o
|
Line 961 Finally, all screens must be commented o
|
|
|
It is also desirable to add |
It is also desirable to add |
|
|
powerd=YES |
powerd=YES |
|
|
in rc.conf. This way, the domain will be properly shut down if |
in rc.conf. This way, the domain will be properly shut down if |
`xm shutdown -R` or `xm shutdown -H` is used on the domain0. |
`xm shutdown -R` or `xm shutdown -H` is used on the dom0. |
|
|
Your domain should be now ready to work, enjoy. |
It is not strictly necessary to have a kernel (as /netbsd) in the domU |
|
file system. However, various programs (e.g. netstat) will use that |
|
kernel to look up symbols to read from kernel virtual memory. If |
|
/netbsd is not the running kernel, those lookups will fail. (This is |
|
not really a Xen-specific issue, but because the domU kernel is |
|
obtained from the dom0, it is far more likely to be out of sync or |
|
missing with Xen.) |
|
|
Creating an unprivileged Linux domain (domU) |
Creating an unprivileged Linux domain (domU) |
-------------------------------------------- |
-------------------------------------------- |
Line 639 the example below)
|
Line 986 the example below)
|
disk = [ 'phy:/dev/wd0e,0x1,w' ] |
disk = [ 'phy:/dev/wd0e,0x1,w' ] |
|
|
does matter to Linux. It wants a Linux device number here (e.g. 0x300 |
does matter to Linux. It wants a Linux device number here (e.g. 0x300 |
for hda). Linux builds device numbers as: (major \<\< 8 + minor). So, |
for hda). Linux builds device numbers as: (major \<\< 8 + minor). |
hda1 which has major 3 and minor 1 on a Linux system will have device |
So, hda1 which has major 3 and minor 1 on a Linux system will have |
number 0x301. Alternatively, devices names can be used (hda, hdb, ...) |
device number 0x301. Alternatively, devices names can be used (hda, |
as xentools has a table to map these names to devices numbers. To export |
hdb, ...) as xentools has a table to map these names to devices |
a partition to a Linux guest we can use: |
numbers. To export a partition to a Linux guest we can use: |
|
|
disk = [ 'phy:/dev/wd0e,0x300,w' ] |
disk = [ 'phy:/dev/wd0e,0x300,w' ] |
root = "/dev/hda1 ro" |
root = "/dev/hda1 ro" |
|
|
and it will appear as /dev/hda on the Linux system, and be used as root |
and it will appear as /dev/hda on the Linux system, and be used as root |
partition. |
partition. |
|
|
To install the Linux system on the partition to be exported to the guest |
To install the Linux system on the partition to be exported to the |
domain, the following method can be used: install sysutils/e2fsprogs |
guest domain, the following method can be used: install |
from pkgsrc. Use mke2fs to format the partition that will be the root |
sysutils/e2fsprogs from pkgsrc. Use mke2fs to format the partition |
partition of your Linux domain, and mount it. Then copy the files from a |
that will be the root partition of your Linux domain, and mount it. |
working Linux system, make adjustments in `/etc` (fstab, network |
Then copy the files from a working Linux system, make adjustments in |
config). It should also be possible to extract binary packages such as |
`/etc` (fstab, network config). It should also be possible to extract |
.rpm or .deb directly to the mounted partition using the appropriate |
binary packages such as .rpm or .deb directly to the mounted partition |
tool, possibly running under NetBSD's Linux emulation. Once the |
using the appropriate tool, possibly running under NetBSD's Linux |
filesystem has been populated, umount it. If desirable, the filesystem |
emulation. Once the file system has been populated, umount it. If |
can be converted to ext3 using tune2fs -j. It should now be possible to |
desirable, the file system can be converted to ext3 using tune2fs -j. |
boot the Linux guest domain, using one of the vmlinuz-\*-xenU kernels |
It should now be possible to boot the Linux guest domain, using one of |
available in the Xen binary distribution. |
the vmlinuz-\*-xenU kernels available in the Xen binary distribution. |
|
|
To get the linux console right, you need to add: |
To get the Linux console right, you need to add: |
|
|
extra = "xencons=tty1" |
extra = "xencons=tty1" |
|
|
to your configuration since not all linux distributions auto-attach a |
to your configuration since not all Linux distributions auto-attach a |
tty to the xen console. |
tty to the xen console. |
|
|
Creating an unprivileged Solaris domain (domU) |
Creating an unprivileged Solaris domain (domU) |
---------------------------------------------- |
---------------------------------------------- |
|
|
Download an Opensolaris [release](http://opensolaris.org/os/downloads/) |
See possibly outdated |
or [development snapshot](http://genunix.org/) DVD image. Attach the DVD |
[Solaris domU instructions](/ports/xen/howto-solaris/). |
image to a MAN.VND.4 device. Copy the kernel and ramdisk filesystem |
|
image to your dom0 filesystem. |
|
|
PCI passthrough: Using PCI devices in guest domains |
dom0# mkdir /root/solaris |
--------------------------------------------------- |
dom0# vnconfig vnd0 osol-1002-124-x86.iso |
|
dom0# mount /dev/vnd0a /mnt |
The dom0 can give other domains access to selected PCI |
|
devices. This can allow, for example, a non-privileged domain to have |
## for a 64-bit guest |
access to a physical network interface or disk controller. However, |
dom0# cp /mnt/boot/amd64/x86.microroot /root/solaris |
keep in mind that giving a domain access to a PCI device most likely |
dom0# cp /mnt/platform/i86xpv/kernel/amd64/unix /root/solaris |
will give the domain read/write access to the whole physical memory, |
|
as PCs don't have an IOMMU to restrict memory access to DMA-capable |
## for a 32-bit guest |
device. Also, it's not possible to export ISA devices to non-dom0 |
dom0# cp /mnt/boot/x86.microroot /root/solaris |
domains, which means that the primary VGA adapter can't be exported. |
dom0# cp /mnt/platform/i86xpv/kernel/unix /root/solaris |
A guest domain trying to access the VGA registers will panic. |
|
|
dom0# umount /mnt |
If the dom0 is NetBSD, it has to be running Xen 3.1, as support has |
|
not been ported to later versions at this time. |
|
|
Keep the MAN.VND.4 configured. For some reason the boot process stalls |
For a PCI device to be exported to a domU, is has to be attached to |
unless the DVD image is attached to the guest as a "phy" device. Create |
the "pciback" driver in dom0. Devices passed to the dom0 via the |
an initial configuration file with the following contents. Substitute |
pciback.hide boot parameter will attach to "pciback" instead of the |
*/dev/wd0k* with an empty partition at least 8 GB large. |
usual driver. The list of devices is specified as "(bus:dev.func)", |
|
|
memory = 640 |
|
name = 'solaris' |
|
disk = [ 'phy:/dev/wd0k,0,w' ] |
|
disk += [ 'phy:/dev/vnd0d,6:cdrom,r' ] |
|
vif = [ 'bridge=bridge0' ] |
|
kernel = '/root/solaris/unix' |
|
ramdisk = '/root/solaris/x86.microroot' |
|
# for a 64-bit guest |
|
extra = '/platform/i86xpv/kernel/amd64/unix - nowin -B install_media=cdrom' |
|
# for a 32-bit guest |
|
#extra = '/platform/i86xpv/kernel/unix - nowin -B install_media=cdrom' |
|
|
|
|
|
Start the guest. |
|
|
|
dom0# xm create -c solaris.cfg |
|
Started domain solaris |
|
v3.3.2 chgset 'unavailable' |
|
SunOS Release 5.11 Version snv_124 64-bit |
|
Copyright 1983-2009 Sun Microsystems, Inc. All rights reserved. |
|
Use is subject to license terms. |
|
Hostname: opensolaris |
|
Remounting root read/write |
|
Probing for device nodes ... |
|
WARNING: emlxs: ddi_modopen drv/fct failed: err 2 |
|
Preparing live image for use |
|
Done mounting Live image |
|
|
|
|
|
Make sure the network is configured. Note that it can take a minute for |
|
the xnf0 interface to appear. |
|
|
|
opensolaris console login: jack |
|
Password: jack |
|
Sun Microsystems Inc. SunOS 5.11 snv_124 November 2008 |
|
jack@opensolaris:~$ pfexec sh |
|
sh-3.2# ifconfig -a |
|
sh-3.2# exit |
|
|
|
|
|
Set a password for VNC and start the VNC server which provides the X11 |
|
display where the installation program runs. |
|
|
|
jack@opensolaris:~$ vncpasswd |
|
Password: solaris |
|
Verify: solaris |
|
jack@opensolaris:~$ cp .Xclients .vnc/xstartup |
|
jack@opensolaris:~$ vncserver :1 |
|
|
|
|
|
From a remote machine connect to the VNC server. Use `ifconfig xnf0` on |
|
the guest to find the correct IP address to use. |
|
|
|
remote$ vncviewer 172.18.2.99:1 |
|
|
|
|
|
It is also possible to launch the installation on a remote X11 display. |
|
|
|
jack@opensolaris:~$ export DISPLAY=172.18.1.1:0 |
|
jack@opensolaris:~$ pfexec gui-install |
|
|
|
|
|
After the GUI installation is complete you will be asked to reboot. |
|
Before that you need to determine the ZFS ID for the new boot filesystem |
|
and update the configuration file accordingly. Return to the guest |
|
console. |
|
|
|
jack@opensolaris:~$ pfexec zdb -vvv rpool | grep bootfs |
|
bootfs = 43 |
|
^C |
|
jack@opensolaris:~$ |
|
|
|
|
|
The final configuration file should look like this. Note in particular |
|
the last line. |
|
|
|
memory = 640 |
|
name = 'solaris' |
|
disk = [ 'phy:/dev/wd0k,0,w' ] |
|
vif = [ 'bridge=bridge0' ] |
|
kernel = '/root/solaris/unix' |
|
ramdisk = '/root/solaris/x86.microroot' |
|
extra = '/platform/i86xpv/kernel/amd64/unix -B zfs-bootfs=rpool/43,bootpath="/xpvd/xdf@0:a"' |
|
|
|
|
|
Restart the guest to verify it works correctly. |
|
|
|
dom0# xm destroy solaris |
|
dom0# xm create -c solaris.cfg |
|
Using config file "./solaris.cfg". |
|
v3.3.2 chgset 'unavailable' |
|
Started domain solaris |
|
SunOS Release 5.11 Version snv_124 64-bit |
|
Copyright 1983-2009 Sun Microsystems, Inc. All rights reserved. |
|
Use is subject to license terms. |
|
WARNING: emlxs: ddi_modopen drv/fct failed: err 2 |
|
Hostname: osol |
|
Configuring devices. |
|
Loading smf(5) service descriptions: 160/160 |
|
svccfg import warnings. See /var/svc/log/system-manifest-import:default.log . |
|
Reading ZFS config: done. |
|
Mounting ZFS filesystems: (6/6) |
|
Creating new rsa public/private host key pair |
|
Creating new dsa public/private host key pair |
|
|
|
osol console login: |
|
|
|
|
|
Using PCI devices in guest domains |
|
---------------------------------- |
|
|
|
The domain0 can give other domains access to selected PCI devices. This |
|
can allow, for example, a non-privileged domain to have access to a |
|
physical network interface or disk controller. However, keep in mind |
|
that giving a domain access to a PCI device most likely will give the |
|
domain read/write access to the whole physical memory, as PCs don't have |
|
an IOMMU to restrict memory access to DMA-capable device. Also, it's not |
|
possible to export ISA devices to non-domain0 domains (which means that |
|
the primary VGA adapter can't be exported. A guest domain trying to |
|
access the VGA registers will panic). |
|
|
|
This functionality is only available in NetBSD-5.1 (and later) domain0 |
|
and domU. If the domain0 is NetBSD, it has to be running Xen 3.1, as |
|
support has not been ported to later versions at this time. |
|
|
|
For a PCI device to be exported to a domU, is has to be attached to the |
|
`pciback` driver in domain0. Devices passed to the domain0 via the |
|
pciback.hide boot parameter will attach to `pciback` instead of the |
|
usual driver. The list of devices is specified as `(bus:dev.func)`, |
|
where bus and dev are 2-digit hexadecimal numbers, and func a |
where bus and dev are 2-digit hexadecimal numbers, and func a |
single-digit number: |
single-digit number: |
|
|
pciback.hide=(00:0a.0)(00:06.0) |
pciback.hide=(00:0a.0)(00:06.0) |
|
|
pciback devices should show up in the domain0's boot messages, and the |
pciback devices should show up in the dom0's boot messages, and the |
devices should be listed in the `/kern/xen/pci` directory. |
devices should be listed in the `/kern/xen/pci` directory. |
|
|
PCI devices to be exported to a domU are listed in the `pci` array of |
PCI devices to be exported to a domU are listed in the "pci" array of |
the domU's config file, with the format `'0000:bus:dev.func'` |
the domU's config file, with the format "0000:bus:dev.func". |
|
|
|
pci = [ '0000:00:06.0', '0000:00:0a.0' ] |
|
|
|
In the domU an "xpci" device will show up, to which one or more pci |
|
buses will attach. Then the PCI drivers will attach to PCI buses as |
|
usual. Note that the default NetBSD DOMU kernels do not have "xpci" |
|
or any PCI drivers built in by default; you have to build your own |
|
kernel to use PCI devices in a domU. Here's a kernel config example; |
|
note that only the "xpci" lines are unusual. |
|
|
|
include "arch/i386/conf/XEN3_DOMU" |
|
|
|
# Add support for PCI buses to the XEN3_DOMU kernel |
|
xpci* at xenbus ? |
|
pci* at xpci ? |
|
|
|
# PCI USB controllers |
|
uhci* at pci? dev ? function ? # Universal Host Controller (Intel) |
|
|
|
# USB bus support |
|
usb* at uhci? |
|
|
|
# USB Hubs |
|
uhub* at usb? |
|
uhub* at uhub? port ? configuration ? interface ? |
|
|
|
# USB Mass Storage |
|
umass* at uhub? port ? configuration ? interface ? |
|
wd* at umass? |
|
# SCSI controllers |
|
ahc* at pci? dev ? function ? # Adaptec [23]94x, aic78x0 SCSI |
|
|
|
# SCSI bus support (for both ahc and umass) |
|
scsibus* at scsi? |
|
|
|
# SCSI devices |
|
sd* at scsibus? target ? lun ? # SCSI disk drives |
|
cd* at scsibus? target ? lun ? # SCSI CD-ROM drives |
|
|
|
|
|
NetBSD as a domU in a VPS |
|
========================= |
|
|
|
The bulk of the HOWTO is about using NetBSD as a dom0 on your own |
|
hardware. This section explains how to deal with Xen in a domU as a |
|
virtual private server where you do not control or have access to the |
|
dom0. This is not intended to be an exhaustive list of VPS providers; |
|
only a few are mentioned that specifically support NetBSD. |
|
|
|
VPS operators provide varying degrees of access and mechanisms for |
|
configuration. The big issue is usually how one controls which kernel |
|
is booted, because the kernel is nominally in the dom0 file system (to |
|
which VPS users do not normally have access). A second issue is how |
|
to install NetBSD. |
|
A VPS user may want to compile a kernel for security updates, to run |
|
npf, run IPsec, or any other reason why someone would want to change |
|
their kernel. |
|
|
|
One approach is to have an administrative interface to upload a kernel, |
|
or to select from a prepopulated list. Other approaches are pygrub |
|
(deprecated) and pvgrub, which are ways to have a bootloader obtain a |
|
kernel from the domU file system. This is closer to a regular physical |
|
computer, where someone who controls a machine can replace the kernel. |
|
|
|
A second issue is multiple CPUs. With NetBSD 6, domUs support |
|
multiple vcpus, and it is typical for VPS providers to enable multiple |
|
CPUs for NetBSD domUs. |
|
|
|
pygrub |
|
------- |
|
|
|
pygrub runs in the dom0 and looks into the domU file system. This |
|
implies that the domU must have a kernel in a file system in a format |
|
known to pygrub. As of 2014, pygrub seems to be of mostly historical |
|
interest. |
|
|
|
pvgrub |
|
------ |
|
|
|
pvgrub is a version of grub that uses PV operations instead of BIOS |
|
calls. It is booted from the dom0 as the domU kernel, and then reads |
|
/grub/menu.lst and loads a kernel from the domU file system. |
|
|
|
[Panix](http://www.panix.com/) lets users use pvgrub. Panix reports |
|
that pvgrub works with FFsv2 with 16K/2K and 32K/4K block/frag sizes |
|
(and hence with defaults from "newfs -O 2"). See [Panix's pvgrub |
|
page](http://www.panix.com/v-colo/grub.html), which describes only |
|
Linux but should be updated to cover NetBSD :-). |
|
|
|
[prgmr.com](http://prgmr.com/) also lets users with pvgrub to boot |
|
their own kernel. See then [prgmr.com NetBSD |
|
HOWTO](http://wiki.prgmr.com/mediawiki/index.php/NetBSD_as_a_DomU) |
|
(which is in need of updating). |
|
|
|
It appears that [grub's FFS |
|
code](http://xenbits.xensource.com/hg/xen-unstable.hg/file/bca284f67702/tools/libfsimage/ufs/fsys_ufs.c) |
|
does not support all aspects of modern FFS, but there are also reports |
|
that FFSv2 works fine. At prgmr, typically one has an ext2 or FAT |
|
partition for the kernel with the intent that grub can understand it, |
|
which leads to /netbsd not being the actual kernel. One must remember |
|
to update the special boot partition. |
|
|
|
Amazon |
|
------ |
|
|
|
See the [Amazon EC2 page](../amazon_ec2/). |
|
|
|
Using npf |
|
--------- |
|
|
pci = [ '0000:00:06.0', '0000:00:0a.0' ] |
In standard kernels, npf is a module, and thus cannot be loaded in a |
|
DOMU kernel. |
|
|
In the domU an `xpci` device will show up, to which one or more pci |
TODO: Explain how to compile npf into a custom kernel, answering (but |
busses will attach. Then the PCI drivers will attach to PCI busses as |
note that the problem was caused by not booting the right kernel) |
usual. Note that the default NetBSD DOMU kernels do not have `xpci` or |
[this email to |
any PCI drivers built in by default; you have to build your own kernel |
netbsd-users](https://mail-index.netbsd.org/netbsd-users/2014/12/26/msg015576.html). |
to use PCI devices in a domU. Here's a kernel config example: |
|
|
TODO items for improving NetBSD/xen |
include "arch/i386/conf/XEN3_DOMU" |
=================================== |
#include "arch/i386/conf/XENU" # in NetBSD 3.0 |
|
|
* Make the NetBSD dom0 kernel work with SMP. |
# Add support for PCI busses to the XEN3_DOMU kernel |
* Test the Xen 4.5 packages adequately to be able to recommend them as |
xpci* at xenbus ? |
the standard approach. |
pci* at xpci ? |
* Get PCI passthrough working on Xen 4.5 |
|
* Get pvgrub into pkgsrc, either via xentools or separately. |
# Now add PCI and related devices to be used by this domain |
* grub |
# USB Controller and Devices |
* Check/add support to pkgsrc grub2 for UFS2 and arbitrary |
|
fragsize/blocksize (UFS2 support may be present; the point is to |
# PCI USB controllers |
make it so that with any UFS1/UFS2 file system setup that works |
uhci* at pci? dev ? function ? # Universal Host Controller (Intel) |
with NetBSD grub will also work). |
|
See [pkg/40258](https://gnats.netbsd.org/40258). |
# USB bus support |
* Push patches upstream. |
usb* at uhci? |
* Get UFS2 patches into pvgrub. |
|
* Add support for PV ops to a version of /boot, and make it usable as |
# USB Hubs |
a kernel in Xen, similar to pvgrub. |
uhub* at usb? |
* Solve somehow the issue with modules for GENERIC not being loadable |
uhub* at uhub? port ? configuration ? interface ? |
in a Xen dom0 or domU kernel. |
|
|
# USB Mass Storage |
Random pointers |
umass* at uhub? port ? configuration ? interface ? |
=============== |
wd* at umass? |
|
# SCSI controllers |
This section contains links from elsewhere not yet integrated into the |
ahc* at pci? dev ? function ? # Adaptec [23]94x, aic78x0 SCSI |
HOWTO, and other guides. |
|
|
# SCSI bus support (for both ahc and umass) |
* http://www.lumbercartel.ca/library/xen/ |
scsibus* at scsi? |
* http://pbraun.nethence.com/doc/sysutils/xen_netbsd_dom0.html |
|
* https://gmplib.org/~tege/xen.html |
# SCSI devices |
|
sd* at scsibus? target ? lun ? # SCSI disk drives |
|
cd* at scsibus? target ? lun ? # SCSI CD-ROM drives |
|
|
|
Links and further information |
|
============================= |
|
|
|
- The [HowTo on Installing into RAID-1](http://mail-index.NetBSD.org/port-xen/2006/03/01/0010.html) |
|
explains how to set up booting a dom0 with Xen using grub |
|
with NetBSD's RAIDframe. (This is obsolete with the use of |
|
NetBSD's native boot.) |
|
- An example of how to use NetBSD's native bootloader to load |
|
NetBSD/Xen instead of Grub can be found in the i386/amd64 boot(8) |
|
and boot.cfg(5) manpages. |
|