version 1.53, 2014/12/26 23:58:18
|
version 1.77, 2015/01/17 01:36:44
|
Line 2 Introduction
|
Line 2 Introduction
|
============ |
============ |
|
|
[![[Xen |
[![[Xen |
screenshot]](http://www.netbsd.org/gallery/in-Action/hubertf-xens.png)](../../gallery/in-Action/hubertf-xen.png) |
screenshot]](http://www.netbsd.org/gallery/in-Action/hubertf-xens.png)](http://www.netbsd.org/gallery/in-Action/hubertf-xen.png) |
|
|
Xen is a virtual machine monitor or hypervisor for x86 hardware |
Xen is a hypervisor (or virtual machine monitor) for x86 hardware |
(i686-class or higher), which supports running multiple guest |
(i686-class or higher), which supports running multiple guest |
operating systems on a single physical machine. With Xen, one uses |
operating systems on a single physical machine. Xen is a Type 1 or |
the Xen kernel to control the CPU, memory and console, a dom0 |
bare-metal hypervisor; one uses the Xen kernel to control the CPU, |
operating system which mediates access to other hardware (e.g., disks, |
memory and console, a dom0 operating system which mediates access to |
network, USB), and one or more domU operating systems which operate in |
other hardware (e.g., disks, network, USB), and one or more domU |
an unprivileged virtualized environment. IO requests from the domU |
operating systems which operate in an unprivileged virtualized |
systems are forwarded by the hypervisor (Xen) to the dom0 to be |
environment. IO requests from the domU systems are forwarded by the |
fulfilled. |
hypervisor (Xen) to the dom0 to be fulfilled. |
|
|
Xen supports two styles of guests. The original is Para-Virtualized |
Xen supports two styles of guests. The original is Para-Virtualized |
(PV) which means that the guest OS does not attempt to access hardware |
(PV) which means that the guest OS does not attempt to access hardware |
Line 49 specific PCI devices can be made availab
|
Line 49 specific PCI devices can be made availab
|
of the dom0. This can be useful to let a domU run X11, or access some |
of the dom0. This can be useful to let a domU run X11, or access some |
network interface or other peripheral. |
network interface or other peripheral. |
|
|
|
NetBSD used to support Xen2; this has been removed. |
|
|
Prerequisites |
Prerequisites |
------------- |
------------- |
|
|
Line 63 architecture. This HOWTO presumes famil
|
Line 65 architecture. This HOWTO presumes famil
|
on i386/amd64 hardware and installing software from pkgsrc. |
on i386/amd64 hardware and installing software from pkgsrc. |
See also the [Xen website](http://www.xenproject.org/). |
See also the [Xen website](http://www.xenproject.org/). |
|
|
History |
|
------- |
|
|
|
NetBSD used to support Xen2; this has been removed. |
|
|
|
Before NetBSD's native bootloader could support Xen, the use of |
|
grub was recommended. If necessary, see the |
|
[old grub information](/ports/xen/howto-grub/). |
|
|
|
Versions of Xen and NetBSD |
Versions of Xen and NetBSD |
========================== |
========================== |
|
|
Line 158 Build problems
|
Line 151 Build problems
|
Ideally, all versions of Xen in pkgsrc would build on all versions of |
Ideally, all versions of Xen in pkgsrc would build on all versions of |
NetBSD on both i386 and amd64. However, that isn't the case. Besides |
NetBSD on both i386 and amd64. However, that isn't the case. Besides |
aging code and aging compilers, qemu (included in xentools for HVM |
aging code and aging compilers, qemu (included in xentools for HVM |
support) is difficult to build. The following are known to fail: |
support) is difficult to build. The following are known to work or FAIL: |
|
|
xenkernel3 netbsd-6 i386 |
|
xentools42 netbsd-6 i386 |
|
|
|
The following are known to work: |
|
|
|
|
xenkernel3 netbsd-5 amd64 |
|
xentools3 netbsd-5 amd64 |
|
xentools3=hvm netbsd-5 amd64 ???? |
|
xenkernel33 netbsd-5 amd64 |
|
xentools33 netbsd-5 amd64 |
xenkernel41 netbsd-5 amd64 |
xenkernel41 netbsd-5 amd64 |
xentools41 netbsd-5 amd64 |
xentools41 netbsd-5 amd64 |
|
xenkernel42 netbsd-5 amd64 |
|
xentools42 netbsd-5 amd64 |
|
|
|
xenkernel3 netbsd-6 i386 FAIL |
|
xentools3 netbsd-6 i386 |
|
xentools3-hvm netbsd-6 i386 FAIL (dependencies fail) |
|
xenkernel33 netbsd-6 i386 |
|
xentools33 netbsd-6 i386 |
xenkernel41 netbsd-6 i386 |
xenkernel41 netbsd-6 i386 |
xentools41 netbsd-6 i386 |
xentools41 netbsd-6 i386 |
|
xenkernel42 netbsd-6 i386 |
|
xentools42 netbsd-6 i386 *MIXED |
|
|
|
(all 3 and 33 seem to FAIL) |
|
xenkernel41 netbsd-7 i386 |
|
xentools41 netbsd-7 i386 |
|
xenkernel42 netbsd-7 i386 |
|
xentools42 netbsd-7 i386 ??FAIL |
|
|
|
(*On netbsd-6 i386, there is a xentools42 in the 2014Q3 official builds, |
|
but it does not build for gdt.) |
|
|
NetBSD as a dom0 |
NetBSD as a dom0 |
================ |
================ |
Line 262 For debugging, one may copy xen-debug.gz
|
Line 274 For debugging, one may copy xen-debug.gz
|
to DIAGNOSTIC and DEBUG in NetBSD. xen-debug.gz is basically only |
to DIAGNOSTIC and DEBUG in NetBSD. xen-debug.gz is basically only |
useful with a serial console. Then, place a NetBSD XEN3_DOM0 kernel |
useful with a serial console. Then, place a NetBSD XEN3_DOM0 kernel |
in /, copied from releasedir/amd64/binary/kernel/netbsd-XEN3_DOM0.gz |
in /, copied from releasedir/amd64/binary/kernel/netbsd-XEN3_DOM0.gz |
of a NetBSD build. Both xen and NetBSD may be left compressed. (If |
of a NetBSD build. If using i386, use |
using i386, use releasedir/i386/binary/kernel/netbsd-XEN3PAE_DOM0.gz.) |
releasedir/i386/binary/kernel/netbsd-XEN3PAE_DOM0.gz. (If using Xen |
|
3.1 and i386, you may use XEN3_DOM0 with the non-PAE Xen. But you |
With Xen as the kernel, you must provide a dom0 NetBSD kernel to be |
should not use Xen 3.1.) Both xen and the NetBSD kernel may be (and |
used as a module; place this in /. Suitable kernels are provided in |
typically are) left compressed. |
releasedir/binary/kernel: |
|
|
In a dom0 kernel, kernfs is mandatory for xend to comunicate with the |
i386 XEN3_DOM0 |
kernel, so ensure that /kern is in fstab. TODO: Say this is default, |
i386 XEN3PAE_DOM0 |
or file a PR and give a reference. |
amd64 XEN3_DOM0 |
|
|
|
The first one is only for use with Xen 3.1 and i386-mode Xen (and you |
|
should not do this). Current Xen always uses PAE on i386, but you |
|
should generally use amd64 for the dom0. In a dom0 kernel, kernfs is |
|
mandatory for xend to comunicate with the kernel, so ensure that /kern |
|
is in fstab. TODO: Say this is default, or file a PR and give a |
|
reference. |
|
|
|
Because you already installed NetBSD, you have a working boot setup |
Because you already installed NetBSD, you have a working boot setup |
with an MBR bootblock, either bootxx_ffsv1 or bootxx_ffsv2 at the |
with an MBR bootblock, either bootxx_ffsv1 or bootxx_ffsv2 at the |
beginning of your root filesystem, /boot present, and likely |
beginning of your root filesystem, /boot present, and likely |
/boot.cfg. (If not, fix before continuing!) |
/boot.cfg. (If not, fix before continuing!) |
|
|
See boot.cfg(5) for an example. The basic line is |
Add a line to to /boot.cfg to boot Xen. See boot.cfg(5) for an |
|
example. The basic line is |
|
|
menu=Xen:load /netbsd-XEN3_DOM0.gz console=pc;multiboot /xen.gz dom0_mem=256M |
menu=Xen:load /netbsd-XEN3_DOM0.gz console=pc;multiboot /xen.gz dom0_mem=256M |
|
|
which specifies that the dom0 should have 256M, leaving the rest to be |
which specifies that the dom0 should have 256M, leaving the rest to be |
allocated for domUs. In an attempt to add performance, one can also |
allocated for domUs. To use a serial console, use |
add |
|
|
menu=Xen:load /netbsd-XEN3_DOM0.gz console=com0;multiboot /xen.gz dom0_mem=256M console=com1 com1=9600,8n1 |
|
|
|
which will use the first serial port for Xen (which counts starting |
|
from 1), forcing speed/parity, and also for NetBSD (which counts |
|
starting at 0). In an attempt to add performance, one can also add |
|
|
dom0_max_vcpus=1 dom0_vcpus_pin |
dom0_max_vcpus=1 dom0_vcpus_pin |
|
|
Line 302 As with non-Xen systems, you should have
|
Line 312 As with non-Xen systems, you should have
|
kernel that works without Xen) and fallback versions of the non-Xen |
kernel that works without Xen) and fallback versions of the non-Xen |
kernel, Xen, and the dom0 kernel. |
kernel, Xen, and the dom0 kernel. |
|
|
|
Now, reboot so that you are running a DOM0 kernel under Xen, rather |
|
than GENERIC without Xen. |
|
|
|
Using grub (historic) |
|
--------------------- |
|
|
|
Before NetBSD's native bootloader could support Xen, the use of |
|
grub was recommended. If necessary, see the |
|
[old grub information](/ports/xen/howto-grub/). |
|
|
The [HowTo on Installing into |
The [HowTo on Installing into |
RAID-1](http://mail-index.NetBSD.org/port-xen/2006/03/01/0010.html) |
RAID-1](http://mail-index.NetBSD.org/port-xen/2006/03/01/0010.html) |
explains how to set up booting a dom0 with Xen using grub with |
explains how to set up booting a dom0 with Xen using grub with |
Line 313 Configuring Xen
|
Line 333 Configuring Xen
|
|
|
Xen logs will be in /var/log/xen. |
Xen logs will be in /var/log/xen. |
|
|
Now, you have a system that will boot Xen and the dom0 kernel, and |
Now, you have a system that will boot Xen and the dom0 kernel, but not |
just run the dom0 kernel. There will be no domUs, and none can be |
do anything else special. Make sure that you have rebooted into Xen. |
started because you still have to configure the dom0 tools. The |
There will be no domUs, and none can be started because you still have |
daemons which should be run vary with Xen version and with whether one |
to configure the dom0 tools. The daemons which should be run vary |
is using xm or xl. Note that xend is for supporting "xm", and should |
with Xen version and with whether one is using xm or xl. Note that |
only be used if you plan on using "xm". Do NOT enable xend if you |
xend is for supporting "xm", and should only be used if you plan on |
plan on using "xl" as it will cause problems. |
using "xm". Do NOT enable xend if you plan on using "xl" as it will |
|
cause problems. |
|
|
The installation of NetBSD should already have created devices for xen |
The installation of NetBSD should already have created devices for xen |
(xencons, xenevt), but if they are not present, create them: |
(xencons, xenevt), but if they are not present, create them: |
Line 339 For 4.1 (and thus xm; xl is believed not
|
Line 360 For 4.1 (and thus xm; xl is believed not
|
xencommons=YES |
xencommons=YES |
xend=YES |
xend=YES |
|
|
TODO: Explain why if xm is preferred on 4.1, rc.d/xendomains has xl. |
(If you are using xentools41 from before 2014-12-26, change |
Or fix the package. |
rc.d/xendomains to use xm rather than xl.) |
|
|
For 4.2 with xm, add to rc.conf |
For 4.2 with xm, add to rc.conf |
|
|
Line 407 and adjusts /etc.
|
Line 428 and adjusts /etc.
|
Note that one must update both the non-Xen kernel typically used for |
Note that one must update both the non-Xen kernel typically used for |
rescue purposes and the DOM0 kernel used with Xen. |
rescue purposes and the DOM0 kernel used with Xen. |
|
|
To convert from grub to /boot, install an mbr bootblock with fdisk, |
Converting from grub to /boot |
bootxx_ with installboot, /boot and /boot.cfg. This really should be |
----------------------------- |
no different than completely reinstalling boot blocks on a non-Xen |
|
system. |
These instructions were [TODO: will be] used to convert a system from |
|
grub to /boot. The system was originally installed in February of |
|
2006 with a RAID1 setup and grub to boot Xen 2, and has been updated |
|
over time. Before these commands, it was running NetBSD 6 i386, Xen |
|
4.1 and grub, much like the message linked earlier in the grub |
|
section. |
|
|
|
# Install mbr bootblocks on both disks. |
|
fdisk -i /dev/rwd0d |
|
fdisk -i /dev/rwd1d |
|
# Install NetBSD primary boot loader (/ is FFSv1) into RAID1 components. |
|
installboot -v /dev/rwd0d /usr/mdec/bootxx_ffsv1 |
|
installboot -v /dev/rwd1d /usr/mdec/bootxx_ffsv1 |
|
# Install secondary boot loader |
|
cp -p /usr/mdec/boot / |
|
# Create boog.cfg following earlier guidance: |
|
menu=Xen:load /netbsd-XEN3PAE_DOM0.gz console=pc;multiboot /xen.gz dom0_mem=256M |
|
menu=Xen.ok:load /netbsd-XEN3PAE_DOM0.ok.gz console=pc;multiboot /xen.ok.gz dom0_mem=256M |
|
menu=GENERIC:boot |
|
menu=GENERIC single-user:boot -s |
|
menu=GENERIC.ok:boot netbsd.ok |
|
menu=GENERIC.ok single-user:boot netbsd.ok -s |
|
menu=Drop to boot prompt:prompt |
|
default=1 |
|
timeout=30 |
|
|
|
TODO: actually do this and fix it if necessary. |
|
|
Updating Xen versions |
Updating Xen versions |
--------------------- |
--------------------- |
Line 431 Unprivileged domains (domU)
|
Line 478 Unprivileged domains (domU)
|
This section describes general concepts about domUs. It does not |
This section describes general concepts about domUs. It does not |
address specific domU operating systems or how to install them. The |
address specific domU operating systems or how to install them. The |
config files for domUs are typically in /usr/pkg/etc/xen, and are |
config files for domUs are typically in /usr/pkg/etc/xen, and are |
typically named so that the file anme, domU name and the domU's host |
typically named so that the file name, domU name and the domU's host |
name match. |
name match. |
|
|
The domU is provided with cpu and memory by Xen, configured by the |
The domU is provided with cpu and memory by Xen, configured by the |
Line 505 anyplace, reasonable places to store dom
|
Line 552 anyplace, reasonable places to store dom
|
(so they are near the dom0 kernel), in /usr/pkg/etc/xen (near the |
(so they are near the dom0 kernel), in /usr/pkg/etc/xen (near the |
config files), or in /u0/xen (where the vdisks are). |
config files), or in /u0/xen (where the vdisks are). |
|
|
|
Note that loading the domU kernel from the dom0 implies that boot |
|
blocks, /boot, /boot.cfg, and so on are all ignored in the domU. |
See the VPS section near the end for discussion of alternate ways to |
See the VPS section near the end for discussion of alternate ways to |
obtain domU kernels. |
obtain domU kernels. |
|
|
Line 828 NetBSD as a domU in a VPS
|
Line 877 NetBSD as a domU in a VPS
|
The bulk of the HOWTO is about using NetBSD as a dom0 on your own |
The bulk of the HOWTO is about using NetBSD as a dom0 on your own |
hardware. This section explains how to deal with Xen in a domU as a |
hardware. This section explains how to deal with Xen in a domU as a |
virtual private server where you do not control or have access to the |
virtual private server where you do not control or have access to the |
dom0. |
dom0. This is not intended to be an exhaustive list of VPS providers; |
|
only a few are mentioned that specifically support NetBSD. |
|
|
VPS operators provide varying degrees of access and mechanisms for |
VPS operators provide varying degrees of access and mechanisms for |
configuration. The big issue is usually how one controls which kernel |
configuration. The big issue is usually how one controls which kernel |
is booted, because the kernel is nominally in the dom0 filesystem (to |
is booted, because the kernel is nominally in the dom0 filesystem (to |
which VPS users do not normally have acesss). |
which VPS users do not normally have acesss). A second issue is how |
|
to install NetBSD. |
A VPS user may want to compile a kernel for security updates, to run |
A VPS user may want to compile a kernel for security updates, to run |
npf, run IPsec, or any other reason why someone would want to change |
npf, run IPsec, or any other reason why someone would want to change |
their kernel. |
their kernel. |
|
|
One approach is to have an adminstrative interface to upload a kernel, |
One approach is to have an adminstrative interface to upload a kernel, |
or to select from a prepopulated list. |
or to select from a prepopulated list. Other approaches are pygrub |
|
(deprecated) and pvgrub, which are ways to have a bootloader obtain a |
|
kernel from the domU filesystem. This is closer to a regular physical |
|
computer, where someone who controls a machine can replace the kernel. |
|
|
|
A second issue is multiple CPUs. With NetBSD 6, domUs support |
|
multiple vcpus, and it is typical for VPS providers to enable multiple |
|
CPUs for NetBSD domUs. |
|
|
Otehr approaches are pvgrub and py-grub, which are ways to start a |
pygrub |
bootloader from the dom0 instead of the actual domU kernel, and for |
------- |
that loader to then load a kernel from the domU filesystem. This is |
|
closer to a regular physical computer, where someone who controls a |
|
machine can replace the kernel. |
|
|
|
prmgr and pvgrub |
pygrub runs in the dom0 and looks into the domU filesystem. This |
---------------- |
implies that the domU must have a kernel in a filesystem in a format |
|
known to pygrub. As of 2014, pygrub seems to be of mostly historical |
|
interest. |
|
|
TODO: Perhaps reference panix, prmgr, amazon as interesting examples. |
pvgrub |
Explain what prmgr does. |
------ |
|
|
|
pvgrub is a version of grub that uses PV operations instead of BIOS |
|
calls. It is booted from the dom0 as the domU kernel, and then reads |
|
/grub/menu.lst and loads a kernel from the domU filesystem. |
|
|
|
[Panix](http://www.panix.com/) lets users use pvgrub. Panix reports |
|
that pvgrub works with FFsv2 with 16K/2K and 32K/4K block/frag sizes |
|
(and hence with defaults from "newfs -O 2"). See [Panix's pvgrub |
|
page](http://www.panix.com/v-colo/grub.html), which describes only |
|
Linux but should be updated to cover NetBSD :-). |
|
|
|
[prgmr.com](http://prgmr.com/) also lets users with pvgrub to boot |
|
their own kernel. See then [prgmr.com NetBSD |
|
HOWTO](http://wiki.prgmr.com/mediawiki/index.php/NetBSD_as_a_DomU) |
|
(which is in need of updating). |
|
|
|
It appears that [grub's FFS |
|
code](http://xenbits.xensource.com/hg/xen-unstable.hg/file/bca284f67702/tools/libfsimage/ufs/fsys_ufs.c) |
|
does not support all aspects of modern FFS, but there are also reports |
|
that FFSv2 works fine. At prgmr, typically one has an ext2 or FAT |
|
partition for the kernel with the intent that grub can understand it, |
|
which leads to /netbsd not being the actual kernel. One must remember |
|
to update the special boot partiion. |
|
|
|
Amazon |
|
------ |
|
|
|
TODO: add link to NetBSD amazon howto. |
|
|
Using npf |
Using npf |
--------- |
--------- |
Line 860 Using npf
|
Line 944 Using npf
|
In standard kernels, npf is a module, and thus cannot be loadeed in a |
In standard kernels, npf is a module, and thus cannot be loadeed in a |
DOMU kernel. |
DOMU kernel. |
|
|
TODO: explain how to compile npf into a custom kernel, answering: |
TODO: explain how to compile npf into a custom kernel, answering (but |
|
note that the problem was caused by not booting the right kernel): |
http://mail-index.netbsd.org/netbsd-users/2014/12/26/msg015576.html |
http://mail-index.netbsd.org/netbsd-users/2014/12/26/msg015576.html |
|
|
|
TODO items for improving NetBSD/xen |
|
=================================== |
|
|
|
* Package Xen 4.4. |
|
* Get PCI passthrough working on Xen 4.2 (or 4.4). |
|
* Get pvgrub into pkgsrc, either via xentools or separately. |
|
* grub |
|
* Check/add support to pkgsrc grub2 for UFS2 and arbitrary |
|
fragsize/blocksize (UFS2 support may be present; the point is to |
|
make it so that with any UFS1/UFS2 filesystem setup that works |
|
with NetBSD grub will also work). |
|
See [pkg/40258](http://gnats.netbsd.org/40258). |
|
* Push patches upstream. |
|
* Get UFS2 patches into pvgrub. |
|
* Add support for PV ops to a version of /boot, and make it usable as |
|
a kernel in Xen, similar to pvgrub. |