version 1.33, 2014/12/24 15:54:50
|
version 1.69, 2014/12/31 00:33:48
|
Line 2 Introduction
|
Line 2 Introduction
|
============ |
============ |
|
|
[![[Xen |
[![[Xen |
screenshot]](http://www.netbsd.org/gallery/in-Action/hubertf-xens.png)](../../gallery/in-Action/hubertf-xen.png) |
screenshot]](http://www.netbsd.org/gallery/in-Action/hubertf-xens.png)](http://www.netbsd.org/gallery/in-Action/hubertf-xen.png) |
|
|
Xen is a virtual machine monitor or hypervisor for x86 hardware |
Xen is a hypervisor (or virtual machine monitor) for x86 hardware |
(i686-class or higher), which supports running multiple guest |
(i686-class or higher), which supports running multiple guest |
operating systems on a single physical machine. With Xen, one uses |
operating systems on a single physical machine. Xen is a Type 1 or |
the Xen kernel to control the CPU, memory and console, a dom0 |
bare-metal hypervisor; one uses the Xen kernel to control the CPU, |
operating system which mediates access to other hardware (e.g., disks, |
memory and console, a dom0 operating system which mediates access to |
network, USB), and one or more domU operating systems which operate in |
other hardware (e.g., disks, network, USB), and one or more domU |
an unprivileged virtualized environment. IO requests from the domU |
operating systems which operate in an unprivileged virtualized |
systems are forwarded by the hypervisor (Xen) to the dom0 to be |
environment. IO requests from the domU systems are forwarded by the |
fulfilled. |
hypervisor (Xen) to the dom0 to be fulfilled. |
|
|
Xen supports two styles of guests. The original is Para-Virtualized |
Xen supports two styles of guests. The original is Para-Virtualized |
(PV) which means that the guest OS does not attempt to access hardware |
(PV) which means that the guest OS does not attempt to access hardware |
Line 49 specific PCI devices can be made availab
|
Line 49 specific PCI devices can be made availab
|
of the dom0. This can be useful to let a domU run X11, or access some |
of the dom0. This can be useful to let a domU run X11, or access some |
network interface or other peripheral. |
network interface or other peripheral. |
|
|
|
NetBSD used to support Xen2; this has been removed. |
|
|
Prerequisites |
Prerequisites |
------------- |
------------- |
|
|
Line 63 architecture. This HOWTO presumes famil
|
Line 65 architecture. This HOWTO presumes famil
|
on i386/amd64 hardware and installing software from pkgsrc. |
on i386/amd64 hardware and installing software from pkgsrc. |
See also the [Xen website](http://www.xenproject.org/). |
See also the [Xen website](http://www.xenproject.org/). |
|
|
History |
|
------- |
|
|
|
NetBSD used to support Xen2; this has been removed. |
|
|
|
Before NetBSD's native bootloader could support Xen, the use of |
|
grub was recommended. If necessary, see the |
|
[old grub information](/ports/xen/howto-grub/). |
|
|
|
Versions of Xen and NetBSD |
Versions of Xen and NetBSD |
========================== |
========================== |
|
|
Line 152 Therefore, this HOWTO recommends running
|
Line 145 Therefore, this HOWTO recommends running
|
xl, the NetBSD 6 stable branch, and to use an amd64 kernel as the |
xl, the NetBSD 6 stable branch, and to use an amd64 kernel as the |
dom0. Either the i386 or amd64 of NetBSD may be used as domUs. |
dom0. Either the i386 or amd64 of NetBSD may be used as domUs. |
|
|
|
Build problems |
|
-------------- |
|
|
|
Ideally, all versions of Xen in pkgsrc would build on all versions of |
|
NetBSD on both i386 and amd64. However, that isn't the case. Besides |
|
aging code and aging compilers, qemu (included in xentools for HVM |
|
support) is difficult to build. The following are known to work or FAIL: |
|
|
|
xenkernel3 netbsd-5 amd64 |
|
xentools3 netbsd-5 amd64 |
|
xentools3=hvm netbsd-5 amd64 ???? |
|
xenkernel33 netbsd-5 amd64 |
|
xentools33 netbsd-5 amd64 |
|
xenkernel41 netbsd-5 amd64 |
|
xentools41 netbsd-5 amd64 |
|
xenkernel42 netbsd-5 amd64 |
|
xentools42 netbsd-5 amd64 |
|
|
|
xenkernel3 netbsd-6 i386 FAIL |
|
xentools3 netbsd-6 i386 |
|
xentools3-hvm netbsd-6 i386 FAIL (dependencies fail) |
|
xenkernel33 netbsd-6 i386 |
|
xentools33 netbsd-6 i386 |
|
xenkernel41 netbsd-6 i386 |
|
xentools41 netbsd-6 i386 |
|
xenkernel42 netbsd-6 i386 |
|
xentools42 netbsd-6 i386 *MIXED |
|
|
|
(all 3 and 33 seem to FAIL) |
|
xenkernel41 netbsd-7 i386 |
|
xentools41 netbsd-7 i386 |
|
xenkernel42 netbsd-7 i386 |
|
xentools42 netbsd-7 i386 ??FAIL |
|
|
|
(*On netbsd-6 i386, there is a xentools42 in the 2014Q3 official builds, |
|
but it does not build for gdt.) |
|
|
NetBSD as a dom0 |
NetBSD as a dom0 |
================ |
================ |
|
|
Line 163 NetBSD, which is not yet a dom0, and the
|
Line 193 NetBSD, which is not yet a dom0, and the
|
NetBSD install to a dom0 install by just changing the kernel and boot |
NetBSD install to a dom0 install by just changing the kernel and boot |
configuration. |
configuration. |
|
|
|
For experimenting with Xen, a machine with as little as 1G of RAM and |
|
100G of disk can work. For running many domUs in productions, far |
|
more will be needed. |
|
|
Styles of dom0 operation |
Styles of dom0 operation |
------------------------ |
------------------------ |
|
|
Line 182 alternately with little problems, simply
|
Line 216 alternately with little problems, simply
|
Xen daemons when not running Xen. |
Xen daemons when not running Xen. |
|
|
Note that NetBSD as dom0 does not support multiple CPUs. This will |
Note that NetBSD as dom0 does not support multiple CPUs. This will |
limit the performance of the Xen/dom0 workstation approach. |
limit the performance of the Xen/dom0 workstation approach. In theory |
|
the only issue is that the "backend drivers" are not yet MPSAFE: |
|
http://mail-index.netbsd.org/netbsd-users/2014/08/29/msg015195.html |
|
|
Installation of NetBSD |
Installation of NetBSD |
---------------------- |
---------------------- |
Line 241 in /, copied from releasedir/amd64/binar
|
Line 277 in /, copied from releasedir/amd64/binar
|
of a NetBSD build. Both xen and NetBSD may be left compressed. (If |
of a NetBSD build. Both xen and NetBSD may be left compressed. (If |
using i386, use releasedir/i386/binary/kernel/netbsd-XEN3PAE_DOM0.gz.) |
using i386, use releasedir/i386/binary/kernel/netbsd-XEN3PAE_DOM0.gz.) |
|
|
In a dom0 kernel, kernfs is mandatory for xend to comunicate with the |
With Xen as the kernel, you must provide a dom0 NetBSD kernel to be |
kernel, so ensure that /kern is in fstab. |
used as a module; place this in /. Suitable kernels are provided in |
|
releasedir/binary/kernel: |
|
|
|
i386 XEN3_DOM0 |
|
i386 XEN3PAE_DOM0 |
|
amd64 XEN3_DOM0 |
|
|
|
The first one is only for use with Xen 3.1 and i386-mode Xen (and you |
|
should not do this). Current Xen always uses PAE on i386, but you |
|
should generally use amd64 for the dom0. In a dom0 kernel, kernfs is |
|
mandatory for xend to comunicate with the kernel, so ensure that /kern |
|
is in fstab. TODO: Say this is default, or file a PR and give a |
|
reference. |
|
|
Because you already installed NetBSD, you have a working boot setup |
Because you already installed NetBSD, you have a working boot setup |
with an MBR bootblock, either bootxx_ffsv1 or bootxx_ffsv2 at the |
with an MBR bootblock, either bootxx_ffsv1 or bootxx_ffsv2 at the |
Line 251 beginning of your root filesystem, /boot
|
Line 299 beginning of your root filesystem, /boot
|
|
|
See boot.cfg(5) for an example. The basic line is |
See boot.cfg(5) for an example. The basic line is |
|
|
"menu=Xen:load /netbsd-XEN3_DOM0.gz console=pc;multiboot /xen.gz dom0_mem=256M" |
menu=Xen:load /netbsd-XEN3_DOM0.gz console=pc;multiboot /xen.gz dom0_mem=256M |
|
|
which specifies that the dom0 should have 256M, leaving the rest to be |
which specifies that the dom0 should have 256M, leaving the rest to be |
allocated for domUs. |
allocated for domUs. In an attempt to add performance, one can also |
|
add |
|
|
|
dom0_max_vcpus=1 dom0_vcpus_pin |
|
|
|
to force only one vcpu to be provided (since NetBSD dom0 can't use |
|
more) and to pin that vcpu to a physical cpu. TODO: benchmark this. |
|
|
As with non-Xen systems, you should have a line to boot /netbsd (a |
As with non-Xen systems, you should have a line to boot /netbsd (a |
kernel that works without Xen) and fallback versions of the non-Xen |
kernel that works without Xen) and fallback versions of the non-Xen |
kernel, Xen, and the dom0 kernel. |
kernel, Xen, and the dom0 kernel. |
|
|
|
Using grub (historic) |
|
--------------------- |
|
|
|
Before NetBSD's native bootloader could support Xen, the use of |
|
grub was recommended. If necessary, see the |
|
[old grub information](/ports/xen/howto-grub/). |
|
|
The [HowTo on Installing into |
The [HowTo on Installing into |
RAID-1](http://mail-index.NetBSD.org/port-xen/2006/03/01/0010.html) |
RAID-1](http://mail-index.NetBSD.org/port-xen/2006/03/01/0010.html) |
explains how to set up booting a dom0 with Xen using grub with |
explains how to set up booting a dom0 with Xen using grub with |
Line 269 boot.)
|
Line 330 boot.)
|
Configuring Xen |
Configuring Xen |
--------------- |
--------------- |
|
|
|
Xen logs will be in /var/log/xen. |
|
|
Now, you have a system that will boot Xen and the dom0 kernel, and |
Now, you have a system that will boot Xen and the dom0 kernel, and |
just run the dom0 kernel. There will be no domUs, and none can be |
just run the dom0 kernel. There will be no domUs, and none can be |
started because you still have to configure the dom0 tools. The |
started because you still have to configure the dom0 tools. The |
Line 277 is using xm or xl. Note that xend is fo
|
Line 340 is using xm or xl. Note that xend is fo
|
only be used if you plan on using "xm". Do NOT enable xend if you |
only be used if you plan on using "xm". Do NOT enable xend if you |
plan on using "xl" as it will cause problems. |
plan on using "xl" as it will cause problems. |
|
|
|
The installation of NetBSD should already have created devices for xen |
|
(xencons, xenevt), but if they are not present, create them: |
|
|
|
cd /dev && sh MAKEDEV xen |
|
|
TODO: Give 3.1 advice (or remove it from pkgsrc). |
TODO: Give 3.1 advice (or remove it from pkgsrc). |
|
|
For 3.3 (and thus xm), add to rc.conf (but note that you should have |
For 3.3 (and thus xm), add to rc.conf (but note that you should have |
Line 287 installed 4.1 or 4.2):
|
Line 355 installed 4.1 or 4.2):
|
|
|
For 4.1 (and thus xm; xl is believed not to work well), add to rc.conf: |
For 4.1 (and thus xm; xl is believed not to work well), add to rc.conf: |
|
|
xend=YES |
|
xencommons=YES |
xencommons=YES |
|
xend=YES |
|
|
TODO: Explain why if xm is preferred on 4.1, rc.d/xendomains has xl. |
(If you are using xentools41 from before 2014-12-26, change |
Or fix the package. |
rc.d/xendomains to use xm rather than xl.) |
|
|
For 4.2 with xm, add to rc.conf |
For 4.2 with xm, add to rc.conf |
|
|
xend=YES |
|
xencommons=YES |
xencommons=YES |
|
xend=YES |
|
|
For 4.2 with xl (preferred), add to rc.conf: |
For 4.2 with xl (preferred), add to rc.conf: |
|
|
TODO: explain if there is a xend replacement |
|
xencommons=YES |
xencommons=YES |
|
TODO: explain if there is a xend replacement |
|
|
TODO: Recommend for/against xen-watchdog. |
TODO: Recommend for/against xen-watchdog. |
|
|
After you have configured the daemons and rebooted, run the following |
After you have configured the daemons and either started them (in the |
to inspect Xen's boot messages, available resources, and running |
order given) or rebooted, run the following (or use xl) to inspect |
domains: |
Xen's boot messages, available resources, and running domains: |
xm dmesg |
|
xm info |
# xm dmesg |
xm list |
[xen's boot info] |
|
# xm info |
|
[available memory, etc.] |
|
# xm list |
|
Name Id Mem(MB) CPU State Time(s) Console |
|
Domain-0 0 64 0 r---- 58.1 |
|
|
|
anita (for testing NetBSD) |
|
-------------------------- |
|
|
|
With the setup so far, one should be able to run anita (see |
|
pkgsrc/sysutils/py-anita) to test NetBSD releases, by doing (as root, |
|
because anita must create a domU): |
|
|
|
anita --vmm=xm test file:///usr/obj/i386/ |
|
|
|
Alternatively, one can use --vmm=xl to use xl-based domU creation instead. |
|
TODO: check this. |
|
|
|
Xen-specific NetBSD issues |
|
-------------------------- |
|
|
|
There are (at least) two additional things different about NetBSD as a |
|
dom0 kernel compared to hardware. |
|
|
|
One is that modules are not usable in DOM0 kernels, so one must |
|
compile in what's needed. It's not really that modules cannot work, |
|
but that modules must be built for XEN3_DOM0 because some of the |
|
defines change and the normal module builds don't do this. Basically, |
|
enabling Xen changes the kernel ABI, and the module build system |
|
doesn't cope with this. |
|
|
|
The other difference is that XEN3_DOM0 does not have exactly the same |
|
options as GENERIC. While it is debatable whether or not this is a |
|
bug, users should be aware of this and can simply add missing config |
|
items if desired. |
|
|
Updating NetBSD in a dom0 |
Updating NetBSD in a dom0 |
------------------------- |
------------------------- |
Line 323 and adjusts /etc.
|
Line 426 and adjusts /etc.
|
Note that one must update both the non-Xen kernel typically used for |
Note that one must update both the non-Xen kernel typically used for |
rescue purposes and the DOM0 kernel used with Xen. |
rescue purposes and the DOM0 kernel used with Xen. |
|
|
To convert from grub to /boot, install an mbr bootblock with fdisk, |
Converting from grub to /boot |
bootxx_ with installboot, /boot and /boot.cfg. This really should be |
----------------------------- |
no different than completely reinstalling boot blocks on a non-Xen |
|
system. |
These instructions were [TODO: will be] used to convert a system from |
|
grub to /boot. The system was originally installed in February of |
|
2006 with a RAID1 setup and grub to boot Xen 2, and has been updated |
|
over time. Before these commands, it was running NetBSD 6 i386, Xen |
|
4.1 and grub, much like the message linked earlier in the grub |
|
section. |
|
|
|
# Install mbr bootblocks on both disks. |
|
fdisk -i /dev/rwd0d |
|
fdisk -i /dev/rwd1d |
|
# Install NetBSD primary boot loader (/ is FFSv1) into RAID1 components. |
|
installboot -v /dev/rwd0d /usr/mdec/bootxx_ffsv1 |
|
installboot -v /dev/rwd1d /usr/mdec/bootxx_ffsv1 |
|
# Install secondary boot loader |
|
cp -p /usr/mdec/boot / |
|
# Create boog.cfg following earlier guidance: |
|
menu=Xen:load /netbsd-XEN3PAE_DOM0.gz console=pc;multiboot /xen.gz dom0_mem=256M |
|
menu=Xen.ok:load /netbsd-XEN3PAE_DOM0.ok.gz console=pc;multiboot /xen.ok.gz dom0_mem=256M |
|
menu=GENERIC:boot |
|
menu=GENERIC single-user:boot -s |
|
menu=GENERIC.ok:boot netbsd.ok |
|
menu=GENERIC.ok single-user:boot netbsd.ok -s |
|
menu=Drop to boot prompt:prompt |
|
default=1 |
|
timeout=30 |
|
|
|
TODO: actually do this and fix it if necessary. |
|
|
Updating Xen versions |
Updating Xen versions |
--------------------- |
--------------------- |
Line 347 Unprivileged domains (domU)
|
Line 476 Unprivileged domains (domU)
|
This section describes general concepts about domUs. It does not |
This section describes general concepts about domUs. It does not |
address specific domU operating systems or how to install them. The |
address specific domU operating systems or how to install them. The |
config files for domUs are typically in /usr/pkg/etc/xen, and are |
config files for domUs are typically in /usr/pkg/etc/xen, and are |
typically named so that the file anme, domU name and the domU's host |
typically named so that the file name, domU name and the domU's host |
name match. |
name match. |
|
|
The domU is provided with cpu and memory by Xen, configured by the |
The domU is provided with cpu and memory by Xen, configured by the |
Line 357 mediated by Xen, and configured in the d
|
Line 486 mediated by Xen, and configured in the d
|
Entropy in domUs can be an issue; physical disks and network are on |
Entropy in domUs can be an issue; physical disks and network are on |
the dom0. NetBSD's /dev/random system works, but is often challenged. |
the dom0. NetBSD's /dev/random system works, but is often challenged. |
|
|
|
Config files |
|
------------ |
|
|
|
There is no good order to present config files and the concepts |
|
surrounding what is being configured. We first show an example config |
|
file, and then in the various sections give details. |
|
|
|
See (at least in xentools41) /usr/pkg/share/examples/xen/xmexample*, |
|
for a large number of well-commented examples, mostly for running |
|
GNU/Linux. |
|
|
|
The following is an example minimal domain configuration file |
|
"/usr/pkg/etc/xen/foo". It is (with only a name change) an actual |
|
known working config file on Xen 4.1 (NetBSD 5 amd64 dom0 and NetBSD 5 |
|
i386 domU). The domU serves as a network file server. |
|
|
|
# -*- mode: python; -*- |
|
|
|
kernel = "/netbsd-XEN3PAE_DOMU-i386-foo.gz" |
|
memory = 1024 |
|
vif = [ 'mac=aa:00:00:d1:00:09,bridge=bridge0' ] |
|
disk = [ 'file:/n0/xen/foo-wd0,0x0,w', |
|
'file:/n0/xen/foo-wd1,0x1,w' ] |
|
|
|
The domain will have the same name as the file. The kernel has the |
|
host/domU name in it, so that on the dom0 one can update the various |
|
domUs independently. The vif line causes an interface to be provided, |
|
with a specific mac address (do not reuse MAC addresses!), in bridge |
|
mode. Two disks are provided, and they are both writable; the bits |
|
are stored in files and Xen attaches them to a vnd(4) device in the |
|
dom0 on domain creation. The system treates xbd0 as the boot device |
|
without needing explicit configuration. |
|
|
|
By default xm looks for domain config files in /usr/pkg/etc/xen. Note |
|
that "xm create" takes the name of a config file, while other commands |
|
take the name of a domain. To create the domain, connect to the |
|
console, create the domain while attaching the console, shutdown the |
|
domain, and see if it has finished stopping, do (or xl with Xen >= |
|
4.2): |
|
|
|
xm create foo |
|
xm console foo |
|
xm create -c foo |
|
xm shutdown foo |
|
xm list |
|
|
|
Typing ^] will exit the console session. Shutting down a domain is |
|
equivalent to pushing the power button; a NetBSD domU will receive a |
|
power-press event and do a clean shutdown. Shutting down the dom0 |
|
will trigger controlled shutdowns of all configured domUs. |
|
|
|
domU kernels |
|
------------ |
|
|
|
On a physical computer, the BIOS reads sector 0, and a chain of boot |
|
loaders finds and loads a kernel. Normally this comes from the root |
|
filesystem. With Xen domUs, the process is totally different. The |
|
normal path is for the domU kernel to be a file in the dom0's |
|
filesystem. At the request of the dom0, Xen loads that kernel into a |
|
new domU instance and starts execution. While domU kernels can be |
|
anyplace, reasonable places to store domU kernels on the dom0 are in / |
|
(so they are near the dom0 kernel), in /usr/pkg/etc/xen (near the |
|
config files), or in /u0/xen (where the vdisks are). |
|
|
|
Note that loading the domU kernel from the dom0 implies that boot |
|
blocks, /boot, /boot.cfg, and so on are all ignored in the domU. |
|
See the VPS section near the end for discussion of alternate ways to |
|
obtain domU kernels. |
|
|
CPU and memory |
CPU and memory |
-------------- |
-------------- |
|
|
A domain is provided with some number of vcpus, less than the |
A domain is provided with some number of vcpus, less than the number |
number of cpus seen by the hypervisor. For a dom0, this is controlled |
of cpus seen by the hypervisor. (For a dom0, this is controlled by |
by the boot argument "dom0_max_vcpus=1". For a domU, it is controlled |
the boot argument "dom0_max_vcpus=1".) For a domU, it is controlled |
from the config file. |
from the config file by the "vcpus = N" directive. |
|
|
A domain is provided with memory, In the straightforward case, the sum |
A domain is provided with memory; this is controlled in the config |
of the the memory allocated to the dom0 and all domUs must be less |
file by "memory = N" (in megabytes). In the straightforward case, the |
|
sum of the the memory allocated to the dom0 and all domUs must be less |
than the available memory. |
than the available memory. |
|
|
Xen also provides a "balloon" driver, which can be used to let domains |
Xen also provides a "balloon" driver, which can be used to let domains |
Line 386 improves performance. The other is that
|
Line 585 improves performance. The other is that
|
failed to work. TODO: give working/notworking NetBSD versions for |
failed to work. TODO: give working/notworking NetBSD versions for |
sparse vnd. Note that the use of file/vnd for Xen is not really |
sparse vnd. Note that the use of file/vnd for Xen is not really |
different than creating a file-backed virtual disk for some other |
different than creating a file-backed virtual disk for some other |
purpose, except that xentools handles the vnconfig commands. |
purpose, except that xentools handles the vnconfig commands. To |
|
create an empty 4G virtual disk, simply do |
|
|
|
dd if=/dev/zero of=foo-xbd0 bs=1m count=4096 |
|
|
With the lvm style, one creates logical devices. They are then used |
With the lvm style, one creates logical devices. They are then used |
similarly to vnds. |
similarly to vnds. TODO: Add an example with lvm. |
|
|
Virtual Networking |
In domU config files, the disks are defined as a sequence of 3-tuples. |
------------------ |
The first element is "method:/path/to/disk". Common methods are |
|
"file:" for file-backed vnd. and "phy:" for something that is already |
|
a (TODO: character or block) device. |
|
|
|
The second element is an artifact of how virtual disks are passed to |
|
Linux, and a source of confusion with NetBSD Xen usage. Linux domUs |
|
are given a device name to associate with the disk, and values like |
|
"hda1" or "sda1" are common. In a NetBSD domU, the first disk appears |
|
as xbd0, the second as xbd1, and so on. However, xm/xl demand a |
|
second argument. The name given is converted to a major/minor by |
|
calling stat(2) on the name in /dev and this is passed to the domU. |
|
In the general case, the dom0 and domU can be different operating |
|
systems, and it is an unwarranted assumption that they have consistent |
|
numbering in /dev, or even that the dom0 OS has a /dev. With NetBSD |
|
as both dom0 and domU, using values of 0x0 for the first disk and 0x1 |
|
for the second works fine and avoids this issue. For a GNU/Linux |
|
guest, one can create /dev/hda1 in /dev, or to pass 0x301 for |
|
/dev/hda1. |
|
|
TODO: explain xvif concept, and that it's general. |
The third element is "w" for writable disks, and "r" for read-only |
|
disks. |
|
|
There are two normal styles: bridging and NAT. |
Virtual Networking |
|
------------------ |
|
|
With bridging, the domU perceives itself to be on the same network as |
Xen provides virtual ethernets, each of which connects the dom0 and a |
the dom0. For server virtualization, this is usually best. |
domU. For each virtual network, there is an interface "xvifN.M" in |
|
the dom0, and in domU index N, a matching interface xennetM (NetBSD |
|
name). The interfaces behave as if there is an Ethernet with two |
|
adaptors connected. From this primitive, one can construct various |
|
configurations. We focus on two common and useful cases for which |
|
there are existing scripts: bridging and NAT. |
|
|
|
With bridging (in the example above), the domU perceives itself to be |
|
on the same network as the dom0. For server virtualization, this is |
|
usually best. Bridging is accomplished by creating a bridge(4) device |
|
and adding the dom0's physical interface and the various xvifN.0 |
|
interfaces to the bridge. One specifies "bridge=bridge0" in the domU |
|
config file. The bridge must be set up already in the dom0; an |
|
example /etc/ifconfig.bridge0 is: |
|
|
|
create |
|
up |
|
!brconfig bridge0 add wm0 |
|
|
With NAT, the domU perceives itself to be behind a NAT running on the |
With NAT, the domU perceives itself to be behind a NAT running on the |
dom0. This is often appropriate when running Xen on a workstation. |
dom0. This is often appropriate when running Xen on a workstation. |
|
TODO: NAT appears to be configured by "vif = [ '' ]". |
|
|
One can construct arbitrary other configurations, but there is no |
The MAC address specified is the one used for the interface in the new |
script support. |
domain. The interface in dom0 will use this address XOR'd with |
|
00:00:00:01:00:00. Random MAC addresses are assigned if not given. |
|
|
Sizing domains |
Sizing domains |
-------------- |
-------------- |
Line 419 create a new file and vnconfig it (or lv
|
Line 659 create a new file and vnconfig it (or lv
|
just like updating physical disks, but without having to be there and |
just like updating physical disks, but without having to be there and |
without those pesky connectors. |
without those pesky connectors. |
|
|
Config files |
Starting domains automatically |
------------ |
------------------------------ |
|
|
TODO: give example config files. Use both lvm and vnd. |
|
|
|
TODO: explain the mess with 3 arguments for disks and how to cope (0x1). |
To start domains foo at bar at boot and shut them down cleanly on dom0 |
|
shutdown, in rc.conf add: |
|
|
Starting domains |
xendomains="foo bar" |
---------------- |
|
|
|
TODO: Explain "xm start" and "xl start". Explain rc.d/xendomains. |
|
|
|
TODO: Explain why 4.1 rc.d/xendomains has xl, when one should use xm |
TODO: Explain why 4.1 rc.d/xendomains has xl, when one should use xm |
on 4.1. |
on 4.1. Or fix the xentools41 package to have xm |
|
|
Creating specific unprivileged domains (domU) |
Creating specific unprivileged domains (domU) |
============================================= |
============================================= |
|
|
Creating domUs is almost entirely independent of operating system. We |
Creating domUs is almost entirely independent of operating system. We |
first explain NetBSD, and then differences for Linux and Solaris. |
have already presented the basics of config files. Note that you must |
|
have already completed the dom0 setup so that "xl list" (or "xm list") |
|
works. |
|
|
Creating an unprivileged NetBSD domain (domU) |
Creating an unprivileged NetBSD domain (domU) |
--------------------------------------------- |
--------------------------------------------- |
|
|
Once you have *domain0* running, you need to start the xen tool daemon |
See the earlier config file, and adjust memory. Decide on how much |
(`/usr/pkg/share/examples/rc.d/xend start`) and the xen backend daemon |
storage you will provide, and prepare it (file or lvm). |
(`/usr/pkg/share/examples/rc.d/xenbackendd start` for Xen3\*, |
|
`/usr/pkg/share/examples/rc.d/xencommons start` for Xen4.\*). Make sure |
|
that `/dev/xencons` and `/dev/xenevt` exist before starting `xend`. You |
|
can create them with this command: |
|
|
|
# cd /dev && sh MAKEDEV xen |
|
|
|
xend will write logs to `/var/log/xend.log` and |
|
`/var/log/xend-debug.log`. You can then control xen with the xm tool. |
|
'xm list' will show something like: |
|
|
|
# xm list |
|
Name Id Mem(MB) CPU State Time(s) Console |
|
Domain-0 0 64 0 r---- 58.1 |
|
|
|
'xm create' allows you to create a new domain. It uses a config file in |
|
PKG\_SYSCONFDIR for its parameters. By default, this file will be in |
|
`/usr/pkg/etc/xen/`. On creation, a kernel has to be specified, which |
|
will be executed in the new domain (this kernel is in the *domain0* file |
|
system, not on the new domain virtual disk; but please note, you should |
|
install the same kernel into *domainU* as `/netbsd` in order to make |
|
your system tools, like savecore(8), work). A suitable kernel is |
|
provided as part of the i386 and amd64 binary sets: XEN3\_DOMU. |
|
|
|
Here is an /usr/pkg/etc/xen/nbsd example config file: |
|
|
|
# -*- mode: python; -*- |
|
#============================================================================ |
|
# Python defaults setup for 'xm create'. |
|
# Edit this file to reflect the configuration of your system. |
|
#============================================================================ |
|
|
|
#---------------------------------------------------------------------------- |
|
# Kernel image file. This kernel will be loaded in the new domain. |
|
kernel = "/home/bouyer/netbsd-XEN3_DOMU" |
|
#kernel = "/home/bouyer/netbsd-INSTALL_XEN3_DOMU" |
|
|
|
# Memory allocation (in megabytes) for the new domain. |
|
memory = 128 |
|
|
|
# A handy name for your new domain. This will appear in 'xm list', |
|
# and you can use this as parameters for xm in place of the domain |
|
# number. All domains must have different names. |
|
# |
|
name = "nbsd" |
|
|
|
# The number of virtual CPUs this domain has. |
|
# |
|
vcpus = 1 |
|
|
|
#---------------------------------------------------------------------------- |
|
# Define network interfaces for the new domain. |
|
|
|
# Number of network interfaces (must be at least 1). Default is 1. |
|
nics = 1 |
|
|
|
# Define MAC and/or bridge for the network interfaces. |
|
# |
|
# The MAC address specified in ``mac'' is the one used for the interface |
|
# in the new domain. The interface in domain0 will use this address XOR'd |
|
# with 00:00:00:01:00:00 (i.e. aa:00:00:51:02:f0 in our example). Random |
|
# MACs are assigned if not given. |
|
# |
|
# ``bridge'' is a required parameter, which will be passed to the |
|
# vif-script called by xend(8) when a new domain is created to configure |
|
# the new xvif interface in domain0. |
|
# |
|
# In this example, the xvif is added to bridge0, which should have been |
|
# set up prior to the new domain being created -- either in the |
|
# ``network'' script or using a /etc/ifconfig.bridge0 file. |
|
# |
|
vif = [ 'mac=aa:00:00:50:02:f0, bridge=bridge0' ] |
|
|
|
#---------------------------------------------------------------------------- |
|
# Define the disk devices you want the domain to have access to, and |
|
# what you want them accessible as. |
|
# |
|
# Each disk entry is of the form: |
|
# |
|
# phy:DEV,VDEV,MODE |
|
# |
|
# where DEV is the device, VDEV is the device name the domain will see, |
|
# and MODE is r for read-only, w for read-write. You can also create |
|
# file-backed domains using disk entries of the form: |
|
# |
|
# file:PATH,VDEV,MODE |
|
# |
|
# where PATH is the path to the file used as the virtual disk, and VDEV |
|
# and MODE have the same meaning as for ``phy'' devices. |
|
# |
|
# VDEV doesn't really matter for a NetBSD guest OS (it's just used as an index), |
|
# but it does for Linux. |
|
# Worse, the device has to exist in /dev/ of domain0, because xm will |
|
# try to stat() it. This means that in order to load a Linux guest OS |
|
# from a NetBSD domain0, you'll have to create /dev/hda1, /dev/hda2, ... |
|
# on domain0, with the major/minor from Linux :( |
|
# Alternatively it's possible to specify the device number in hex, |
|
# e.g. 0x301 for /dev/hda1, 0x302 for /dev/hda2, etc ... |
|
|
|
disk = [ 'phy:/dev/wd0e,0x1,w' ] |
While the kernel will be obtained from the dom0 filesystem, the same |
#disk = [ 'file:/var/xen/nbsd-disk,0x01,w' ] |
file should be present in the domU as /netbsd so that tools like |
#disk = [ 'file:/var/xen/nbsd-disk,0x301,w' ] |
savecore(8) can work. (This is helpful but not necessary.) |
|
|
|
The kernel must be specifically for Xen and for use as a domU. The |
|
i386 and amd64 provide the following kernels: |
|
|
|
i386 XEN3_DOMU |
|
i386 XEN3PAE_DOMU |
|
amd64 XEN3_DOMU |
|
|
|
Unless using Xen 3.1 (and you shouldn't) with i386-mode Xen, you must |
|
use the PAE version of the i386 kernel. |
|
|
|
This will boot NetBSD, but this is not that useful if the disk is |
|
empty. One approach is to unpack sets onto the disk outside of xen |
|
(by mounting it, just as you would prepare a physical disk for a |
|
system you can't run the installer on). |
|
|
|
A second approach is to run an INSTALL kernel, which has a miniroot |
|
and can load sets from the network. To do this, copy the INSTALL |
|
kernel to / and change the kernel line in the config file to: |
|
|
#---------------------------------------------------------------------------- |
kernel = "/home/bouyer/netbsd-INSTALL_XEN3_DOMU" |
# Set the kernel command line for the new domain. |
|
|
|
# Set root device. This one does matter for NetBSD |
Then, start the domain as "xl create -c configname". |
root = "xbd0" |
|
# extra parameters passed to the kernel |
|
# this is where you can set boot flags like -s, -a, etc ... |
|
#extra = "" |
|
|
|
#---------------------------------------------------------------------------- |
|
# Set according to whether you want the domain restarted when it exits. |
|
# The default is False. |
|
#autorestart = True |
|
|
|
# end of nbsd config file ==================================================== |
|
|
|
When a new domain is created, xen calls the |
|
`/usr/pkg/etc/xen/vif-bridge` script for each virtual network interface |
|
created in *domain0*. This can be used to automatically configure the |
|
xvif?.? interfaces in *domain0*. In our example, these will be bridged |
|
with the bridge0 device in *domain0*, but the bridge has to exist first. |
|
To do this, create the file `/etc/ifconfig.bridge0` and make it look |
|
like this: |
|
|
|
create |
|
!brconfig $int add ex0 up |
|
|
|
(replace `ex0` with the name of your physical interface). Then bridge0 |
|
will be created on boot. See the bridge(4) man page for details. |
|
|
|
So, here is a suitable `/usr/pkg/etc/xen/vif-bridge` for xvif?.? (a |
|
working vif-bridge is also provided with xentools20) configuring: |
|
|
|
#!/bin/sh |
|
#============================================================================ |
|
# $NetBSD: howto.mdwn,v 1.32 2014/12/24 15:31:36 gdt Exp $ |
|
# |
|
# /usr/pkg/etc/xen/vif-bridge |
|
# |
|
# Script for configuring a vif in bridged mode with a dom0 interface. |
|
# The xend(8) daemon calls a vif script when bringing a vif up or down. |
|
# The script name to use is defined in /usr/pkg/etc/xen/xend-config.sxp |
|
# in the ``vif-script'' field. |
|
# |
|
# Usage: vif-bridge up|down [var=value ...] |
|
# |
|
# Actions: |
|
# up Adds the vif interface to the bridge. |
|
# down Removes the vif interface from the bridge. |
|
# |
|
# Variables: |
|
# domain name of the domain the interface is on (required). |
|
# vifq vif interface name (required). |
|
# mac vif MAC address (required). |
|
# bridge bridge to add the vif to (required). |
|
# |
|
# Example invocation: |
|
# |
|
# vif-bridge up domain=VM1 vif=xvif1.0 mac="ee:14:01:d0:ec:af" bridge=bridge0 |
|
# |
|
#============================================================================ |
|
|
|
# Exit if anything goes wrong |
|
set -e |
|
|
|
echo "vif-bridge $*" |
|
|
|
# Operation name. |
|
OP=$1; shift |
|
|
|
# Pull variables in args into environment |
|
for arg ; do export "${arg}" ; done |
|
|
|
# Required parameters. Fail if not set. |
|
domain=${domain:?} |
|
vif=${vif:?} |
|
mac=${mac:?} |
|
bridge=${bridge:?} |
|
|
|
# Optional parameters. Set defaults. |
|
ip=${ip:-''} # default to null (do nothing) |
|
|
|
# Are we going up or down? |
|
case $OP in |
|
up) brcmd='add' ;; |
|
down) brcmd='delete' ;; |
|
*) |
|
echo 'Invalid command: ' $OP |
|
echo 'Valid commands are: up, down' |
|
exit 1 |
|
;; |
|
esac |
|
|
|
# Don't do anything if the bridge is "null". |
|
if [ "${bridge}" = "null" ] ; then |
|
exit |
|
fi |
|
|
|
# Don't do anything if the bridge doesn't exist. |
|
if ! ifconfig -l | grep "${bridge}" >/dev/null; then |
|
exit |
|
fi |
|
|
|
# Add/remove vif to/from bridge. |
|
ifconfig x${vif} $OP |
|
brconfig ${bridge} ${brcmd} x${vif} |
|
|
|
Now, running |
|
|
|
xm create -c /usr/pkg/etc/xen/nbsd |
|
|
|
should create a domain and load a NetBSD kernel in it. (Note: `-c` |
|
causes xm to connect to the domain's console once created.) The kernel |
|
will try to find its root file system on xbd0 (i.e., wd0e) which hasn't |
|
been created yet. wd0e will be seen as a disk device in the new domain, |
|
so it will be 'sub-partitioned'. We could attach a ccd to wd0e in |
|
*domain0* and partition it, newfs and extract the NetBSD/i386 or amd64 |
|
tarballs there, but there's an easier way: load the |
|
`netbsd-INSTALL_XEN3_DOMU` kernel provided in the NetBSD binary sets. |
|
Like other install kernels, it contains a ramdisk with sysinst, so you |
|
can install NetBSD using sysinst on your new domain. |
|
|
|
If you want to install NetBSD/Xen with a CDROM image, the following line |
Alternatively, if you want to install NetBSD/Xen with a CDROM image, the following |
should be used in the `/usr/pkg/etc/xen/nbsd` file: |
line should be used in the config file. |
|
|
disk = [ 'phy:/dev/wd0e,0x1,w', 'phy:/dev/cd0a,0x2,r' ] |
disk = [ 'phy:/dev/wd0e,0x1,w', 'phy:/dev/cd0a,0x2,r' ] |
|
|
After booting the domain, the option to install via CDROM may be |
After booting the domain, the option to install via CDROM may be |
selected. The CDROM device should be changed to `xbd1d`. |
selected. The CDROM device should be changed to `xbd1d`. |
|
|
Once done installing, `halt -p` the new domain (don't reboot or halt, it |
Once done installing, "halt -p" the new domain (don't reboot or halt, |
would reload the INSTALL\_XEN3\_DOMU kernel even if you changed the |
it would reload the INSTALL_XEN3_DOMU kernel even if you changed the |
config file), switch the config file back to the XEN3\_DOMU kernel, and |
config file), switch the config file back to the XEN3_DOMU kernel, |
start the new domain again. Now it should be able to use `root on xbd0a` |
and start the new domain again. Now it should be able to use "root on |
and you should have a second, functional NetBSD system on your xen |
xbd0a" and you should have a, functional NetBSD domU. |
installation. |
|
|
|
|
TODO: check if this is still accurate. |
When the new domain is booting you'll see some warnings about *wscons* |
When the new domain is booting you'll see some warnings about *wscons* |
and the pseudo-terminals. These can be fixed by editing the files |
and the pseudo-terminals. These can be fixed by editing the files |
`/etc/ttys` and `/etc/wscons.conf`. You must disable all terminals in |
`/etc/ttys` and `/etc/wscons.conf`. You must disable all terminals in |
Line 700 Finally, all screens must be commented o
|
Line 741 Finally, all screens must be commented o
|
|
|
It is also desirable to add |
It is also desirable to add |
|
|
powerd=YES |
powerd=YES |
|
|
in rc.conf. This way, the domain will be properly shut down if |
in rc.conf. This way, the domain will be properly shut down if |
`xm shutdown -R` or `xm shutdown -H` is used on the domain0. |
`xm shutdown -R` or `xm shutdown -H` is used on the dom0. |
|
|
Your domain should be now ready to work, enjoy. |
Your domain should be now ready to work, enjoy. |
|
|
Line 719 the example below)
|
Line 760 the example below)
|
disk = [ 'phy:/dev/wd0e,0x1,w' ] |
disk = [ 'phy:/dev/wd0e,0x1,w' ] |
|
|
does matter to Linux. It wants a Linux device number here (e.g. 0x300 |
does matter to Linux. It wants a Linux device number here (e.g. 0x300 |
for hda). Linux builds device numbers as: (major \<\< 8 + minor). So, |
for hda). Linux builds device numbers as: (major \<\< 8 + minor). |
hda1 which has major 3 and minor 1 on a Linux system will have device |
So, hda1 which has major 3 and minor 1 on a Linux system will have |
number 0x301. Alternatively, devices names can be used (hda, hdb, ...) |
device number 0x301. Alternatively, devices names can be used (hda, |
as xentools has a table to map these names to devices numbers. To export |
hdb, ...) as xentools has a table to map these names to devices |
a partition to a Linux guest we can use: |
numbers. To export a partition to a Linux guest we can use: |
|
|
disk = [ 'phy:/dev/wd0e,0x300,w' ] |
disk = [ 'phy:/dev/wd0e,0x300,w' ] |
root = "/dev/hda1 ro" |
root = "/dev/hda1 ro" |
|
|
and it will appear as /dev/hda on the Linux system, and be used as root |
and it will appear as /dev/hda on the Linux system, and be used as root |
partition. |
partition. |
|
|
To install the Linux system on the partition to be exported to the guest |
To install the Linux system on the partition to be exported to the |
domain, the following method can be used: install sysutils/e2fsprogs |
guest domain, the following method can be used: install |
from pkgsrc. Use mke2fs to format the partition that will be the root |
sysutils/e2fsprogs from pkgsrc. Use mke2fs to format the partition |
partition of your Linux domain, and mount it. Then copy the files from a |
that will be the root partition of your Linux domain, and mount it. |
working Linux system, make adjustments in `/etc` (fstab, network |
Then copy the files from a working Linux system, make adjustments in |
config). It should also be possible to extract binary packages such as |
`/etc` (fstab, network config). It should also be possible to extract |
.rpm or .deb directly to the mounted partition using the appropriate |
binary packages such as .rpm or .deb directly to the mounted partition |
tool, possibly running under NetBSD's Linux emulation. Once the |
using the appropriate tool, possibly running under NetBSD's Linux |
filesystem has been populated, umount it. If desirable, the filesystem |
emulation. Once the filesystem has been populated, umount it. If |
can be converted to ext3 using tune2fs -j. It should now be possible to |
desirable, the filesystem can be converted to ext3 using tune2fs -j. |
boot the Linux guest domain, using one of the vmlinuz-\*-xenU kernels |
It should now be possible to boot the Linux guest domain, using one of |
available in the Xen binary distribution. |
the vmlinuz-\*-xenU kernels available in the Xen binary distribution. |
|
|
To get the linux console right, you need to add: |
To get the linux console right, you need to add: |
|
|
Line 754 tty to the xen console.
|
Line 795 tty to the xen console.
|
Creating an unprivileged Solaris domain (domU) |
Creating an unprivileged Solaris domain (domU) |
---------------------------------------------- |
---------------------------------------------- |
|
|
Download an Opensolaris [release](http://opensolaris.org/os/downloads/) |
See possibly outdated |
or [development snapshot](http://genunix.org/) DVD image. Attach the DVD |
[Solaris domU instructions](/ports/xen/howto-solaris/). |
image to a MAN.VND.4 device. Copy the kernel and ramdisk filesystem |
|
image to your dom0 filesystem. |
|
|
PCI passthrough: Using PCI devices in guest domains |
dom0# mkdir /root/solaris |
--------------------------------------------------- |
dom0# vnconfig vnd0 osol-1002-124-x86.iso |
|
dom0# mount /dev/vnd0a /mnt |
The dom0 can give other domains access to selected PCI |
|
devices. This can allow, for example, a non-privileged domain to have |
## for a 64-bit guest |
access to a physical network interface or disk controller. However, |
dom0# cp /mnt/boot/amd64/x86.microroot /root/solaris |
keep in mind that giving a domain access to a PCI device most likely |
dom0# cp /mnt/platform/i86xpv/kernel/amd64/unix /root/solaris |
will give the domain read/write access to the whole physical memory, |
|
as PCs don't have an IOMMU to restrict memory access to DMA-capable |
## for a 32-bit guest |
device. Also, it's not possible to export ISA devices to non-dom0 |
dom0# cp /mnt/boot/x86.microroot /root/solaris |
domains, which means that the primary VGA adapter can't be exported. |
dom0# cp /mnt/platform/i86xpv/kernel/unix /root/solaris |
A guest domain trying to access the VGA registers will panic. |
|
|
dom0# umount /mnt |
If the dom0 is NetBSD, it has to be running Xen 3.1, as support has |
|
not been ported to later versions at this time. |
|
|
Keep the MAN.VND.4 configured. For some reason the boot process stalls |
For a PCI device to be exported to a domU, is has to be attached to |
unless the DVD image is attached to the guest as a "phy" device. Create |
the "pciback" driver in dom0. Devices passed to the dom0 via the |
an initial configuration file with the following contents. Substitute |
pciback.hide boot parameter will attach to "pciback" instead of the |
*/dev/wd0k* with an empty partition at least 8 GB large. |
usual driver. The list of devices is specified as "(bus:dev.func)", |
|
|
memory = 640 |
|
name = 'solaris' |
|
disk = [ 'phy:/dev/wd0k,0,w' ] |
|
disk += [ 'phy:/dev/vnd0d,6:cdrom,r' ] |
|
vif = [ 'bridge=bridge0' ] |
|
kernel = '/root/solaris/unix' |
|
ramdisk = '/root/solaris/x86.microroot' |
|
# for a 64-bit guest |
|
extra = '/platform/i86xpv/kernel/amd64/unix - nowin -B install_media=cdrom' |
|
# for a 32-bit guest |
|
#extra = '/platform/i86xpv/kernel/unix - nowin -B install_media=cdrom' |
|
|
|
|
|
Start the guest. |
|
|
|
dom0# xm create -c solaris.cfg |
|
Started domain solaris |
|
v3.3.2 chgset 'unavailable' |
|
SunOS Release 5.11 Version snv_124 64-bit |
|
Copyright 1983-2009 Sun Microsystems, Inc. All rights reserved. |
|
Use is subject to license terms. |
|
Hostname: opensolaris |
|
Remounting root read/write |
|
Probing for device nodes ... |
|
WARNING: emlxs: ddi_modopen drv/fct failed: err 2 |
|
Preparing live image for use |
|
Done mounting Live image |
|
|
|
|
|
Make sure the network is configured. Note that it can take a minute for |
|
the xnf0 interface to appear. |
|
|
|
opensolaris console login: jack |
|
Password: jack |
|
Sun Microsystems Inc. SunOS 5.11 snv_124 November 2008 |
|
jack@opensolaris:~$ pfexec sh |
|
sh-3.2# ifconfig -a |
|
sh-3.2# exit |
|
|
|
|
|
Set a password for VNC and start the VNC server which provides the X11 |
|
display where the installation program runs. |
|
|
|
jack@opensolaris:~$ vncpasswd |
|
Password: solaris |
|
Verify: solaris |
|
jack@opensolaris:~$ cp .Xclients .vnc/xstartup |
|
jack@opensolaris:~$ vncserver :1 |
|
|
|
|
|
From a remote machine connect to the VNC server. Use `ifconfig xnf0` on |
|
the guest to find the correct IP address to use. |
|
|
|
remote$ vncviewer 172.18.2.99:1 |
|
|
|
|
|
It is also possible to launch the installation on a remote X11 display. |
|
|
|
jack@opensolaris:~$ export DISPLAY=172.18.1.1:0 |
|
jack@opensolaris:~$ pfexec gui-install |
|
|
|
|
|
After the GUI installation is complete you will be asked to reboot. |
|
Before that you need to determine the ZFS ID for the new boot filesystem |
|
and update the configuration file accordingly. Return to the guest |
|
console. |
|
|
|
jack@opensolaris:~$ pfexec zdb -vvv rpool | grep bootfs |
|
bootfs = 43 |
|
^C |
|
jack@opensolaris:~$ |
|
|
|
|
|
The final configuration file should look like this. Note in particular |
|
the last line. |
|
|
|
memory = 640 |
|
name = 'solaris' |
|
disk = [ 'phy:/dev/wd0k,0,w' ] |
|
vif = [ 'bridge=bridge0' ] |
|
kernel = '/root/solaris/unix' |
|
ramdisk = '/root/solaris/x86.microroot' |
|
extra = '/platform/i86xpv/kernel/amd64/unix -B zfs-bootfs=rpool/43,bootpath="/xpvd/xdf@0:a"' |
|
|
|
|
|
Restart the guest to verify it works correctly. |
|
|
|
dom0# xm destroy solaris |
|
dom0# xm create -c solaris.cfg |
|
Using config file "./solaris.cfg". |
|
v3.3.2 chgset 'unavailable' |
|
Started domain solaris |
|
SunOS Release 5.11 Version snv_124 64-bit |
|
Copyright 1983-2009 Sun Microsystems, Inc. All rights reserved. |
|
Use is subject to license terms. |
|
WARNING: emlxs: ddi_modopen drv/fct failed: err 2 |
|
Hostname: osol |
|
Configuring devices. |
|
Loading smf(5) service descriptions: 160/160 |
|
svccfg import warnings. See /var/svc/log/system-manifest-import:default.log . |
|
Reading ZFS config: done. |
|
Mounting ZFS filesystems: (6/6) |
|
Creating new rsa public/private host key pair |
|
Creating new dsa public/private host key pair |
|
|
|
osol console login: |
|
|
|
|
|
Using PCI devices in guest domains |
|
---------------------------------- |
|
|
|
The domain0 can give other domains access to selected PCI devices. This |
|
can allow, for example, a non-privileged domain to have access to a |
|
physical network interface or disk controller. However, keep in mind |
|
that giving a domain access to a PCI device most likely will give the |
|
domain read/write access to the whole physical memory, as PCs don't have |
|
an IOMMU to restrict memory access to DMA-capable device. Also, it's not |
|
possible to export ISA devices to non-domain0 domains (which means that |
|
the primary VGA adapter can't be exported. A guest domain trying to |
|
access the VGA registers will panic). |
|
|
|
This functionality is only available in NetBSD-5.1 (and later) domain0 |
|
and domU. If the domain0 is NetBSD, it has to be running Xen 3.1, as |
|
support has not been ported to later versions at this time. |
|
|
|
For a PCI device to be exported to a domU, is has to be attached to the |
|
`pciback` driver in domain0. Devices passed to the domain0 via the |
|
pciback.hide boot parameter will attach to `pciback` instead of the |
|
usual driver. The list of devices is specified as `(bus:dev.func)`, |
|
where bus and dev are 2-digit hexadecimal numbers, and func a |
where bus and dev are 2-digit hexadecimal numbers, and func a |
single-digit number: |
single-digit number: |
|
|
pciback.hide=(00:0a.0)(00:06.0) |
pciback.hide=(00:0a.0)(00:06.0) |
|
|
pciback devices should show up in the domain0's boot messages, and the |
pciback devices should show up in the dom0's boot messages, and the |
devices should be listed in the `/kern/xen/pci` directory. |
devices should be listed in the `/kern/xen/pci` directory. |
|
|
PCI devices to be exported to a domU are listed in the `pci` array of |
PCI devices to be exported to a domU are listed in the "pci" array of |
the domU's config file, with the format `'0000:bus:dev.func'` |
the domU's config file, with the format "0000:bus:dev.func". |
|
|
pci = [ '0000:00:06.0', '0000:00:0a.0' ] |
pci = [ '0000:00:06.0', '0000:00:0a.0' ] |
|
|
In the domU an `xpci` device will show up, to which one or more pci |
In the domU an "xpci" device will show up, to which one or more pci |
busses will attach. Then the PCI drivers will attach to PCI busses as |
busses will attach. Then the PCI drivers will attach to PCI busses as |
usual. Note that the default NetBSD DOMU kernels do not have `xpci` or |
usual. Note that the default NetBSD DOMU kernels do not have "xpci" |
any PCI drivers built in by default; you have to build your own kernel |
or any PCI drivers built in by default; you have to build your own |
to use PCI devices in a domU. Here's a kernel config example: |
kernel to use PCI devices in a domU. Here's a kernel config example; |
|
note that only the "xpci" lines are unusual. |
|
|
|
include "arch/i386/conf/XEN3_DOMU" |
|
|
|
# Add support for PCI busses to the XEN3_DOMU kernel |
|
xpci* at xenbus ? |
|
pci* at xpci ? |
|
|
|
# PCI USB controllers |
|
uhci* at pci? dev ? function ? # Universal Host Controller (Intel) |
|
|
|
# USB bus support |
|
usb* at uhci? |
|
|
|
# USB Hubs |
|
uhub* at usb? |
|
uhub* at uhub? port ? configuration ? interface ? |
|
|
|
# USB Mass Storage |
|
umass* at uhub? port ? configuration ? interface ? |
|
wd* at umass? |
|
# SCSI controllers |
|
ahc* at pci? dev ? function ? # Adaptec [23]94x, aic78x0 SCSI |
|
|
|
# SCSI bus support (for both ahc and umass) |
|
scsibus* at scsi? |
|
|
|
# SCSI devices |
|
sd* at scsibus? target ? lun ? # SCSI disk drives |
|
cd* at scsibus? target ? lun ? # SCSI CD-ROM drives |
|
|
include "arch/i386/conf/XEN3_DOMU" |
|
#include "arch/i386/conf/XENU" # in NetBSD 3.0 |
|
|
|
# Add support for PCI busses to the XEN3_DOMU kernel |
NetBSD as a domU in a VPS |
xpci* at xenbus ? |
========================= |
pci* at xpci ? |
|
|
|
# Now add PCI and related devices to be used by this domain |
|
# USB Controller and Devices |
|
|
|
# PCI USB controllers |
The bulk of the HOWTO is about using NetBSD as a dom0 on your own |
uhci* at pci? dev ? function ? # Universal Host Controller (Intel) |
hardware. This section explains how to deal with Xen in a domU as a |
|
virtual private server where you do not control or have access to the |
|
dom0. |
|
|
# USB bus support |
VPS operators provide varying degrees of access and mechanisms for |
usb* at uhci? |
configuration. The big issue is usually how one controls which kernel |
|
is booted, because the kernel is nominally in the dom0 filesystem (to |
|
which VPS users do not normally have acesss). |
|
|
|
A VPS user may want to compile a kernel for security updates, to run |
|
npf, run IPsec, or any other reason why someone would want to change |
|
their kernel. |
|
|
|
One approach is to have an adminstrative interface to upload a kernel, |
|
or to select from a prepopulated list. Other approaches are pygrub |
|
(deprecated) and pvgrub, which are ways to have a bootloader obtain a |
|
kernel from the domU filesystem. This is closer to a regular physical |
|
computer, where someone who controls a machine can replace the kernel. |
|
|
# USB Hubs |
pygrub |
uhub* at usb? |
------- |
uhub* at uhub? port ? configuration ? interface ? |
|
|
|
# USB Mass Storage |
pygrub runs in the dom0 and looks into the domU filesystem. This |
umass* at uhub? port ? configuration ? interface ? |
implies that the domU must have a kernel in a filesystem in a format |
wd* at umass? |
known to pygrub. As of 2014, pygrub seems to be of mostly historical |
# SCSI controllers |
interest. |
ahc* at pci? dev ? function ? # Adaptec [23]94x, aic78x0 SCSI |
|
|
|
# SCSI bus support (for both ahc and umass) |
pvgrub |
scsibus* at scsi? |
------ |
|
|
# SCSI devices |
pvgrub is a version of grub that uses PV operations instead of BIOS |
sd* at scsibus? target ? lun ? # SCSI disk drives |
calls. It is booted from the dom0 as the domU kernel, and then reads |
cd* at scsibus? target ? lun ? # SCSI CD-ROM drives |
/grub/menu.lst and loads a kernel from the domU filesystem. |
|
|
|
[prgmr.com](http://prgmr.com/) uses this approach to let users choose |
|
their own operating system and kernel. See then [prgmr.com NetBSD |
|
HOWTO](http://wiki.prgmr.com/mediawiki/index.php/NetBSD_as_a_DomU). |
|
|
|
Because [grub's FFS code](http://xenbits.xensource.com/hg/xen-unstable.hg/file/bca284f67702/tools/libfsimage/ufs/fsys_ufs.c) |
|
appears not to support all aspects of modern FFS, |
|
typically one has an ext2 or FAT partition for the kernel, so that |
|
grub can understand it, which leads to /netbsd not being the actual |
|
kernel. One must remember to update the special boot partiion. |
|
|
|
Amazon |
|
------ |
|
|
NetBSD as a domU in a VPS |
TODO: add link to NetBSD amazon howto. |
========================= |
|
|
|
The bulk of the HOWTO is about using NetBSD as a dom0 on your own |
Using npf |
hardware. This section explains how to deal with Xen in a domU as a |
--------- |
virtual private server where you do not control or have access to the |
|
dom0. |
|
|
|
TODO: Perhaps reference panix, prmgr, amazon as interesting examples. |
In standard kernels, npf is a module, and thus cannot be loadeed in a |
|
DOMU kernel. |
|
|
TODO: Somewhere, discuss pvgrub and py-grub to load the domU kernel |
TODO: explain how to compile npf into a custom kernel, answering (but |
from the domU filesystem. |
note that the problem was caused by not booting the right kernel): |
|
http://mail-index.netbsd.org/netbsd-users/2014/12/26/msg015576.html |
|
|
|
TODO items for improving NetBSD/xen |
|
=================================== |
|
|
|
* Package Xen 4.4. |
|
* Get PCI passthrough working on Xen 4.2 (or 4.4). |
|
* Get pvgrub into pkgsrc, either via xentools or separately. |
|
* grub |
|
* Add support to pkgsrc grub2 for UFS2 and arbitrary |
|
fragsize/blocksize (UFS2 support may be present; the point is to |
|
make it so that with any UFS1/UFS2 filesystem setup that works |
|
with NetBSD grub will also work). |
|
* Push patches upstream. |
|
* Get UFS2 patches into pvgrub. |
|
* Add support for PV ops to a version of /boot, and make it usable as |
|
a kernel in Xen, similar to pvgrub. |