version 1.110, 2016/12/20 04:09:54
|
version 1.111, 2016/12/20 08:18:01
|
Line 30 style is less efficient but can run unmo
|
Line 30 style is less efficient but can run unmo
|
Generally any amd64 machine will work with Xen and PV guests. In |
Generally any amd64 machine will work with Xen and PV guests. In |
theory i386 computers without amd64 support can be used for Xen <= |
theory i386 computers without amd64 support can be used for Xen <= |
4.2, but we have no recent reports of this working (this is a hint). |
4.2, but we have no recent reports of this working (this is a hint). |
For HVM guests, the VT or VMX cpu feature (Intel) or SVM/HVM/VT |
For HVM guests, the VT or VMX CPU feature (Intel) or SVM/HVM/VT |
(amd64) is needed; "cpuctl identify 0" will show this. TODO: Clean up |
(amd64) is needed; "cpuctl identify 0" will show this. TODO: Clean up |
and check the above features. |
and check the above features. |
|
|
Line 49 specific PCI devices can be made availab
|
Line 49 specific PCI devices can be made availab
|
of the dom0. This can be useful to let a domU run X11, or access some |
of the dom0. This can be useful to let a domU run X11, or access some |
network interface or other peripheral. |
network interface or other peripheral. |
|
|
NetBSD 6 and earlier supported Xen 2; support was removed fro NetBSD |
NetBSD 6 and earlier supported Xen 2; support was removed from NetBSD |
7. Xen 2 has been removed from pkgsrc. |
7. Xen 2 has been removed from pkgsrc. |
|
|
Prerequisites |
Prerequisites |
Line 77 versions of NetBSD are intentionally ign
|
Line 77 versions of NetBSD are intentionally ign
|
Xen |
Xen |
--- |
--- |
|
|
In NetBSD, xen is provided in pkgsrc, via matching pairs of packages |
In NetBSD, Xen is provided in pkgsrc, via matching pairs of packages |
xenkernel and xentools. We will refer only to the kernel versions, |
xenkernel and xentools. We will refer only to the kernel versions, |
but note that both packages must be installed together and must have |
but note that both packages must be installed together and must have |
matching versions. |
matching versions. |
Line 158 run Xen today supports amd64.)
|
Line 158 run Xen today supports amd64.)
|
|
|
Xen, the dom0 kernel, and each domU kernel can be either i386 or |
Xen, the dom0 kernel, and each domU kernel can be either i386 or |
amd64. When building a xenkernel package, one obtains i386 on an i386 |
amd64. When building a xenkernel package, one obtains i386 on an i386 |
host, and amd64 on an amd64 host. If the xen kernel is i386, then the |
host, and amd64 on an amd64 host. If the Xen kernel is i386, then the |
dom0 kernel and all domU kernels must be i386. With an amd64 xen |
dom0 kernel and all domU kernels must be i386. With an amd64 Xen |
kernel, an amd64 dom0 kernel is known to work, and an i386PAE dom0 |
kernel, an amd64 dom0 kernel is known to work, and an i386PAE dom0 |
kernel should in theory work. An amd64 xen/dom0 is known to support |
kernel should in theory work. An amd64 Xen/dom0 is known to support |
both i386PAE and amd64 domUs. |
both i386PAE and amd64 domUs. |
|
|
i386 dom0 and domU kernels must be PAE (except for Xen 3.1); these are |
i386 dom0 and domU kernels must be PAE (except for Xen 3.1); these are |
Line 173 Because of the above, the standard appro
|
Line 173 Because of the above, the standard appro
|
dom0. |
dom0. |
|
|
Xen 4.2 is the last version to support i386 as a host. TODO: Clarify |
Xen 4.2 is the last version to support i386 as a host. TODO: Clarify |
if this is about the CPU, the xen kernel, or the dom0 kernel having to |
if this is about the CPU, the Xen kernel, or the dom0 kernel having to |
be amd64. |
be amd64. |
|
|
|
|
Line 284 However, the partitioning approach is ve
|
Line 284 However, the partitioning approach is ve
|
If you want to use RAIDframe for the dom0, there are no special issues |
If you want to use RAIDframe for the dom0, there are no special issues |
for Xen. Typically one provides RAID storage for the dom0, and the |
for Xen. Typically one provides RAID storage for the dom0, and the |
domU systems are unaware of RAID. The 2nd-stage loader bootxx_* skips |
domU systems are unaware of RAID. The 2nd-stage loader bootxx_* skips |
over a RAID1 header to find /boot from a filesystem within a RAID |
over a RAID1 header to find /boot from a file system within a RAID |
partition; this is no different when booting Xen. |
partition; this is no different when booting Xen. |
|
|
There are 4 styles of providing backing storage for the virtual disks |
There are 4 styles of providing backing storage for the virtual disks |
Line 300 for domU disks. This is almost as effic
|
Line 300 for domU disks. This is almost as effic
|
and more flexible. Hence raw disk partitions should typically not |
and more flexible. Hence raw disk partitions should typically not |
be used. |
be used. |
|
|
One can use files in the dom0 filesystem, typically created by dd'ing |
One can use files in the dom0 file system, typically created by dd'ing |
/dev/zero to create a specific size. This is somewhat less efficient, |
/dev/zero to create a specific size. This is somewhat less efficient, |
but very convenient, as one can cp the files for backup, or move them |
but very convenient, as one can cp the files for backup, or move them |
between dom0 hosts. |
between dom0 hosts. |
Line 333 releasedir/i386/binary/kernel/netbsd-XEN
|
Line 333 releasedir/i386/binary/kernel/netbsd-XEN
|
should not use Xen 3.1.) Both xen and the NetBSD kernel may be (and |
should not use Xen 3.1.) Both xen and the NetBSD kernel may be (and |
typically are) left compressed. |
typically are) left compressed. |
|
|
In a dom0 kernel, kernfs is mandatory for xend to comunicate with the |
In a dom0 kernel, kernfs is mandatory for xend to communicate with the |
kernel, so ensure that /kern is in fstab. TODO: Say this is default, |
kernel, so ensure that /kern is in fstab. TODO: Say this is default, |
or file a PR and give a reference. |
or file a PR and give a reference. |
|
|
Because you already installed NetBSD, you have a working boot setup |
Because you already installed NetBSD, you have a working boot setup |
with an MBR bootblock, either bootxx_ffsv1 or bootxx_ffsv2 at the |
with an MBR bootblock, either bootxx_ffsv1 or bootxx_ffsv2 at the |
beginning of your root filesystem, /boot present, and likely |
beginning of your root file system, /boot present, and likely |
/boot.cfg. (If not, fix before continuing!) |
/boot.cfg. (If not, fix before continuing!) |
|
|
Add a line to to /boot.cfg to boot Xen. See boot.cfg(5) for an |
Add a line to to /boot.cfg to boot Xen. See boot.cfg(5) for an |
Line 359 starting at 0). In an attempt to add pe
|
Line 359 starting at 0). In an attempt to add pe
|
dom0_max_vcpus=1 dom0_vcpus_pin |
dom0_max_vcpus=1 dom0_vcpus_pin |
|
|
to force only one vcpu to be provided (since NetBSD dom0 can't use |
to force only one vcpu to be provided (since NetBSD dom0 can't use |
more) and to pin that vcpu to a physical cpu. TODO: benchmark this. |
more) and to pin that vcpu to a physical CPU. TODO: benchmark this. |
|
|
Xen has [many boot |
Xen has [many boot |
options](http://xenbits.xenproject.org/docs/4.5-testing/misc/xen-command-line.html), |
options](http://xenbits.xenproject.org/docs/4.5-testing/misc/xen-command-line.html), |
and other tham dom0 memory and max_vcpus, they are generally not |
and other than dom0 memory and max_vcpus, they are generally not |
necessary. |
necessary. |
|
|
As with non-Xen systems, you should have a line to boot /netbsd (a |
As with non-Xen systems, you should have a line to boot /netbsd (a |
Line 411 note that you should be using 4.x):
|
Line 411 note that you should be using 4.x):
|
|
|
For "xl" (4.x), you should enabled xend and xencommons (xenstored). |
For "xl" (4.x), you should enabled xend and xencommons (xenstored). |
Trying to boot 4.x without xencommons=YES will result in a hang; it is |
Trying to boot 4.x without xencommons=YES will result in a hang; it is |
necessary to hig ^C on the console to let the machine finish booting. |
necessary to hit ^C on the console to let the machine finish booting. |
TODO: explain why xend is installed by the package. |
TODO: explain why xend is installed by the package. |
|
|
xencommons=YES |
xencommons=YES |
Line 465 Xen-specific NetBSD issues
|
Line 465 Xen-specific NetBSD issues
|
There are (at least) two additional things different about NetBSD as a |
There are (at least) two additional things different about NetBSD as a |
dom0 kernel compared to hardware. |
dom0 kernel compared to hardware. |
|
|
One is that the module ABI is different because some of the #ddefines |
One is that the module ABI is different because some of the #defines |
change, so one must build modules for Xen. As of netbsd-7, the build |
change, so one must build modules for Xen. As of netbsd-7, the build |
system does this automatically. TODO: check this. (Before building |
system does this automatically. TODO: check this. (Before building |
Xen modules was added, it was awkward to use modules to the point |
Xen modules was added, it was awkward to use modules to the point |
Line 497 over time. Before these commands, it wa
|
Line 497 over time. Before these commands, it wa
|
4.1 and grub, much like the message linked earlier in the grub |
4.1 and grub, much like the message linked earlier in the grub |
section. |
section. |
|
|
# Install mbr bootblocks on both disks. |
# Install MBR bootblocks on both disks. |
fdisk -i /dev/rwd0d |
fdisk -i /dev/rwd0d |
fdisk -i /dev/rwd1d |
fdisk -i /dev/rwd1d |
# Install NetBSD primary boot loader (/ is FFSv1) into RAID1 components. |
# Install NetBSD primary boot loader (/ is FFSv1) into RAID1 components. |
Line 505 section.
|
Line 505 section.
|
installboot -v /dev/rwd1d /usr/mdec/bootxx_ffsv1 |
installboot -v /dev/rwd1d /usr/mdec/bootxx_ffsv1 |
# Install secondary boot loader |
# Install secondary boot loader |
cp -p /usr/mdec/boot / |
cp -p /usr/mdec/boot / |
# Create boog.cfg following earlier guidance: |
# Create boot.cfg following earlier guidance: |
menu=Xen:load /netbsd-XEN3PAE_DOM0.gz console=pc;multiboot /xen.gz dom0_mem=256M |
menu=Xen:load /netbsd-XEN3PAE_DOM0.gz console=pc;multiboot /xen.gz dom0_mem=256M |
menu=Xen.ok:load /netbsd-XEN3PAE_DOM0.ok.gz console=pc;multiboot /xen.ok.gz dom0_mem=256M |
menu=Xen.ok:load /netbsd-XEN3PAE_DOM0.ok.gz console=pc;multiboot /xen.ok.gz dom0_mem=256M |
menu=GENERIC:boot |
menu=GENERIC:boot |
Line 590 config files for domUs are typically in
|
Line 590 config files for domUs are typically in
|
typically named so that the file name, domU name and the domU's host |
typically named so that the file name, domU name and the domU's host |
name match. |
name match. |
|
|
The domU is provided with cpu and memory by Xen, configured by the |
The domU is provided with CPU and memory by Xen, configured by the |
dom0. The domU is provided with disk and network by the dom0, |
dom0. The domU is provided with disk and network by the dom0, |
mediated by Xen, and configured in the dom0. |
mediated by Xen, and configured in the dom0. |
|
|
Line 627 domUs independently. The vif line cause
|
Line 627 domUs independently. The vif line cause
|
with a specific mac address (do not reuse MAC addresses!), in bridge |
with a specific mac address (do not reuse MAC addresses!), in bridge |
mode. Two disks are provided, and they are both writable; the bits |
mode. Two disks are provided, and they are both writable; the bits |
are stored in files and Xen attaches them to a vnd(4) device in the |
are stored in files and Xen attaches them to a vnd(4) device in the |
dom0 on domain creation. The system treates xbd0 as the boot device |
dom0 on domain creation. The system treats xbd0 as the boot device |
without needing explicit configuration. |
without needing explicit configuration. |
|
|
By default xm looks for domain config files in /usr/pkg/etc/xen. Note |
By default xm looks for domain config files in /usr/pkg/etc/xen. Note |
Line 653 domU kernels
|
Line 653 domU kernels
|
|
|
On a physical computer, the BIOS reads sector 0, and a chain of boot |
On a physical computer, the BIOS reads sector 0, and a chain of boot |
loaders finds and loads a kernel. Normally this comes from the root |
loaders finds and loads a kernel. Normally this comes from the root |
filesystem. With Xen domUs, the process is totally different. The |
file system. With Xen domUs, the process is totally different. The |
normal path is for the domU kernel to be a file in the dom0's |
normal path is for the domU kernel to be a file in the dom0's |
filesystem. At the request of the dom0, Xen loads that kernel into a |
file system. At the request of the dom0, Xen loads that kernel into a |
new domU instance and starts execution. While domU kernels can be |
new domU instance and starts execution. While domU kernels can be |
anyplace, reasonable places to store domU kernels on the dom0 are in / |
anyplace, reasonable places to store domU kernels on the dom0 are in / |
(so they are near the dom0 kernel), in /usr/pkg/etc/xen (near the |
(so they are near the dom0 kernel), in /usr/pkg/etc/xen (near the |
Line 670 CPU and memory
|
Line 670 CPU and memory
|
-------------- |
-------------- |
|
|
A domain is provided with some number of vcpus, less than the number |
A domain is provided with some number of vcpus, less than the number |
of cpus seen by the hypervisor. (For a dom0, this is controlled by |
of CPUs seen by the hypervisor. (For a dom0, this is controlled by |
the boot argument "dom0_max_vcpus=1".) For a domU, it is controlled |
the boot argument "dom0_max_vcpus=1".) For a domU, it is controlled |
from the config file by the "vcpus = N" directive. |
from the config file by the "vcpus = N" directive. |
|
|
Line 734 disks.
|
Line 734 disks.
|
Virtual Networking |
Virtual Networking |
------------------ |
------------------ |
|
|
Xen provides virtual ethernets, each of which connects the dom0 and a |
Xen provides virtual Ethernets, each of which connects the dom0 and a |
domU. For each virtual network, there is an interface "xvifN.M" in |
domU. For each virtual network, there is an interface "xvifN.M" in |
the dom0, and in domU index N, a matching interface xennetM (NetBSD |
the dom0, and in domU index N, a matching interface xennetM (NetBSD |
name). The interfaces behave as if there is an Ethernet with two |
name). The interfaces behave as if there is an Ethernet with two |
adaptors connected. From this primitive, one can construct various |
adapters connected. From this primitive, one can construct various |
configurations. We focus on two common and useful cases for which |
configurations. We focus on two common and useful cases for which |
there are existing scripts: bridging and NAT. |
there are existing scripts: bridging and NAT. |
|
|
Line 782 shutdown, in rc.conf add:
|
Line 782 shutdown, in rc.conf add:
|
|
|
xendomains="foo bar" |
xendomains="foo bar" |
|
|
Note that earlier versions of the xentools41 xendomains rc.d scripth |
Note that earlier versions of the xentools41 xendomains rc.d script |
usd xl, when one should use xm with 4.1. |
used xl, when one should use xm with 4.1. |
|
|
Creating specific unprivileged domains (domU) |
Creating specific unprivileged domains (domU) |
============================================= |
============================================= |
Line 799 Creating an unprivileged NetBSD domain (
|
Line 799 Creating an unprivileged NetBSD domain (
|
See the earlier config file, and adjust memory. Decide on how much |
See the earlier config file, and adjust memory. Decide on how much |
storage you will provide, and prepare it (file or lvm). |
storage you will provide, and prepare it (file or lvm). |
|
|
While the kernel will be obtained from the dom0 filesystem, the same |
While the kernel will be obtained from the dom0 file system, the same |
file should be present in the domU as /netbsd so that tools like |
file should be present in the domU as /netbsd so that tools like |
savecore(8) can work. (This is helpful but not necessary.) |
savecore(8) can work. (This is helpful but not necessary.) |
|
|
Line 862 in rc.conf. This way, the domain will be
|
Line 862 in rc.conf. This way, the domain will be
|
`xm shutdown -R` or `xm shutdown -H` is used on the dom0. |
`xm shutdown -R` or `xm shutdown -H` is used on the dom0. |
|
|
It is not strictly necessary to have a kernel (as /netbsd) in the domU |
It is not strictly necessary to have a kernel (as /netbsd) in the domU |
filesystem. However, various programs (e.g. netstat) will use that |
file system. However, various programs (e.g. netstat) will use that |
kernel to look up symbols to read from kernel virtual memory. If |
kernel to look up symbols to read from kernel virtual memory. If |
/netbsd is not the running kernel, those lookups will fail. (This is |
/netbsd is not the running kernel, those lookups will fail. (This is |
not really a Xen-specific issue, but because the domU kernel is |
not really a Xen-specific issue, but because the domU kernel is |
Line 901 Then copy the files from a working Linux
|
Line 901 Then copy the files from a working Linux
|
`/etc` (fstab, network config). It should also be possible to extract |
`/etc` (fstab, network config). It should also be possible to extract |
binary packages such as .rpm or .deb directly to the mounted partition |
binary packages such as .rpm or .deb directly to the mounted partition |
using the appropriate tool, possibly running under NetBSD's Linux |
using the appropriate tool, possibly running under NetBSD's Linux |
emulation. Once the filesystem has been populated, umount it. If |
emulation. Once the file system has been populated, umount it. If |
desirable, the filesystem can be converted to ext3 using tune2fs -j. |
desirable, the file system can be converted to ext3 using tune2fs -j. |
It should now be possible to boot the Linux guest domain, using one of |
It should now be possible to boot the Linux guest domain, using one of |
the vmlinuz-\*-xenU kernels available in the Xen binary distribution. |
the vmlinuz-\*-xenU kernels available in the Xen binary distribution. |
|
|
To get the linux console right, you need to add: |
To get the Linux console right, you need to add: |
|
|
extra = "xencons=tty1" |
extra = "xencons=tty1" |
|
|
to your configuration since not all linux distributions auto-attach a |
to your configuration since not all Linux distributions auto-attach a |
tty to the xen console. |
tty to the xen console. |
|
|
Creating an unprivileged Solaris domain (domU) |
Creating an unprivileged Solaris domain (domU) |
Line 954 the domU's config file, with the format
|
Line 954 the domU's config file, with the format
|
pci = [ '0000:00:06.0', '0000:00:0a.0' ] |
pci = [ '0000:00:06.0', '0000:00:0a.0' ] |
|
|
In the domU an "xpci" device will show up, to which one or more pci |
In the domU an "xpci" device will show up, to which one or more pci |
busses will attach. Then the PCI drivers will attach to PCI busses as |
buses will attach. Then the PCI drivers will attach to PCI buses as |
usual. Note that the default NetBSD DOMU kernels do not have "xpci" |
usual. Note that the default NetBSD DOMU kernels do not have "xpci" |
or any PCI drivers built in by default; you have to build your own |
or any PCI drivers built in by default; you have to build your own |
kernel to use PCI devices in a domU. Here's a kernel config example; |
kernel to use PCI devices in a domU. Here's a kernel config example; |
Line 962 note that only the "xpci" lines are unus
|
Line 962 note that only the "xpci" lines are unus
|
|
|
include "arch/i386/conf/XEN3_DOMU" |
include "arch/i386/conf/XEN3_DOMU" |
|
|
# Add support for PCI busses to the XEN3_DOMU kernel |
# Add support for PCI buses to the XEN3_DOMU kernel |
xpci* at xenbus ? |
xpci* at xenbus ? |
pci* at xpci ? |
pci* at xpci ? |
|
|
Line 1001 only a few are mentioned that specifical
|
Line 1001 only a few are mentioned that specifical
|
|
|
VPS operators provide varying degrees of access and mechanisms for |
VPS operators provide varying degrees of access and mechanisms for |
configuration. The big issue is usually how one controls which kernel |
configuration. The big issue is usually how one controls which kernel |
is booted, because the kernel is nominally in the dom0 filesystem (to |
is booted, because the kernel is nominally in the dom0 file system (to |
which VPS users do not normally have acesss). A second issue is how |
which VPS users do not normally have access). A second issue is how |
to install NetBSD. |
to install NetBSD. |
A VPS user may want to compile a kernel for security updates, to run |
A VPS user may want to compile a kernel for security updates, to run |
npf, run IPsec, or any other reason why someone would want to change |
npf, run IPsec, or any other reason why someone would want to change |
their kernel. |
their kernel. |
|
|
One approach is to have an adminstrative interface to upload a kernel, |
One approach is to have an administrative interface to upload a kernel, |
or to select from a prepopulated list. Other approaches are pygrub |
or to select from a prepopulated list. Other approaches are pygrub |
(deprecated) and pvgrub, which are ways to have a bootloader obtain a |
(deprecated) and pvgrub, which are ways to have a bootloader obtain a |
kernel from the domU filesystem. This is closer to a regular physical |
kernel from the domU file system. This is closer to a regular physical |
computer, where someone who controls a machine can replace the kernel. |
computer, where someone who controls a machine can replace the kernel. |
|
|
A second issue is multiple CPUs. With NetBSD 6, domUs support |
A second issue is multiple CPUs. With NetBSD 6, domUs support |
Line 1021 CPUs for NetBSD domUs.
|
Line 1021 CPUs for NetBSD domUs.
|
pygrub |
pygrub |
------- |
------- |
|
|
pygrub runs in the dom0 and looks into the domU filesystem. This |
pygrub runs in the dom0 and looks into the domU file system. This |
implies that the domU must have a kernel in a filesystem in a format |
implies that the domU must have a kernel in a file system in a format |
known to pygrub. As of 2014, pygrub seems to be of mostly historical |
known to pygrub. As of 2014, pygrub seems to be of mostly historical |
interest. |
interest. |
|
|
Line 1031 pvgrub
|
Line 1031 pvgrub
|
|
|
pvgrub is a version of grub that uses PV operations instead of BIOS |
pvgrub is a version of grub that uses PV operations instead of BIOS |
calls. It is booted from the dom0 as the domU kernel, and then reads |
calls. It is booted from the dom0 as the domU kernel, and then reads |
/grub/menu.lst and loads a kernel from the domU filesystem. |
/grub/menu.lst and loads a kernel from the domU file system. |
|
|
[Panix](http://www.panix.com/) lets users use pvgrub. Panix reports |
[Panix](http://www.panix.com/) lets users use pvgrub. Panix reports |
that pvgrub works with FFsv2 with 16K/2K and 32K/4K block/frag sizes |
that pvgrub works with FFsv2 with 16K/2K and 32K/4K block/frag sizes |
Line 1050 does not support all aspects of modern F
|
Line 1050 does not support all aspects of modern F
|
that FFSv2 works fine. At prgmr, typically one has an ext2 or FAT |
that FFSv2 works fine. At prgmr, typically one has an ext2 or FAT |
partition for the kernel with the intent that grub can understand it, |
partition for the kernel with the intent that grub can understand it, |
which leads to /netbsd not being the actual kernel. One must remember |
which leads to /netbsd not being the actual kernel. One must remember |
to update the special boot partiion. |
to update the special boot partition. |
|
|
Amazon |
Amazon |
------ |
------ |
Line 1079 TODO items for improving NetBSD/xen
|
Line 1079 TODO items for improving NetBSD/xen
|
* grub |
* grub |
* Check/add support to pkgsrc grub2 for UFS2 and arbitrary |
* Check/add support to pkgsrc grub2 for UFS2 and arbitrary |
fragsize/blocksize (UFS2 support may be present; the point is to |
fragsize/blocksize (UFS2 support may be present; the point is to |
make it so that with any UFS1/UFS2 filesystem setup that works |
make it so that with any UFS1/UFS2 file system setup that works |
with NetBSD grub will also work). |
with NetBSD grub will also work). |
See [pkg/40258](http://gnats.netbsd.org/40258). |
See [pkg/40258](http://gnats.netbsd.org/40258). |
* Push patches upstream. |
* Push patches upstream. |