File:  [NetBSD Developer Wiki] / wikisrc / ports / xen / howto.mdwn
Revision 1.145: download - view: text, annotated - select for diffs
Thu Jul 26 10:36:26 2018 UTC (2 years, 9 months ago) by maxv
Branches: MAIN
CVS tags: HEAD
Reduce the noise:

 - Sum up the "Xen" section in a single, readable table.

 - Remove the "Status" section, it is pointless and probably

 - Remove the "Using npf" section. We do have support for kernel
   modules now, and basically, that kind of stuff shouldn't be in
   a Xen howto.

 - Remove the "missing kernel modules support" from the TODO list,
   we do have that now. By the way, we should probably remove this
   TODO section. This page is supposed to be a HowTo, not a page
   where we put some random things.

    1: [[!meta title="Xen HowTo"]]
    3: Introduction
    4: ============
    6: [![[Xen
    7: screenshot]](](
    9: Xen is a hypervisor (or virtual machine monitor) for x86 hardware
   10: (i686-class or higher), which supports running multiple guest
   11: operating systems on a single physical machine.  Xen is a Type 1 or
   12: bare-metal hypervisor; one uses the Xen kernel to control the CPU,
   13: memory and console, a dom0 operating system which mediates access to
   14: other hardware (e.g., disks, network, USB), and one or more domU
   15: operating systems which operate in an unprivileged virtualized
   16: environment.  IO requests from the domU systems are forwarded by the
   17: hypervisor (Xen) to the dom0 to be fulfilled.
   19: Xen supports two styles of guests.  The original is Para-Virtualized
   20: (PV) which means that the guest OS does not attempt to access hardware
   21: directly, but instead makes hypercalls to the hypervisor.  This is
   22: analogous to a user-space program making system calls.  (The dom0
   23: operating system uses PV calls for some functions, such as updating
   24: memory mapping page tables, but has direct hardware access for disk
   25: and network.)   PV guests must be specifically coded for Xen.
   27: The more recent style is HVM, which means that the guest does not have
   28: code for Xen and need not be aware that it is running under Xen.
   29: Attempts to access hardware registers are trapped and emulated.  This
   30: style is less efficient but can run unmodified guests.
   32: Generally any machine that runs NetBSD/amd64 will work with Xen and PV
   33: guests.  In theory i386 computers (without x86_64/amd64 support) can
   34: be used for Xen <= 4.2, but we have no recent reports of this working
   35: (this is a hint).  For HVM guests, hardware support is needed, but it
   36: is common on recent machines.  For Intel CPUs, one needs the VT-x
   37: extension, shown in "cpuctl identify 0" as VMX.  For AMD CPUs, one
   38: needs the AMD-V extensions, shown in "cpuctl identify 0" as SVM.
   39: There are further features for IOMMU virtualization, Intel's VT-d and
   40: AMD's AMD-Vi.  TODO: Explain whether Xen on NetBSD makes use of these
   41: features.  TODO: Review by someone who really understands this.
   43: Note that a FreeBSD dom0 requires VT-x and VT-d (or equivalent); this
   44: is because the FreeBSD dom0 does not run in PV mode.
   46: At boot, the dom0 kernel is loaded as a module with Xen as the kernel.
   47: The dom0 can start one or more domUs.  (Booting is explained in detail
   48: in the dom0 section.)
   50: NetBSD supports Xen in that it can serve as dom0, be used as a domU,
   51: and that Xen kernels and tools are available in pkgsrc.  This HOWTO
   52: attempts to address both the case of running a NetBSD dom0 on hardware
   53: and running domUs under it (NetBSD and other), and also running NetBSD
   54: as a domU in a VPS.
   56: Xen 3.1 in pkgsrc used to support "PCI passthrough", which means that
   57: specific PCI devices can be made available to a specific domU instead
   58: of the dom0.  This can be useful to let a domU run X11, or access some
   59: network interface or other peripheral.
   61: NetBSD 6 and earlier supported Xen 2; support was removed from NetBSD
   62: 7.  Xen 2 has been removed from pkgsrc.
   64: Prerequisites
   65: -------------
   67: Installing NetBSD/Xen is not extremely difficult, but it is more
   68: complex than a normal installation of NetBSD.
   69: In general, this HOWTO is occasionally overly restrictive about how
   70: things must be done, guiding the reader to stay on the established
   71: path when there are no known good reasons to stray.
   73: This HOWTO presumes a basic familiarity with the Xen system
   74: architecture, with installing NetBSD on i386/amd64 hardware, and with
   75: installing software from pkgsrc.  See also the [Xen
   76: website](
   78: Versions of Xen and NetBSD
   79: ==========================
   81: Most of the installation concepts and instructions are independent
   82: of Xen version and NetBSD version.  This section gives advice on
   83: which version to choose.  Versions not in pkgsrc and older unsupported
   84: versions of NetBSD are intentionally ignored.
   86: The term "amd64" is used to refer to both the NetBSD port and to the
   87: hardware architecture on which it runs.  (Such hardware is made by
   88: both Intel and AMD, and in 2016 a normal PC has this CPU
   89: architecture.)
   91: Xen
   92: ---
   94: In NetBSD, Xen is provided in pkgsrc, via matching pairs of packages
   95: xenkernel and xentools.  We will refer only to the kernel versions,
   96: but note that both packages must be installed together and must have
   97: matching versions.
   99: Versions available in pkgsrc:
  101: [[!table data="""
  102: Xen Version	|Package Name	|Xen CPU Support	|EOL'ed By Upstream
  103: 4.2		|xenkernel42	|32bit, 64bit		|Yes
  104: 4.5		|xenkernel45	|64bit			|Yes
  105: 4.6		|xenkernel46	|64bit			|Partially
  106: 4.8		|xenkernel48	|64bit			|No
  107: 4.11		|xenkernel411	|64bit			|No
  108: """]]
  110: See also the [Xen Security Advisory page](
  112: Note: Xen 4.2 was the last version to support 32bit CPUs.
  114: Note that NetBSD support is called XEN3.  It works with Xen 3 and Xen
  115: 4 because the hypercall interface has been stable.
  117: Xen command program
  118: -------------------
  120: Early Xen used a program called xm to manipulate the system from the
  121: dom0.  Starting in 4.1, a replacement program with similar behavior
  122: called xl is provided, but it does not work well in 4.1.  In 4.2, both
  123: xm and xl work fine.  4.4 is the last version that has xm.
  125: You must make a global choice to use xm or xl, because it affects not
  126: only which command you use, but the command used by rc.d scripts
  127: (specifically xendomains) and which daemons should be run.  The
  128: xentools packages provide xm for 3.1, 3.3 and 4.1 and xl for 4.2 and up.
  130: In 4.2, you can choose to use xm by simply changing the ctl_command
  131: variable and setting xend=YES in rc.conf.
  133: With xl, virtual devices are configured in parallel, which can cause
  134: problems if they are written assuming serial operation (e.g., updating
  135: firewall rules without explicit locking).  There is now locking for
  136: the provided scripts, which works for normal casses (e.g, file-backed
  137: xbd, where a vnd must be allocated).  But, as of 201612, it has not
  138: been adequately tested for a complex custom setup with a large number
  139: of interfaces.
  141: NetBSD
  142: ------
  144: The netbsd-7, netbsd-8, and -current branches are all reasonable
  145: choices, with more or less the same considerations for non-Xen use.
  146: Therefore, netbsd-7 is recommended as the stable version of the most
  147: recent release for production use.  (Note that netbsd-7 (and therefore
  148: 8/current) have a important scheduler fix (in November of 2015)
  149: affecting contention between dom0 and domUs; see
  150: for a
  151: description.)  For production, netbsd-7 is appropriate.  For learning,
  152: netbsd-8 is appropriate.  For developing Xen, netbsd-current may be
  153: appropriate.
  155: As of NetBSD 6, a NetBSD domU will support multiple vcpus.  There is
  156: no SMP support for NetBSD as dom0.  (The dom0 itself doesn't really
  157: need SMP for dom0 functions; the lack of support is really a problem
  158: when using a dom0 as a normal computer.)
  160: Architecture
  161: ------------
  163: Xen itself can run on i386 (Xen < 4.2) or amd64 hardware (all Xen
  164: versions).  (Practically, almost any computer where one would want to
  165: run Xen today supports amd64.)
  167: Xen, the dom0 system, and each domU system can be either i386 or
  168: amd64.  When building a xenkernel package, one obtains an i386 Xen
  169: kernel on an i386 host, and an amd64 Xen kernel on an amd64 host.  If
  170: the Xen kernel is i386, then the dom0 kernel and all domU kernels must
  171: be i386.  With an amd64 Xen kernel, an amd64 dom0 kernel is known to
  172: work, and an i386 dom0 kernel should in theory work.  An amd64
  173: Xen/dom0 is known to support both i386 and amd64 domUs.
  175: i386 dom0 and domU kernels must be PAE (except for an i386 Xen 3.1
  176: kernel, where one can use non-PAE for dom0 and all domUs); PAE kernels
  177: are included in the NetBSD default build.  (Note that emacs (at least)
  178: fails if run on i386 with PAE when built without, and vice versa,
  179: presumably due to bugs in the undump code.)
  181: Because of the above, the standard approach is to use an amd64 Xen
  182: kernel and NetBSD/amd64 for the dom0.  For domUs, NetBSD/i386 (with
  183: the PAE kernel) and NetBSD/amd64 are in widespread use, and there is
  184: little to no Xen-specific reason to prefer one over the other.
  186: Note that to use an i386 dom0 with Xen 4.5 or higher, one must build
  187: (or obtain from pre-built packages) an amd64 Xen kernel and install
  188: that on the system.  (One must also use a PAE i386 kernel, but this is
  189: also required with an i386 Xen kernel.).  Almost no one in the
  190: NetBSD/Xen community does this, and the standard, well-tested,
  191: approach is to use an amd64 dom0.
  193: A [posting on
  194: xen-devel](
  195: explained that PV system call overhead was higher on amd64, and thus
  196: there is some notion that i386 guests are faster.  It goes on to
  197: caution that the total situation is complex and not entirely
  198: understood. On top of that caution, the post is about Linux, not
  199: NetBSD.  TODO: Include link to benchmarks, if someone posts them.
  201: Stability
  202: ---------
  204: Mostly, NetBSD as a dom0 or domU is quite stable. However, just like every
  205: other architecture, there are some open PRs indicating problems.
  207: Note also that there are issues with sparse vnd(4) instances, but
  208: these are not about Xen -- they just are noticed with sparse vnd(4)
  209: instances in support of virtual disks in a dom0.
  211: Recommendation
  212: --------------
  214: Therefore, this HOWTO recommends running xenkernel46, xl, the NetBSD 7
  215: stable branch, and therefore to use an amd64 kernel as the dom0.
  216: Either the i386PAE or amd64 version of NetBSD may be used as domUs.
  218: A tentative replacement recommendation is xenkernel48, xl, and NetBSD
  219: 8.
  221: Because bugs are fixed quite often, and because of Xen security
  222: advisories, it is good to stay up to date with NetBSD (tracking a
  223: stable branch), with the Xen kernel (tracking a Xen version via
  224: pkgsrc), and with the Xen tools.  Specifically, NetBSD (-7 and
  225: -current) got an important fix affecting dom0/domU timesharing in
  226: November, 2015, and xentools46 got a fix to enable Ubuntu guests to
  227: boot in December, 2016.
  229: NetBSD as a dom0
  230: ================
  232: NetBSD can be used as a dom0 and works very well.  The following
  233: sections address installation, updating NetBSD, and updating Xen.
  234: Note that it doesn't make sense to talk about installing a dom0 OS
  235: without also installing Xen itself.  We first address installing
  236: NetBSD, which is not yet a dom0, and then adding Xen, pivoting the
  237: NetBSD install to a dom0 install by just changing the kernel and boot
  238: configuration.
  240: For experimenting with Xen, a machine with as little as 1G of RAM and
  241: 100G of disk can work.  For running many domUs in productions, far
  242: more will be needed; e.g. 4-8G and 1T of disk is reasonable for a
  243: half-dozen domUs of 512M and 32G each.  Basically, the RAM and disk
  244: have to be bigger than the sum of the RAM/disk needs of the dom0 and
  245: all the domUs.
  247: In 2018-05, trouble booting a dom0 was reported with 256M of RAM: with
  248: 512M it worked reliably.  This does not make sense, but if you see
  249: "not ELF" after Xen boots, try increasing dom0 RAM.
  251: Styles of dom0 operation
  252: ------------------------
  254: There are two basic ways to use Xen.  The traditional method is for
  255: the dom0 to do absolutely nothing other than providing support to some
  256: number of domUs.  Such a system was probably installed for the sole
  257: purpose of hosting domUs, and sits in a server room on a UPS.
  259: The other way is to put Xen under a normal-usage computer, so that the
  260: dom0 is what the computer would have been without Xen, perhaps a
  261: desktop or laptop.  Then, one can run domUs at will.  Purists will
  262: deride this as less secure than the previous approach, and for a
  263: computer whose purpose is to run domUs, they are right.  But Xen and a
  264: dom0 (without domUs) is not meaningfully less secure than the same
  265: things running without Xen.  One can boot Xen or boot regular NetBSD
  266: alternately with little problems, simply refraining from starting the
  267: Xen daemons when not running Xen.
  269: Note that NetBSD as dom0 does not support multiple CPUs.  This will
  270: limit the performance of the Xen/dom0 workstation approach.  In theory
  271: the only issue is that the "backend drivers" are not yet MPSAFE:
  274: Installation of NetBSD
  275: ----------------------
  277: First,
  278: [install NetBSD/amd64](/guide/inst/)
  279: just as you would if you were not using Xen.
  280: However, the partitioning approach is very important.
  282: If you want to use RAIDframe for the dom0, there are no special issues
  283: for Xen.  Typically one provides RAID storage for the dom0, and the
  284: domU systems are unaware of RAID.  The 2nd-stage loader bootxx_* skips
  285: over a RAID1 header to find /boot from a file system within a RAID
  286: partition; this is no different when booting Xen.
  288: There are 4 styles of providing backing storage for the virtual disks
  289: used by domUs: raw partitions, LVM, file-backed vnd(4), and SAN.
  291: With raw partitions, one has a disklabel (or gpt) partition sized for
  292: each virtual disk to be used by the domU.  (If you are able to predict
  293: how domU usage will evolve, please add an explanation to the HOWTO.
  294: Seriously, needs tend to change over time.)
  296: One can use [lvm(8)](/guide/lvm/) to create logical devices to use
  297: for domU disks.  This is almost as efficient as raw disk partitions
  298: and more flexible.  Hence raw disk partitions should typically not
  299: be used.
  301: One can use files in the dom0 file system, typically created by dd'ing
  302: /dev/zero to create a specific size.  This is somewhat less efficient,
  303: but very convenient, as one can cp the files for backup, or move them
  304: between dom0 hosts.
  306: Finally, in theory one can place the files backing the domU disks in a
  307: SAN.  (This is an invitation for someone who has done this to add a
  308: HOWTO page.)
  310: Installation of Xen
  311: -------------------
  313: In the dom0, install sysutils/xenkernel42 and sysutils/xentools42 from
  314: pkgsrc (or another matching pair).  See [the pkgsrc
  315: documentation]( for help with
  316: pkgsrc.  Ensure that your packages are recent; the HOWTO does not
  317: contemplate old builds.
  320: For Xen 3.1, support for HVM guests is in sysutils/xentool3-hvm.  More
  321: recent versions have HVM support integrated in the main xentools
  322: package.  It is entirely reasonable to run only PV guests.
  324: Next you need to install the selected Xen kernel itself, which is
  325: installed by pkgsrc as "/usr/pkg/xen*-kernel/xen.gz".  Copy it to /.
  326: For debugging, one may copy xen-debug.gz; this is conceptually similar
  327: to DIAGNOSTIC and DEBUG in NetBSD.  xen-debug.gz is basically only
  328: useful with a serial console.  Then, place a NetBSD XEN3_DOM0 kernel
  329: in /, copied from releasedir/amd64/binary/kernel/netbsd-XEN3_DOM0.gz
  330: of a NetBSD build.  If using i386, use
  331: releasedir/i386/binary/kernel/netbsd-XEN3PAE_DOM0.gz.  (If using Xen
  332: 3.1 and i386, you may use XEN3_DOM0 with the non-PAE Xen.  But you
  333: should not use Xen 3.1.)  Both xen and the NetBSD kernel may be (and
  334: typically are) left compressed.
  336: In a dom0, kernfs is mandatory for xend to communicate with the
  337: kernel, so ensure that /kern is in fstab.  (A standard NetBSD install
  338: should already mount /kern.)
  340: Because you already installed NetBSD, you have a working boot setup
  341: with an MBR bootblock, either bootxx_ffsv1 or bootxx_ffsv2 at the
  342: beginning of your root file system, have /boot, and likely also
  343: /boot.cfg.  (If not, fix before continuing!)
  345: Add a line to to /boot.cfg to boot Xen.  See boot.cfg(5) for an
  346: example.  The basic line is
  348:         menu=Xen:load /netbsd-XEN3_DOM0.gz console=pc;multiboot /xen.gz dom0_mem=512M
  350: which specifies that the dom0 should have 512M, leaving the rest to be
  351: allocated for domUs.  To use a serial console, use
  353:         menu=Xen:load /netbsd-XEN3_DOM0.gz;multiboot /xen.gz dom0_mem=512M console=com1 com1=9600,8n1
  355: which will use the first serial port for Xen (which counts starting
  356: from 1, unlike NetBSD which counts starting from 0), forcing
  357: speed/parity.  Because the NetBSD command line lacks a
  358: "console=pc" argument, it will use the default "xencons" console device,
  359: which directs the console I/O through Xen to the same console device Xen
  360: itself uses (in this case, the serial port).
  362: In an attempt to add performance, one can also add
  364:         dom0_max_vcpus=1 dom0_vcpus_pin
  366: to force only one vcpu to be provided (since NetBSD dom0 can't use
  367: more) and to pin that vcpu to a physical CPU.  TODO: benchmark this.
  369: Xen has [many boot
  370: options](,
  371: and other than dom0 memory and max_vcpus, they are generally not
  372: necessary.
  374: As with non-Xen systems, you should have a line to boot /netbsd (a
  375: kernel that works without Xen).  Consider a line to boot /netbsd.ok (a
  376: fallback version of the non-Xen kernel, updated manually when you are
  377: sure /netbsd is ok).  Consider also a line to boot fallback versions
  378: of Xen and the dom0 kernel, but note that non-Xen NetBSD can be used
  379: to resolve Xen booting issues.
  381: Probably you want a default=N line to choose Xen in the absence of
  382: intervention.
  384: Now, reboot so that you are running a DOM0 kernel under Xen, rather
  385: than GENERIC without Xen.
  387: Using grub (historic)
  388: ---------------------
  390: Before NetBSD's native bootloader could support Xen, the use of
  391: grub was recommended.  If necessary, see the
  392: [old grub information](/ports/xen/howto-grub).
  394: The [HowTo on Installing into
  395: RAID-1](
  396: explains how to set up booting a dom0 with Xen using grub with
  397: NetBSD's RAIDframe.  (This is obsolete with the use of NetBSD's native
  398: boot.  Now, just create a system with RAID-1, and alter /boot.cfg as
  399: described above.)
  401: Configuring Xen
  402: ---------------
  404: Xen logs will be in /var/log/xen.
  406: Now, you have a system that will boot Xen and the dom0 kernel, but not
  407: do anything else special.  Make sure that you have rebooted into Xen.
  408: There will be no domUs, and none can be started because you still have
  409: to configure the dom0 daemons.
  411: The daemons which should be run vary with Xen version and with whether
  412: one is using xm or xl.  The Xen 3.1, 3.3 and 4.1 packages use xm.  Xen
  413: 4.2 and up packages use xl.  To use xm with 4.2, edit xendomains to
  414: use xm instead.
  416: For 3.1 and 3.3, you should enable xend and xenbackendd:
  418:         xend=YES
  419:         xenbackendd=YES
  421: For 4.1 and up, you should enable xencommons.  Not enabling xencommons
  422: will result in a hang; it is necessary to hit ^C on the console to let
  423: the machine finish booting.  If you are using xm (default in 4.1, or
  424: if you changed xendomains in 4.2), you should also enable xend:
  426:         xend=YES # only if using xm, and only installed <= 4.2
  427:         xencommons=YES
  429: TODO: Recommend for/against xen-watchdog.
  431: After you have configured the daemons and either started them (in the
  432: order given) or rebooted, use xm or xl to inspect Xen's boot messages,
  433: available resources, and running domains.  An example with xl follows:
  435:         # xl dmesg
  436: 	[xen's boot info]
  437:         # xl info
  438: 	[available memory, etc.]
  439:         # xl list
  440:         Name              Id  Mem(MB)  CPU  State  Time(s)  Console
  441:         Domain-0           0       64    0  r----     58.1
  443: ### Issues with xencommons
  445: xencommons starts xenstored, which stores data on behalf of dom0 and
  446: domUs.  It does not currently work to stop and start xenstored.
  447: Certainly all domUs should be shutdown first, following the sort order
  448: of the rc.d scripts.  However, the dom0 sets up state with xenstored,
  449: and is not notified when xenstored exits, leading to not recreating
  450: the state when the new xenstored starts.  Until there's a mechanism to
  451: make this work, one should not expect to be able to restart xenstored
  452: (and thus xencommons).  There is currently no reason to expect that
  453: this will get fixed any time soon.
  455: ### No-longer needed advice about devices
  457: The installation of NetBSD should already have created devices for xen
  458: (xencons, xenevt, xsd_kva), but if they are not present, create them:
  460:         cd /dev && sh MAKEDEV xen
  462: anita (for testing NetBSD)
  463: --------------------------
  465: With the setup so far (assuming 4.2/xl), one should be able to run
  466: anita (see pkgsrc/misc/py-anita) to test NetBSD releases, by doing (as
  467: root, because anita must create a domU):
  469:         anita --vmm=xl test file:///usr/obj/i386/
  471: Alternatively, one can use --vmm=xm to use xm-based domU creation
  472: instead (and must, on Xen <= 4.1).   TODO: confirm that anita xl really works.
  474: Xen-specific NetBSD issues
  475: --------------------------
  477: There are (at least) two additional things different about NetBSD as a
  478: dom0 kernel compared to hardware.
  480: One is that the module ABI is different because some of the #defines
  481: change, so one must build modules for Xen.  As of netbsd-7, the build
  482: system does this automatically.  TODO: check this.  (Before building
  483: Xen modules was added, it was awkward to use modules to the point
  484: where it was considered that it did not work.)
  486: The other difference is that XEN3_DOM0 does not have exactly the same
  487: options as GENERIC.  While it is debatable whether or not this is a
  488: bug, users should be aware of this and can simply add missing config
  489: items if desired.
  491: Updating NetBSD in a dom0
  492: -------------------------
  494: This is just like updating NetBSD on bare hardware, assuming the new
  495: version supports the version of Xen you are running.  Generally, one
  496: replaces the kernel and reboots, and then overlays userland binaries
  497: and adjusts /etc.
  499: Note that one must update both the non-Xen kernel typically used for
  500: rescue purposes and the DOM0 kernel used with Xen.
  502: Converting from grub to /boot
  503: -----------------------------
  505: These instructions were [TODO: will be] used to convert a system from
  506: grub to /boot.  The system was originally installed in February of
  507: 2006 with a RAID1 setup and grub to boot Xen 2, and has been updated
  508: over time.  Before these commands, it was running NetBSD 6 i386, Xen
  509: 4.1 and grub, much like the message linked earlier in the grub
  510: section.
  512:         # Install MBR bootblocks on both disks. 
  513:         fdisk -i /dev/rwd0d
  514:         fdisk -i /dev/rwd1d
  515:         # Install NetBSD primary boot loader (/ is FFSv1) into RAID1 components.
  516:         installboot -v /dev/rwd0d /usr/mdec/bootxx_ffsv1
  517:         installboot -v /dev/rwd1d /usr/mdec/bootxx_ffsv1
  518:         # Install secondary boot loader
  519:         cp -p /usr/mdec/boot /
  520:         # Create boot.cfg following earlier guidance:
  521:         menu=Xen:load /netbsd-XEN3PAE_DOM0.gz console=pc;multiboot /xen.gz dom0_mem=512M
  522:         menu=Xen.ok:load /netbsd-XEN3PAE_DOM0.ok.gz console=pc;multiboot /xen.ok.gz dom0_mem=512M
  523:         menu=GENERIC:boot
  524:         menu=GENERIC single-user:boot -s
  525:         menu=GENERIC.ok:boot netbsd.ok
  526:         menu=GENERIC.ok single-user:boot netbsd.ok -s
  527:         menu=Drop to boot prompt:prompt
  528:         default=1
  529:         timeout=30
  531: TODO: actually do this and fix it if necessary.
  533: Upgrading Xen versions
  534: ---------------------
  536: Minor version upgrades are trivial.  Just rebuild/replace the
  537: xenkernel version and copy the new xen.gz to / (where /boot.cfg
  538: references it), and reboot.
  540: Major version upgrades are conceptually not difficult, but can run
  541: into all the issues found when installing Xen.  Assuming migration
  542: from 4.1 to 4.2, remove the xenkernel41 and xentools41 packages and
  543: install the xenkernel42 and xentools42 packages.  Copy the 4.2 xen.gz
  544: to /.
  546: Ensure that the contents of /etc/rc.d/xen* are correct.  Specifically,
  547: they must match the package you just installed and not be left over
  548: from some previous installation.
  550: Enable the correct set of daemons; see the configuring section above.
  551: (Upgrading from 3.x to 4.x without doing this will result in a hang.)
  553: Ensure that the domU config files are valid for the new version.
  554: Specifically, for 4.x remove autorestart=True, and ensure that disks
  555: are specified with numbers as the second argument, as the examples
  556: above show, and not NetBSD device names.
  558: Hardware known to work
  559: ----------------------
  561: Arguably, this section is misplaced, and there should be a page of
  562: hardware that runs NetBSD/amd64 well, with the mostly-well-founded
  563: assumption that NetBSD/xen runs fine on any modern hardware that
  564: NetBSD/amd64 runs well on.  Until then, we give motherboard/CPU (and
  565: sometimes RAM) pairs/triples to aid those choosing a motherboard.
  566: Note that Xen systems usually do not run X, so a listing here does not
  567: imply that X works at all.
  569:         Supermicro X9SRL-F, Xeon E5-1650 v2, 96 GiB ECC
  570:         Supermicro ??, Atom C2758 (8 core), 32 GiB ECC
  571:         ASUS M5A78L-M/USB3 AM3+ microATX, AMD Piledriver X8 4000MHz, 16 GiB ECC
  573: Older hardware:
  575:         Intel D915GEV, Pentium4 CPU 3.40GHz, 4GB 533MHz Synchronous DDR2
  576:         INTEL DG33FB, "Intel(R) Core(TM)2 Duo CPU     E6850  @ 3.00GHz"
  577:         INTEL DG33FB, "Intel(R) Core(TM)2 Duo CPU     E8400  @ 3.00GHz"
  579: Running Xen under qemu
  580: ----------------------
  582: The astute reader will note that this section is somewhat twisted.
  583: However, it can be useful to run Xen under qemu either because the
  584: version of NetBSD as a dom0 does not run on the hardware in use, or to
  585: generate automated test cases involving Xen.
  587: In 2015-01, the following combination was reported to mostly work:
  589:         host OS: NetBSD/amd64 6.1.4
  590:         qemu: 2.2.0 from pkgsrc
  591:         Xen kernel: xenkernel42-4.2.5nb1 from pkgsrc
  592:         dom0 kernel: NetBSD/amd64 6.1.5
  593:         Xen tools: xentools42-4.2.5 from pkgsrc
  595: See [PR 47720]( for a problem with dom0
  596: shutdown.
  598: Unprivileged domains (domU)
  599: ===========================
  601: This section describes general concepts about domUs.  It does not
  602: address specific domU operating systems or how to install them.  The
  603: config files for domUs are typically in /usr/pkg/etc/xen, and are
  604: typically named so that the file name, domU name and the domU's host
  605: name match.
  607: The domU is provided with CPU and memory by Xen, configured by the
  608: dom0.  The domU is provided with disk and network by the dom0,
  609: mediated by Xen, and configured in the dom0.
  611: Entropy in domUs can be an issue; physical disks and network are on
  612: the dom0.  NetBSD's /dev/random system works, but is often challenged.
  614: Config files
  615: ------------
  617: There is no good order to present config files and the concepts
  618: surrounding what is being configured.  We first show an example config
  619: file, and then in the various sections give details.
  621: See (at least in xentools41) /usr/pkg/share/examples/xen/xmexample*,
  622: for a large number of well-commented examples, mostly for running
  623: GNU/Linux.
  625: The following is an example minimal domain configuration file
  626: "/usr/pkg/etc/xen/foo".  It is (with only a name change) an actual
  627: known working config file on Xen 4.1 (NetBSD 5 amd64 dom0 and NetBSD 5
  628: i386 domU).  The domU serves as a network file server.
  630:         # -*- mode: python; -*-
  632:         kernel = "/netbsd-XEN3PAE_DOMU-i386-foo.gz"
  633:         memory = 1024
  634:         vif = [ 'mac=aa:00:00:d1:00:09,bridge=bridge0' ]
  635:         disk = [ 'file:/n0/xen/foo-wd0,0x0,w',
  636:                  'file:/n0/xen/foo-wd1,0x1,w' ]
  638: The domain will have the same name as the file.  The kernel has the
  639: host/domU name in it, so that on the dom0 one can update the various
  640: domUs independently.  The vif line causes an interface to be provided,
  641: with a specific mac address (do not reuse MAC addresses!), in bridge
  642: mode.  Two disks are provided, and they are both writable; the bits
  643: are stored in files and Xen attaches them to a vnd(4) device in the
  644: dom0 on domain creation.  The system treats xbd0 as the boot device
  645: without needing explicit configuration.
  647: By default xm looks for domain config files in /usr/pkg/etc/xen.  Note
  648: that "xm create" takes the name of a config file, while other commands
  649: take the name of a domain.  To create the domain, connect to the
  650: console, create the domain while attaching the console, shutdown the
  651: domain, and see if it has finished stopping, do (or xl with Xen >=
  652: 4.2):
  654:         xm create foo
  655:         xm console foo
  656:         xm create -c foo
  657:         xm shutdown foo
  658:         xm list
  660: Typing ^] will exit the console session.  Shutting down a domain is
  661: equivalent to pushing the power button; a NetBSD domU will receive a
  662: power-press event and do a clean shutdown.  Shutting down the dom0
  663: will trigger controlled shutdowns of all configured domUs.
  665: domU kernels
  666: ------------
  668: On a physical computer, the BIOS reads sector 0, and a chain of boot
  669: loaders finds and loads a kernel.  Normally this comes from the root
  670: file system.  With Xen domUs, the process is totally different.  The
  671: normal path is for the domU kernel to be a file in the dom0's
  672: file system.  At the request of the dom0, Xen loads that kernel into a
  673: new domU instance and starts execution.  While domU kernels can be
  674: anyplace, reasonable places to store domU kernels on the dom0 are in /
  675: (so they are near the dom0 kernel), in /usr/pkg/etc/xen (near the
  676: config files), or in /u0/xen (where the vdisks are).
  678: Note that loading the domU kernel from the dom0 implies that boot
  679: blocks, /boot, /boot.cfg, and so on are all ignored in the domU.
  680: See the VPS section near the end for discussion of alternate ways to
  681: obtain domU kernels.
  683: CPU and memory
  684: --------------
  686: A domain is provided with some number of vcpus, less than the number
  687: of CPUs seen by the hypervisor.  (For a dom0, this is controlled by
  688: the boot argument "dom0_max_vcpus=1".)  For a domU, it is controlled
  689: from the config file by the "vcpus = N" directive.
  691: A domain is provided with memory; this is controlled in the config
  692: file by "memory = N" (in megabytes).  In the straightforward case, the
  693: sum of the the memory allocated to the dom0 and all domUs must be less
  694: than the available memory.
  696: Xen also provides a "balloon" driver, which can be used to let domains
  697: use more memory temporarily.  TODO: Explain better, and explain how
  698: well it works with NetBSD.
  700: Virtual disks
  701: -------------
  703: With the file/vnd style, typically one creates a directory,
  704: e.g. /u0/xen, on a disk large enough to hold virtual disks for all
  705: domUs.  Then, for each domU disk, one writes zeros to a file that then
  706: serves to hold the virtual disk's bits; a suggested name is foo-xbd0
  707: for the first virtual disk for the domU called foo.  Writing zeros to
  708: the file serves two purposes.  One is that preallocating the contents
  709: improves performance.  The other is that vnd on sparse files has
  710: failed to work.  TODO: give working/notworking NetBSD versions for
  711: sparse vnd and gnats reference.  Note that the use of file/vnd for Xen
  712: is not really different than creating a file-backed virtual disk for
  713: some other purpose, except that xentools handles the vnconfig
  714: commands.  To create an empty 4G virtual disk, simply do
  716:         dd if=/dev/zero of=foo-xbd0 bs=1m count=4096
  718: Do not use qemu-img-xen, because this will create sparse file.  There
  719: have been recent (2015) reports of sparse vnd(4) devices causing
  720: lockups, but there is apparently no PR.
  722: With the lvm style, one creates logical devices.  They are then used
  723: similarly to vnds.  TODO: Add an example with lvm.
  725: In domU config files, the disks are defined as a sequence of 3-tuples.
  726: The first element is "method:/path/to/disk".  Common methods are
  727: "file:" for file-backed vnd. and "phy:" for something that is already
  728: a (TODO: character or block) device.
  730: The second element is an artifact of how virtual disks are passed to
  731: Linux, and a source of confusion with NetBSD Xen usage.  Linux domUs
  732: are given a device name to associate with the disk, and values like
  733: "hda1" or "sda1" are common.  In a NetBSD domU, the first disk appears
  734: as xbd0, the second as xbd1, and so on.  However, xm/xl demand a
  735: second argument.  The name given is converted to a major/minor by
  736: calling stat(2) on the name in /dev and this is passed to the domU.
  737: In the general case, the dom0 and domU can be different operating
  738: systems, and it is an unwarranted assumption that they have consistent
  739: numbering in /dev, or even that the dom0 OS has a /dev.  With NetBSD
  740: as both dom0 and domU, using values of 0x0 for the first disk and 0x1
  741: for the second works fine and avoids this issue.  For a GNU/Linux
  742: guest, one can create /dev/hda1 in /dev, or to pass 0x301 for
  743: /dev/hda1.
  745: The third element is "w" for writable disks, and "r" for read-only
  746: disks.
  748: Note that NetBSD by default creates only vnd[0123].  If you need more
  749: than 4 total virtual disks at a time, run e.g. "./MAKEDEV vnd4" in the
  750: dom0.
  752: Note that NetBSD by default creates only xbd[0123].  If you need more
  753: virtual disks in a domU, run e.g. "./MAKEDEV xbd4" in the domU.
  755: Virtual Networking
  756: ------------------
  758: Xen provides virtual Ethernets, each of which connects the dom0 and a
  759: domU.  For each virtual network, there is an interface "xvifN.M" in
  760: the dom0, and in domU index N, a matching interface xennetM (NetBSD
  761: name).  The interfaces behave as if there is an Ethernet with two
  762: adapters connected.  From this primitive, one can construct various
  763: configurations.  We focus on two common and useful cases for which
  764: there are existing scripts: bridging and NAT.
  766: With bridging (in the example above), the domU perceives itself to be
  767: on the same network as the dom0.  For server virtualization, this is
  768: usually best.  Bridging is accomplished by creating a bridge(4) device
  769: and adding the dom0's physical interface and the various xvifN.0
  770: interfaces to the bridge.  One specifies "bridge=bridge0" in the domU
  771: config file.  The bridge must be set up already in the dom0; an
  772: example /etc/ifconfig.bridge0 is:
  774:         create
  775:         up
  776:         !brconfig bridge0 add wm0
  778: With NAT, the domU perceives itself to be behind a NAT running on the
  779: dom0.  This is often appropriate when running Xen on a workstation.
  780: TODO: NAT appears to be configured by "vif = [ '' ]".
  782: The MAC address specified is the one used for the interface in the new
  783: domain.  The interface in dom0 will use this address XOR'd with
  784: 00:00:00:01:00:00.  Random MAC addresses are assigned if not given.
  786: Sizing domains
  787: --------------
  789: Modern x86 hardware has vast amounts of resources.  However, many
  790: virtual servers can function just fine on far less.  A system with
  791: 512M of RAM and a 4G disk can be a reasonable choice.  Note that it is
  792: far easier to adjust virtual resources than physical ones.  For
  793: memory, it's just a config file edit and a reboot.  For disk, one can
  794: create a new file and vnconfig it (or lvm), and then dump/restore,
  795: just like updating physical disks, but without having to be there and
  796: without those pesky connectors.
  798: Starting domains automatically
  799: ------------------------------
  801: To start domains foo at bar at boot and shut them down cleanly on dom0
  802: shutdown, in rc.conf add:
  804:         xendomains="foo bar"
  806: Note that earlier versions of the xentools41 xendomains rc.d script
  807: used xl, when one should use xm with 4.1.
  809: Creating specific unprivileged domains (domU)
  810: =============================================
  812: Creating domUs is almost entirely independent of operating system.  We
  813: have already presented the basics of config files.  Note that you must
  814: have already completed the dom0 setup so that "xl list" (or "xm list")
  815: works.
  817: Creating an unprivileged NetBSD domain (domU)
  818: ---------------------------------------------
  820: See the earlier config file, and adjust memory.  Decide on how much
  821: storage you will provide, and prepare it (file or lvm).
  823: While the kernel will be obtained from the dom0 file system, the same
  824: file should be present in the domU as /netbsd so that tools like
  825: savecore(8) can work.   (This is helpful but not necessary.)
  827: The kernel must be specifically for Xen and for use as a domU.  The
  828: i386 and amd64 provide the following kernels:
  830:         i386 XEN3_DOMU
  831:         i386 XEN3PAE_DOMU
  832:         amd64 XEN3_DOMU
  834: Unless using Xen 3.1 (and you shouldn't) with i386-mode Xen, you must
  835: use the PAE version of the i386 kernel.
  837: This will boot NetBSD, but this is not that useful if the disk is
  838: empty.  One approach is to unpack sets onto the disk outside of xen
  839: (by mounting it, just as you would prepare a physical disk for a
  840: system you can't run the installer on).
  842: A second approach is to run an INSTALL kernel, which has a miniroot
  843: and can load sets from the network.  To do this, copy the INSTALL
  844: kernel to / and change the kernel line in the config file to:
  846:         kernel = "/home/bouyer/netbsd-INSTALL_XEN3_DOMU"
  848: Then, start the domain as "xl create -c configname".
  850: Alternatively, if you want to install NetBSD/Xen with a CDROM image, the following
  851: line should be used in the config file.
  853:     disk = [ 'phy:/dev/wd0e,0x1,w', 'phy:/dev/cd0a,0x2,r' ]
  855: After booting the domain, the option to install via CDROM may be
  856: selected.  The CDROM device should be changed to `xbd1d`.
  858: Once done installing, "halt -p" the new domain (don't reboot or halt,
  859: it would reload the INSTALL_XEN3_DOMU kernel even if you changed the
  860: config file), switch the config file back to the XEN3_DOMU kernel,
  861: and start the new domain again. Now it should be able to use "root on
  862: xbd0a" and you should have a, functional NetBSD domU.
  864: TODO: check if this is still accurate.
  865: When the new domain is booting you'll see some warnings about *wscons*
  866: and the pseudo-terminals. These can be fixed by editing the files
  867: `/etc/ttys` and `/etc/wscons.conf`. You must disable all terminals in
  868: `/etc/ttys`, except *console*, like this:
  870:     console "/usr/libexec/getty Pc"         vt100   on secure
  871:     ttyE0   "/usr/libexec/getty Pc"         vt220   off secure
  872:     ttyE1   "/usr/libexec/getty Pc"         vt220   off secure
  873:     ttyE2   "/usr/libexec/getty Pc"         vt220   off secure
  874:     ttyE3   "/usr/libexec/getty Pc"         vt220   off secure
  876: Finally, all screens must be commented out from `/etc/wscons.conf`.
  878: It is also desirable to add
  880:         powerd=YES
  882: in rc.conf. This way, the domain will be properly shut down if
  883: `xm shutdown -R` or `xm shutdown -H` is used on the dom0.
  885: It is not strictly necessary to have a kernel (as /netbsd) in the domU
  886: file system.  However, various programs (e.g. netstat) will use that
  887: kernel to look up symbols to read from kernel virtual memory.  If
  888: /netbsd is not the running kernel, those lookups will fail.  (This is
  889: not really a Xen-specific issue, but because the domU kernel is
  890: obtained from the dom0, it is far more likely to be out of sync or
  891: missing with Xen.)
  893: Creating an unprivileged Linux domain (domU)
  894: --------------------------------------------
  896: Creating unprivileged Linux domains isn't much different from
  897: unprivileged NetBSD domains, but there are some details to know.
  899: First, the second parameter passed to the disk declaration (the '0x1' in
  900: the example below)
  902:     disk = [ 'phy:/dev/wd0e,0x1,w' ]
  904: does matter to Linux. It wants a Linux device number here (e.g. 0x300
  905: for hda).  Linux builds device numbers as: (major \<\< 8 + minor).
  906: So, hda1 which has major 3 and minor 1 on a Linux system will have
  907: device number 0x301.  Alternatively, devices names can be used (hda,
  908: hdb, ...)  as xentools has a table to map these names to devices
  909: numbers.  To export a partition to a Linux guest we can use:
  911:         disk = [ 'phy:/dev/wd0e,0x300,w' ]
  912:         root = "/dev/hda1 ro"
  914: and it will appear as /dev/hda on the Linux system, and be used as root
  915: partition.
  917: To install the Linux system on the partition to be exported to the
  918: guest domain, the following method can be used: install
  919: sysutils/e2fsprogs from pkgsrc.  Use mke2fs to format the partition
  920: that will be the root partition of your Linux domain, and mount it.
  921: Then copy the files from a working Linux system, make adjustments in
  922: `/etc` (fstab, network config).  It should also be possible to extract
  923: binary packages such as .rpm or .deb directly to the mounted partition
  924: using the appropriate tool, possibly running under NetBSD's Linux
  925: emulation.  Once the file system has been populated, umount it.  If
  926: desirable, the file system can be converted to ext3 using tune2fs -j.
  927: It should now be possible to boot the Linux guest domain, using one of
  928: the vmlinuz-\*-xenU kernels available in the Xen binary distribution.
  930: To get the Linux console right, you need to add:
  932:     extra = "xencons=tty1"
  934: to your configuration since not all Linux distributions auto-attach a
  935: tty to the xen console.
  937: Creating an unprivileged Solaris domain (domU)
  938: ----------------------------------------------
  940: See possibly outdated
  941: [Solaris domU instructions](/ports/xen/howto-solaris/).
  944: PCI passthrough: Using PCI devices in guest domains
  945: ---------------------------------------------------
  947: The dom0 can give other domains access to selected PCI
  948: devices. This can allow, for example, a non-privileged domain to have
  949: access to a physical network interface or disk controller.  However,
  950: keep in mind that giving a domain access to a PCI device most likely
  951: will give the domain read/write access to the whole physical memory,
  952: as PCs don't have an IOMMU to restrict memory access to DMA-capable
  953: device.  Also, it's not possible to export ISA devices to non-dom0
  954: domains, which means that the primary VGA adapter can't be exported.
  955: A guest domain trying to access the VGA registers will panic.
  957: If the dom0 is NetBSD, it has to be running Xen 3.1, as support has
  958: not been ported to later versions at this time.
  960: For a PCI device to be exported to a domU, is has to be attached to
  961: the "pciback" driver in dom0.  Devices passed to the dom0 via the
  962: pciback.hide boot parameter will attach to "pciback" instead of the
  963: usual driver.  The list of devices is specified as "(bus:dev.func)",
  964: where bus and dev are 2-digit hexadecimal numbers, and func a
  965: single-digit number:
  967:         pciback.hide=(00:0a.0)(00:06.0)
  969: pciback devices should show up in the dom0's boot messages, and the
  970: devices should be listed in the `/kern/xen/pci` directory.
  972: PCI devices to be exported to a domU are listed in the "pci" array of
  973: the domU's config file, with the format "0000:bus:dev.func".
  975:         pci = [ '0000:00:06.0', '0000:00:0a.0' ]
  977: In the domU an "xpci" device will show up, to which one or more pci
  978: buses will attach.  Then the PCI drivers will attach to PCI buses as
  979: usual.  Note that the default NetBSD DOMU kernels do not have "xpci"
  980: or any PCI drivers built in by default; you have to build your own
  981: kernel to use PCI devices in a domU.  Here's a kernel config example;
  982: note that only the "xpci" lines are unusual.
  984:         include         "arch/i386/conf/XEN3_DOMU"
  986:         # Add support for PCI buses to the XEN3_DOMU kernel
  987:         xpci* at xenbus ?
  988:         pci* at xpci ?
  990:         # PCI USB controllers
  991:         uhci*   at pci? dev ? function ?        # Universal Host Controller (Intel)
  993:         # USB bus support
  994:         usb*    at uhci?
  996:         # USB Hubs
  997:         uhub*   at usb?
  998:         uhub*   at uhub? port ? configuration ? interface ?
 1000:         # USB Mass Storage
 1001:         umass*  at uhub? port ? configuration ? interface ?
 1002:         wd*     at umass?
 1003:         # SCSI controllers
 1004:         ahc*    at pci? dev ? function ?        # Adaptec [23]94x, aic78x0 SCSI
 1006:         # SCSI bus support (for both ahc and umass)
 1007:         scsibus* at scsi?
 1009:         # SCSI devices
 1010:         sd*     at scsibus? target ? lun ?      # SCSI disk drives
 1011:         cd*     at scsibus? target ? lun ?      # SCSI CD-ROM drives
 1014: NetBSD as a domU in a VPS
 1015: =========================
 1017: The bulk of the HOWTO is about using NetBSD as a dom0 on your own
 1018: hardware.  This section explains how to deal with Xen in a domU as a
 1019: virtual private server where you do not control or have access to the
 1020: dom0.  This is not intended to be an exhaustive list of VPS providers;
 1021: only a few are mentioned that specifically support NetBSD.
 1023: VPS operators provide varying degrees of access and mechanisms for
 1024: configuration.  The big issue is usually how one controls which kernel
 1025: is booted, because the kernel is nominally in the dom0 file system (to
 1026: which VPS users do not normally have access).  A second issue is how
 1027: to install NetBSD.
 1028: A VPS user may want to compile a kernel for security updates, to run
 1029: npf, run IPsec, or any other reason why someone would want to change
 1030: their kernel.
 1032: One approach is to have an administrative interface to upload a kernel,
 1033: or to select from a prepopulated list.  Other approaches are pygrub
 1034: (deprecated) and pvgrub, which are ways to have a bootloader obtain a
 1035: kernel from the domU file system.  This is closer to a regular physical
 1036: computer, where someone who controls a machine can replace the kernel.
 1038: A second issue is multiple CPUs.  With NetBSD 6, domUs support
 1039: multiple vcpus, and it is typical for VPS providers to enable multiple
 1040: CPUs for NetBSD domUs.
 1042: pygrub
 1043: -------
 1045: pygrub runs in the dom0 and looks into the domU file system.  This
 1046: implies that the domU must have a kernel in a file system in a format
 1047: known to pygrub.  As of 2014, pygrub seems to be of mostly historical
 1048: interest.
 1050: pvgrub
 1051: ------
 1053: pvgrub is a version of grub that uses PV operations instead of BIOS
 1054: calls.  It is booted from the dom0 as the domU kernel, and then reads
 1055: /grub/menu.lst and loads a kernel from the domU file system.
 1057: [Panix]( lets users use pvgrub.  Panix reports
 1058: that pvgrub works with FFsv2 with 16K/2K and 32K/4K block/frag sizes
 1059: (and hence with defaults from "newfs -O 2").  See [Panix's pvgrub
 1060: page](, which describes only
 1061: Linux but should be updated to cover NetBSD :-).
 1063: []( also lets users with pvgrub to boot
 1064: their own kernel.  See then [ NetBSD
 1065: HOWTO](
 1066: (which is in need of updating).
 1068: It appears that [grub's FFS
 1069: code](
 1070: does not support all aspects of modern FFS, but there are also reports
 1071: that FFSv2 works fine.  At prgmr, typically one has an ext2 or FAT
 1072: partition for the kernel with the intent that grub can understand it,
 1073: which leads to /netbsd not being the actual kernel.  One must remember
 1074: to update the special boot partition.
 1076: Amazon
 1077: ------
 1079: See the [Amazon EC2 page](/amazon_ec2/).
 1081: TODO items for improving NetBSD/xen
 1082: ===================================
 1084: * Make the NetBSD dom0 kernel work with SMP.
 1085: * Test the Xen 4.5 packages adequately to be able to recommend them as
 1086:   the standard approach.
 1087: * Get PCI passthrough working on Xen 4.5
 1088: * Get pvgrub into pkgsrc, either via xentools or separately.
 1089: * grub
 1090:   * Check/add support to pkgsrc grub2 for UFS2 and arbitrary
 1091:     fragsize/blocksize (UFS2 support may be present; the point is to
 1092:     make it so that with any UFS1/UFS2 file system setup that works
 1093:     with NetBSD grub will also work).
 1094:     See [pkg/40258](
 1095:   * Push patches upstream.
 1096:   * Get UFS2 patches into pvgrub.
 1097: * Add support for PV ops to a version of /boot, and make it usable as
 1098:   a kernel in Xen, similar to pvgrub.
 1100: Random pointers
 1101: ===============
 1103: This section contains links from elsewhere not yet integrated into the
 1104: HOWTO, and other guides.
 1106: *
 1107: *
 1108: *

CVSweb for NetBSD wikisrc <> software: FreeBSD-CVSweb