wikisrc/ports/xen/howto.mdwn - view

File: [NetBSD Developer Wiki] / wikisrc / ports / xen / howto.mdwn
Revision 1.145: download - view: text, annotated - select for diffs
Thu Jul 26 10:36:26 2018 UTC (16 months, 2 weeks ago) by maxv
Branches: MAIN
CVS tags: HEAD

Reduce the noise:

 - Sum up the "Xen" section in a single, readable table.

 - Remove the "Status" section, it is pointless and probably
   inaccurate.

 - Remove the "Using npf" section. We do have support for kernel
   modules now, and basically, that kind of stuff shouldn't be in
   a Xen howto.

 - Remove the "missing kernel modules support" from the TODO list,
   we do have that now. By the way, we should probably remove this
   TODO section. This page is supposed to be a HowTo, not a page
   where we put some random things.

1: [[!meta title="Xen HowTo"]] 2: 3: Introduction 4: ============ 5: 6: [![[Xen 7: screenshot]](https://www.netbsd.org/gallery/in-Action/hubertf-xens.png)](https://www.netbsd.org/gallery/in-Action/hubertf-xen.png) 8: 9: Xen is a hypervisor (or virtual machine monitor) for x86 hardware 10: (i686-class or higher), which supports running multiple guest 11: operating systems on a single physical machine. Xen is a Type 1 or 12: bare-metal hypervisor; one uses the Xen kernel to control the CPU, 13: memory and console, a dom0 operating system which mediates access to 14: other hardware (e.g., disks, network, USB), and one or more domU 15: operating systems which operate in an unprivileged virtualized 16: environment. IO requests from the domU systems are forwarded by the 17: hypervisor (Xen) to the dom0 to be fulfilled. 18: 19: Xen supports two styles of guests. The original is Para-Virtualized 20: (PV) which means that the guest OS does not attempt to access hardware 21: directly, but instead makes hypercalls to the hypervisor. This is 22: analogous to a user-space program making system calls. (The dom0 23: operating system uses PV calls for some functions, such as updating 24: memory mapping page tables, but has direct hardware access for disk 25: and network.) PV guests must be specifically coded for Xen. 26: 27: The more recent style is HVM, which means that the guest does not have 28: code for Xen and need not be aware that it is running under Xen. 29: Attempts to access hardware registers are trapped and emulated. This 30: style is less efficient but can run unmodified guests. 31: 32: Generally any machine that runs NetBSD/amd64 will work with Xen and PV 33: guests. In theory i386 computers (without x86_64/amd64 support) can 34: be used for Xen <= 4.2, but we have no recent reports of this working 35: (this is a hint). For HVM guests, hardware support is needed, but it 36: is common on recent machines. For Intel CPUs, one needs the VT-x 37: extension, shown in "cpuctl identify 0" as VMX. For AMD CPUs, one 38: needs the AMD-V extensions, shown in "cpuctl identify 0" as SVM. 39: There are further features for IOMMU virtualization, Intel's VT-d and 40: AMD's AMD-Vi. TODO: Explain whether Xen on NetBSD makes use of these 41: features. TODO: Review by someone who really understands this. 42: 43: Note that a FreeBSD dom0 requires VT-x and VT-d (or equivalent); this 44: is because the FreeBSD dom0 does not run in PV mode. 45: 46: At boot, the dom0 kernel is loaded as a module with Xen as the kernel. 47: The dom0 can start one or more domUs. (Booting is explained in detail 48: in the dom0 section.) 49: 50: NetBSD supports Xen in that it can serve as dom0, be used as a domU, 51: and that Xen kernels and tools are available in pkgsrc. This HOWTO 52: attempts to address both the case of running a NetBSD dom0 on hardware 53: and running domUs under it (NetBSD and other), and also running NetBSD 54: as a domU in a VPS. 55: 56: Xen 3.1 in pkgsrc used to support "PCI passthrough", which means that 57: specific PCI devices can be made available to a specific domU instead 58: of the dom0. This can be useful to let a domU run X11, or access some 59: network interface or other peripheral. 60: 61: NetBSD 6 and earlier supported Xen 2; support was removed from NetBSD 62: 7. Xen 2 has been removed from pkgsrc. 63: 64: Prerequisites 65: ------------- 66: 67: Installing NetBSD/Xen is not extremely difficult, but it is more 68: complex than a normal installation of NetBSD. 69: In general, this HOWTO is occasionally overly restrictive about how 70: things must be done, guiding the reader to stay on the established 71: path when there are no known good reasons to stray. 72: 73: This HOWTO presumes a basic familiarity with the Xen system 74: architecture, with installing NetBSD on i386/amd64 hardware, and with 75: installing software from pkgsrc. See also the [Xen 76: website](http://www.xenproject.org/). 77: 78: Versions of Xen and NetBSD 79: ========================== 80: 81: Most of the installation concepts and instructions are independent 82: of Xen version and NetBSD version. This section gives advice on 83: which version to choose. Versions not in pkgsrc and older unsupported 84: versions of NetBSD are intentionally ignored. 85: 86: The term "amd64" is used to refer to both the NetBSD port and to the 87: hardware architecture on which it runs. (Such hardware is made by 88: both Intel and AMD, and in 2016 a normal PC has this CPU 89: architecture.) 90: 91: Xen 92: --- 93: 94: In NetBSD, Xen is provided in pkgsrc, via matching pairs of packages 95: xenkernel and xentools. We will refer only to the kernel versions, 96: but note that both packages must be installed together and must have 97: matching versions. 98: 99: Versions available in pkgsrc: 100: 101: [[!table data=""" 102: Xen Version |Package Name |Xen CPU Support |EOL'ed By Upstream 103: 4.2 |xenkernel42 |32bit, 64bit |Yes 104: 4.5 |xenkernel45 |64bit |Yes 105: 4.6 |xenkernel46 |64bit |Partially 106: 4.8 |xenkernel48 |64bit |No 107: 4.11 |xenkernel411 |64bit |No 108: """]] 109: 110: See also the [Xen Security Advisory page](http://xenbits.xen.org/xsa/). 111: 112: Note: Xen 4.2 was the last version to support 32bit CPUs. 113: 114: Note that NetBSD support is called XEN3. It works with Xen 3 and Xen 115: 4 because the hypercall interface has been stable. 116: 117: Xen command program 118: ------------------- 119: 120: Early Xen used a program called xm to manipulate the system from the 121: dom0. Starting in 4.1, a replacement program with similar behavior 122: called xl is provided, but it does not work well in 4.1. In 4.2, both 123: xm and xl work fine. 4.4 is the last version that has xm. 124: 125: You must make a global choice to use xm or xl, because it affects not 126: only which command you use, but the command used by rc.d scripts 127: (specifically xendomains) and which daemons should be run. The 128: xentools packages provide xm for 3.1, 3.3 and 4.1 and xl for 4.2 and up. 129: 130: In 4.2, you can choose to use xm by simply changing the ctl_command 131: variable and setting xend=YES in rc.conf. 132: 133: With xl, virtual devices are configured in parallel, which can cause 134: problems if they are written assuming serial operation (e.g., updating 135: firewall rules without explicit locking). There is now locking for 136: the provided scripts, which works for normal casses (e.g, file-backed 137: xbd, where a vnd must be allocated). But, as of 201612, it has not 138: been adequately tested for a complex custom setup with a large number 139: of interfaces. 140: 141: NetBSD 142: ------ 143: 144: The netbsd-7, netbsd-8, and -current branches are all reasonable 145: choices, with more or less the same considerations for non-Xen use. 146: Therefore, netbsd-7 is recommended as the stable version of the most 147: recent release for production use. (Note that netbsd-7 (and therefore 148: 8/current) have a important scheduler fix (in November of 2015) 149: affecting contention between dom0 and domUs; see 150: https://releng.netbsd.org/cgi-bin/req-7.cgi?show=1040 for a 151: description.) For production, netbsd-7 is appropriate. For learning, 152: netbsd-8 is appropriate. For developing Xen, netbsd-current may be 153: appropriate. 154: 155: As of NetBSD 6, a NetBSD domU will support multiple vcpus. There is 156: no SMP support for NetBSD as dom0. (The dom0 itself doesn't really 157: need SMP for dom0 functions; the lack of support is really a problem 158: when using a dom0 as a normal computer.) 159: 160: Architecture 161: ------------ 162: 163: Xen itself can run on i386 (Xen < 4.2) or amd64 hardware (all Xen 164: versions). (Practically, almost any computer where one would want to 165: run Xen today supports amd64.) 166: 167: Xen, the dom0 system, and each domU system can be either i386 or 168: amd64. When building a xenkernel package, one obtains an i386 Xen 169: kernel on an i386 host, and an amd64 Xen kernel on an amd64 host. If 170: the Xen kernel is i386, then the dom0 kernel and all domU kernels must 171: be i386. With an amd64 Xen kernel, an amd64 dom0 kernel is known to 172: work, and an i386 dom0 kernel should in theory work. An amd64 173: Xen/dom0 is known to support both i386 and amd64 domUs. 174: 175: i386 dom0 and domU kernels must be PAE (except for an i386 Xen 3.1 176: kernel, where one can use non-PAE for dom0 and all domUs); PAE kernels 177: are included in the NetBSD default build. (Note that emacs (at least) 178: fails if run on i386 with PAE when built without, and vice versa, 179: presumably due to bugs in the undump code.) 180: 181: Because of the above, the standard approach is to use an amd64 Xen 182: kernel and NetBSD/amd64 for the dom0. For domUs, NetBSD/i386 (with 183: the PAE kernel) and NetBSD/amd64 are in widespread use, and there is 184: little to no Xen-specific reason to prefer one over the other. 185: 186: Note that to use an i386 dom0 with Xen 4.5 or higher, one must build 187: (or obtain from pre-built packages) an amd64 Xen kernel and install 188: that on the system. (One must also use a PAE i386 kernel, but this is 189: also required with an i386 Xen kernel.). Almost no one in the 190: NetBSD/Xen community does this, and the standard, well-tested, 191: approach is to use an amd64 dom0. 192: 193: A [posting on 194: xen-devel](https://lists.xen.org/archives/html/xen-devel/2012-07/msg00085.html) 195: explained that PV system call overhead was higher on amd64, and thus 196: there is some notion that i386 guests are faster. It goes on to 197: caution that the total situation is complex and not entirely 198: understood. On top of that caution, the post is about Linux, not 199: NetBSD. TODO: Include link to benchmarks, if someone posts them. 200: 201: Stability 202: --------- 203: 204: Mostly, NetBSD as a dom0 or domU is quite stable. However, just like every 205: other architecture, there are some open PRs indicating problems. 206: 207: Note also that there are issues with sparse vnd(4) instances, but 208: these are not about Xen -- they just are noticed with sparse vnd(4) 209: instances in support of virtual disks in a dom0. 210: 211: Recommendation 212: -------------- 213: 214: Therefore, this HOWTO recommends running xenkernel46, xl, the NetBSD 7 215: stable branch, and therefore to use an amd64 kernel as the dom0. 216: Either the i386PAE or amd64 version of NetBSD may be used as domUs. 217: 218: A tentative replacement recommendation is xenkernel48, xl, and NetBSD 219: 8. 220: 221: Because bugs are fixed quite often, and because of Xen security 222: advisories, it is good to stay up to date with NetBSD (tracking a 223: stable branch), with the Xen kernel (tracking a Xen version via 224: pkgsrc), and with the Xen tools. Specifically, NetBSD (-7 and 225: -current) got an important fix affecting dom0/domU timesharing in 226: November, 2015, and xentools46 got a fix to enable Ubuntu guests to 227: boot in December, 2016. 228: 229: NetBSD as a dom0 230: ================ 231: 232: NetBSD can be used as a dom0 and works very well. The following 233: sections address installation, updating NetBSD, and updating Xen. 234: Note that it doesn't make sense to talk about installing a dom0 OS 235: without also installing Xen itself. We first address installing 236: NetBSD, which is not yet a dom0, and then adding Xen, pivoting the 237: NetBSD install to a dom0 install by just changing the kernel and boot 238: configuration. 239: 240: For experimenting with Xen, a machine with as little as 1G of RAM and 241: 100G of disk can work. For running many domUs in productions, far 242: more will be needed; e.g. 4-8G and 1T of disk is reasonable for a 243: half-dozen domUs of 512M and 32G each. Basically, the RAM and disk 244: have to be bigger than the sum of the RAM/disk needs of the dom0 and 245: all the domUs. 246: 247: In 2018-05, trouble booting a dom0 was reported with 256M of RAM: with 248: 512M it worked reliably. This does not make sense, but if you see 249: "not ELF" after Xen boots, try increasing dom0 RAM. 250: 251: Styles of dom0 operation 252: ------------------------ 253: 254: There are two basic ways to use Xen. The traditional method is for 255: the dom0 to do absolutely nothing other than providing support to some 256: number of domUs. Such a system was probably installed for the sole 257: purpose of hosting domUs, and sits in a server room on a UPS. 258: 259: The other way is to put Xen under a normal-usage computer, so that the 260: dom0 is what the computer would have been without Xen, perhaps a 261: desktop or laptop. Then, one can run domUs at will. Purists will 262: deride this as less secure than the previous approach, and for a 263: computer whose purpose is to run domUs, they are right. But Xen and a 264: dom0 (without domUs) is not meaningfully less secure than the same 265: things running without Xen. One can boot Xen or boot regular NetBSD 266: alternately with little problems, simply refraining from starting the 267: Xen daemons when not running Xen. 268: 269: Note that NetBSD as dom0 does not support multiple CPUs. This will 270: limit the performance of the Xen/dom0 workstation approach. In theory 271: the only issue is that the "backend drivers" are not yet MPSAFE: 272: https://mail-index.netbsd.org/netbsd-users/2014/08/29/msg015195.html 273: 274: Installation of NetBSD 275: ---------------------- 276: 277: First, 278: [install NetBSD/amd64](/guide/inst/) 279: just as you would if you were not using Xen. 280: However, the partitioning approach is very important. 281: 282: If you want to use RAIDframe for the dom0, there are no special issues 283: for Xen. Typically one provides RAID storage for the dom0, and the 284: domU systems are unaware of RAID. The 2nd-stage loader bootxx_* skips 285: over a RAID1 header to find /boot from a file system within a RAID 286: partition; this is no different when booting Xen. 287: 288: There are 4 styles of providing backing storage for the virtual disks 289: used by domUs: raw partitions, LVM, file-backed vnd(4), and SAN. 290: 291: With raw partitions, one has a disklabel (or gpt) partition sized for 292: each virtual disk to be used by the domU. (If you are able to predict 293: how domU usage will evolve, please add an explanation to the HOWTO. 294: Seriously, needs tend to change over time.) 295: 296: One can use [lvm(8)](/guide/lvm/) to create logical devices to use 297: for domU disks. This is almost as efficient as raw disk partitions 298: and more flexible. Hence raw disk partitions should typically not 299: be used. 300: 301: One can use files in the dom0 file system, typically created by dd'ing 302: /dev/zero to create a specific size. This is somewhat less efficient, 303: but very convenient, as one can cp the files for backup, or move them 304: between dom0 hosts. 305: 306: Finally, in theory one can place the files backing the domU disks in a 307: SAN. (This is an invitation for someone who has done this to add a 308: HOWTO page.) 309: 310: Installation of Xen 311: ------------------- 312: 313: In the dom0, install sysutils/xenkernel42 and sysutils/xentools42 from 314: pkgsrc (or another matching pair). See [the pkgsrc 315: documentation](https://www.NetBSD.org/docs/pkgsrc/) for help with 316: pkgsrc. Ensure that your packages are recent; the HOWTO does not 317: contemplate old builds. 318: 319: 320: For Xen 3.1, support for HVM guests is in sysutils/xentool3-hvm. More 321: recent versions have HVM support integrated in the main xentools 322: package. It is entirely reasonable to run only PV guests. 323: 324: Next you need to install the selected Xen kernel itself, which is 325: installed by pkgsrc as "/usr/pkg/xen*-kernel/xen.gz". Copy it to /. 326: For debugging, one may copy xen-debug.gz; this is conceptually similar 327: to DIAGNOSTIC and DEBUG in NetBSD. xen-debug.gz is basically only 328: useful with a serial console. Then, place a NetBSD XEN3_DOM0 kernel 329: in /, copied from releasedir/amd64/binary/kernel/netbsd-XEN3_DOM0.gz 330: of a NetBSD build. If using i386, use 331: releasedir/i386/binary/kernel/netbsd-XEN3PAE_DOM0.gz. (If using Xen 332: 3.1 and i386, you may use XEN3_DOM0 with the non-PAE Xen. But you 333: should not use Xen 3.1.) Both xen and the NetBSD kernel may be (and 334: typically are) left compressed. 335: 336: In a dom0, kernfs is mandatory for xend to communicate with the 337: kernel, so ensure that /kern is in fstab. (A standard NetBSD install 338: should already mount /kern.) 339: 340: Because you already installed NetBSD, you have a working boot setup 341: with an MBR bootblock, either bootxx_ffsv1 or bootxx_ffsv2 at the 342: beginning of your root file system, have /boot, and likely also 343: /boot.cfg. (If not, fix before continuing!) 344: 345: Add a line to to /boot.cfg to boot Xen. See boot.cfg(5) for an 346: example. The basic line is 347: 348: menu=Xen:load /netbsd-XEN3_DOM0.gz console=pc;multiboot /xen.gz dom0_mem=512M 349: 350: which specifies that the dom0 should have 512M, leaving the rest to be 351: allocated for domUs. To use a serial console, use 352: 353: menu=Xen:load /netbsd-XEN3_DOM0.gz;multiboot /xen.gz dom0_mem=512M console=com1 com1=9600,8n1 354: 355: which will use the first serial port for Xen (which counts starting 356: from 1, unlike NetBSD which counts starting from 0), forcing 357: speed/parity. Because the NetBSD command line lacks a 358: "console=pc" argument, it will use the default "xencons" console device, 359: which directs the console I/O through Xen to the same console device Xen 360: itself uses (in this case, the serial port). 361: 362: In an attempt to add performance, one can also add 363: 364: dom0_max_vcpus=1 dom0_vcpus_pin 365: 366: to force only one vcpu to be provided (since NetBSD dom0 can't use 367: more) and to pin that vcpu to a physical CPU. TODO: benchmark this. 368: 369: Xen has [many boot 370: options](http://xenbits.xenproject.org/docs/4.5-testing/misc/xen-command-line.html), 371: and other than dom0 memory and max_vcpus, they are generally not 372: necessary. 373: 374: As with non-Xen systems, you should have a line to boot /netbsd (a 375: kernel that works without Xen). Consider a line to boot /netbsd.ok (a 376: fallback version of the non-Xen kernel, updated manually when you are 377: sure /netbsd is ok). Consider also a line to boot fallback versions 378: of Xen and the dom0 kernel, but note that non-Xen NetBSD can be used 379: to resolve Xen booting issues. 380: 381: Probably you want a default=N line to choose Xen in the absence of 382: intervention. 383: 384: Now, reboot so that you are running a DOM0 kernel under Xen, rather 385: than GENERIC without Xen. 386: 387: Using grub (historic) 388: --------------------- 389: 390: Before NetBSD's native bootloader could support Xen, the use of 391: grub was recommended. If necessary, see the 392: [old grub information](/ports/xen/howto-grub). 393: 394: The [HowTo on Installing into 395: RAID-1](https://mail-index.NetBSD.org/port-xen/2006/03/01/0010.html) 396: explains how to set up booting a dom0 with Xen using grub with 397: NetBSD's RAIDframe. (This is obsolete with the use of NetBSD's native 398: boot. Now, just create a system with RAID-1, and alter /boot.cfg as 399: described above.) 400: 401: Configuring Xen 402: --------------- 403: 404: Xen logs will be in /var/log/xen. 405: 406: Now, you have a system that will boot Xen and the dom0 kernel, but not 407: do anything else special. Make sure that you have rebooted into Xen. 408: There will be no domUs, and none can be started because you still have 409: to configure the dom0 daemons. 410: 411: The daemons which should be run vary with Xen version and with whether 412: one is using xm or xl. The Xen 3.1, 3.3 and 4.1 packages use xm. Xen 413: 4.2 and up packages use xl. To use xm with 4.2, edit xendomains to 414: use xm instead. 415: 416: For 3.1 and 3.3, you should enable xend and xenbackendd: 417: 418: xend=YES 419: xenbackendd=YES 420: 421: For 4.1 and up, you should enable xencommons. Not enabling xencommons 422: will result in a hang; it is necessary to hit ^C on the console to let 423: the machine finish booting. If you are using xm (default in 4.1, or 424: if you changed xendomains in 4.2), you should also enable xend: 425: 426: xend=YES # only if using xm, and only installed <= 4.2 427: xencommons=YES 428: 429: TODO: Recommend for/against xen-watchdog. 430: 431: After you have configured the daemons and either started them (in the 432: order given) or rebooted, use xm or xl to inspect Xen's boot messages, 433: available resources, and running domains. An example with xl follows: 434: 435: # xl dmesg 436: [xen's boot info] 437: # xl info 438: [available memory, etc.] 439: # xl list 440: Name Id Mem(MB) CPU State Time(s) Console 441: Domain-0 0 64 0 r---- 58.1 442: 443: ### Issues with xencommons 444: 445: xencommons starts xenstored, which stores data on behalf of dom0 and 446: domUs. It does not currently work to stop and start xenstored. 447: Certainly all domUs should be shutdown first, following the sort order 448: of the rc.d scripts. However, the dom0 sets up state with xenstored, 449: and is not notified when xenstored exits, leading to not recreating 450: the state when the new xenstored starts. Until there's a mechanism to 451: make this work, one should not expect to be able to restart xenstored 452: (and thus xencommons). There is currently no reason to expect that 453: this will get fixed any time soon. 454: 455: ### No-longer needed advice about devices 456: 457: The installation of NetBSD should already have created devices for xen 458: (xencons, xenevt, xsd_kva), but if they are not present, create them: 459: 460: cd /dev && sh MAKEDEV xen 461: 462: anita (for testing NetBSD) 463: -------------------------- 464: 465: With the setup so far (assuming 4.2/xl), one should be able to run 466: anita (see pkgsrc/misc/py-anita) to test NetBSD releases, by doing (as 467: root, because anita must create a domU): 468: 469: anita --vmm=xl test file:///usr/obj/i386/ 470: 471: Alternatively, one can use --vmm=xm to use xm-based domU creation 472: instead (and must, on Xen <= 4.1). TODO: confirm that anita xl really works. 473: 474: Xen-specific NetBSD issues 475: -------------------------- 476: 477: There are (at least) two additional things different about NetBSD as a 478: dom0 kernel compared to hardware. 479: 480: One is that the module ABI is different because some of the #defines 481: change, so one must build modules for Xen. As of netbsd-7, the build 482: system does this automatically. TODO: check this. (Before building 483: Xen modules was added, it was awkward to use modules to the point 484: where it was considered that it did not work.) 485: 486: The other difference is that XEN3_DOM0 does not have exactly the same 487: options as GENERIC. While it is debatable whether or not this is a 488: bug, users should be aware of this and can simply add missing config 489: items if desired. 490: 491: Updating NetBSD in a dom0 492: ------------------------- 493: 494: This is just like updating NetBSD on bare hardware, assuming the new 495: version supports the version of Xen you are running. Generally, one 496: replaces the kernel and reboots, and then overlays userland binaries 497: and adjusts /etc. 498: 499: Note that one must update both the non-Xen kernel typically used for 500: rescue purposes and the DOM0 kernel used with Xen. 501: 502: Converting from grub to /boot 503: ----------------------------- 504: 505: These instructions were [TODO: will be] used to convert a system from 506: grub to /boot. The system was originally installed in February of 507: 2006 with a RAID1 setup and grub to boot Xen 2, and has been updated 508: over time. Before these commands, it was running NetBSD 6 i386, Xen 509: 4.1 and grub, much like the message linked earlier in the grub 510: section. 511: 512: # Install MBR bootblocks on both disks. 513: fdisk -i /dev/rwd0d 514: fdisk -i /dev/rwd1d 515: # Install NetBSD primary boot loader (/ is FFSv1) into RAID1 components. 516: installboot -v /dev/rwd0d /usr/mdec/bootxx_ffsv1 517: installboot -v /dev/rwd1d /usr/mdec/bootxx_ffsv1 518: # Install secondary boot loader 519: cp -p /usr/mdec/boot / 520: # Create boot.cfg following earlier guidance: 521: menu=Xen:load /netbsd-XEN3PAE_DOM0.gz console=pc;multiboot /xen.gz dom0_mem=512M 522: menu=Xen.ok:load /netbsd-XEN3PAE_DOM0.ok.gz console=pc;multiboot /xen.ok.gz dom0_mem=512M 523: menu=GENERIC:boot 524: menu=GENERIC single-user:boot -s 525: menu=GENERIC.ok:boot netbsd.ok 526: menu=GENERIC.ok single-user:boot netbsd.ok -s 527: menu=Drop to boot prompt:prompt 528: default=1 529: timeout=30 530: 531: TODO: actually do this and fix it if necessary. 532: 533: Upgrading Xen versions 534: --------------------- 535: 536: Minor version upgrades are trivial. Just rebuild/replace the 537: xenkernel version and copy the new xen.gz to / (where /boot.cfg 538: references it), and reboot. 539: 540: Major version upgrades are conceptually not difficult, but can run 541: into all the issues found when installing Xen. Assuming migration 542: from 4.1 to 4.2, remove the xenkernel41 and xentools41 packages and 543: install the xenkernel42 and xentools42 packages. Copy the 4.2 xen.gz 544: to /. 545: 546: Ensure that the contents of /etc/rc.d/xen* are correct. Specifically, 547: they must match the package you just installed and not be left over 548: from some previous installation. 549: 550: Enable the correct set of daemons; see the configuring section above. 551: (Upgrading from 3.x to 4.x without doing this will result in a hang.) 552: 553: Ensure that the domU config files are valid for the new version. 554: Specifically, for 4.x remove autorestart=True, and ensure that disks 555: are specified with numbers as the second argument, as the examples 556: above show, and not NetBSD device names. 557: 558: Hardware known to work 559: ---------------------- 560: 561: Arguably, this section is misplaced, and there should be a page of 562: hardware that runs NetBSD/amd64 well, with the mostly-well-founded 563: assumption that NetBSD/xen runs fine on any modern hardware that 564: NetBSD/amd64 runs well on. Until then, we give motherboard/CPU (and 565: sometimes RAM) pairs/triples to aid those choosing a motherboard. 566: Note that Xen systems usually do not run X, so a listing here does not 567: imply that X works at all. 568: 569: Supermicro X9SRL-F, Xeon E5-1650 v2, 96 GiB ECC 570: Supermicro ??, Atom C2758 (8 core), 32 GiB ECC 571: ASUS M5A78L-M/USB3 AM3+ microATX, AMD Piledriver X8 4000MHz, 16 GiB ECC 572: 573: Older hardware: 574: 575: Intel D915GEV, Pentium4 CPU 3.40GHz, 4GB 533MHz Synchronous DDR2 576: INTEL DG33FB, "Intel(R) Core(TM)2 Duo CPU E6850 @ 3.00GHz" 577: INTEL DG33FB, "Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz" 578: 579: Running Xen under qemu 580: ---------------------- 581: 582: The astute reader will note that this section is somewhat twisted. 583: However, it can be useful to run Xen under qemu either because the 584: version of NetBSD as a dom0 does not run on the hardware in use, or to 585: generate automated test cases involving Xen. 586: 587: In 2015-01, the following combination was reported to mostly work: 588: 589: host OS: NetBSD/amd64 6.1.4 590: qemu: 2.2.0 from pkgsrc 591: Xen kernel: xenkernel42-4.2.5nb1 from pkgsrc 592: dom0 kernel: NetBSD/amd64 6.1.5 593: Xen tools: xentools42-4.2.5 from pkgsrc 594: 595: See [PR 47720](https://gnats.netbsd.org/47720) for a problem with dom0 596: shutdown. 597: 598: Unprivileged domains (domU) 599: =========================== 600: 601: This section describes general concepts about domUs. It does not 602: address specific domU operating systems or how to install them. The 603: config files for domUs are typically in /usr/pkg/etc/xen, and are 604: typically named so that the file name, domU name and the domU's host 605: name match. 606: 607: The domU is provided with CPU and memory by Xen, configured by the 608: dom0. The domU is provided with disk and network by the dom0, 609: mediated by Xen, and configured in the dom0. 610: 611: Entropy in domUs can be an issue; physical disks and network are on 612: the dom0. NetBSD's /dev/random system works, but is often challenged. 613: 614: Config files 615: ------------ 616: 617: There is no good order to present config files and the concepts 618: surrounding what is being configured. We first show an example config 619: file, and then in the various sections give details. 620: 621: See (at least in xentools41) /usr/pkg/share/examples/xen/xmexample*, 622: for a large number of well-commented examples, mostly for running 623: GNU/Linux. 624: 625: The following is an example minimal domain configuration file 626: "/usr/pkg/etc/xen/foo". It is (with only a name change) an actual 627: known working config file on Xen 4.1 (NetBSD 5 amd64 dom0 and NetBSD 5 628: i386 domU). The domU serves as a network file server. 629: 630: # -*- mode: python; -*- 631: 632: kernel = "/netbsd-XEN3PAE_DOMU-i386-foo.gz" 633: memory = 1024 634: vif = [ 'mac=aa:00:00:d1:00:09,bridge=bridge0' ] 635: disk = [ 'file:/n0/xen/foo-wd0,0x0,w', 636: 'file:/n0/xen/foo-wd1,0x1,w' ] 637: 638: The domain will have the same name as the file. The kernel has the 639: host/domU name in it, so that on the dom0 one can update the various 640: domUs independently. The vif line causes an interface to be provided, 641: with a specific mac address (do not reuse MAC addresses!), in bridge 642: mode. Two disks are provided, and they are both writable; the bits 643: are stored in files and Xen attaches them to a vnd(4) device in the 644: dom0 on domain creation. The system treats xbd0 as the boot device 645: without needing explicit configuration. 646: 647: By default xm looks for domain config files in /usr/pkg/etc/xen. Note 648: that "xm create" takes the name of a config file, while other commands 649: take the name of a domain. To create the domain, connect to the 650: console, create the domain while attaching the console, shutdown the 651: domain, and see if it has finished stopping, do (or xl with Xen >= 652: 4.2): 653: 654: xm create foo 655: xm console foo 656: xm create -c foo 657: xm shutdown foo 658: xm list 659: 660: Typing ^] will exit the console session. Shutting down a domain is 661: equivalent to pushing the power button; a NetBSD domU will receive a 662: power-press event and do a clean shutdown. Shutting down the dom0 663: will trigger controlled shutdowns of all configured domUs. 664: 665: domU kernels 666: ------------ 667: 668: On a physical computer, the BIOS reads sector 0, and a chain of boot 669: loaders finds and loads a kernel. Normally this comes from the root 670: file system. With Xen domUs, the process is totally different. The 671: normal path is for the domU kernel to be a file in the dom0's 672: file system. At the request of the dom0, Xen loads that kernel into a 673: new domU instance and starts execution. While domU kernels can be 674: anyplace, reasonable places to store domU kernels on the dom0 are in / 675: (so they are near the dom0 kernel), in /usr/pkg/etc/xen (near the 676: config files), or in /u0/xen (where the vdisks are). 677: 678: Note that loading the domU kernel from the dom0 implies that boot 679: blocks, /boot, /boot.cfg, and so on are all ignored in the domU. 680: See the VPS section near the end for discussion of alternate ways to 681: obtain domU kernels. 682: 683: CPU and memory 684: -------------- 685: 686: A domain is provided with some number of vcpus, less than the number 687: of CPUs seen by the hypervisor. (For a dom0, this is controlled by 688: the boot argument "dom0_max_vcpus=1".) For a domU, it is controlled 689: from the config file by the "vcpus = N" directive. 690: 691: A domain is provided with memory; this is controlled in the config 692: file by "memory = N" (in megabytes). In the straightforward case, the 693: sum of the the memory allocated to the dom0 and all domUs must be less 694: than the available memory. 695: 696: Xen also provides a "balloon" driver, which can be used to let domains 697: use more memory temporarily. TODO: Explain better, and explain how 698: well it works with NetBSD. 699: 700: Virtual disks 701: ------------- 702: 703: With the file/vnd style, typically one creates a directory, 704: e.g. /u0/xen, on a disk large enough to hold virtual disks for all 705: domUs. Then, for each domU disk, one writes zeros to a file that then 706: serves to hold the virtual disk's bits; a suggested name is foo-xbd0 707: for the first virtual disk for the domU called foo. Writing zeros to 708: the file serves two purposes. One is that preallocating the contents 709: improves performance. The other is that vnd on sparse files has 710: failed to work. TODO: give working/notworking NetBSD versions for 711: sparse vnd and gnats reference. Note that the use of file/vnd for Xen 712: is not really different than creating a file-backed virtual disk for 713: some other purpose, except that xentools handles the vnconfig 714: commands. To create an empty 4G virtual disk, simply do 715: 716: dd if=/dev/zero of=foo-xbd0 bs=1m count=4096 717: 718: Do not use qemu-img-xen, because this will create sparse file. There 719: have been recent (2015) reports of sparse vnd(4) devices causing 720: lockups, but there is apparently no PR. 721: 722: With the lvm style, one creates logical devices. They are then used 723: similarly to vnds. TODO: Add an example with lvm. 724: 725: In domU config files, the disks are defined as a sequence of 3-tuples. 726: The first element is "method:/path/to/disk". Common methods are 727: "file:" for file-backed vnd. and "phy:" for something that is already 728: a (TODO: character or block) device. 729: 730: The second element is an artifact of how virtual disks are passed to 731: Linux, and a source of confusion with NetBSD Xen usage. Linux domUs 732: are given a device name to associate with the disk, and values like 733: "hda1" or "sda1" are common. In a NetBSD domU, the first disk appears 734: as xbd0, the second as xbd1, and so on. However, xm/xl demand a 735: second argument. The name given is converted to a major/minor by 736: calling stat(2) on the name in /dev and this is passed to the domU. 737: In the general case, the dom0 and domU can be different operating 738: systems, and it is an unwarranted assumption that they have consistent 739: numbering in /dev, or even that the dom0 OS has a /dev. With NetBSD 740: as both dom0 and domU, using values of 0x0 for the first disk and 0x1 741: for the second works fine and avoids this issue. For a GNU/Linux 742: guest, one can create /dev/hda1 in /dev, or to pass 0x301 for 743: /dev/hda1. 744: 745: The third element is "w" for writable disks, and "r" for read-only 746: disks. 747: 748: Note that NetBSD by default creates only vnd[0123]. If you need more 749: than 4 total virtual disks at a time, run e.g. "./MAKEDEV vnd4" in the 750: dom0. 751: 752: Note that NetBSD by default creates only xbd[0123]. If you need more 753: virtual disks in a domU, run e.g. "./MAKEDEV xbd4" in the domU. 754: 755: Virtual Networking 756: ------------------ 757: 758: Xen provides virtual Ethernets, each of which connects the dom0 and a 759: domU. For each virtual network, there is an interface "xvifN.M" in 760: the dom0, and in domU index N, a matching interface xennetM (NetBSD 761: name). The interfaces behave as if there is an Ethernet with two 762: adapters connected. From this primitive, one can construct various 763: configurations. We focus on two common and useful cases for which 764: there are existing scripts: bridging and NAT. 765: 766: With bridging (in the example above), the domU perceives itself to be 767: on the same network as the dom0. For server virtualization, this is 768: usually best. Bridging is accomplished by creating a bridge(4) device 769: and adding the dom0's physical interface and the various xvifN.0 770: interfaces to the bridge. One specifies "bridge=bridge0" in the domU 771: config file. The bridge must be set up already in the dom0; an 772: example /etc/ifconfig.bridge0 is: 773: 774: create 775: up 776: !brconfig bridge0 add wm0 777: 778: With NAT, the domU perceives itself to be behind a NAT running on the 779: dom0. This is often appropriate when running Xen on a workstation. 780: TODO: NAT appears to be configured by "vif = [ '' ]". 781: 782: The MAC address specified is the one used for the interface in the new 783: domain. The interface in dom0 will use this address XOR'd with 784: 00:00:00:01:00:00. Random MAC addresses are assigned if not given. 785: 786: Sizing domains 787: -------------- 788: 789: Modern x86 hardware has vast amounts of resources. However, many 790: virtual servers can function just fine on far less. A system with 791: 512M of RAM and a 4G disk can be a reasonable choice. Note that it is 792: far easier to adjust virtual resources than physical ones. For 793: memory, it's just a config file edit and a reboot. For disk, one can 794: create a new file and vnconfig it (or lvm), and then dump/restore, 795: just like updating physical disks, but without having to be there and 796: without those pesky connectors. 797: 798: Starting domains automatically 799: ------------------------------ 800: 801: To start domains foo at bar at boot and shut them down cleanly on dom0 802: shutdown, in rc.conf add: 803: 804: xendomains="foo bar" 805: 806: Note that earlier versions of the xentools41 xendomains rc.d script 807: used xl, when one should use xm with 4.1. 808: 809: Creating specific unprivileged domains (domU) 810: ============================================= 811: 812: Creating domUs is almost entirely independent of operating system. We 813: have already presented the basics of config files. Note that you must 814: have already completed the dom0 setup so that "xl list" (or "xm list") 815: works. 816: 817: Creating an unprivileged NetBSD domain (domU) 818: --------------------------------------------- 819: 820: See the earlier config file, and adjust memory. Decide on how much 821: storage you will provide, and prepare it (file or lvm). 822: 823: While the kernel will be obtained from the dom0 file system, the same 824: file should be present in the domU as /netbsd so that tools like 825: savecore(8) can work. (This is helpful but not necessary.) 826: 827: The kernel must be specifically for Xen and for use as a domU. The 828: i386 and amd64 provide the following kernels: 829: 830: i386 XEN3_DOMU 831: i386 XEN3PAE_DOMU 832: amd64 XEN3_DOMU 833: 834: Unless using Xen 3.1 (and you shouldn't) with i386-mode Xen, you must 835: use the PAE version of the i386 kernel. 836: 837: This will boot NetBSD, but this is not that useful if the disk is 838: empty. One approach is to unpack sets onto the disk outside of xen 839: (by mounting it, just as you would prepare a physical disk for a 840: system you can't run the installer on). 841: 842: A second approach is to run an INSTALL kernel, which has a miniroot 843: and can load sets from the network. To do this, copy the INSTALL 844: kernel to / and change the kernel line in the config file to: 845: 846: kernel = "/home/bouyer/netbsd-INSTALL_XEN3_DOMU" 847: 848: Then, start the domain as "xl create -c configname". 849: 850: Alternatively, if you want to install NetBSD/Xen with a CDROM image, the following 851: line should be used in the config file. 852: 853: disk = [ 'phy:/dev/wd0e,0x1,w', 'phy:/dev/cd0a,0x2,r' ] 854: 855: After booting the domain, the option to install via CDROM may be 856: selected. The CDROM device should be changed to `xbd1d`. 857: 858: Once done installing, "halt -p" the new domain (don't reboot or halt, 859: it would reload the INSTALL_XEN3_DOMU kernel even if you changed the 860: config file), switch the config file back to the XEN3_DOMU kernel, 861: and start the new domain again. Now it should be able to use "root on 862: xbd0a" and you should have a, functional NetBSD domU. 863: 864: TODO: check if this is still accurate. 865: When the new domain is booting you'll see some warnings about *wscons* 866: and the pseudo-terminals. These can be fixed by editing the files 867: `/etc/ttys` and `/etc/wscons.conf`. You must disable all terminals in 868: `/etc/ttys`, except *console*, like this: 869: 870: console "/usr/libexec/getty Pc" vt100 on secure 871: ttyE0 "/usr/libexec/getty Pc" vt220 off secure 872: ttyE1 "/usr/libexec/getty Pc" vt220 off secure 873: ttyE2 "/usr/libexec/getty Pc" vt220 off secure 874: ttyE3 "/usr/libexec/getty Pc" vt220 off secure 875: 876: Finally, all screens must be commented out from `/etc/wscons.conf`. 877: 878: It is also desirable to add 879: 880: powerd=YES 881: 882: in rc.conf. This way, the domain will be properly shut down if 883: `xm shutdown -R` or `xm shutdown -H` is used on the dom0. 884: 885: It is not strictly necessary to have a kernel (as /netbsd) in the domU 886: file system. However, various programs (e.g. netstat) will use that 887: kernel to look up symbols to read from kernel virtual memory. If 888: /netbsd is not the running kernel, those lookups will fail. (This is 889: not really a Xen-specific issue, but because the domU kernel is 890: obtained from the dom0, it is far more likely to be out of sync or 891: missing with Xen.) 892: 893: Creating an unprivileged Linux domain (domU) 894: -------------------------------------------- 895: 896: Creating unprivileged Linux domains isn't much different from 897: unprivileged NetBSD domains, but there are some details to know. 898: 899: First, the second parameter passed to the disk declaration (the '0x1' in 900: the example below) 901: 902: disk = [ 'phy:/dev/wd0e,0x1,w' ] 903: 904: does matter to Linux. It wants a Linux device number here (e.g. 0x300 905: for hda). Linux builds device numbers as: (major \<\< 8 + minor). 906: So, hda1 which has major 3 and minor 1 on a Linux system will have 907: device number 0x301. Alternatively, devices names can be used (hda, 908: hdb, ...) as xentools has a table to map these names to devices 909: numbers. To export a partition to a Linux guest we can use: 910: 911: disk = [ 'phy:/dev/wd0e,0x300,w' ] 912: root = "/dev/hda1 ro" 913: 914: and it will appear as /dev/hda on the Linux system, and be used as root 915: partition. 916: 917: To install the Linux system on the partition to be exported to the 918: guest domain, the following method can be used: install 919: sysutils/e2fsprogs from pkgsrc. Use mke2fs to format the partition 920: that will be the root partition of your Linux domain, and mount it. 921: Then copy the files from a working Linux system, make adjustments in 922: `/etc` (fstab, network config). It should also be possible to extract 923: binary packages such as .rpm or .deb directly to the mounted partition 924: using the appropriate tool, possibly running under NetBSD's Linux 925: emulation. Once the file system has been populated, umount it. If 926: desirable, the file system can be converted to ext3 using tune2fs -j. 927: It should now be possible to boot the Linux guest domain, using one of 928: the vmlinuz-\*-xenU kernels available in the Xen binary distribution. 929: 930: To get the Linux console right, you need to add: 931: 932: extra = "xencons=tty1" 933: 934: to your configuration since not all Linux distributions auto-attach a 935: tty to the xen console. 936: 937: Creating an unprivileged Solaris domain (domU) 938: ---------------------------------------------- 939: 940: See possibly outdated 941: [Solaris domU instructions](/ports/xen/howto-solaris/). 942: 943: 944: PCI passthrough: Using PCI devices in guest domains 945: --------------------------------------------------- 946: 947: The dom0 can give other domains access to selected PCI 948: devices. This can allow, for example, a non-privileged domain to have 949: access to a physical network interface or disk controller. However, 950: keep in mind that giving a domain access to a PCI device most likely 951: will give the domain read/write access to the whole physical memory, 952: as PCs don't have an IOMMU to restrict memory access to DMA-capable 953: device. Also, it's not possible to export ISA devices to non-dom0 954: domains, which means that the primary VGA adapter can't be exported. 955: A guest domain trying to access the VGA registers will panic. 956: 957: If the dom0 is NetBSD, it has to be running Xen 3.1, as support has 958: not been ported to later versions at this time. 959: 960: For a PCI device to be exported to a domU, is has to be attached to 961: the "pciback" driver in dom0. Devices passed to the dom0 via the 962: pciback.hide boot parameter will attach to "pciback" instead of the 963: usual driver. The list of devices is specified as "(bus:dev.func)", 964: where bus and dev are 2-digit hexadecimal numbers, and func a 965: single-digit number: 966: 967: pciback.hide=(00:0a.0)(00:06.0) 968: 969: pciback devices should show up in the dom0's boot messages, and the 970: devices should be listed in the `/kern/xen/pci` directory. 971: 972: PCI devices to be exported to a domU are listed in the "pci" array of 973: the domU's config file, with the format "0000:bus:dev.func". 974: 975: pci = [ '0000:00:06.0', '0000:00:0a.0' ] 976: 977: In the domU an "xpci" device will show up, to which one or more pci 978: buses will attach. Then the PCI drivers will attach to PCI buses as 979: usual. Note that the default NetBSD DOMU kernels do not have "xpci" 980: or any PCI drivers built in by default; you have to build your own 981: kernel to use PCI devices in a domU. Here's a kernel config example; 982: note that only the "xpci" lines are unusual. 983: 984: include "arch/i386/conf/XEN3_DOMU" 985: 986: # Add support for PCI buses to the XEN3_DOMU kernel 987: xpci* at xenbus ? 988: pci* at xpci ? 989: 990: # PCI USB controllers 991: uhci* at pci? dev ? function ? # Universal Host Controller (Intel) 992: 993: # USB bus support 994: usb* at uhci? 995: 996: # USB Hubs 997: uhub* at usb? 998: uhub* at uhub? port ? configuration ? interface ? 999: 1000: # USB Mass Storage 1001: umass* at uhub? port ? configuration ? interface ? 1002: wd* at umass? 1003: # SCSI controllers 1004: ahc* at pci? dev ? function ? # Adaptec [23]94x, aic78x0 SCSI 1005: 1006: # SCSI bus support (for both ahc and umass) 1007: scsibus* at scsi? 1008: 1009: # SCSI devices 1010: sd* at scsibus? target ? lun ? # SCSI disk drives 1011: cd* at scsibus? target ? lun ? # SCSI CD-ROM drives 1012: 1013: 1014: NetBSD as a domU in a VPS 1015: ========================= 1016: 1017: The bulk of the HOWTO is about using NetBSD as a dom0 on your own 1018: hardware. This section explains how to deal with Xen in a domU as a 1019: virtual private server where you do not control or have access to the 1020: dom0. This is not intended to be an exhaustive list of VPS providers; 1021: only a few are mentioned that specifically support NetBSD. 1022: 1023: VPS operators provide varying degrees of access and mechanisms for 1024: configuration. The big issue is usually how one controls which kernel 1025: is booted, because the kernel is nominally in the dom0 file system (to 1026: which VPS users do not normally have access). A second issue is how 1027: to install NetBSD. 1028: A VPS user may want to compile a kernel for security updates, to run 1029: npf, run IPsec, or any other reason why someone would want to change 1030: their kernel. 1031: 1032: One approach is to have an administrative interface to upload a kernel, 1033: or to select from a prepopulated list. Other approaches are pygrub 1034: (deprecated) and pvgrub, which are ways to have a bootloader obtain a 1035: kernel from the domU file system. This is closer to a regular physical 1036: computer, where someone who controls a machine can replace the kernel. 1037: 1038: A second issue is multiple CPUs. With NetBSD 6, domUs support 1039: multiple vcpus, and it is typical for VPS providers to enable multiple 1040: CPUs for NetBSD domUs. 1041: 1042: pygrub 1043: ------- 1044: 1045: pygrub runs in the dom0 and looks into the domU file system. This 1046: implies that the domU must have a kernel in a file system in a format 1047: known to pygrub. As of 2014, pygrub seems to be of mostly historical 1048: interest. 1049: 1050: pvgrub 1051: ------ 1052: 1053: pvgrub is a version of grub that uses PV operations instead of BIOS 1054: calls. It is booted from the dom0 as the domU kernel, and then reads 1055: /grub/menu.lst and loads a kernel from the domU file system. 1056: 1057: [Panix](http://www.panix.com/) lets users use pvgrub. Panix reports 1058: that pvgrub works with FFsv2 with 16K/2K and 32K/4K block/frag sizes 1059: (and hence with defaults from "newfs -O 2"). See [Panix's pvgrub 1060: page](http://www.panix.com/v-colo/grub.html), which describes only 1061: Linux but should be updated to cover NetBSD :-). 1062: 1063: [prgmr.com](http://prgmr.com/) also lets users with pvgrub to boot 1064: their own kernel. See then [prgmr.com NetBSD 1065: HOWTO](http://wiki.prgmr.com/mediawiki/index.php/NetBSD_as_a_DomU) 1066: (which is in need of updating). 1067: 1068: It appears that [grub's FFS 1069: code](http://xenbits.xensource.com/hg/xen-unstable.hg/file/bca284f67702/tools/libfsimage/ufs/fsys_ufs.c) 1070: does not support all aspects of modern FFS, but there are also reports 1071: that FFSv2 works fine. At prgmr, typically one has an ext2 or FAT 1072: partition for the kernel with the intent that grub can understand it, 1073: which leads to /netbsd not being the actual kernel. One must remember 1074: to update the special boot partition. 1075: 1076: Amazon 1077: ------ 1078: 1079: See the [Amazon EC2 page](/amazon_ec2/). 1080: 1081: TODO items for improving NetBSD/xen 1082: =================================== 1083: 1084: * Make the NetBSD dom0 kernel work with SMP. 1085: * Test the Xen 4.5 packages adequately to be able to recommend them as 1086: the standard approach. 1087: * Get PCI passthrough working on Xen 4.5 1088: * Get pvgrub into pkgsrc, either via xentools or separately. 1089: * grub 1090: * Check/add support to pkgsrc grub2 for UFS2 and arbitrary 1091: fragsize/blocksize (UFS2 support may be present; the point is to 1092: make it so that with any UFS1/UFS2 file system setup that works 1093: with NetBSD grub will also work). 1094: See [pkg/40258](https://gnats.netbsd.org/40258). 1095: * Push patches upstream. 1096: * Get UFS2 patches into pvgrub. 1097: * Add support for PV ops to a version of /boot, and make it usable as 1098: a kernel in Xen, similar to pvgrub. 1099: 1100: Random pointers 1101: =============== 1102: 1103: This section contains links from elsewhere not yet integrated into the 1104: HOWTO, and other guides. 1105: 1106: * http://www.lumbercartel.ca/library/xen/ 1107: * http://pbraun.nethence.com/doc/sysutils/xen_netbsd_dom0.html 1108: * https://gmplib.org/~tege/xen.html