--- wikisrc/ports/xen/howto.mdwn 2020/11/15 14:31:58 1.180 +++ wikisrc/ports/xen/howto.mdwn 2020/11/15 15:09:31 1.186 @@ -8,7 +8,31 @@ systems which operate in an unprivileged from the domU systems are forwarded by the Xen hypervisor to the dom0 to be fulfilled. -Xen supports different styles of guests; see [PV on HVM](https://wiki.xen.org/wiki/PV_on_HVM) and [PVH(v2)](https://wiki.xenproject.org/wiki/PVH_(v2\)_Domu) for upstream documentation. +This HOWTO presumes a basic familiarity with the Xen system +architecture, with installing NetBSD on amd64 hardware, and with +installing software from pkgsrc. See also the [Xen +website](http://www.xenproject.org/). + +[[!toc]] + +# Overview + +The basic concept of Xen is that the hypervisor (xenkernel) runs on +the hardware, and runs a privileged domain ("dom0") that can access +disks/networking/etc. One then runs additonal unprivileged domains +(each a "domU"), presumably to do something useful. + +This HOWTO addresses how to run a NetBSD dom0 (and hence also build +xen itself). It also addresses how to run domUs in that environment, +and how to deal with having a domU in a Xen environment run by someone +else and/or not running NetBSD. + +There are many choices one can make; the HOWTO recommends the standard +approach and limits discussion of alternatives in many cases. + +## Guest Styles + +Xen supports different styles of guests. [[!table data=""" Style of guest |Supported by NetBSD @@ -21,6 +45,7 @@ PVH |current-only (domU, dom0 not yet) In Para-Virtualized (PV) mode, the guest OS does not attempt to access hardware directly, but instead makes hypercalls to the hypervisor; PV guests must be specifically coded for Xen. +See [PV](https://wiki.xen.org/wiki/Paravirtualization_(PV\)). In HVM mode, no guest modification is required; however, hardware support is required, such as VT-x on Intel CPUs and SVM on AMD CPUs. @@ -28,27 +53,32 @@ The dom0 runs qemu to emulate hardware. In PVHVM mode, the guest runs as HVM, but additionally can use PV drivers for efficiency. +See [PV on HVM](https://wiki.xen.org/wiki/PV_on_HVM). There have been two PVH modes: original PVH and PVHv2. Original PVH was based on PV mode and is no longer relevant at all. PVHv2 is basically lightweight HVM with PV drivers. A critical feature of it is that qemu is not needed; the hypervisor can do the emulation that is required. Thus, a dom0 can be PVHv2. - The source code uses PVH and config files use pvh; this refers to PVHv2. +See [PVH(v2)](https://wiki.xenproject.org/wiki/PVH_(v2\)_Domu). -At boot, the dom0 kernel is loaded as a module with Xen as the kernel. +At system boot, the dom0 kernel is loaded as a module with Xen as the kernel. The dom0 can start one or more domUs. (Booting is explained in detail in the dom0 section.) -This HOWTO presumes a basic familiarity with the Xen system -architecture, with installing NetBSD on i386/amd64 hardware, and with -installing software from pkgsrc. See also the [Xen -website](http://www.xenproject.org/). +## CPU Architecture -[[!toc]] +Xen runs on x86_64 hardware (the NetBSD amd64 port). + +There is a concept of Xen running on ARM, but there are no reports of this working with NetBSD. + +The dom0 system should be amd64. (Instructions for i386PAE dom0 have been removed from the HOWTO.) + +The domU can be i386PAE or amd64. +i386PAE at one point was considered as [faster](https://lists.xen.org/archives/html/xen-devel/2012-07/msg00085.html) than amd64. -# Versions and Support +## Xen Versions In NetBSD, Xen is provided in pkgsrc, via matching pairs of packages xenkernel and xentools. We will refer only to the kernel versions, @@ -58,37 +88,32 @@ matching versions. Versions available in pkgsrc: [[!table data=""" -Xen Version |Package Name |Xen CPU Support |xm? |EOL'ed By Upstream -4.11 |xenkernel411 |x86_64 | |No -4.13 |xenkernel413 |x86_64 | |No +Xen Version |Package Name |Xen CPU Support |EOL'ed By Upstream +4.11 |xenkernel411 |x86_64 |No +4.13 |xenkernel413 |x86_64 |No """]] See also the [Xen Security Advisory page](http://xenbits.xen.org/xsa/). -Multiprocessor (SMP) support in NetBSD differs depending on the domain: +Older Xen had a python-based management tool called xm, now replaced +by xl. -[[!table data=""" -Domain |Supports SMP -dom0 |No -domU |Yes -"""]] +## NetBSD versions -Note: NetBSD support is called XEN3. However, it does support Xen 4, -because the hypercall interface has remained identical. +Xen has been supported in NetBSD for a long time, at least since 2005. +Initially Xen was PV only. -Older Xen had a python-based management tool called xm, now replaced -by xl. +NetBSD 8 and up support PV and HVM modes. -Architecture ------------- +Support for PVHVM and PVH is available only in NetBSD-current. -Xen 4.5 and later runs on x86_64 hardware (the NetBSD amd64 port). -There is a concept of Xen running on ARM, but there are no reports of this working with NetBSD. +NetBSD as a dom0 does not run SMP, because some drivers are not yet +safe for this. \todo Link to more information about what needs work. -The dom0 system should be amd64. (Instructions for i386PAE dom0 have been removed from the HOWTO.) +NetBSD, when run as a domU, can and does typically run SMP. -The domU can be i386PAE or amd64. -i386PAE at one point was considered as [faster](https://lists.xen.org/archives/html/xen-devel/2012-07/msg00085.html) than amd64. +Note: NetBSD support is called XEN3. However, it does support Xen 4, +because the hypercall interface has remained identical. # Creating a dom0 @@ -100,22 +125,23 @@ In 2018-05, trouble booting a dom0 was r 512M it worked reliably. This does not make sense, but if you see "not ELF" after Xen boots, try increasing dom0 RAM. -Installation of NetBSD ----------------------- +## Installation of NetBSD [Install NetBSD/amd64](/guide/inst/) just as you would if you were not using Xen. +Therefore, use the most recent release, or a build from the most recent stable branch. -Installation of Xen -------------------- +## Installation of Xen -We will consider that you chose to use Xen 4.13, with NetBSD/amd64 as -dom0. In the dom0, install xenkernel48 and xentools48 from pkgsrc. +Use the most recent version of Xen in pkgsrc, unless the DESCR says that it ss not suitable. +Therefore, choose 4.13. +In the dom0, install xenkernel413 and xentools413 from pkgsrc. -Once this is done, install the Xen kernel itself: +Once this is done, copy the Xen kernel from where pkgsrc puts it to +where the boot process will be able to find it: [[!template id=programlisting text=""" -# cp /usr/pkg/xen48-kernel/xen.gz / +# cp -p /usr/pkg/xen413-kernel/xen.gz / """]] Then, place a NetBSD XEN3_DOM0 kernel in the `/` directory. Such kernel @@ -123,7 +149,7 @@ can either be compiled manually, or down example at: [[!template id=programlisting text=""" -ftp.netbsd.org/pub/NetBSD/NetBSD-8.0/amd64/binary/kernel/netbsd-XEN3_DOM0.gz +ftp.netbsd.org/pub/NetBSD/NetBSD-9.1/amd64/binary/kernel/netbsd-XEN3_DOM0.gz """]] Add a line to /boot.cfg to boot Xen: @@ -153,13 +179,12 @@ more) and to pin that vcpu to a physical and other than dom0 memory and max_vcpus, they are generally not necessary. -Copy the boot scripts into `/etc/rc.d`: - -[[!template id=programlisting text=""" -# cp /usr/pkg/share/examples/rc.d/xen* /etc/rc.d/ -"""]] +Ensure that the boot scripts installed in +`/usr/pkg/share/examples/rc.d` are in `/etc/rc.d`, either because you +have `PKG_RCD_SCRIPTS=yes`, or manually. (This is not special to Xen, +but a normal part of pkgsrc usage.) -Enable `xencommons`: +Set `xencommons=YES` in rc.conf: [[!template id=filecontent name="/etc/rc.conf" text=""" xencommons=YES @@ -197,8 +222,7 @@ make this work, one should not expect to (and thus xencommons). There is currently no reason to expect that this will get fixed any time soon. -anita (for testing NetBSD) --------------------------- +## anita (for testing NetBSD) With the setup so far, one should be able to run anita (see pkgsrc/misc/py-anita) to test NetBSD releases, by doing (as @@ -208,8 +232,7 @@ root, because anita must create a domU): anita --vmm=xl test file:///usr/obj/i386/ """]] -Xen-specific NetBSD issues --------------------------- +## Xen-specific NetBSD issues There are (at least) two additional things different about NetBSD as a dom0 kernel compared to hardware. @@ -223,8 +246,7 @@ options as GENERIC. While it is debatab bug, users should be aware of this and can simply add missing config items if desired. -Updating NetBSD in a dom0 -------------------------- +## Updating NetBSD in a dom0 This is just like updating NetBSD on bare hardware, assuming the new version supports the version of Xen you are running. Generally, one @@ -234,8 +256,7 @@ and adjusts `/etc`. Note that one must update both the non-Xen kernel typically used for rescue purposes and the DOM0 kernel used with Xen. -Converting from grub to /boot ------------------------------ +## Converting from grub to /boot These instructions were used to convert a system from grub to /boot. The system was originally installed in February of @@ -265,8 +286,7 @@ default=1 timeout=30 """]] -Upgrading Xen versions ---------------------- +## Upgrading Xen versions Minor version upgrades are trivial. Just rebuild/replace the xenkernel version and copy the new xen.gz to `/` (where `/boot.cfg` @@ -287,8 +307,7 @@ mediated by Xen, and configured in the d Entropy in domUs can be an issue; physical disks and network are on the dom0. NetBSD's /dev/random system works, but is often challenged. -Config files ------------- +## Config files See /usr/pkg/share/examples/xen/xlexample* for a small number of well-commented examples, mostly for running @@ -334,8 +353,7 @@ equivalent to pushing the power button; power-press event and do a clean shutdown. Shutting down the dom0 will trigger controlled shutdowns of all configured domUs. -CPU and memory --------------- +## CPU and memory A domain is provided with some number of vcpus, up to the number of CPUs seen by the hypervisor. For a domU, it is controlled @@ -349,8 +367,7 @@ than the available memory. Xen also provides a "balloon" driver, which can be used to let domains use more memory temporarily. -Virtual disks -------------- +## Virtual disks In domU config files, the disks are defined as a sequence of 3-tuples: @@ -499,7 +516,8 @@ It is also desirable to add powerd=YES in rc.conf. This way, the domain will be properly shut down if -`xm shutdown -R` or `xm shutdown -H` is used on the dom0. +`xl shutdown -R` or `xl shutdown -H` is used on the dom0. +\todo Check the translation to xl. It is not strictly necessary to have a kernel (as /netbsd) in the domU file system. However, various programs (e.g. netstat) will use that @@ -702,8 +720,9 @@ NetBSD apparently only use specific vers \todo Explain better. -pvgrub ------- +## Boot methods + +### pvgrub pvgrub is a version of grub that uses PV operations instead of BIOS calls. It is booted from the dom0 as the domU kernel, and then reads @@ -728,8 +747,7 @@ partition for the kernel with the intent which leads to /netbsd not being the actual kernel. One must remember to update the special boot partition. -pygrub -------- +### pygrub pygrub runs in the dom0 and looks into the domU file system. This implies that the domU must have a kernel in a file system in a format @@ -743,7 +761,8 @@ historical DomU images using it still wo As of 2014, pygrub seems to be of mostly historical interest. New DomUs should use pvgrub. -Amazon ------- +## Specific Providers + +### Amazon See the [Amazon EC2 page](/amazon_ec2/).