Diff for /wikisrc/ports/xen/howto.mdwn between versions 1.49 and 1.56

version 1.49, 2014/12/26 20:25:19 version 1.56, 2014/12/27 00:25:48
Line 49  specific PCI devices can be made availab Line 49  specific PCI devices can be made availab
 of the dom0.  This can be useful to let a domU run X11, or access some  of the dom0.  This can be useful to let a domU run X11, or access some
 network interface or other peripheral.  network interface or other peripheral.
   
   NetBSD used to support Xen2; this has been removed.
   
 Prerequisites  Prerequisites
 -------------  -------------
   
Line 63  architecture.  This HOWTO presumes famil Line 65  architecture.  This HOWTO presumes famil
 on i386/amd64 hardware and installing software from pkgsrc.  on i386/amd64 hardware and installing software from pkgsrc.
 See also the [Xen website](http://www.xenproject.org/).  See also the [Xen website](http://www.xenproject.org/).
   
 History  
 -------  
   
 NetBSD used to support Xen2; this has been removed.  
   
 Before NetBSD's native bootloader could support Xen, the use of  
 grub was recommended.  If necessary, see the  
 [old grub information](/ports/xen/howto-grub/).  
   
 Versions of Xen and NetBSD  Versions of Xen and NetBSD
 ==========================  ==========================
   
Line 204  alternately with little problems, simply Line 197  alternately with little problems, simply
 Xen daemons when not running Xen.  Xen daemons when not running Xen.
   
 Note that NetBSD as dom0 does not support multiple CPUs.  This will  Note that NetBSD as dom0 does not support multiple CPUs.  This will
 limit the performance of the Xen/dom0 workstation approach.  limit the performance of the Xen/dom0 workstation approach.  In theory
   the only issue is that the "backend drivers" are not yet MPSAFE:
     http://mail-index.netbsd.org/netbsd-users/2014/08/29/msg015195.html
   
 Installation of NetBSD  Installation of NetBSD
 ----------------------  ----------------------
Line 300  As with non-Xen systems, you should have Line 295  As with non-Xen systems, you should have
 kernel that works without Xen) and fallback versions of the non-Xen  kernel that works without Xen) and fallback versions of the non-Xen
 kernel, Xen, and the dom0 kernel.  kernel, Xen, and the dom0 kernel.
   
   Using grub (historic)
   ---------------------
   
   Before NetBSD's native bootloader could support Xen, the use of
   grub was recommended.  If necessary, see the
   [old grub information](/ports/xen/howto-grub/).
   
 The [HowTo on Installing into  The [HowTo on Installing into
 RAID-1](http://mail-index.NetBSD.org/port-xen/2006/03/01/0010.html)  RAID-1](http://mail-index.NetBSD.org/port-xen/2006/03/01/0010.html)
 explains how to set up booting a dom0 with Xen using grub with  explains how to set up booting a dom0 with Xen using grub with
Line 309  boot.) Line 311  boot.)
 Configuring Xen  Configuring Xen
 ---------------  ---------------
   
   Xen logs will be in /var/log/xen.
   
 Now, you have a system that will boot Xen and the dom0 kernel, and  Now, you have a system that will boot Xen and the dom0 kernel, and
 just run the dom0 kernel.  There will be no domUs, and none can be  just run the dom0 kernel.  There will be no domUs, and none can be
 started because you still have to configure the dom0 tools.  The  started because you still have to configure the dom0 tools.  The
Line 332  installed 4.1 or 4.2): Line 336  installed 4.1 or 4.2):
   
 For 4.1 (and thus xm; xl is believed not to work well), add to rc.conf:  For 4.1 (and thus xm; xl is believed not to work well), add to rc.conf:
   
         xend=YES  
         xencommons=YES          xencommons=YES
           xend=YES
   
 TODO: Explain why if xm is preferred on 4.1, rc.d/xendomains has xl.  (If you are using xentools41 from before 2014-12-26, change
 Or fix the package.  rc.d/xendomains to use xm rather than xl.)
   
 For 4.2 with xm, add to rc.conf  For 4.2 with xm, add to rc.conf
   
         xend=YES  
         xencommons=YES          xencommons=YES
           xend=YES
   
 For 4.2 with xl (preferred), add to rc.conf:  For 4.2 with xl (preferred), add to rc.conf:
   
         TODO: explain if there is a xend replacement  
         xencommons=YES          xencommons=YES
           TODO: explain if there is a xend replacement
   
 TODO: Recommend for/against xen-watchdog.  TODO: Recommend for/against xen-watchdog.
   
 After you have configured the daemons and either started them or  After you have configured the daemons and either started them (in the
 rebooted, run the following (or use xl) to inspect Xen's boot  order given) or rebooted, run the following (or use xl) to inspect
 messages, available resources, and running domains:  Xen's boot messages, available resources, and running domains:
   
         # xm dmesg          # xm dmesg
         [xen's boot info]          [xen's boot info]
Line 403  and adjusts /etc. Line 407  and adjusts /etc.
 Note that one must update both the non-Xen kernel typically used for  Note that one must update both the non-Xen kernel typically used for
 rescue purposes and the DOM0 kernel used with Xen.  rescue purposes and the DOM0 kernel used with Xen.
   
 To convert from grub to /boot, install an mbr bootblock with fdisk,  Converting from grub to /boot
 bootxx_ with installboot, /boot and /boot.cfg.  This really should be  -----------------------------
 no different than completely reinstalling boot blocks on a non-Xen  
 system.  These instructions were [TODO: will be] used to convert a system from
   grub to /boot.  The system was originally installed in February of
   2006 with a RAID1 setup and grub to boot Xen 2, and has been updated
   over time.  Before these commands, it was running NetBSD 6 i386, Xen
   4.1 and grub, much like the message linked earlier in the grub
   section.
   
           # Install mbr bootblocks on both disks. 
           fdisk -i /dev/rwd0d
           fdisk -i /dev/rwd1d
           # Install NetBSD primary boot loader (/ is FFSv1) into RAID1 components.
           installboot -v /dev/rwd0d /usr/mdec/bootxx_ffsv1
           installboot -v /dev/rwd1d /usr/mdec/bootxx_ffsv1
           # Install secondary boot loader
           cp -p /usr/mdec/boot /
           # Create boog.cfg following earlier guidance:
           menu=Xen:load /netbsd-XEN3PAE_DOM0.gz console=pc;multiboot /xen.gz dom0_mem=256M
           menu=Xen.ok:load /netbsd-XEN3PAE_DOM0.ok.gz console=pc;multiboot /xen.ok.gz dom0_mem=256M
           menu=GENERIC:boot
           menu=GENERIC single-user:boot -s
           menu=GENERIC.ok:boot netbsd.ok
           menu=GENERIC.ok single-user:boot netbsd.ok -s
           menu=Drop to boot prompt:prompt
           default=1
           timeout=30
   
   TODO: actually do this and fix it if necessary.
   
 Updating Xen versions  Updating Xen versions
 ---------------------  ---------------------
Line 593  dom0.  This is often appropriate when ru Line 623  dom0.  This is often appropriate when ru
 TODO: NAT appears to be configured by "vif = [ '' ]".  TODO: NAT appears to be configured by "vif = [ '' ]".
   
 The MAC address specified is the one used for the interface in the new  The MAC address specified is the one used for the interface in the new
 domain.  The interface in domain0 will use this address XOR'd with  domain.  The interface in dom0 will use this address XOR'd with
 00:00:00:01:00:00.  Random MAC addresses are assigned if not given.  00:00:00:01:00:00.  Random MAC addresses are assigned if not given.
   
 Sizing domains  Sizing domains
Line 693  It is also desirable to add Line 723  It is also desirable to add
         powerd=YES          powerd=YES
   
 in rc.conf. This way, the domain will be properly shut down if  in rc.conf. This way, the domain will be properly shut down if
 `xm shutdown -R` or `xm shutdown -H` is used on the domain0.  `xm shutdown -R` or `xm shutdown -H` is used on the dom0.
   
 Your domain should be now ready to work, enjoy.  Your domain should be now ready to work, enjoy.
   
Line 744  tty to the xen console. Line 774  tty to the xen console.
 Creating an unprivileged Solaris domain (domU)  Creating an unprivileged Solaris domain (domU)
 ----------------------------------------------  ----------------------------------------------
   
 Download an Opensolaris [release](http://opensolaris.org/os/downloads/)  See possibly outdated
 or [development snapshot](http://genunix.org/) DVD image. Attach the DVD  [Solaris domU instructions](/ports/xen/howto-solaris/).
 image to a MAN.VND.4 device. Copy the kernel and ramdisk filesystem  
 image to your dom0 filesystem.  
   
     dom0# mkdir /root/solaris  
     dom0# vnconfig vnd0 osol-1002-124-x86.iso  
     dom0# mount /dev/vnd0a /mnt  
   
     ## for a 64-bit guest  
     dom0# cp /mnt/boot/amd64/x86.microroot /root/solaris  
     dom0# cp /mnt/platform/i86xpv/kernel/amd64/unix /root/solaris  
   
     ## for a 32-bit guest  
     dom0# cp /mnt/boot/x86.microroot /root/solaris  
     dom0# cp /mnt/platform/i86xpv/kernel/unix /root/solaris  
   
     dom0# umount /mnt  
             
   
 Keep the MAN.VND.4 configured. For some reason the boot process stalls  
 unless the DVD image is attached to the guest as a "phy" device. Create  
 an initial configuration file with the following contents. Substitute  
 */dev/wd0k* with an empty partition at least 8 GB large.  
   
     memory = 640  
     name = 'solaris'  
     disk = [ 'phy:/dev/wd0k,0,w' ]  
     disk += [ 'phy:/dev/vnd0d,6:cdrom,r' ]  
     vif = [ 'bridge=bridge0' ]  
     kernel = '/root/solaris/unix'  
     ramdisk = '/root/solaris/x86.microroot'  
     # for a 64-bit guest  
     extra = '/platform/i86xpv/kernel/amd64/unix - nowin -B install_media=cdrom'  
     # for a 32-bit guest  
     #extra = '/platform/i86xpv/kernel/unix - nowin -B install_media=cdrom'  
             
   
 Start the guest.  
   
     dom0# xm create -c solaris.cfg  
     Started domain solaris  
                           v3.3.2 chgset 'unavailable'  
     SunOS Release 5.11 Version snv_124 64-bit  
     Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.  
     Use is subject to license terms.  
     Hostname: opensolaris  
     Remounting root read/write  
     Probing for device nodes ...  
     WARNING: emlxs: ddi_modopen drv/fct failed: err 2  
     Preparing live image for use  
     Done mounting Live image  
             
   
 Make sure the network is configured. Note that it can take a minute for  
 the xnf0 interface to appear.  
   
     opensolaris console login: jack  
     Password: jack  
     Sun Microsystems Inc.   SunOS 5.11      snv_124 November 2008  
     jack@opensolaris:~$ pfexec sh  
     sh-3.2# ifconfig -a  
     sh-3.2# exit  
             
   
 Set a password for VNC and start the VNC server which provides the X11  
 display where the installation program runs.  
   
     jack@opensolaris:~$ vncpasswd  
     Password: solaris  
     Verify: solaris  
     jack@opensolaris:~$ cp .Xclients .vnc/xstartup  
     jack@opensolaris:~$ vncserver :1  
             
   
 From a remote machine connect to the VNC server. Use `ifconfig xnf0` on  
 the guest to find the correct IP address to use.  
   
     remote$ vncviewer 172.18.2.99:1  
             
   
 It is also possible to launch the installation on a remote X11 display.  
   
     jack@opensolaris:~$ export DISPLAY=172.18.1.1:0  
     jack@opensolaris:~$ pfexec gui-install  
              
   
 After the GUI installation is complete you will be asked to reboot.  
 Before that you need to determine the ZFS ID for the new boot filesystem  
 and update the configuration file accordingly. Return to the guest  
 console.  
   
     jack@opensolaris:~$ pfexec zdb -vvv rpool | grep bootfs  
                     bootfs = 43  
     ^C  
     jack@opensolaris:~$  
              
   
 The final configuration file should look like this. Note in particular  
 the last line.  
   
     memory = 640  
     name = 'solaris'  
     disk = [ 'phy:/dev/wd0k,0,w' ]  
     vif = [ 'bridge=bridge0' ]  
     kernel = '/root/solaris/unix'  
     ramdisk = '/root/solaris/x86.microroot'  
     extra = '/platform/i86xpv/kernel/amd64/unix -B zfs-bootfs=rpool/43,bootpath="/xpvd/xdf@0:a"'  
              
   
 Restart the guest to verify it works correctly.  
   
     dom0# xm destroy solaris  
     dom0# xm create -c solaris.cfg  
     Using config file "./solaris.cfg".  
     v3.3.2 chgset 'unavailable'  
     Started domain solaris  
     SunOS Release 5.11 Version snv_124 64-bit  
     Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.  
     Use is subject to license terms.  
     WARNING: emlxs: ddi_modopen drv/fct failed: err 2  
     Hostname: osol  
     Configuring devices.  
     Loading smf(5) service descriptions: 160/160  
     svccfg import warnings. See /var/svc/log/system-manifest-import:default.log .  
     Reading ZFS config: done.  
     Mounting ZFS filesystems: (6/6)  
     Creating new rsa public/private host key pair  
     Creating new dsa public/private host key pair  
   
     osol console login:  
              
   
 Using PCI devices in guest domains  
 ----------------------------------  
   
 The domain0 can give other domains access to selected PCI devices. This  
 can allow, for example, a non-privileged domain to have access to a  
 physical network interface or disk controller. However, keep in mind  
 that giving a domain access to a PCI device most likely will give the  
 domain read/write access to the whole physical memory, as PCs don't have  
 an IOMMU to restrict memory access to DMA-capable device. Also, it's not  
 possible to export ISA devices to non-domain0 domains (which means that  
 the primary VGA adapter can't be exported. A guest domain trying to  
 access the VGA registers will panic).  
   
 This functionality is only available in NetBSD-5.1 (and later) domain0  
 and domU. If the domain0 is NetBSD, it has to be running Xen 3.1, as  
 support has not been ported to later versions at this time.  
   
 For a PCI device to be exported to a domU, is has to be attached to the  
 `pciback` driver in domain0. Devices passed to the domain0 via the  
 pciback.hide boot parameter will attach to `pciback` instead of the  
 usual driver. The list of devices is specified as `(bus:dev.func)`,  
 where bus and dev are 2-digit hexadecimal numbers, and func a  
 single-digit number:  
   
     pciback.hide=(00:0a.0)(00:06.0)  
   
 pciback devices should show up in the domain0's boot messages, and the  
 devices should be listed in the `/kern/xen/pci` directory.  
   
 PCI devices to be exported to a domU are listed in the `pci` array of  
 the domU's config file, with the format `'0000:bus:dev.func'`  
   
     pci = [ '0000:00:06.0', '0000:00:0a.0' ]  
   
 In the domU an `xpci` device will show up, to which one or more pci  PCI passthrough: Using PCI devices in guest domains
 busses will attach. Then the PCI drivers will attach to PCI busses as  ---------------------------------------------------
 usual. Note that the default NetBSD DOMU kernels do not have `xpci` or  
 any PCI drivers built in by default; you have to build your own kernel  
 to use PCI devices in a domU. Here's a kernel config example:  
   
     include         "arch/i386/conf/XEN3_DOMU"  The dom0 can give other domains access to selected PCI
     #include         "arch/i386/conf/XENU"           # in NetBSD 3.0  devices. This can allow, for example, a non-privileged domain to have
   access to a physical network interface or disk controller.  However,
     # Add support for PCI busses to the XEN3_DOMU kernel  keep in mind that giving a domain access to a PCI device most likely
     xpci* at xenbus ?  will give the domain read/write access to the whole physical memory,
     pci* at xpci ?  as PCs don't have an IOMMU to restrict memory access to DMA-capable
   device.  Also, it's not possible to export ISA devices to non-dom0
     # Now add PCI and related devices to be used by this domain  domains, which means that the primary VGA adapter can't be exported.
     # USB Controller and Devices  A guest domain trying to access the VGA registers will panic.
   
     # PCI USB controllers  If the dom0 is NetBSD, it has to be running Xen 3.1, as support has
     uhci*   at pci? dev ? function ?        # Universal Host Controller (Intel)  not been ported to later versions at this time.
   
   For a PCI device to be exported to a domU, is has to be attached to
   the "pciback" driver in dom0.  Devices passed to the dom0 via the
   pciback.hide boot parameter will attach to "pciback" instead of the
   usual driver.  The list of devices is specified as "(bus:dev.func)",
   where bus and dev are 2-digit hexadecimal numbers, and func a
   single-digit number:
   
     # USB bus support          pciback.hide=(00:0a.0)(00:06.0)
     usb*    at uhci?  
   
     # USB Hubs  pciback devices should show up in the dom0's boot messages, and the
     uhub*   at usb?  devices should be listed in the `/kern/xen/pci` directory.
     uhub*   at uhub? port ? configuration ? interface ?  
   
     # USB Mass Storage  PCI devices to be exported to a domU are listed in the "pci" array of
     umass*  at uhub? port ? configuration ? interface ?  the domU's config file, with the format "0000:bus:dev.func".
     wd*     at umass?  
     # SCSI controllers  
     ahc*    at pci? dev ? function ?        # Adaptec [23]94x, aic78x0 SCSI  
   
     # SCSI bus support (for both ahc and umass)          pci = [ '0000:00:06.0', '0000:00:0a.0' ]
     scsibus* at scsi?  
   
     # SCSI devices  In the domU an "xpci" device will show up, to which one or more pci
     sd*     at scsibus? target ? lun ?      # SCSI disk drives  busses will attach.  Then the PCI drivers will attach to PCI busses as
     cd*     at scsibus? target ? lun ?      # SCSI CD-ROM drives  usual.  Note that the default NetBSD DOMU kernels do not have "xpci"
   or any PCI drivers built in by default; you have to build your own
   kernel to use PCI devices in a domU.  Here's a kernel config example;
   note that only the "xpci" lines are unusual.
   
           include         "arch/i386/conf/XEN3_DOMU"
   
           # Add support for PCI busses to the XEN3_DOMU kernel
           xpci* at xenbus ?
           pci* at xpci ?
   
           # PCI USB controllers
           uhci*   at pci? dev ? function ?        # Universal Host Controller (Intel)
   
           # USB bus support
           usb*    at uhci?
   
           # USB Hubs
           uhub*   at usb?
           uhub*   at uhub? port ? configuration ? interface ?
   
           # USB Mass Storage
           umass*  at uhub? port ? configuration ? interface ?
           wd*     at umass?
           # SCSI controllers
           ahc*    at pci? dev ? function ?        # Adaptec [23]94x, aic78x0 SCSI
   
           # SCSI bus support (for both ahc and umass)
           scsibus* at scsi?
   
           # SCSI devices
           sd*     at scsibus? target ? lun ?      # SCSI disk drives
           cd*     at scsibus? target ? lun ?      # SCSI CD-ROM drives
   
   
 NetBSD as a domU in a VPS  NetBSD as a domU in a VPS
Line 959  hardware.  This section explains how to  Line 856  hardware.  This section explains how to 
 virtual private server where you do not control or have access to the  virtual private server where you do not control or have access to the
 dom0.  dom0.
   
 TODO: Perhaps reference panix, prmgr, amazon as interesting examples.  VPS operators provide varying degrees of access and mechanisms for
   configuration.  The big issue is usually how one controls which kernel
   is booted, because the kernel is nominally in the dom0 filesystem (to
   which VPS users do not normally have acesss).
   
   A VPS user may want to compile a kernel for security updates, to run
   npf, run IPsec, or any other reason why someone would want to change
   their kernel.
   
   One approach is to have an adminstrative interface to upload a kernel,
   or to select from a prepopulated list.
   
   Otehr approaches are pvgrub and py-grub, which are ways to start a
   bootloader from the dom0 instead of the actual domU kernel, and for
   that loader to then load a kernel from the domU filesystem.  This is
   closer to a regular physical computer, where someone who controls a
   machine can replace the kernel.
   
 TODO: Somewhere, discuss pvgrub and py-grub to load the domU kernel  prmgr and pvgrub
 from the domU filesystem.  ----------------
   
   TODO: Perhaps reference panix, prmgr, amazon as interesting examples.
   Explain what prmgr does.
   
 Using npf  Using npf
 ---------  ---------

Removed from v.1.49  
changed lines
  Added in v.1.56


CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb