Return to RootOnZFS.mdwn CVS log

Up to [NetBSD Developer Wiki] / wikisrc / wiki

Annotation of wikisrc/wiki/RootOnZFS.mdwn, revision 1.1

1.1     ! wiki        1: # Root On ZFS
        !             2: 
        !             3: NetBSD-9 gained much improved ZFS support.
        !             4: However, one feature it's still missing is the ability to have your system root on ZFS.
        !             5: For that, we need to teach our boot loader about ZFS, but until then we can work around this limitation by using a FFS boot partition and a ZFS root ramdisk.
        !             6: You'll also need to use NetBSD-9.99.47 or newer.
        !             7: 
        !             8: The idea is simple - the boot loader loads the NetBSD kernel from FFS and the ZFS root ramdisk.
        !             9: The ramdisk then mounts the boot partition and copies the requires ZFS modules from it into the ramdisk.
        !            10: ZFS is then initialised to load the modules and then umounts the boot partition.
        !            11: This step saves you from having to load the modules in boot.cfg.
        !            12: Because the initial root is on a ramdisk, we can then load the boot partition from inside the ZFS root to maintain it.
        !            13: 
        !            14: Once ZFS is initialised, the ZFS root is then mounted and sysctl then instructs init to chroot to it and the normal boot procedure then starts.
        !            15: Once started, the kernel and modules can updated by mounting the boot partition to /altroot.
        !            16: 
        !            17: ## Assumptions
        !            18: 
        !            19: The ramdisk does make some assumptions as there is no way to pass variables from boot.cfg:
        !            20: 
        !            21:   *  The boot partition can be referenced as `NAME=boot` - for GPT this is as simple as giving it a label.
        !            22:   *  The ZFS root pool is called `rpool` and the root filesystem is called `ROOT`.
        !            23: 
        !            24: ## Generic setup
        !            25: 
        !            26: Until the NetBSD installer can be updated, here are the manual steps from fresh:
        !            27: 
        !            28:   *  Build the ramdisk (`cd src/distrib/amd64/ramdisks/ramdisk-zfsroot; nbmake-amd64`)
        !            29:   *  Boot the NetBSD installer
        !            30:   *  Create a small (I used 2G) FFS partition and a normal swap partition
        !            31:   *  Create a partition for ZFS
        !            32:   *  Finish the NetBSD installation - install the etc, base and maybe rescue sets
        !            33:   *  Reboot into your minimal NetBSD install
        !            34:   *  Label the boot partition boot - if you don't do this then the ramdisk will moan it can't load modules
        !            35: 
        !            36: ## ZFS Setup
        !            37: 
        !            38: So far, so good. Now we need to make the needed adjustments to change root:
        !            39: 
        !            40:   *  Create a ZFS pool called `rpool` on your ZFS partition
        !            41:   *  Create a ZFS filesystem called `ROOT` in `rpool`
        !            42:   *  Set the mountpoint of `rpool/ROOT` to legacy so that the mount command can deal with it
        !            43:   *  Create any other ZFS filesystems you want to - if you create `/usr` or `/var` or anything in these they **must** to be legacy mount points as well
        !            44:   *  `zfs umount rpool/ROOT`
        !            45:   *  Add `rpool/ROOT /altroot zfs rw` to /etc/fstab
        !            46:   *  `mount /altroot`
        !            47:   *  Extract the sets you need to `/altroot`. At a minimum you will need etc and base.
        !            48:   *  Copy `/etc/wscons.conf`, `/etc/fstab` and any other config files sysinst might have created to `/etc/rc.conf` to `/altroot/etc`
        !            49:   *  Set `zfs=YES` in `/altroot/etc/rc.conf`
        !            50:   *  Remove `rpool/ROOT` from `/altroot/etc/fstab`
        !            51:   *  Add `name=boot /altroot ffs rw,noauto` to `/altroot/etc/fstab` along with any ZFS filesytems you created in '/usr' and '/var'
        !            52:   *  Add the filesystems as well to `critical_filesystems_local` in `/altroot/etc/rc.conf` - such as `critical_filesystems_local="/usr /var /var/log"`
        !            53:   *  Copy the `ramdisk-zfsroot.fs` to `/`
        !            54:   *  Edit `/boot.cfg` and add `menu=Boot ZFS Root:fs /ramdisk-zfsroot.fs;boot`
        !            55: 
        !            56: And done! You can now reboot and enjoy your root on ZFS and all the benefits it brings.
        !            57: When updating the kernel, remember to `mount /altroot` and update it there as well as the modules.
        !            58: 
        !            59: ## Future work
        !            60: 
        !            61: All these steps could be done in the installer.
        !            62: This is probably an easier task than adding ZFS support to the bootloader. But don't let that stop you if you are stuck for something to do!
        !            63: 
        !            64: # DANGER NETBSD ROOT ON ZFS USER, DANGER!
        !            65: 
        !            66: There seems to be an issue accessing `/dev` nodes on ZFS.
        !            67: While -current has enough fixes to work somewhat, accessing the boot partition whilst while root is on ZFS will hang, panic and maybe even corrupt your boot partition.
        !            68: As such, you'll have to reboot into single user. From here you can mount the ZFS partition to `/altroot` and copy the updated kernel and modules from there to the boot partition.
        !            69: Ironically, this is what the ramdisk approach was designed to actively avoid. Hopefully we can get it resolved soon.