File:  [NetBSD Developer Wiki] / wikisrc / wiki / RootOnZFS.mdwn
Revision 1.1: download - view: text, annotated - select for diffs
Sat Feb 22 19:17:04 2020 UTC (2 years, 3 months ago) by wiki
Branches: MAIN
CVS tags: HEAD
web commit by roy

# Root On ZFS

NetBSD-9 gained much improved ZFS support.
However, one feature it's still missing is the ability to have your system root on ZFS.
For that, we need to teach our boot loader about ZFS, but until then we can work around this limitation by using a FFS boot partition and a ZFS root ramdisk.
You'll also need to use NetBSD-9.99.47 or newer.

The idea is simple - the boot loader loads the NetBSD kernel from FFS and the ZFS root ramdisk.
The ramdisk then mounts the boot partition and copies the requires ZFS modules from it into the ramdisk.
ZFS is then initialised to load the modules and then umounts the boot partition.
This step saves you from having to load the modules in boot.cfg.
Because the initial root is on a ramdisk, we can then load the boot partition from inside the ZFS root to maintain it.

Once ZFS is initialised, the ZFS root is then mounted and sysctl then instructs init to chroot to it and the normal boot procedure then starts.
Once started, the kernel and modules can updated by mounting the boot partition to /altroot.

## Assumptions

The ramdisk does make some assumptions as there is no way to pass variables from boot.cfg:

  *  The boot partition can be referenced as `NAME=boot` - for GPT this is as simple as giving it a label.
  *  The ZFS root pool is called `rpool` and the root filesystem is called `ROOT`.

## Generic setup

Until the NetBSD installer can be updated, here are the manual steps from fresh:

  *  Build the ramdisk (`cd src/distrib/amd64/ramdisks/ramdisk-zfsroot; nbmake-amd64`)
  *  Boot the NetBSD installer
  *  Create a small (I used 2G) FFS partition and a normal swap partition
  *  Create a partition for ZFS
  *  Finish the NetBSD installation - install the etc, base and maybe rescue sets
  *  Reboot into your minimal NetBSD install
  *  Label the boot partition boot - if you don't do this then the ramdisk will moan it can't load modules

## ZFS Setup

So far, so good. Now we need to make the needed adjustments to change root:

  *  Create a ZFS pool called `rpool` on your ZFS partition
  *  Create a ZFS filesystem called `ROOT` in `rpool`
  *  Set the mountpoint of `rpool/ROOT` to legacy so that the mount command can deal with it
  *  Create any other ZFS filesystems you want to - if you create `/usr` or `/var` or anything in these they **must** to be legacy mount points as well
  *  `zfs umount rpool/ROOT`
  *  Add `rpool/ROOT /altroot zfs rw` to /etc/fstab
  *  `mount /altroot`
  *  Extract the sets you need to `/altroot`. At a minimum you will need etc and base.
  *  Copy `/etc/wscons.conf`, `/etc/fstab` and any other config files sysinst might have created to `/etc/rc.conf` to `/altroot/etc`
  *  Set `zfs=YES` in `/altroot/etc/rc.conf`
  *  Remove `rpool/ROOT` from `/altroot/etc/fstab`
  *  Add `name=boot /altroot ffs rw,noauto` to `/altroot/etc/fstab` along with any ZFS filesytems you created in '/usr' and '/var'
  *  Add the filesystems as well to `critical_filesystems_local` in `/altroot/etc/rc.conf` - such as `critical_filesystems_local="/usr /var /var/log"`
  *  Copy the `ramdisk-zfsroot.fs` to `/`
  *  Edit `/boot.cfg` and add `menu=Boot ZFS Root:fs /ramdisk-zfsroot.fs;boot`

And done! You can now reboot and enjoy your root on ZFS and all the benefits it brings.
When updating the kernel, remember to `mount /altroot` and update it there as well as the modules.

## Future work

All these steps could be done in the installer.
This is probably an easier task than adding ZFS support to the bootloader. But don't let that stop you if you are stuck for something to do!

# DANGER NETBSD ROOT ON ZFS USER, DANGER!

There seems to be an issue accessing `/dev` nodes on ZFS.
While -current has enough fixes to work somewhat, accessing the boot partition whilst while root is on ZFS will hang, panic and maybe even corrupt your boot partition.
As such, you'll have to reboot into single user. From here you can mount the ZFS partition to `/altroot` and copy the updated kernel and modules from there to the boot partition.
Ironically, this is what the ramdisk approach was designed to actively avoid. Hopefully we can get it resolved soon.

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb