ZFS on NetBSD

This page attempts to do two things: provide enough orientation and pointers to standard ZFS documentation for NetBSD users who are new to ZFS, and to describe NetBSD-specific ZFS information. It is emphatically not a tutorial or an introduction to ZFS.

Many things are marked with \todo because they need a better explanation, and some have question marks, indicating that the statement needs verification.

Documentation Pointers

See the man pages for zfs(8) and zpool(8).

Status of ZFS in NetBSD

Sources of ZFS code

\todo Verify/fix.

Currently, there are multiple ZFS projects and codebases:

OpenZFS is a coordinating project to align open ZFS codebases. There is a notion of a shared core codebase and OS-specific adaptation code. As of early 2020, it appears that there are two co-root repositories: illumos and zfsonlinux, with a notion that fixes and improvements should be cross-shared. This is different from the situation through late 2019 where sharing was done via a designated sharing repository, and appears to be a new, more efficient, procedure among cooperating people, rather than any kind of fork.

\todo Explain how FreeBSD code relates to zfsonlinux (imported/merged and fixes pushed upstream?).

\todo Explain how the NetBSD code relates (imported from FreeBSD? intent to continue to track?)

\todo Explain if NetBSD has a plan to join zfsonlinux as a first-class member, vs tracking via FreeBSD.

\todo Explain how OpenZFS on OS X relates (tracks zfsonlinux?).

See FreeBSD's history.

NetBSD code history

\todo This section really needs help.

Before NetBSD 8, NetBSD imported ZFS code from ?, which was version ?

Before NetBSD 9, NetBSD imported updated ZFS code from FreeBSD. That FreeBSD code came from ?, which was version ?

NetBSD 8 and earlier

While there is some ZFS code, it is old, and seems to have significant problems. If one wants to use ZFS, first upgrade to NetBSD 9. It is unlikely that anyone is interested in helping, other than telling you to upgrade to 9.

(Reports of how well NetBSD 8 works are welcome on netbsd-users, if it can actually be recommended for use.)

NetBSD 9

There have been fixes since 9.0 RELEASE. It is best to upgrade along the netbsd-9 branch, but the release should be ok. Most aspects work solidly.

\todo Explain this in terms of versions of FreeBSD OpenZFS and/or zfsonlinux.

\todo This supports pool version 28/5000 (really true?). Of the feature flags found in modern OpenZFS, \todo are supported.

Generally, fixes to ZFS in current will be pulled up to 9, but new features typically will not be.

NetBSD current

The ZFS code in current is very similar to that in 9.

There is initial support for ZFS root, via booting from ffs and pivoting.

One can make a ccd using a zvol as a component. See the zvol section below.

Things that aren't supported yet

\todo hotswap (maybe - not clear exactly what this means)

\todo direct boot into zfs root (via boot blocks reading zfs)

Architectures

Most people seem to be using amd64.

To build zfs, one puts MKZFS=yes in mk.conf. This is default on amd64 and aarch64 on netbsd-9. In current, it is also default on sparc64.

More or less, zfs can be enabled on an architecture when it is known to build and run reliably. (Of course, users are welcome to build it and report.)

NetBSD-specific information

rc.conf

The main configuration is to put zfs=YES in rc.conf, so that the rc.d scripts bring up ZFS and mount ZFS file systems.

pool locations

One can add disks or parts of disks into pools. Methods of specifying areas to be included include:

legacy vs ? mount points

\todo Explain, if this is NetBSD specific. Explain consequences, as this seems to have something to do with mount ordering.

mount order

NetBSD 9 mounts other file systems and then ZFS file systems. This can be a problem if /usr/pkgsrc is on ZFS and /usr/pkgsrc/distfiles is on NFS. A workaround is to use noauto and do the mounts in /etc/rc.local.

NetBSD current after 20200301 mounts ZFS first. \todo Explain consequences.

NFS

\todo Verify if this is accurate.

zfs filesystems can be exported over NFS. While there are zfs commands that appear to be about controlling exports, they simply print information that can be added to exports(5).

This is reported to work on 9.0 STABLE, but to cause a panic on current (20200302). See misc/55042.

zvol

Within a ZFS pool, the standard approach is to have file systems, but one can also create a zvol, which is a block device of a certain size.

\todo The zvol will appear as /dev/???? and can be used in many respects like a slice. However, the system will not read disklabels and gpt labels from a zvol; in this respect it is more like a disklabel partition or wedge than a disk drive.

\todo Explain that one can export a zvol via iscsi.

\todo Explain if one can swap on a zvol.

\todo Explain that one can use ccd to create a normal-looking disk from a zvol. This allows reading a GPT label from the zvol, which is useful in case the zvol had been exported via iscsi and some other system created a label.

TRIM

There is some notion of TRIM and zfs using it.

\todo Explain how this relates to NetBSD.

Memory usage

Basically, ZFS uses lots of memory and most people run it on systems with large amounts of memory. NetBSD works well on systems with comparatively small amounts of memory. So a natural question is how well ZFS works on one's VAX with 2M of RAM :-)

More seriously, one might ask if it is reasonable to run ZFS on a RPI3 with 1G of RAM, or even if it is reasonable on a system with 4G.

\todo Give ballpark level for minimum sane RAM, and the amount which is cleanly enough.

For now, a good guess is that a 4G system with only 1T of disk is probably ok, and that 1G is very likely not ok.

Besides RAM, zfs requires that architecture kernel stack size is at least 12KB or more - some operations cause stack overflow with 8KB kernel stack. On NetBSD, the architectures with 16KB kernel stack are amd64, sparc64, powerpc, and experimental ia64, hppa. mac68k and sh3 have 12KB kernel stack. All others use only 8KB stack, which is not enough to run zfs.

FreeBSD has some documentation about memory use. There is a notion of a minimum of 1G (used for ZFS), and using 1G for 1T of storage, and more if deduplication is enabled. FreeBSD considers all i386 systems to be low memory; this appears to be a clue.

\todo Explain if the FreeBSD sysctl list applies, or if not what we should do instead.

Interoperability with other systems

\todo Explain pool version and feature flags relationship to FreeBSD, Linux, OpenIndiana/Illumos/?, and ?

\todo Explain how to configure a pool in terms of version/features for use with particula other systems.

Quick Start

See the FreeBSD Quickstart Guide; only the first item is NetBSD specific.