File:  [NetBSD Developer Wiki] / wikisrc / zfs.mdwn
Revision 1.26: download - view: text, annotated - select for diffs
Wed Jun 24 15:19:56 2020 UTC (3 months, 3 weeks ago) by wiki
Branches: MAIN
CVS tags: HEAD
web commit by jdolecek: Document that zfs REQUIRES 12KB or more kernel stack to work.

# ZFS on NetBSD

This page attempts to do two things: provide enough orientation and
pointers to standard ZFS documentation for NetBSD users who are new to
ZFS, and to describe NetBSD-specific ZFS information.  It is
emphatically not a tutorial or an introduction to ZFS.

Many things are marked with \todo because they need a better
explanation, and some have question marks, indicating that the
statement needs verification.

# Documentation Pointers

See the man pages for zfs(8) and zpool(8).

  - [Oracle ZFS Administration Manual](https://docs.oracle.com/cd/E26505_01/html/E37384/index.html)
  - [FreeBSD Handbook ZFS Chapter](https://www.freebsd.org/doc/handbook/zfs.html)
  - [OpenZFS admin docs index page](https://github.com/openzfs/zfs/wiki/Admin-Documentation)

  - [Wikipedia](https://en.wikipedia.org/wiki/ZFS)

# Status of ZFS in NetBSD

## Sources of ZFS code

\todo Verify/fix.

Currently, there are multiple ZFS projects and codebases:

  - ZFS as released under the CDDL (common ancestor)
  - [OpenZFS](http://www.open-zfs.org/wiki/Main_Page) [github wiki](https://github.com/openzfs/zfs/wiki)
  - [zfsonlinux](https://zfsonlinux.org/)
  - [OpenZFS on OS X ](https://openzfsonosx.org/) [repo](https://github.com/openzfsonosx)
  - proprietary ZFS in Solaris (not relevant in open source)

OpenZFS is a coordinating project to align open ZFS codebases.  There
is a notion of a shared core codebase and OS-specific adaptation code.
As of early 2020, it appears that there are two co-root repositories:
illumos and zfsonlinux, with a notion that fixes and improvements
should be cross-shared.  This is different from the situation through
late 2019 where sharing was done via a designated sharing repository,
and appears to be a new, more efficient, procedure among cooperating
people, rather than any kind of fork.

  - [deprecation of OpenZFS repo](https://github.com/openzfs/openzfs/)
  - [zfsonlinux relationship to OpenZFS](https://github.com/openzfs/zfs/wiki/OpenZFS-Patches)

\todo Explain how FreeBSD code relates to zfsonlinux (imported/merged and fixes pushed upstream?).

\todo Explain how the NetBSD code relates (imported from FreeBSD? intent to continue to track?)

\todo Explain if NetBSD has a plan to join zfsonlinux as a first-class
member, vs tracking via FreeBSD.

\todo Explain how OpenZFS on OS X relates (tracks zfsonlinux?).

See [FreeBSD's history](https://wiki.freebsd.org/ZFSTuningGuide).

## NetBSD code history

\todo This section really needs help.

Before NetBSD 8, NetBSD imported ZFS code from ?, which was version ?

Before NetBSD 9, NetBSD imported updated ZFS code from FreeBSD.  That
FreeBSD code came from ?, which was version ?

## NetBSD 8 and earlier

While there is some ZFS code, it is old, and seems to have significant
problems.  If one wants to use ZFS, first upgrade to NetBSD 9.  It is
unlikely that anyone is interested in helping, other than telling you
to upgrade to 9.

(Reports of how well NetBSD 8 works are welcome on netbsd-users, if it
can actually be recommended for use.)

## NetBSD 9

There have been fixes since 9.0 RELEASE.  It is best to upgrade along
the netbsd-9 branch, but the release should be ok.  Most aspects work solidly.

\todo Explain this in terms of versions of FreeBSD OpenZFS and/or
zfsonlinux.

\todo This supports pool version 28/5000 (really true?).  Of the
feature flags found in modern OpenZFS, \todo are supported.

Generally, fixes to ZFS in current will be pulled up to 9, but new
features typically will not be.

## NetBSD current

The ZFS code in current is very similar to that in 9.

There is initial support for [[ZFS root|wiki/RootOnZFS]], via booting from
ffs and pivoting.

One can make a ccd using a zvol as a component.  See the zvol section below.

## Things that aren't supported yet

\todo hotswap (maybe - not clear exactly what this means)

\todo direct boot into zfs root (via boot blocks reading zfs)

## Architectures

Most people seem to be using amd64.

To build zfs, one puts MKZFS=yes in mk.conf.  This is default on amd64
and aarch64 on netbsd-9.  In current, it is also default on sparc64.

More or less, zfs can be enabled on an architecture when it is known
to build and run reliably.  (Of course, users are welcome to build it
and report.)

# NetBSD-specific information

## rc.conf

The main configuration is to put zfs=YES in rc.conf, so that the rc.d
scripts bring up ZFS and mount ZFS file systems.

## pool locations

One can add disks or parts of disks into pools.  Methods of specifying
areas to be included include:

  - entire disks (e.g., /dev/rwd0d on amd64)
  - disklabel partitions (e.g., /dev/sd0e)
  - wedges (e.g., /dev/dk0)

## legacy vs ? mount points

\todo Explain, if this is NetBSD specific.  Explain consequences, as
this seems to have something to do with mount ordering.

## mount order

NetBSD 9 mounts other file systems and then ZFS file systems.  This can
be a problem if /usr/pkgsrc is on ZFS and /usr/pkgsrc/distfiles is on
NFS.  A workaround is to use noauto and do the mounts in
/etc/rc.local.

NetBSD current after 20200301 mounts ZFS first. \todo Explain
consequences.

## NFS

\todo Verify if this is accurate.

zfs filesystems can be exported over NFS.  While there are zfs
commands that appear to be about controlling exports, they simply
print information that can be added to exports(5).

This is reported to work on 9.0 STABLE, but to cause a panic on
current (20200302).  See [misc/55042](http://gnats.netbsd.org/55042).

## zvol

Within a ZFS pool, the standard approach is to have file systems, but
one can also create a zvol, which is a block device of a certain size.

\todo The zvol will appear as /dev/???? and can be used in many
respects like a slice.  However, the system will not read disklabels
and gpt labels from a zvol; in this respect it is more like a disklabel
partition or wedge than a disk drive.

\todo Explain that one can export a zvol via iscsi.

\todo Explain if one can swap on a zvol.

\todo Explain that one can use ccd to create a normal-looking disk
from a zvol.  This allows reading a GPT label from the zvol, which is
useful in case the zvol had been exported via iscsi and some other
system created a label.

## TRIM

There is some notion of TRIM and zfs using it.

\todo Explain how this relates to NetBSD.

# Memory usage

Basically, ZFS uses lots of memory and most people run it on systems
with large amounts of memory.  NetBSD works well on systems with
comparatively small amounts of memory.  So a natural question is how
well ZFS works on one's VAX with 2M of RAM :-)

More seriously, one might ask if it is reasonable to run ZFS on a RPI3
with 1G of RAM, or even if it is reasonable on a system with 4G.

\todo Give ballpark level for minimum sane RAM, and the amount which
is cleanly enough.

For now, a good guess is that a 4G system with only 1T of disk is
probably ok, and that 1G is very likely not ok.

Besides RAM, zfs requires that architecture kernel stack size is at least 12KB or more - some
operations cause stack overflow with 8KB kernel stack. On NetBSD, the architectures
with 16KB kernel stack are amd64, sparc64, powerpc, and experimental ia64, hppa. mac68k and sh3 have 12KB kernel
stack. All others use only 8KB stack, which is not enough to run zfs.

FreeBSD has some documentation about memory use.  There is a notion of
a minimum of 1G (used for ZFS), and using 1G for 1T of storage, and
more if deduplication is enabled.  FreeBSD considers all i386 systems
to be low memory; this appears to be a clue.

\todo Explain if the FreeBSD sysctl list applies, or if not what we
should do instead.

  - [FreeBSD low memory documentation](https://www.freebsd.org/doc/handbook/zfs-advanced.html)

# Interoperability with other systems

\todo Explain pool version and feature flags relationship to FreeBSD,
Linux, OpenIndiana/Illumos/?, and ?

\todo Explain how to configure a pool in terms of version/features for
use with particula other systems.

# Quick Start

See the [FreeBSD Quickstart
Guide](https://www.freebsd.org/doc/handbook/zfs-quickstart.html); only
the first item is NetBSD specific.

  - Put zfs=YES in rc.conf.

  - Create a pool as "zpool create pool1 /dev/dk0".

  - df and see /pool1

  - Really, read the FreeBSD docs and the other linked documentation above.

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb