ZFS combines a volume manager, RAID and a filesystem in one system, and checksums every block of data. It detects silent corruption (“bit rot”), repairs it from redundant copies, and adds instant snapshots, efficient replication and transparent compression. It’s a favourite for storage and backup servers, and on dedicated servers with several drives.

Is ZFS right for this server?

Good fit:

  • dedicated servers with 2+ drives where data integrity matters;
  • backup targets and file storage;
  • hosts that benefit from snapshots and replication (zfs send).

Less suited:

  • small single-disk VPSes. It works, but you lose self-healing (no redundancy) and ZFS’s RAM-hungry cache competes with your applications;
  • systems where you can’t install out-of-tree kernel modules.

Install (Ubuntu and Debian)

sudo apt install zfsutils-linux          # Ubuntu
sudo apt install linux-headers-amd64 zfsutils-linux zfs-dkms   # Debian (contrib repo enabled)

On RHEL-family systems, use the OpenZFS repository (zfs-release package). Check:

zfs version

Create a pool

Identify disks by stable IDs, not sdb and sdc:

ls -l /dev/disk/by-id/

Mirror (like RAID 1):

sudo zpool create -o ashift=12 tank mirror \
  /dev/disk/by-id/ata-DISK_SERIAL_1 /dev/disk/by-id/ata-DISK_SERIAL_2

RAIDZ2 (like RAID 6, survives two failures):

sudo zpool create -o ashift=12 tank raidz2 /dev/disk/by-id/disk{1..6}
  • ashift=12 means 4 KiB sectors. That’s right for virtually all modern disks and SSDs, and it can’t be changed later.
  • The pool is mounted at /tank automatically.

Check it:

zpool status
zpool list

Datasets: ZFS’s “filesystems”

Instead of partitions, create datasets. Each has its own properties and snapshots, and all of them share the pool’s space:

sudo zfs set compression=lz4 tank          # inherited by children
sudo zfs set atime=off tank
sudo zfs create tank/backups
sudo zfs create tank/www
sudo zfs create -o recordsize=16k tank/mysql        # match the InnoDB page size
sudo zfs set quota=200G tank/www
zfs list

compression=lz4 (or zstd) is almost always worth it. It saves space and often increases speed, because less data is read from disk. See the ratio with zfs get compressratio.

Change the mount point with sudo zfs set mountpoint=/var/www tank/www.

Snapshots

Snapshots are instant and take no space at first. They only grow as data changes:

sudo zfs snapshot tank/www@before-upgrade
zfs list -t snapshot

Browse a snapshot’s files (read-only) in the hidden .zfs directory:

ls /tank/www/.zfs/snapshot/before-upgrade/

Restore one file by copying it back, or roll the whole dataset back:

sudo zfs rollback tank/www@before-upgrade      # discards later changes
sudo zfs destroy tank/www@before-upgrade

Automate snapshots with sanoid or zfs-auto-snapshot (hourly, daily and monthly with retention).

Replication: zfs send / receive

Send a snapshot to another ZFS machine, the efficient way to back up:

sudo zfs send tank/www@daily-1 | ssh backup sudo zfs receive pool/www
sudo zfs send -i @daily-1 tank/www@daily-2 | ssh backup sudo zfs receive pool/www   # incremental

Only changed blocks travel. syncoid (from the sanoid project) wraps this into one command. Combined with off-site storage, this is a strong backup setup. See the 3-2-1 rule.

Scrubs: find and fix corruption

A scrub reads every block and verifies checksums. With redundancy, ZFS repairs bad blocks automatically:

sudo zpool scrub tank
zpool status tank           # progress and any errors found

Ubuntu and Debian schedule a monthly scrub automatically. Check zpool status regularly, or set up the ZED daemon to email you about errors (ZED_EMAIL_ADDR in /etc/zfs/zed.d/zed.rc).

Replace a failed disk

zpool status                       # shows the FAULTED / UNAVAIL device
sudo zpool replace tank /dev/disk/by-id/OLD /dev/disk/by-id/NEW
zpool status                       # resilver progress

Memory: the ARC

ZFS caches data in RAM (the ARC). By default it can use up to about half of RAM on Linux. It gives memory back under pressure, but on servers that also run databases or applications, cap it:

echo "options zfs zfs_arc_max=4294967296" | sudo tee /etc/modprobe.d/zfs.conf   # 4 GiB
sudo update-initramfs -u
# apply now without reboot:
echo 4294967296 | sudo tee /sys/module/zfs/parameters/zfs_arc_max

Watch it with arc_summary. Note that free shows the ARC as “used”, not “cache”.

Things to avoid

  • Dedup: it needs a lot of RAM and is rarely worth it. Use compression instead.
  • Filling the pool past ~80–90%: performance drops sharply. Set quotas or alerts.
  • Hardware RAID underneath ZFS: give ZFS the raw disks (HBA/IT mode), so it can see and repair errors.
  • Adding a single disk to a redundant pool with zpool add: it becomes a non-redundant vdev, and you can’t easily remove it. Double-check add versus attach.

Quick reference

Task Command
Pool status zpool status
Create mirror zpool create -o ashift=12 tank mirror d1 d2
Create dataset zfs create tank/name
Compression zfs set compression=lz4 tank
Snapshot zfs snapshot tank/name@label
Roll back zfs rollback tank/name@label
Replicate zfs send … | ssh host zfs receive …
Scrub zpool scrub tank

Related: Btrfs and software RAID with mdadm.

  • Linux
  • Storage
  • ZFS