ZFS on Linux with Encryption Part 2: The Compiling

First off: Warning. I don’t know what the stability of this feature is. It’s been in the code for a couple of months, it hasn’t been widely used. I’ve been testing it, and so far it’s worked as expected. 

In exploring native encryption, I attempted to get it on Linux/ZFS using the instruction on this site: https://blog.heckel.xyz/2017/01/08/zfs-encryption-openzfs-zfs-on-linux/. While I’m sure they worked at the time, the code in the referenced non-standard repos has changed and I couldn’t get anything to compile correctly.

After trying for about a day, I realized (later than I care to admit) that I should have just tried the standard repos. They worked like a charm. The instructions below compiled and successfully installed ZFS on Linux with dataset encryption on both Ubuntu 17.10 and CentOS 7.4 in the November/December 2017 time frame.

Compiling ZFS with Native Encryption

The first step is to make sure a development environment is installed on your Linux system. Make sure you have compiler packages, etc. installed. Here’s a few packages for CentOS you’ll need (you’ll need similar packages/libraries for whatever platform you run).

• openssl-devel
• attr, libattr-devel
• libblkid-devel
• zlib-devel
• libuuid-devel

The builds were pretty good at telling you what packages you needed if they were missing, so of course install any that are requested.

You’ll need to build the SPL code and the ZFS code.

First, build the SPL code.

git clone https://github.com/zfsonlinux/spl
cd spl
./autogen
./configure
make
make install

Then the ZFS code:

git clone https://github.com/zfsonlinux/zfs
cd zfs
./autogen
./configure --prefix=/usr  # <-- This puts the binaries in /usr/sbin instead of /usr/local/sbin
make
make install

If you try the zfs command right away, you’ll probably get something similar to the following:

/sbin/zfs: error while loading shared libraries: libnvpair.so.1: cannot open shared object file: No such file or directory

Running ldconfig usually fixes that.

You might need to modprobe zfs to get the modules loaded, especially if you end up rebooting. There’s of course ways to auto-load the modules depending on your distribution.

Creating the zpool

zpool create -o ashift=12 storage raidz2 /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg

The -o ashift=12 is important if you have 4K sector drives, which these 8 TB WD Reds are. If you don’t throw that option in, your performance will suffer, big-time. I found my pool performed about 25% of what it did when ashift=12 was selected.

I was doing copy tests with Samba and getting only 25-30 MB/s. Once I destroyed the zpool and used ashift=12 for a new zpool on the same drives, I was able to get ~120 MB/s, which is the practical limit for a 1 Gigabit link (1,000 Gbit / 8 = 125 MB/s). Local copies were faster. Figure this out ahead of time, because to set the ashift you have to do zpool destroy, which does what it sounds like it does: Destroys the pool (and data).

The zpool will be called “storage” (yes, original) so of course use whatever name you prefer. raidz2 uses a double-parity system, so out of 6 drives, I would get a pool with the space of 4 of them (roughly 32 TBs).

The rest are the devices themselves. You don’t need to partition the drives, ZFS does it automatically.

Encryption is done on a dataset by dataset basis, which is nice to be able to have some storage be encrypted and other parts not. To create an encrypted dataset, first enable the feature in the zpool.

zpool set feature@encryption=enabled storage

Then create a new dataset under the storage zpool using a passphrase (you can also use a keyfile, but I’m opting for a passphrase):

zfs create -o encryption=on -o keylocation=prompt -o keyformat=passphrase storage/encrypted

Anything you put in /storage/encrypted/ will now be encrypted at rest.

When the system comes up, the zpool could be automatically imported (or you have to import it manually) but the /storage/encrypted/ dataset won’t be automatically added.

# zpool import storage
# zfs mount storage/encrypted -l
# Enter passphrase for 'storage/encrypted':

Once you enter the passphrase, the dataset is mounted.