Fork me on GitHub

Gentoo Hardened ZFS rootfs with dm-crypt/luks 0.6.5

Disclaimer

  1. Keep in mind that ZFS on Linux is supported upstream, for differing values of support
  2. I do not care much for hibernate, normal suspending works.
  3. This is for a laptop/desktop, so I choose multilib.
  4. IANAL
  5. If you patch the kernel to add in ZFS support directly, you cannot share the binary, the cddl and gpl2 are not compatible in that way.

Initialization

Make sure your installation media supports zfs on linux and installing whatever bootloader is required (uefi needs media that supports it as well). It does not matter too much if it is older media, as long as it supports what you need, Gentoo is nice like that. Here is an iso that works well for me at this link Live DVDs newer then 12.1 should also have support.

Formatting

I will be assuming the following.

  1. /boot on /dev/sda1
  2. cryptroot on /dev/sda2
  3. An optional bios_boot partition for grub (gpt stuff)
  4. swap inside cryptroot OR not used.

When using GPT for partitioning, create the first partition at 1M, just to make sure you are on a sector boundryMost newer drives are 4k advanced format drives. Because of this you need ashift=12, some/most newer SSDs need ashift=13 or greatercompression set to lz4 will make your system incompatible with upstream (oracle) zfs, if you want to stay compatible then just set compression=on

General Setup

# setup encrypted partition
# aes-xts-plain64 was chosen due to speed, xts-essiv SHOULD be more secure, but about half as slow, on aes-ni I was getting about 200MBps
cryptsetup luksFormat -l 512 -c aes-xts-plain64 -h sha512 /dev/sda2
cryptsetup luksOpen /dev/sda2 cryptroot

# setup ZFS
# a deeper look into this can be found at [this link](https://github.com/ryao/zfs-overlay/blob/master/zfs-install "ryao's repo")
zpool create -f -o ashift=12 -o cachefile=/tmp/zpool.cache -O normalization=formD -m none -R /mnt/gentoo mypool /dev/mapper/cryptroot
zfs create -o mountpoint=none -o compression=lz4 mypool/ROOT
# rootfs
zfs create -o mountpoint=/ mypool/ROOT/rootfs
# system mountpoints were seperated so that we can set nodev and nosuid as mount options
zfs create -o mountpoint=/opt mypool/ROOT/rootfs/OPT
zfs create -o mountpoint=/usr mypool/ROOT/rootfs/USR
zfs create -o mountpoint=/usr/src -o sync=disabled mypool/ROOT/rootfs/USR/SRC
zfs create -o mountpoint=/var mypool/ROOT/rootfs/VAR
# portage
zfs create -o mountpoint=none -o setuid=off mypool/GENTOO
zfs create -o mountpoint=/usr/portage -o atime=off mypool/GENTOO/portage
zfs create -o mountpoint=/usr/portage/distfiles -o compression=off mypool/GENTOO/distfiles
zfs create -o mountpoint=/usr/portage/packages -o compression=off mypool/GENTOO/packages
zfs create -o mountpoint=/var/tmp/portage -o sync=disabled mypool/GENTOO/build-dir
# homedirs
zfs create -o mountpoint=/home mypool/HOME
zfs create -o mountpoint=/root mypool/HOME/root
# replace USER with your username
zfs create -o mountpoint=/home/USER mypool/HOME/USER
# set the bootfs, some initrams require this
zpool set bootfs=mypool/ROOT/rootfs mypool

cd /mnt/gentoo

# Download the latest stage3 and extract it.
wget ftp://gentoo.osuosl.org/pub/gentoo/releases/amd64/autobuilds/current-stage3-amd64-hardened/stage3-amd64-hardened-*.tar.bz2
tar -xf /mnt/gentoo/stage3-amd64-hardened-*.tar.bz2 -C /mnt/gentoo

# get the latest portage tree
emerge --sync

# copy the zfs cache from the live system to the chroot
mkdir -p /mnt/gentoo/etc/zfs
cp /tmp/zpool.cache /mnt/gentoo/etc/zfs/zpool.cache

Kernel Config

If you are compiling the modules into the kernel staticly, then keep these things in mind.

  • When configuring the kernel, make sure that CONFIG_SPL and CONFIG_ZFS are set to 'Y'.
  • Portage will want to install sys-kernel/spl and sys-fs/zfs-kmod when emerge sys-fs/zfs is run because of dependencies. Also, both are still necessary to make the sys-fs/zfs configure script happy.
  • You do not need to run or install module-rebuild.

Install as normal up until the kernel install.

echo "=sys-kernel/genkernel-3.4.40 ~amd64       #needed for zfs and encryption support" >> /etc/portage/package.accept_keywords
emerge sys-kernel/genkernel
emerge sys-kernel/gentoo-sources                #or hardned-sources

# patch the kernel

# If you want to build the modules into the kernel directly, you will need to patch the kernel directly.  Otherwise, skip the patch commands.
# configure the kernel then prepare for zfs / spl
cd /usr/src/linux
make prepare
make scripts
env EXTRA_ECONF='--enable-linux-builtin' ebuild /usr/portage/sys-kernel/spl/spl-0.6.5.ebuild clean configure
(cd /var/tmp/portage/sys-kernel/spl-0.6.5/work/spl-0.6.5/ && ./copy-builtin /usr/src/linux)
env EXTRA_ECONF='--with-spl=/usr/src/linux --enable-linux-builtin --with-spl-obj=/usr/src/linux' ebuild /usr/portage/sys-fs/zfs-kmod/zfs-kmod-0.6.5.ebuild clean configure
(cd /var/tmp/portage/sys-fs/zfs-kmod-0.6.5  /work/zfs-kmod-0.6.5 && ./copy-builtin /usr/src/linux)
env EXTRA_ECONF='--with-spl=/usr/src/linux --enable-linux-builtin' ebuild /usr/portage/sys-fs/zfs/zfs-0.6.5.ebuild clean merge
mkdir -p /etc/portage/profile
echo 'sys-fs/zfs -kernel-builtin' >> /etc/portage/profile/package.use.mask
echo 'sys-fs/zfs kernel-builtin' >> /etc/portage/package.use

# finish configuring, building and installing the kernel making sure to enable dm-crypt support

# if not building zfs into the kernel, install module-rebuild
emerge module-rebuild

# install SPL and ZFS stuff zfs pulls in spl automatically
mkdir -p /etc/portage/profile
echo 'sys-fs/zfs -kernel-builtin' >> /etc/portage/profile/package.use.mask
echo 'sys-fs/zfs kernel-builtin' >> /etc/portage/package.use
emerge sys-fs/zfs

# Add zfs to the correct runlevel
rc-update add zfs-mount boot

# initrd creation, add '--callback="module-rebuild rebuild"' to the options if not building the modules into the kernel
genkernel --luks --zfs --disklabel initramfs

Finish installing as normal, your kernel line should look like this, and you should also have a the initrd defined.

# kernel line for grub2, libzfs support is not needed in grub2 because you are not mounting the filesystem directly.
linux  /kernel-3.5.0-gentoo real_root=ZFS=mypool/ROOT/rootfs crypt_root=/dev/sda2 dozfs=force ro
initrd /initramfs-genkernel-x86_64-3.5.0

In /etc/fstab, make sure BOOT, ROOT and SWAP lines are commented out and finish the install.

You should now have a working encryped zfs install.

social