Fork me on GitHub

Gentoo Hardened ZFS rootfs with dm-crypt/luks 0.6.2

Disclaimer

  1. Keep in mind that ZFS on Linux is supported upstream, for differing values of support
  2. I do not care much for hibernate, normal suspending works.
  3. This is for a laptop/desktop, so I choose multilib.
  4. IANAL
  5. If you patch the kernel to add in ZFS support directly, you cannot share the binary, the cddl and gpl2 are not compatible in that way.

Initialization

Make sure your installation media supports zfs on linux and installing whatever bootloader is required (uefi needs media that supports it as well). I uploaded an iso that works for me at this link Live DVDs newer then 12.1 should also have support, but the previous link has a stable version of zfsonlinux (not 0.6.2 yet, but good enough for install!). If you need to install the bootloader via uefi, you can use one of the latest Fedora CDs, though the gentoo media should be getting support 'soon'. You can install your system normally up until the formatting begins.

Formatting

I will be assuming the following.

  1. /boot on /dev/sda1
  2. cryptroot on /dev/sda2
  3. An optional bios_boot partition for grub (gpt stuff)
  4. swap inside cryptroot OR not used.

When using GPT for partitioning, create the first partition at 1M, just to make sure you are on a sector boundryMost newer drives are 4k advanced format drives. Because of this you need ashift=12, some/most newer SSDs need ashift=13compression set to lz4 will make your system incompatible with upstream (oracle) zfs, if you want to stay compatible then just set compression=ondue to linux not having the best memory management, zfs on luks can be kinda unstable, I have not had a problem on my laptop, but my servers have been sad

General Setup

#setup encrypted partition
#aes-xts-plain64 was chosen due to speed, xts-essiv SHOULD be more secure, but about half as slow, on aes-ni I was getting about 200MBps
cryptsetup luksFormat -l 512 -c aes-xts-plain64 -h sha512 /dev/sda2
cryptsetup luksOpen /dev/sda2 cryptroot

#setup ZFS
zpool create -f -o ashift=12 -o cachefile=/tmp/zpool.cache -O normalization=formD -m none -R /mnt/gentoo mypool /dev/mapper/cryptroot
zfs create -o mountpoint=none -o compression=lz4 mypool/ROOT
#rootfs
zfs create -o mountpoint=/ mypool/ROOT/rootfs
#system mountpoints were seperated so that we can set nodev and nosuid as mount options
zfs create -o mountpoint=/opt mypool/ROOT/rootfs/OPT
zfs create -o mountpoint=/usr mypool/ROOT/rootfs/USR
zfs create -o mountpoint=/usr/src -o sync=disabled mypool/ROOT/rootfs/USR/SRC
zfs create -o mountpoint=/var mypool/ROOT/rootfs/VAR
#portage
zfs create -o mountpoint=none mypool/GENTOO
zfs create -o mountpoint=/usr/portage mypool/GENTOO/portage
zfs create -o mountpoint=/usr/portage/distfiles -o compression=off mypool/GENTOO/distfiles
zfs create -o mountpoint=/usr/portage/packages -o compression=off mypool/GENTOO/packages
zfs create -o mountpoint=/var/tmp/portage -o sync=disabled mypool/GENTOO/build-dir
#homedirs
zfs create -o mountpoint=/home mypool/HOME
zfs create -o mountpoint=/root mypool/HOME/root
#replace user with your username
zfs create -o mountpoint=/home/USER mypool/HOME/USER

cd /mnt/gentoo

#Download the latest stage3 and extract it.
wget ftp://gentoo.osuosl.org/pub/gentoo/releases/amd64/autobuilds/current-stage3-amd64-hardened/stage3-amd64-hardened-*.tar.bz2
tar -xf /mnt/gentoo/stage3-amd64-hardened-*.tar.bz2 -C /mnt/gentoo

#get the latest portage tree
emerge --sync

#copy the zfs cache from the live system to the chroot
mkdir -p /mnt/gentoo/etc/zfs
cp /tmp/zpool.cache /mnt/gentoo/etc/zfs/zpool.cache

Kernel Config

If you are compiling the modules into the kernel staticly, then keep these things in mind.

  • When configuring the kernel, make sure that CONFIG_SPL and CONFIG_ZFS are set to 'Y'.
  • Portage will want to install sys-kernel/spl when emerge sys-fs/zfs is run because of dependencies. Also, sys-kernel/spl is still necessary to make the sys-fs/zfs configure script happy.
  • You do not need to run or install module-rebuild.
  • There have been some updates to the kernel/userspace ioctl since 0.6.0-rc9 was tagged.
    • An issue occurs if newer userland utilities are used with older kernel modules.

Install as normal up until the kernel install.

echo "=sys-kernel/genkernel-3.4.40 ~amd64       #needed for zfs and encryption support" >> /etc/portage/package.accept_keywords
emerge sys-kernel/genkernel
emerge sys-kernel/gentoo-sources                #or hardned-sources

#patch the kernel

#If you want to build the modules into the kernel directly, you will need to patch the kernel directly.  Otherwise, skip the patch commands.
env EXTRA_ECONF='--enable-linux-builtin' ebuild /usr/portage/sys-kernel/spl/spl-0.6.2.ebuild clean configure
(cd /var/tmp/portage/sys-kernel/spl-0.6.2/work/spl-spl-0.6.2 && ./copy-builtin /usr/src/linux)
env EXTRA_ECONF='--with-spl=/usr/src/linux --enable-linux-builtin' ebuild /usr/portage/sys-fs/zfs-kmod/zfs-kmod-0.6.2.ebuild clean configure
(cd /var/tmp/portage/sys-fs/zfs-kmod-0.6.2/work/zfs-zfs-0.6.2/ && ./copy-builtin /usr/src/linux)
mkdir -p /etc/portage/profile
echo 'sys-fs/zfs -kernel-builtin' >> /etc/portage/profile/package.use.mask
echo 'sys-fs/zfs kernel-builtin' >> /etc/portage/package.use

#finish configuring, building and installing the kernel making sure to enable dm-crypt support

#if not building zfs into the kernel, install module-rebuild
emerge module-rebuild

#install SPL and ZFS stuff zfs pulls in spl automatically
mkdir -p /etc/portage/profile
echo 'sys-fs/zfs -kernel-builtin' >> /etc/portage/profile/package.use.mask
echo 'sys-fs/zfs kernel-builtin' >> /etc/portage/package.use
emerge sys-fs/zfs

# Add zfs to the correct runlevel
rc-update add zfs boot

#initrd creation, add '--callback="module-rebuild rebuild"' to the options if not building the modules into the kernel
genkernel --luks --zfs --disklabel initramfs

Finish installing as normal, your kernel line should look like this, and you should also have a the initrd defined.

#kernel line for grub2, libzfs support is not needed in grub2 because you are not mounting the filesystem directly.
linux  /kernel-3.5.0-gentoo real_root=ZFS=mypool/ROOT/rootfs crypt_root=/dev/sda2 dozfs=force ro
initrd /initramfs-genkernel-x86_64-3.5.0

In /etc/fstab, make sure BOOT, ROOT and SWAP lines are commented out and finish the install.

You should now have a working encryped zfs install.

social