Let's Play a Game

Building Gentoo disk images

Matthew Thode (prometheanfire) — Wed, 24 Apr 2019 00:00:00 -0500

Disclaimer

I'm not responsible if you ruin your system, this guide functions as documentation for future me. Remember to back up your data.

Why this is useful / needed

It's useful to have a way of building a disk image for shipping, either for testing or production usage. The image output formats could be qcow2, raw or compressed tarball, it's up to you to make this what you want it to be.

Pre-work

Install diskimage-builder, for Gentoo you just have to 'emerge' the latest version. I personally keep one around in a virtual environment for testing (this allows me to build musl images as well easily).

The actual setup

What diskimage-builder actually does is take elements and run them. Each elements consists of a set of phases where the element takes actions. All you are really doing is defining the elements and they will insert themselves where needed. It also uses environment variables for tunables, or for other various small tweaks.

This is how I build the images at http://distfiles.gentoo.org/experimental/amd64/openstack/

export GENTOO_PORTAGE_CLEANUP=True
export DIB_INSTALLTYPE_pip_and_virtualenv=package
export DIB_INSTALLTYPE_simple_init=repo
export GENTOO_PYTHON_TARGETS="python3_6"
export GENTOO_PYTHON_ACTIVE_VERSION="python3.6"
export ELEMENTS="gentoo simple-init growroot vm openssh-server block-device-mbr"
export COMMAND="disk-image-create -a amd64 -t qcow2 --image-size 3"
export DATE="$(date -u +%Y%m%d)"

GENTOO_PROFILE=default/linux/amd64/17.0/no-multilib/hardened ${COMMAND} -o "gentoo-openstack-amd64-hardened-nomultilib-${DATE}" ${ELEMENTS}
GENTOO_PROFILE=default/linux/amd64/17.0/no-multilib ${COMMAND} -o "gentoo-openstack-amd64-default-nomultilib-${DATE}" ${ELEMENTS}
GENTOO_PROFILE=default/linux/amd64/17.0/hardened ${COMMAND} -o "gentoo-openstack-amd64-hardened-${DATE}" ${ELEMENTS}
GENTOO_PROFILE=default/linux/amd64/17.0/systemd ${COMMAND} -o "gentoo-openstack-amd64-systemd-${DATE}" ${ELEMENTS}
${COMMAND} -o "gentoo-openstack-amd64-default-${DATE}" ${ELEMENTS}

For musl I've had to do some custom work as I have to build the stage4s locally, but it's largely the same (with the additional need to define a musl overlay.

cd ~/diskimage-builder
cp ~/10-gentoo-image.musl diskimage_builder/elements/gentoo/root.d/10-gentoo-image
pip install -U .
cd ~/

export GENTOO_PORTAGE_CLEANUP=False
export DIB_INSTALLTYPE_pip_and_virtualenv=package
export DIB_INSTALLTYPE_simple_init=repo
export GENTOO_PYTHON_TARGETS="python3_6"
export GENTOO_PYTHON_ACTIVE_VERSION="python3.6"
DATE="$(date +%Y%m%d)"
export GENTOO_OVERLAYS="musl"
export GENTOO_PROFILE=default/linux/amd64/17.0/musl/hardened

disk-image-create -a amd64 -t qcow2 --image-size 3 -o gentoo-openstack-amd64-hardened-musl-"${DATE}" gentoo simple-init growroot vm

cd ~/diskimage-builder
git checkout diskimage_builder/elements/gentoo/root.d/10-gentoo-image
pip install -U .
cd ~/

Generic images

The elements I use are for an OpenStack image, meaning there is no default user/pass, those are set by cloud-init / glean. For a generic image you will want the following elements.

'gentoo growroot devuser vm'

The following environment variables are needed as well (changed to match your needs).

DIB_DEV_USER_PASSWORD=supersecrete DIB_DEV_USER_USERNAME=secrete DIB_DEV_USER_PWDLESS_SUDO=yes DIB_DEV_USER_AUTHORIZED_KEYS=/foo/bar/.ssh/authorized_keys

Fin

All this work was done upstream, if you have a question (or feature request) just ask. I'm on irc (Freenode) as prometheanfire or the same nick at gentoo.org for email.

Native ZFS encryption for your rootfs

Matthew Thode (prometheanfire) — Sat, 10 Feb 2018 00:00:00 -0600

Disclaimer

I'm not responsible if you ruin your system, this guide functions as documentation for future me. Remember to back up your data.

Why do this instead of luks

I wanted to remove a layer from the File to Disk layering, before it was ZFS -> LUKS -> disk, now it's ZFS -> disk.

Prework

I just got a new laptop and wanted to just migrate the data, luckily the old laptop was using ZFS as well, so the data could be sent/received though native ZFS means.

The actual setup

Set up your root pool with the encryption key, it will be inherited by all child datasets, no child datasets will be allowed to be unencrypted.

In my case the pool name was slaanesh-zp00, so I ran the following to create the fresh pool.

zpool create -O encryption=on -O keyformat=passphrase zfstest /dev/zvol/slaanesh-zp00/zfstest

After that just go on and create your datasets as normal, transfer old data as needed (it'll be encrypted as it's written). See https://wiki.gentoo.org/wiki/ZFS for a good general guide on setting up your datasets.

decrypting at boot

If you are using dracut it should just work. No changes to what you pass on the kernel command line are needed. The code is upstream in https://github.com/zfsonlinux/zfs/blob/master/contrib/dracut/90zfs/zfs-load-key.sh.in

notes

Make sure you install from git master, there was a disk format change for encrypted datasets that just went in a week or so ago.

Undervolting your CPU for fun and profit

Matthew Thode (prometheanfire) — Thu, 18 Jan 2018 00:00:00 -0600

Disclaimer

I'm not responsible if you ruin your system, this guide functions as documentation for future me. While this should just cause MCEs or system resets if done wrong it might also be able to ruin things in a more permanent way.

Why do this

It lowers temps generally and if you hit thermal throttling (as happens with my 5th Generation X1 Carbon) you may thermally throttle less often or at a higher frequency.

Prework

Repaste first, it's generally easier to do and can result in better gains (and it's all about them gains). For instance, my Skull Canyon NUC was resetting itself thermally when stress testing. Repasting lowered the max temps by 20°C.

Undervolting - The How

I based my info on https://github.com/mihic/linux-intel-undervolt which seems to work on my Intel Kaby Lake based laptop.

Using the MSR registers to write the values via msr-tools wrmsr binary.

The following python3 snippet is how I got the values to write.

# for a -110mv offset I run the following
format(0xFFE00000&( (round(-110*1.024)&0xFFF) <<21), '08x')

What you are actually writing is actually as follows, with the plane index being for cpu, gpu or cache for 0, 1 or 2 respectively.

constant	plane index	constant	write/read	offset
`80000`	`0`	`1`	`1`	`F1E00000`

So we end up with 0x80000011F1E00000 to write to MSR register 0x150.

Undervolting - The Integration

I made a script in /opt/bin/, where I place my custom system scripts called undervolt.sh (make sure it's executable).

#!/usr/bin/env sh
/usr/sbin/wrmsr 0x150 0x80000011F1E00000  # cpu core  -110
/usr/sbin/wrmsr 0x150 0x80000211F1E00000  # cpu cache -110
/usr/sbin/wrmsr 0x150 0x80000111F4800000  # gpu core  -90

I then made a custom systemd unit in /etc/systemd/system called undervolt.service with the following content.

[Unit]
Description=Undervolt Service

[Service]
ExecStart=/opt/bin/undervolt.sh

[Install]
WantedBy=multi-user.target

I then systemctl enable undervolt.service so I'll get my settings on boot.

The following script was also placed in /usr/lib/systemd/system-sleep/undervolt.sh to get the settings after recovering from sleep, as they are lost at that point.

#!/bin/sh
if [ "${1}" == "post" ]; then
  /opt/bin/undervolt.sh
fi

That's it, everything after this point is what I went through in testing.

Testing for stability

I stress tested with mprime in stress mode for the CPU and glxmark or gputest for the GPU. I only recorded the results for the CPU, as that's what I care about.

No changes:

15-17W package tdp
15W core tdp
3.06-3.22 Ghz

50mv Undervolt:

15-16W package tdp
14-15W core tdp
3.22-3.43 Ghz

70mv undervolt:

15-16W package tdp
14W core tdp
3.22-3.43 Ghz

90mv undervolt:

15W package tdp
13-14W core tdp
3.32-3.54 Ghz

100mv undervolt:

15W package tdp
13W core tdp
3.42-3.67 Ghz

110mv undervolt:

15W package tdp
13W core tdp
3.42-3.67 Ghz

started getting gfx artifacts, switched the gfx undervolt to 100 here

115mv undervolt:

15W package tdp
13W core tdp
3.48-3.72 Ghz

115mv undervolt with repaste:

15-16W package tdp
14-15W core tdp
3.63-3.81 Ghz

120mv undervolt with repaste:

15-16W package tdp
14-15W core tdp
3.63-3.81 Ghz

I decided on 110mv cpu and 90mv gpu undervolt for stability, with proper ventilation I get about 3.7-3.8 Ghz out of a max of 3.9 Ghz.

Other notes: Undervolting made the cpu max speed less spiky.

Gentoo Puppet Portage Package Provider

Matthew Thode (prometheanfire) — Sat, 10 Jun 2017 00:00:00 -0500

Why do this

The previus built in puppet portage package provider (I'm just going to shorten it to PPPP) only supported very simplistic package interactions. Mainly package name (with slot) install and uninstall. This has proven fairly limiting, if you want to install a specific version of a package and lock it down you were forced to call out to exec or editing package.{mask,unmask,keywords} files.

The new provider (which will be built into puppet in 5.0 or puppet-agent-2.0) supports all the package provider attributes.

How do I get this awesome thing

Emerge puppet or puppet-agent with the experimental use flag.

What it can do

You can use the following attributes with the new PPPP.

Name - The full package atom works now, using qatom on the backend.
ensure - now allowing a package purge as well (CONFIG_PROTECT="-*").
install_options - you can now pass options to emerge (--deep or --usepkgonly for example).
uninstall_options - just like install_options

Being able to call out specific versions and per package install options will give much greater flexability.

fin

Here is the pull request that upstream puppet merged.

If you have any questions I'm on freenode as prometheanfire.

Gentoo portage templates

Matthew Thode (prometheanfire) — Sat, 03 Jun 2017 00:00:00 -0500

Why do this

Gentoo is known for being somewhat complex to manage, making clusters of gentoo machines even more complex in most scenarios. Using the following methods the configuration becomes easier.

By the end of this you should be able to have a default hiera configuration for Gentoo while still being able to override it for specific use cases. What makes the method I chose particularly powerful is the ability to delete default vales entirely, not just setting them to something else.

Most of these methods came from my experience with chef that I thought would apply well to other config engines. While some don't like shoving logic to the configuration template engine, I'm open to suggestions.

Requirements

Puppet 4.x or puppet-agent with hiera support.
Puppet's stdlib installed (specifically for delete_values stuff).
(optional) use puppetserver instead of running this oneoff.
Hiera configured to use the following configuration.

Hiera config

:merge_behavior: deeper
:deep_merge_options:
  :merge_hash_arrays: true

Basic Setup

Convert the common portage configuraitons to templates.
Convert the data in those templates to a datastructure.
Use hiera to write the defaults / node overrides.
Call the templates via a puppet module.

Datastructures

The easiest way of explaing how this works is to say that the only data stored in the 'deepest' value is ever going to be True or False. The reason for this is a because using deep_merge in hiera is an additive process and we need a flag to remove things further down the line.

The datastructure itself is fairly simple, here is an excerpt from my setup.

make_conf:
  emerge_default_opts:
    "--quiet-build": true
    "--changed-use": true

If I wanted to disable --quiet-build down the line you would just set the value to False in a higher precidence (the specific node config instead of the general location.

make_conf:
  emerge_default_opts:
    "--quiet-build": false

Configuration Templates

The templates themselves are epp based, not erb (the old method).

package.keywords

For this one I'll also supply how I auto-set the right archetecture, works for amd64 at least.

Hiera data

"app-admin/paxtest ~%{facts.architecture}": true

Template

&lt;%- |$packages| -%&gt;
# THIS FILE WAS GENERATED BY PUPPET, CHANGES WILL BE OVERWRITTEN

&lt;%- keys(delete_values($packages, false)).each |$package| { -%&gt;
&lt;%= "$package" %&gt;
&lt;%- } -%&gt;

This one is the simplest, if a value for the key (paxtest in this case) is set to false, don't use it, the remaining keys are then set as plan text.

package.use

&lt;%- |$packages| -%&gt;
# THIS FILE WAS GENERATED BY PUPPET, CHANGES WILL BE OVERWRITTEN

&lt;%- keys(delete_values($packages, false)).each |$package| { -%&gt;
  &lt;%- if ! empty(keys(delete_values($packages[$package], false))) { -%&gt;
&lt;%= "$package" %&gt; &lt;%= join(keys(delete_values($packages[$package], false)), ' ') %&gt;
  &lt;%- } -%&gt;
&lt;%- } -%&gt;

This one is fairly straight forward as well, for each package that isn't disabled, if there are keys for the package (signifying use flags, needed because we remove the unset flags) then set them. This combines the flags set from all levels in hiera.

make.conf

This will be the most complicated one, but it's also likely to be the most important. I'll explain a bit about it after the paste.

&lt;%- |$config| -%&gt;
# THIS FILE WAS GENERATED BY PUPPET, CHANGES WILL BE OVERWRITTEN

CFLAGS="&lt;%= join(keys(delete_values($config['cflags'], false)), ' ') %&gt;"
CXXFLAGS="&lt;%= join(keys(delete_values($config['cxxflags'], false)), ' ') %&gt;"
CHOST="&lt;%= $config['chost'] %&gt;"
MAKEOPTS="&lt;%= join(keys(delete_values($config['makeopts'], false)), ' ') %&gt;"
CPU_FLAGS_X86="&lt;%= join(keys(delete_values($config['cpu_flags_x86'], false)), ' ') %&gt;"
ABI_X86="&lt;%= join(keys(delete_values($config['abi_x86'], false)), ' ') %&gt;"

USE="&lt;%= join(keys(delete_values($config['use'], false)), ' ') %&gt;"

GENTOO_MIRRORS="&lt;%= join(keys(delete_values($config['gentoo_mirrors'], false)), ' ') %&gt;"
&lt;% if has_key($config, 'portage_binhost') { -%&gt;
  &lt;%- if $config['portage_binhost'] != false { -%&gt;
PORTAGE_BINHOST="&lt;%= $config['portage_binhost'] %&gt;"
  &lt;%- } -%&gt;
&lt;% } -%&gt;

FEATURES="&lt;%= join(keys(delete_values($config['features'], false)), ' ') %&gt;"
EMERGE_DEFAULT_OPTS="&lt;%= join(keys(delete_values($config['emerge_default_opts'], false)), ' ') %&gt;"
PKGDIR="&lt;%= $config['pkgdir'] %&gt;"
PORT_LOGDIR="&lt;%= $config['port_logdir'] %&gt;"
PORTAGE_GPG_DIR="&lt;%= $config['portage_gpg_dir'] %&gt;"
PORTAGE_GPG_KEY='&lt;%= $config['portage_gpg_key'] %&gt;'

GRUB_PLATFORMS="&lt;%= join(keys(delete_values($config['grub_platforms'], false)), ' ') %&gt;"
LINGUAS="&lt;%= join(keys(delete_values($config['linguas'], false)), ' ') %&gt;"
L10N="&lt;%= join(keys(delete_values($config['l10n'], false)), ' ') %&gt;"

PORTAGE_ELOG_CLASSES="&lt;%= join(keys(delete_values($config['portage_elog_classes'], false)), ' ') %&gt;"
PORTAGE_ELOG_SYSTEM="&lt;%= join(keys(delete_values($config['portage_elog_system'], false)), ' ') %&gt;"
PORTAGE_ELOG_MAILURI="&lt;%= $config['portage_elog_mailuri'] %&gt;"
PORTAGE_ELOG_MAILFROM="&lt;%= $config['portage_elog_mailfrom'] %&gt;"

&lt;% if has_key($config, 'accept_licence') { -%&gt;
ACCEPT_LICENSE="&lt;%= join(keys(delete_values($config['accept_licence'], false)), ' ') %&gt;"
&lt;%- } -%&gt;
POLICY_TYPES="&lt;%= join(keys(delete_values($config['policy_types'], false)), ' ') %&gt;"
PAX_MARKINGS="&lt;%= join(keys(delete_values($config['pax_markings'], false)), ' ') %&gt;"

USE_PYTHON="&lt;%= join(keys(delete_values($config['use_python'], false)), ' ') %&gt;"
PYTHON_TARGETS="&lt;%= join(keys(delete_values($config['python_targets'], false)), ' ') %&gt;"
RUBY_TARGETS="&lt;%= join(keys(delete_values($config['ruby_targets'], false)), ' ') %&gt;"
PHP_TARGETS="&lt;%= join(keys(delete_values($config['php_targets'], false)), ' ') %&gt;"

&lt;% if has_key($config, 'source') { -%&gt;
source &lt;%= join(keys(delete_values($config['source'], false)), ' ') %&gt;
&lt;%- } -%&gt;

The basic idea of this is that you pass in the full make.conf datastructre you will generate as a single variable. Everything else is pulled (or elemated from that).

Each option that is selected already has all the options merged, but this could mean both the disabled versions of a given value could be still there, this is removed using the delete_values($config['foo'], false) bit.

The puppet module itself

It's fairly easy to call, just make sure the template is in the template location and do it as follows.

file { '/etc/portage/make.conf':
  content =&gt; epp('portage/portage-make_conf.epp', {'config' =&gt; hiera_hash('portage')['make_conf']})
}

fin

If you have any questions I'm on freenode as prometheanfire.

Gentoo at Fosdem

Matthew Thode (prometheanfire) — Thu, 09 Feb 2017 00:00:00 -0600

At the stand

It was nice to meet everyone and hang out as well. There was an interview with Hacker Public Radio which you can find HERE as well.

Just a short one this time, but it was nice to meet everyone.

Openstack Newton Update

Matthew Thode (prometheanfire) — Tue, 11 Oct 2016 00:00:00 -0500

The short of it

Openstack Newton was packaged early last week (when rc2 was still going on upstream) and the tags for the major projects were packaged the day they released (nova and the like).

I've updated the openstack-meta package to 2016.2.9999 and would recommend people use that.

Heat has also been packaged this time around so you are able to use that if you wish.

I'll link to my keywords and use files so you may use them if you wish as well. Please keep in mind that my use file is for my personal setup (static kernel, vxlan/linuxbridge and postgresql)

Gentoo, Openstack and OSIC

Matthew Thode (prometheanfire) — Mon, 06 Jun 2016 00:00:00 -0500

What to use it for

I recently applied for, and use an allocation from https://osic.org/ do extend more support for running Openstack on Gentoo. The end goal of this is to allow Gentoo to become a gated job within the Openstack test infrastructure. To do that, we need to add support for building an image that can be used.

(pre)work

To speed up the work on adding support for generating an openstack infra Gentoo image I already completed work on adding Gentoo to diskimage builder. You can see images at http://gentoo.osuosl.org/experimental/amd64/openstack/

(actual)work

The actual work has been slow going unfortunately, working with upstreams to add Gentoo support has tended to find other issues that need fixing along the way. The main thing that slowed me down though was the Openstack summit (Newton). That went on at the same time and reveiws were delated at least a week, usually two.

Since then though I've been able to work though some of the issues and have started testing the final image build in diskimage builder.

More to do

The main things left to do is to add gentoo support to the bindep elemet within diskimage builder and finish and other rough edges in other elements (if they exist). After that, Openstack Infra can start caching a Gentoo image and the real work can begin. Adding Gentoo support to the Openstack Ansible project to allow for better deployments.

Of OpenStack and uwsgi

Matthew Thode (prometheanfire) — Wed, 23 Mar 2016 00:00:00 -0500

Why use uwsgi

Not all OpenStack services support uwsgi. However, in the Liberty timeframe it is supported as the primary way to run Keystone api services and recommended way of running Horizon (if you use it). Going forward other openstack services will be movnig to support it as well, for instance I know that Neutron is working on it or have it completed for the Mitaka release.

Basic Setup

Install >=www-servers/uwsgi-2.0.11.2-r1 with the python use flag as it has an updated init script.
Make sure you note the group you want for the webserver to access the uwsgi sockets, I chose nginx.

Configs and permissions

When defaults are available I will only note what needs to change.

uwsgi configs

/etc/conf.d/uwsgi

UWSGI_EMPEROR_PATH="/etc/uwsgi.d/"
UWSGI_EMPEROR_GROUP=nginx
UWSGI_EXTRA_OPTIONS='--need-plugins python27'

/etc/uwsgi.d/keystone-admin.ini

[uwsgi]
master = true
plugins = python27
processes = 10
threads = 2
chmod-socket = 660

socket = /run/uwsgi/keystone_admin.socket
pidfile = /run/uwsgi/keystone_admin.pid
logger = file:/var/log/keystone/uwsgi-admin.log

name = keystone
uid = keystone
gid = nginx

chdir = /var/www/keystone/
wsgi-file = /var/www/keystone/admin

/etc/uwsgi.d/keystone-main.ini

[uwsgi]
master = true
plugins = python27
processes = 4
threads = 2
chmod-socket = 660

socket = /run/uwsgi/keystone_main.socket
pidfile = /run/uwsgi/keystone_main.pid
logger = file:/var/log/keystone/uwsgi-main.log

name = keystone
uid = keystone
gid = nginx

chdir = /var/www/keystone/
wsgi-file = /var/www/keystone/main

I have horizon in use via a virtual environment so enabled vaccum in this config.

/etc/uwsgi.d/horizon.ini

[uwsgi]
master = true  
plugins = python27
processes = 10  
threads = 2  
chmod-socket = 660
vacuum = true

socket = /run/uwsgi/horizon.sock  
pidfile = /run/uwsgi/horizon.pid  
log-syslog = file:/var/log/horizon/horizon.log

name = horizon
uid = horizon
gid = nginx

chdir = /var/www/horizon/
wsgi-file = /var/www/horizon/horizon.wsgi

wsgi scripts

The directories are owned by the serverice they are containing, keystone:keystone or horizon:horizon.

/var/www/keystone/admin perms are 0750 keystone:keystone

# Copyright 2013 OpenStack Foundation
#
#    Licensed under the Apache License, Version 2.0 (the "License"); you may
#    not use this file except in compliance with the License. You may obtain
#    a copy of the License at
#
#         http://www.apache.org/licenses/LICENSE-2.0
#
#    Unless required by applicable law or agreed to in writing, software
#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
#    License for the specific language governing permissions and limitations
#    under the License.

import os

from keystone.server import wsgi as wsgi_server


name = os.path.basename(__file__)

# NOTE(ldbragst): 'application' is required in this context by WSGI spec.
# The following is a reference to Python Paste Deploy documentation
# http://pythonpaste.org/deploy/
application = wsgi_server.initialize_application(name)

/var/www/keystone/main perms are 0750 keystone:keystone

# Copyright 2013 OpenStack Foundation
#
#    Licensed under the Apache License, Version 2.0 (the "License"); you may
#    not use this file except in compliance with the License. You may obtain
#    a copy of the License at
#
#         http://www.apache.org/licenses/LICENSE-2.0
#
#    Unless required by applicable law or agreed to in writing, software
#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
#    License for the specific language governing permissions and limitations
#    under the License.

import os

from keystone.server import wsgi as wsgi_server


name = os.path.basename(__file__)

# NOTE(ldbragst): 'application' is required in this context by WSGI spec.
# The following is a reference to Python Paste Deploy documentation
# http://pythonpaste.org/deploy/
application = wsgi_server.initialize_application(name)

Note that this has paths to where I have my horizon virtual environment.

/var/www/horizon/horizon.wsgi perms are 0750 horizon:horizon

#!/usr/bin/env python
import os
import sys


activate_this = '/home/horizon/horizon/.venv/bin/activate_this.py'
execfile(activate_this, dict(__file__=activate_this))

sys.path.insert(0, '/home/horizon/horizon')
os.environ['DJANGO_SETTINGS_MODULE'] = 'openstack_dashboard.settings'

import django.core.wsgi
application = django.core.wsgi.get_wsgi_application()

Creating Gentoo VM Images

Matthew Thode (prometheanfire) — Thu, 18 Feb 2016 00:00:00 -0600

Initial Setup and Info

This guide uses Openstack's Diskimage-builder tool for generation of images, while you can use this for Openstack, you can also create generic images with it.

Setting up Diskimage-builder is fairly simple, when you use it, it does expect to be run as root.

All you need to do is follow this guide, at it's simplest it's just a couple of git clones and PATH setup.

You will need app-emulation/qemu for generation of qcow2 files.

The current setup utilizes the stage4 images being generated, see this link for more details.

There are currently only 4 profiles supported, however I hope to support musl and selinux profiles 'soon'.

default/linux/amd64/13.0
default/linux/amd64/13.0/no-multilib
hardened/linux/amd64
hardened/linux/amd64/no-multilib

Generating an Openstack image

To use a profile other than default/linux/amd64/13.0 set the GENTOO_PROFILE environment variable to one of the other supported profiles.

disk-image-create -a amd64 -t qcow2 --image-size 2 gentoo simple-init growroot vm is all you need to start. It will output a file named image.qcow2.

For openstack there are two ways you could go for initial setup (post-vm start). The first and most common is cloud-init, but that includes a few python deps that I don't think are really needed. The other is simple-init (glean), which is more limited, but as it's name suggests, simple.

Here is a link to glean (simple-init) for those wanting more info glean

Generating a Generic Image You Can Log Into

Using the devuser element you can set up custom users. You will need to set up some more environment variables though.

Docs can be found here

An example invocation follows, simple-init may be needed so that interfaces get dhcp addresses, though you may wat to set that up manually, your choice.

DIB_DEV_USER_PASSWORD=foobar DIB_DEV_USER_USERNAME=gentoo DIB_DEV_USER_PWDLESS_SUDO=yes DIB_DEV_USER_AUTHORIZED_KEYS=/dev/null disk-image-create -a amd64 -t qcow2 --image-size 2 gentoo simple-init growroot devuser vm

Stage4 tarballs, minimal and cloud

Matthew Thode (prometheanfire) — Fri, 29 Jan 2016 00:00:00 -0600

Where are they

The tarballs can be found in the normal place.

Minimal

This is meant to be just what you need to boot, the disk won't expand itself, it won't even get networking info or set any passwords for you (no default password).

This tarball is suposed to be the base you generate more complex images from, it is what is going to be used by Openstack's diskimage-builder.

The primary things it does is get you a kernel, bootloader and sshd.

stage4-minimal spec

Cloud

This was primarilly targeted for use with openstack but it should work with amazon as well, both use cloud-init.

Network interfaces are expected to use dhcp, a couple of other useful things are installed as well, syslog, logrotate, etc.

By default cloud-init will take data (keys mainly) and set them up for the 'gentoo' user.

stage4-cloud spec

I'll be posting about the work being done to take these stages and build bootable images. At the momebt I do have images available here.

openstack images

Of OpenStack and SSL

Matthew Thode (prometheanfire) — Mon, 18 Jan 2016 00:00:00 -0600

SSL in vanila OpenStack

The nature of OpenStack projects is largely like projects in Gentoo. Even though they are all under the OpenStack umbrella that doesn't mean they all have to work the same, or even work together.

For instance, nova has the ability to do ssl itself, you can define a CA and public/private keypair. Glance (last time I checked) doesn't do ssl yourself so you must offload it. Other service might do ssl themselves but not in the same way nova does it.

This means that the most 'standard' setup would be to not run ssl, but this isn't exactly desirable. So run a ssl reverse proxy.

Basic Setup

OpenStack services are set up on one host (just in this example).
OpenStack services are configed to listen on localhost only.
Public, Internal and Admin URLs need to be defined with https.
Some tuning needs to be done so services work properly, primarily to glance and nova-novnc.
Nginx is used as the reverse proxy.

Configs and Tuning

General Config for All Services/Sites

This is the basic setup for each of the openstack services, the only difference between them will be what goes in the location subsection.

server {
    listen LOCAL_PUBLIC_IPV4:PORT;
    listen [LOCAL_PUBLIC_IPV6]:PORT;
    server_name = name.subdomain.example.com;
    access_log /var/log/nginx/keystone/access.log;
    error_log /var/log/nginx/keystone/error.log;

    ssl on;
    ssl_certificate /etc/nginx/ssl/COMBINED_PUB_PRIV_KEY.pem;
    ssl_certificate_key /etc/nginx/ssl/COMBINED_PUB_PRIV_KEY.pem;
    add_header Public-Key-Pins 'pin-sha256="PUB_KEY_PIN_SHA"; max-age=2592000; includeSubDomains';
    ssl_dhparam /etc/nginx/params.4096;
    resolver TRUSTED_DNS_SERVER;
    resolver_timeout 5s;
    ssl_stapling on;
    ssl_stapling_verify on;
    ssl_trusted_certificate /etc/nginx/ssl/COMBINED_PUB_PRIV_KEY.pem;
    add_header X-XSS-Protection "1; mode=block";
    add_header Content-Security-Policy "default-src 'self' https: wss:;";
    add_header Strict-Transport-Security "max-age=31536000; includeSubdomains; preload";
    add_header X-Frame-Options DENY;
    add_header X-Content-Type-Options nosniff;

    location / {
        # this changes depending on the service
        proxy_pass http://127.0.0.1:PORT;
    }
}

Keystone and Uwsgi

It turns out keystone has switched to uwsgi for it's service backend. This is good because it means we can have the web server connect to that, no more trying to do it all by itself. I'll leave the setting up of uwsgi itself as an exercise to the reader :P

Please keep in mind keystone has two services, admin and main (ports 35357 and 5000 by default).

This config has a few extra things, but it is currently what I know to be 'secure' (similiar config on this blog gets an A+ on all those ssl test things). It's the last location piece that changes the most between services.

location / {
    uwsgi_pass unix:///run/uwsgi/keystone_admin.socket;
    include /etc/nginx/uwsgi_params;
    uwsgi_param SCRIPT_NAME '';
}

Glance

Glance just needs one thing on top of the general proxying that it needs. It needs client_max_body_size 0; in the main server stanza so that you can upload images without being cut off at some low size.

client_max_body_size 0;
location / {
    proxy_pass http://127.0.0.1:9191;
}

Nova

The serviecs for nova just need the basic proxy_pass line. The only exception is novnc, it needs some proxy headers passed.

location / {
    proxy_pass http://127.0.0.1:6080;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_set_header Upgrade $http_upgrade;
}

Rabbitmq

Rabbit is fairly simple, you just need to enable ssl and disable the plaintext port (setting up your config of course).

[
    {ssl, [{versions, ['tlsv1.2', 'tlsv1.1']}]},
    {rabbit, [
        {tcp_listeners, []},
        {ssl_listeners, [5671]},
        {ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/CA_CERT.pem"},
                       {certfile,  "/etc/rabbitmq/ssl/PUB_KEY.pem"},
                       {keyfile,   "PRIV_KEYKEY.key"},
                       {versions, ['tlsv1.2', 'tlsv1.1']}
                      ]
        }]
    }
].

Openstack Configs

The openstack configs can differ slightly but they are all mostly the same now they are using the same libraries (oslo stuff).

General Config

[keystone_authtoken]
auth_uri = https://name.subdomain.example.com:5000
auth_url = https://name.subdomain.example.com:35357

[oslo_messaging_rabbit]
rabbit_host = name.subdomain.example.com
rabbit_port = 5671
rabbit_use_ssl = True

Nova

osapi_compute_listen = 127.0.0.1
metadata_listen = 127.0.0.1
novncproxy_host = 127.0.0.1
enabled_apis = osapi_compute, metadata
[vnc]
novncproxy_base_url = https://name.subdomain.example.com:6080/vnc_auto.html
# the following only on the 'master' host
vncserver_proxyclient_address = 1.2.3.4
vncserver_listen = 1.2.3.4

[glance]
host = name.subdomain.example.com
protocol = https
api_servers = https://name.subdomain.example.com:9292

[neutron]
url = https://name.subdomain.example.com:9696
auth_url = https://name.subdomain.example.com:35357

Cinder

# api-servers get this
osapi_volume_listen = 127.0.0.1

# volume-servers and api-servers get this
glance_api_servers=https://name.subdomain.example.com:9292

Glance

glance_api_servers=https://name.subdomain.example.com:9292

Glance

# api
bind_host = 127.0.0.1
registry_host = name.subdomain.example.com
registry_port = 9191
registry_client_protocol = https

# cache
registry_host = name.subdomain.example.com
registry_port = 9191

# registry
bind_host = 127.0.0.1
rabbit_host = name.subdomain.example.com
rabbit_port = 5671
rabbit_use_ssl = True

# scrubber
registry_host = name.subdomain.example.com
registry_port = 9191

Neutron

# neutron.conf
bind_host = 127.0.0.1
nova_url = https://name.subdomain.example.com:8774/v2

[nova]
auth_url = https://name.subdomain.example.com:35357

# metadata_agent.ini
nova_metadata_ip = name.subdomain.example.com
nova_metadata_protocol = https

OpenStack on Gentoo is awesome

Matthew Thode (prometheanfire) — Thu, 24 Sep 2015 00:00:00 -0500

Some may wonder why to run OpenStack on Gentoo, it's akin to running one extremely complex piece of software on another potentially extremely complex operating system.

I propose Gentoo as the most correct operating system to run OpenStack as Gentoo is best prepared to handle some of the complexities that come with OpenStack.

Things Gentoo does well for OpenStack

Dependency resolution - we can take upstream's dependencies (requirements.txt) and map it directly to Gentoo's packages without changing the versions supported. This is not always the case in other distributions.
Because of this we can better rely on upstream's testing as validation that OpenStack on Gentoo will work properly.
useflags - some dependencies (such as memcached, qemu, etc) are optional depending on what services you are running.
python3 support - as upstream is finally moving to python3 we are easily able to extend support as it occurs.
patching - https://wiki.gentoo.org/wiki//etc/portage/patches allows users to patch anything as needed.
Upstream vanilla Openstack - We are not carrying many (I can count them on one hand) patches that are not upstreamed.
Testing - All the major services have testing support to validate good installs and / or custom patches not breaking things.

Future of OpenStack on Gentoo

The Liberty release 15/10/2015
- Upstream is reversioning the major components of OpenStack, 2015.2.0 will not exist, it will be something like 12.0.0.
- The reversioning will mean some manual intervention if you wish to use Liberty on Gentoo, namely you will need to mask greater than version 2000 of the major components of OpenStack
More services
- I will at least add Heat during the next (Liberty) release cycle, possibly more
- I will investigate readding Horizon, but doubt it'll be easily packagable.
Security
- I'd like to look into some of the selinux policies that are being developed for some of the services. I know nova/qemu has apparmor support.

Limits of testing

You cannot directly test OpenStack packages by emerging with USE="test" FEATURES="test" because there are interdepenecies that cause loops in the depgraph of portage for the (test) dependencies. You can get around it one way though.

# install only the test deps
emerge --onlydeps --oneshot --with-test-deps sys-cluster/nova
# test and install the actual package
USE="test" FEATURES="test" emerge sys-cluster/nova

Gentoo Hardened ZFS rootfs with dm-crypt/luks 0.6.5

Matthew Thode (prometheanfire) — Wed, 02 Sep 2015 00:00:00 -0500

Disclaimer

Keep in mind that ZFS on Linux is supported upstream, for differing values of support
I do not care much for hibernate, normal suspending works.
This is for a laptop/desktop, so I choose multilib.
IANAL
If you patch the kernel to add in ZFS support directly, you cannot share the binary, the cddl and gpl2 are not compatible in that way.

Initialization

Make sure your installation media supports zfs on linux and installing whatever bootloader is required (uefi needs media that supports it as well). It does not matter too much if it is older media, as long as it supports what you need, Gentoo is nice like that. Here is an iso that works well for me at this link Live DVDs newer then 12.1 should also have support.

Formatting

I will be assuming the following.

/boot on /dev/sda1
cryptroot on /dev/sda2
An optional bios_boot partition for grub (gpt stuff)
swap inside cryptroot OR not used.

When using GPT for partitioning, create the first partition at 1M, just to make sure you are on a sector boundry Most newer drives are 4k advanced format drives. Because of this you need ashift=12, some/most newer SSDs need ashift=13 or greater compression set to lz4 will make your system incompatible with upstream (oracle) zfs, if you want to stay compatible then just set compression=on

General Setup

# setup encrypted partition
# aes-xts-plain64 was chosen due to speed, xts-essiv SHOULD be more secure, but about half as slow, on aes-ni I was getting about 200MBps
cryptsetup luksFormat -l 512 -c aes-xts-plain64 -h sha512 /dev/sda2
cryptsetup luksOpen /dev/sda2 cryptroot

# setup ZFS
# a deeper look into this can be found at [this link](https://github.com/ryao/zfs-overlay/blob/master/zfs-install "ryao's repo")
zpool create -f -o ashift=12 -o cachefile=/tmp/zpool.cache -O normalization=formD -m none -R /mnt/gentoo mypool /dev/mapper/cryptroot
zfs create -o mountpoint=none -o compression=lz4 mypool/ROOT
# rootfs
zfs create -o mountpoint=/ mypool/ROOT/rootfs
# system mountpoints were seperated so that we can set nodev and nosuid as mount options
zfs create -o mountpoint=/opt mypool/ROOT/rootfs/OPT
zfs create -o mountpoint=/usr mypool/ROOT/rootfs/USR
zfs create -o mountpoint=/usr/src -o sync=disabled mypool/ROOT/rootfs/USR/SRC
zfs create -o mountpoint=/var mypool/ROOT/rootfs/VAR
# portage
zfs create -o mountpoint=none -o setuid=off mypool/GENTOO
zfs create -o mountpoint=/usr/portage -o atime=off mypool/GENTOO/portage
zfs create -o mountpoint=/usr/portage/distfiles -o compression=off mypool/GENTOO/distfiles
zfs create -o mountpoint=/usr/portage/packages -o compression=off mypool/GENTOO/packages
zfs create -o mountpoint=/var/tmp/portage -o sync=disabled mypool/GENTOO/build-dir
# homedirs
zfs create -o mountpoint=/home mypool/HOME
zfs create -o mountpoint=/root mypool/HOME/root
# replace USER with your username
zfs create -o mountpoint=/home/USER mypool/HOME/USER
# set the bootfs, some initrams require this
zpool set bootfs=mypool/ROOT/rootfs mypool

cd /mnt/gentoo

# Download the latest stage3 and extract it.
wget ftp://gentoo.osuosl.org/pub/gentoo/releases/amd64/autobuilds/current-stage3-amd64-hardened/stage3-amd64-hardened-*.tar.bz2
tar -xf /mnt/gentoo/stage3-amd64-hardened-*.tar.bz2 -C /mnt/gentoo

# get the latest portage tree
emerge --sync

# copy the zfs cache from the live system to the chroot
mkdir -p /mnt/gentoo/etc/zfs
cp /tmp/zpool.cache /mnt/gentoo/etc/zfs/zpool.cache

Kernel Config

If you are compiling the modules into the kernel staticly, then keep these things in mind.

When configuring the kernel, make sure that CONFIG_SPL and CONFIG_ZFS are set to 'Y'.
Portage will want to install sys-kernel/spl and sys-fs/zfs-kmod when emerge sys-fs/zfs is run because of dependencies. Also, both are still necessary to make the sys-fs/zfs configure script happy.
You do not need to run or install module-rebuild.

Install as normal up until the kernel install.

echo "=sys-kernel/genkernel-3.4.40 ~amd64       #needed for zfs and encryption support" &gt;&gt; /etc/portage/package.accept_keywords
emerge sys-kernel/genkernel
emerge sys-kernel/gentoo-sources                #or hardned-sources

# patch the kernel

# If you want to build the modules into the kernel directly, you will need to patch the kernel directly.  Otherwise, skip the patch commands.
# configure the kernel then prepare for zfs / spl
cd /usr/src/linux
make prepare
make scripts
env EXTRA_ECONF='--enable-linux-builtin' ebuild /usr/portage/sys-kernel/spl/spl-0.6.5.ebuild clean configure
(cd /var/tmp/portage/sys-kernel/spl-0.6.5/work/spl-0.6.5/ &amp;&amp; ./copy-builtin /usr/src/linux)
env EXTRA_ECONF='--with-spl=/usr/src/linux --enable-linux-builtin --with-spl-obj=/usr/src/linux' ebuild /usr/portage/sys-fs/zfs-kmod/zfs-kmod-0.6.5.ebuild clean configure
(cd /var/tmp/portage/sys-fs/zfs-kmod-0.6.5  /work/zfs-kmod-0.6.5 &amp;&amp; ./copy-builtin /usr/src/linux)
env EXTRA_ECONF='--with-spl=/usr/src/linux --enable-linux-builtin' ebuild /usr/portage/sys-fs/zfs/zfs-0.6.5.ebuild clean merge
mkdir -p /etc/portage/profile
echo 'sys-fs/zfs -kernel-builtin' &gt;&gt; /etc/portage/profile/package.use.mask
echo 'sys-fs/zfs kernel-builtin' &gt;&gt; /etc/portage/package.use

# finish configuring, building and installing the kernel making sure to enable dm-crypt support

# if not building zfs into the kernel, install module-rebuild
emerge module-rebuild

# install SPL and ZFS stuff zfs pulls in spl automatically
mkdir -p /etc/portage/profile
echo 'sys-fs/zfs -kernel-builtin' &gt;&gt; /etc/portage/profile/package.use.mask
echo 'sys-fs/zfs kernel-builtin' &gt;&gt; /etc/portage/package.use
emerge sys-fs/zfs

# Add zfs to the correct runlevel
rc-update add zfs-mount boot

# initrd creation, add '--callback="module-rebuild rebuild"' to the options if not building the modules into the kernel
genkernel --luks --zfs --disklabel initramfs

Finish installing as normal, your kernel line should look like this, and you should also have a the initrd defined.

# kernel line for grub2, libzfs support is not needed in grub2 because you are not mounting the filesystem directly.
linux  /kernel-3.5.0-gentoo real_root=ZFS=mypool/ROOT/rootfs crypt_root=/dev/sda2 dozfs=force ro
initrd /initramfs-genkernel-x86_64-3.5.0

In /etc/fstab, make sure BOOT, ROOT and SWAP lines are commented out and finish the install.

You should now have a working encryped zfs install.

Gentoo on the Odroid-U3

Matthew Thode (prometheanfire) — Mon, 25 Aug 2014 00:00:00 -0500

Arm cross compiler setup and stuffs

This will set up a way to compile things for arm on your native system (amd64 for me)

emerge dev-embedded/u-boot-tools sys-devel/crossdev
crossdev -S -s4 -t armv7a-hardfloat-linux-gnueabi

Building the kernel

This assumes you have kernel sources, I'm testing 3.17-rc2 since they just got support for the odroid-u3 into upstream.

Also, I tend to build without modules, so keep that in mind.

# get the base config (For me on an odroid-u3
ARCH=arm CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi- make exynos_defconfig
# change it to add what I want/need
ARCH=arm CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi- make menuconfig
# build the kernel
ARCH=arm CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi- make -j10

Setting up the SD Card

I tend to be generous, 10M for the bootloader

parted /dev/sdb mklabel msdos y
parted /dev/sdb mkpart p fat32 10M 200M
parted /dev/sdb mkpart p 200M 100%
parted /dev/sdb toggle 1 boot

mkfs.vfat /dev/sdb1
mkfs.ext4 /dev/sdb2

Building uboot

This may differ between boards, but should general look like the following (I hear vanilla uboot works now)

I used the odroid-v2010.12 branch and one thing to note is that if it sees a zImage on the boot partition it will ONLY use that, kinda of annoying.

git clone git://github.com/hardkernel/u-boot.git
cd u-boot
sed -i -e "s/soft-float/float-abi=hard -mfpu=vfpv3/g" arch/arm/cpu/armv7/config.mk
ARCH=arm CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi- make smdk4412_config
ARCH=arm CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi- make -j1
sudo "sh /home/USER/dev/arm/u-boot/sd_fuse/sd_fusing.sh /dev/sdb"

Copying the kernel/userland

sudo -i
mount /dev/sdb2 /mnt/gentoo
mount /dev/sdb1 /mnt/gentoo/boot
cp /home/USER/dev/linux/arch/arm/boot/dts/exynos4412-odroidu3.dtb /mnt/gentoo/boot/
cp /home/USER/dev/linux/arch/arm/boot/zImage /mnt/gentoo/boot/kernel-3.17-rc2.raw
cd /mnt/gentoo/boot
cat kernel-3.17-rc2.raw exynos4412-odroidu3.dtb &gt; kernel-3.17-rc2

tar -xf /tmp/stage3-armv7a_hardfp-hardened-20140627.tar.bz2 -C /mnt/gentoo/

Setting up userland

I tend to just copy or generate a shadow file and overwrite the root entry in /etc/shadow...

Then set up on when booted

Setting up the bootloader

put this in /mnt/gentoo/boot/boot.txt

setenv initrd_high "0xffffffff"
setenv fdt_high "0xffffffff"
setenv fb_x_res "1920"
setenv fb_y_res "1080"
setenv hdmi_phy_res "1080"
setenv bootcmd "fatload mmc 0:1 0x40008000 kernel-3.17-rc2; bootm 0x40008000"
setenv bootargs "console=tty1 console=ttySAC1,115200n8 fb_x_res=${fb_x_res} fb_y_res=${fb_y_res} hdmi_phy_res=${hdmi_phy_res} root=/dev/mmcblk0p2 rootwait ro mem=2047M"
boot

and run this

mkimage -A arm -T script -C none -n "Boot.scr for odroid-u3" -d boot.txt boot.scr

That should do it :D

I used steev (a fellow gentoo dev) and http://www.funtoo.org/ODROID_U2 as sources.

testing the s3700

Matthew Thode (prometheanfire) — Sun, 02 Mar 2014 00:00:00 -0600

The Setup

Two 100G s3700 drives, one tested with luks, one not.

Filled the drive to test with it filled.

Testing 4k/8k, with luks using --size=8 or --size=9 for 4k and 8k respectivly.

I used the following settings in fio, changing the filename and block size where appropriate.

[global]
bs=4k
ioengine=posixaio
iodepth=32
size=200g
filename=/dev/mapper/testssd
direct=1

[rand-read]
rw=randread
stonewall

[rand-write]
rw=randwrite
stonewall

[seq-read]
rw=read
stonewall

[seq-write]
rw=write
stonewall

Results

There was generally high interupts and context switches, but oddly less so with luks.

Sequential IOPS

4k sequential writes with luks was 73.7% of max ( 12424 vs 16855 )
4k sequential reads with luks was 76.7% of max ( 14471 vs 18864 )
8k sequential writes with luks was 71.0% of max ( 9640 vs 13573 )
8k sequential reads with luks was 71.8% of max ( 10744 vs 14966 )

Random IOPS

4k random writes with luks was 82.2% of max ( 13919 vs 16924
4k random reads with luks was 80.7% of max ( 6260 vs 7756 )
8k random writes with luks was 71.7% of max ( 9718 vs 13557 )
8k random reads with luks was 64.7% of max ( 4222 vs 6526 )

Conclusion

My use case is zfs usage as l2arc/zil cache, I'll be using 8k on luks.

Interupts and Context Switches with ZFS with LUKS

Matthew Thode (prometheanfire) — Thu, 20 Feb 2014 00:00:00 -0600

Why care?

Interupts and Context switches represent overhead and ineffeciency (in general).

The Setup

The Same 15 4TB drives in a raidz3 with a hotspare (16 drives total).

Testing was just a scrub (no fresh boot). Switched between CFQ and NOOP a couple of times.

Results

The picture generally speaks for itself, but in general I noticed a doubling of the context switches and adding 1 to the load level.

I also about a 20% increase in Interupts.

The results

Conclusion

Use NOOP.

ZFS performance on LUKS

Matthew Thode (prometheanfire) — Tue, 18 Feb 2014 00:00:00 -0600

the setup

The system being tested is a 15 drive raidz3 setup with 4TB drives running kernel 3.13.2 (non-hardened for testing).

The encryption algorithm

I was previously using cbc-essiv, this causes I found (via a simple scrub test) that it caused my performance to be about 10% lower then xts-plain64.

cbc-essiv was slow because it needed to read the previous block in order to write the current one.

I chose xts-plain64 even though there are known attacks against it.

From Wikipedia

XTS mode is susceptible to data manipulation and tampering, and applications must employ measures to detect modifications of data if manipulation and tampering is a concern: "...since there are no authentication tags then any ciphertext (original or modified by attacker) will be decrypted as some plaintext and there is no built-in mechanism to detect alterations. The best that can be done is to ensure that any alteration of the ciphertext will completely randomize the plaintext, and rely on the application that uses this transform to include sufficient redundancy in its plaintext to detect and discard such random plaintexts." The mode is also susceptible to traffic analysis, replay, and sector randomization attacks.

I believe that the auth tag is double verified by both luks checksuming and zfs checksuming.
I am not sure about the other issues (traffic analysis, replay, and sector randomization attacks).
I do not currently consider myself up against state actors.

Setting the Elevator

I unknowningly was using cfq, this caused more load spikes, longer scrub times and higher load in general.

ZFSonLinux does set the elevator to NOOP if it is put directly on a hard disk but using LUKS interferes with this.

Here is the results of me testing via scrubs (done only once, fresh boot each time, but good enough for me...).

Elevator	Scrub Time	Average Load15	Link to load
NOOP	10h42m	5.1 (ish)	NOOP Graph
CFQ	11h47m	6.5 (ish)	CFQ Graph

Conclusion

By changing all drives to xts-plain64 and the elevators to noop my scrubs went from 360MB/s to 450MB/s (a 20% gain).

Of ZFS and SELinux

Matthew Thode (prometheanfire) — Thu, 07 Nov 2013 00:00:00 -0600

This patchset has been incuded in zfsonlinux, the selinux policy is still needed

This continues my work in adding zfs to the list of filesystems that SELinux supports

Disclaimer

These patches are against git master (because 0.6.3 should be out 'soon').
IANAL
If you patch the kernel to add in ZFS support directly, you cannot share the binary, the cddl and gpl2 are not compatible in that way.

The Patch

The only patch you should need comes from this pull request. If you wish, you could also use my selinux forked branch).

http://dev.gentoo.org/~prometheanfire/patches/zfs/rootcontext.patch

https://github.com/prometheanfire/zfs/tree/selinux

These pull requests may be intresting to you as well.

what it does

The patch adds selinux support to more inode types (links and directories and stuff). When it detects selinux support, it will also add the mountoption rootcontext= to the mount commands it sends the host.

Building the stuffs

#get the patches
mkdir -p /etc/portage/patches/sys-fs/zfs /etc/portage/patches/sys-fs/zfs-kmod
curl http://dev.gentoo.org/~prometheanfire/patches/zfs/rootcontext.patch -o /etc/portage/patches/sys-fs/zfs/rootcontext.patch
curl http://dev.gentoo.org/~prometheanfire/patches/zfs/rootcontext.patch -o /etc/portage/patches/sys-fs/zfs-kmod/rootcontext.patch

#get the code into the kernel if compiling statically
#replace configure with merge to install normally
#if you wish to use my zfs branch, include the following line to the env ebuild command at the start of it (right after 'env')
#EGIT_BRANCH=selinux zfs_kmod_LIVE_REPO='git://github.com/prometheanfire/zfs.git' zfs_LIVE_REPO='git://github.com/prometheanfire/zfs.git'
#configure the kernel normally and run 'make prepare'
env EXTRA_ECONF='--enable-linux-builtin' ebuild /usr/portage/sys-kernel/spl/spl-9999.ebuild clean configure
(cd /var/tmp/portage/sys-kernel/spl-9999/work/spl-9999 &amp;&amp; ./copy-builtin /usr/src/linux)
env EXTRA_ECONF='--with-spl=/usr/src/linux --enable-linux-builtin' ebuild /usr/portage/sys-fs/zfs-kmod/zfs-kmod-9999.ebuild clean configure
(cd /var/tmp/portage/sys-fs/zfs-kmod-9999/work/zfs-kmod-9999 &amp;&amp; ./copy-builtin /usr/src/linux)

#build and install the kernel, making sure to enable spl/zfs if you are compiling statically.

#merge this no mater what
env EXTRA_ECONF='--with-spl=/usr/src/linux --enable-linux-builtin' ebuild /usr/portage/sys-fs/zfs/zfs-9999.ebuild clean merge

#build the initrd and configure your bootloader

#get the code into the system if NOT compiling statically
env EXTRA_ECONF='--enable-linux-builtin' ebuild /usr/portage/sys-kernel/spl/spl-9999.ebuild clean merge

Labeling the datasets

When you reboot into the new kernel and with the new zfs userland tools you can set the rootcontext as a property directly.

The rootcontext for default is 'system_u:object_r:fs_t'.

zfs set rootcontext=system_u:object_r:portage_ebuild_t node02-zp00/GENTOO/portage

# zfs list -o name,mountpoint,rootcontext
NAME                                  MOUNTPOINT              ROOTCONTEXT
node02-zp00                           none                    default
node02-zp00/GENTOO                    none                    default
node02-zp00/GENTOO/portage            /usr/portage            system_u:object_r:portage_ebuild_t
node02-zp00/GENTOO/portage/distfiles  /usr/portage/distfiles  system_u:object_r:portage_ebuild_t
node02-zp00/GENTOO/portage/packages   /usr/portage/packages   system_u:object_r:portage_ebuild_t
node02-zp00/HOME                      /home                   system_u:object_r:home_root_t
node02-zp00/HOME/root                 /root                   root:object_r:user_home_dir_t
node02-zp00/ROOT                      none                    default
node02-zp00/ROOT/opt                  /opt                    system_u:object_r:usr_t
node02-zp00/ROOT/rootfs               legacy                  system_u:object_r:root_t
node02-zp00/test-vol                  -                       default

Selinux Policy

I am working on getting this upstream, but for now this is needed.

mkdir -p ~/selinux/zfs
cd ~/selinux/zfs
curl http://dev.gentoo.org/~prometheanfire/patches/zfs/zfs.fc -o zfs.fc
curl http://dev.gentoo.org/~prometheanfire/patches/zfs/zfs.te -o zfs.te

#make and insert the module
make -f /usr/share/selinux/strict/include/Makefile zfs.pp
semodule -i zfs.pp

#when it becomes needed (upstream support) run 'semodule -r zfs' to remove the module.

The rest

You will need to relabel most likely (since this adds support for selinux markings on directories and the like).

Treat this like a non-selinux system and follow the migration guide here

Gentoo Hardened ZFS rootfs with dm-crypt/luks 0.6.2

Matthew Thode (prometheanfire) — Tue, 10 Sep 2013 00:00:00 -0500

Disclaimer

Keep in mind that ZFS on Linux is supported upstream, for differing values of support
I do not care much for hibernate, normal suspending works.
This is for a laptop/desktop, so I choose multilib.
IANAL
If you patch the kernel to add in ZFS support directly, you cannot share the binary, the cddl and gpl2 are not compatible in that way.

Initialization

Make sure your installation media supports zfs on linux and installing whatever bootloader is required (uefi needs media that supports it as well). I uploaded an iso that works for me at this link Live DVDs newer then 12.1 should also have support, but the previous link has a stable version of zfsonlinux (not 0.6.2 yet, but good enough for install!). If you need to install the bootloader via uefi, you can use one of the latest Fedora CDs, though the gentoo media should be getting support 'soon'. You can install your system normally up until the formatting begins.

Formatting

I will be assuming the following.

/boot on /dev/sda1
cryptroot on /dev/sda2
An optional bios_boot partition for grub (gpt stuff)
swap inside cryptroot OR not used.

When using GPT for partitioning, create the first partition at 1M, just to make sure you are on a sector boundry Most newer drives are 4k advanced format drives. Because of this you need ashift=12, some/most newer SSDs need ashift=13 compression set to lz4 will make your system incompatible with upstream (oracle) zfs, if you want to stay compatible then just set compression=on due to linux not having the best memory management, zfs on luks can be kinda unstable, I have not had a problem on my laptop, but my servers have been sad

General Setup

#setup encrypted partition
#aes-xts-plain64 was chosen due to speed, xts-essiv SHOULD be more secure, but about half as slow, on aes-ni I was getting about 200MBps
cryptsetup luksFormat -l 512 -c aes-xts-plain64 -h sha512 /dev/sda2
cryptsetup luksOpen /dev/sda2 cryptroot

#setup ZFS
zpool create -f -o ashift=12 -o cachefile=/tmp/zpool.cache -O normalization=formD -m none -R /mnt/gentoo mypool /dev/mapper/cryptroot
zfs create -o mountpoint=none -o compression=lz4 mypool/ROOT
#rootfs
zfs create -o mountpoint=/ mypool/ROOT/rootfs
#system mountpoints were seperated so that we can set nodev and nosuid as mount options
zfs create -o mountpoint=/opt mypool/ROOT/rootfs/OPT
zfs create -o mountpoint=/usr mypool/ROOT/rootfs/USR
zfs create -o mountpoint=/usr/src -o sync=disabled mypool/ROOT/rootfs/USR/SRC
zfs create -o mountpoint=/var mypool/ROOT/rootfs/VAR
#portage
zfs create -o mountpoint=none mypool/GENTOO
zfs create -o mountpoint=/usr/portage mypool/GENTOO/portage
zfs create -o mountpoint=/usr/portage/distfiles -o compression=off mypool/GENTOO/distfiles
zfs create -o mountpoint=/usr/portage/packages -o compression=off mypool/GENTOO/packages
zfs create -o mountpoint=/var/tmp/portage -o sync=disabled mypool/GENTOO/build-dir
#homedirs
zfs create -o mountpoint=/home mypool/HOME
zfs create -o mountpoint=/root mypool/HOME/root
#replace user with your username
zfs create -o mountpoint=/home/USER mypool/HOME/USER

cd /mnt/gentoo

#Download the latest stage3 and extract it.
wget ftp://gentoo.osuosl.org/pub/gentoo/releases/amd64/autobuilds/current-stage3-amd64-hardened/stage3-amd64-hardened-*.tar.bz2
tar -xf /mnt/gentoo/stage3-amd64-hardened-*.tar.bz2 -C /mnt/gentoo

#get the latest portage tree
emerge --sync

#copy the zfs cache from the live system to the chroot
mkdir -p /mnt/gentoo/etc/zfs
cp /tmp/zpool.cache /mnt/gentoo/etc/zfs/zpool.cache

Kernel Config

If you are compiling the modules into the kernel staticly, then keep these things in mind.

When configuring the kernel, make sure that CONFIG_SPL and CONFIG_ZFS are set to 'Y'.
Portage will want to install sys-kernel/spl when emerge sys-fs/zfs is run because of dependencies. Also, sys-kernel/spl is still necessary to make the sys-fs/zfs configure script happy.
You do not need to run or install module-rebuild.
There have been some updates to the kernel/userspace ioctl since 0.6.0-rc9 was tagged.
- An issue occurs if newer userland utilities are used with older kernel modules.

Install as normal up until the kernel install.

echo "=sys-kernel/genkernel-3.4.40 ~amd64       #needed for zfs and encryption support" &gt;&gt; /etc/portage/package.accept_keywords
emerge sys-kernel/genkernel
emerge sys-kernel/gentoo-sources                #or hardned-sources

#patch the kernel

#If you want to build the modules into the kernel directly, you will need to patch the kernel directly.  Otherwise, skip the patch commands.
env EXTRA_ECONF='--enable-linux-builtin' ebuild /usr/portage/sys-kernel/spl/spl-0.6.2.ebuild clean configure
(cd /var/tmp/portage/sys-kernel/spl-0.6.2/work/spl-spl-0.6.2 &amp;&amp; ./copy-builtin /usr/src/linux)
env EXTRA_ECONF='--with-spl=/usr/src/linux --enable-linux-builtin' ebuild /usr/portage/sys-fs/zfs-kmod/zfs-kmod-0.6.2.ebuild clean configure
(cd /var/tmp/portage/sys-fs/zfs-kmod-0.6.2/work/zfs-zfs-0.6.2/ &amp;&amp; ./copy-builtin /usr/src/linux)
mkdir -p /etc/portage/profile
echo 'sys-fs/zfs -kernel-builtin' &gt;&gt; /etc/portage/profile/package.use.mask
echo 'sys-fs/zfs kernel-builtin' &gt;&gt; /etc/portage/package.use

#finish configuring, building and installing the kernel making sure to enable dm-crypt support

#if not building zfs into the kernel, install module-rebuild
emerge module-rebuild

#install SPL and ZFS stuff zfs pulls in spl automatically
mkdir -p /etc/portage/profile
echo 'sys-fs/zfs -kernel-builtin' &gt;&gt; /etc/portage/profile/package.use.mask
echo 'sys-fs/zfs kernel-builtin' &gt;&gt; /etc/portage/package.use
emerge sys-fs/zfs

# Add zfs to the correct runlevel
rc-update add zfs boot

#initrd creation, add '--callback="module-rebuild rebuild"' to the options if not building the modules into the kernel
genkernel --luks --zfs --disklabel initramfs

Finish installing as normal, your kernel line should look like this, and you should also have a the initrd defined.

#kernel line for grub2, libzfs support is not needed in grub2 because you are not mounting the filesystem directly.
linux  /kernel-3.5.0-gentoo real_root=ZFS=mypool/ROOT/rootfs crypt_root=/dev/sda2 dozfs=force ro
initrd /initramfs-genkernel-x86_64-3.5.0

In /etc/fstab, make sure BOOT, ROOT and SWAP lines are commented out and finish the install.

You should now have a working encryped zfs install.

Gentoo Hardened ZFS rootfs with dm-crypt/luks stable edition

Matthew Thode (prometheanfire) — Wed, 03 Apr 2013 00:00:00 -0500

Disclaimer

Keep in mind that ZFS on Linux is supported upstream, for differing values of support
I do not care much for hibernate, normal suspending works.
This is for a laptop/desktop, so I choose multilib.
If you patch the kernel to add in ZFS support directly, you cannot share the binary, the cddl and gpl2 are not compatible in that way.

Initialization

Make sure your installation media supports zfs on linux and installing whatever bootloader is required (uefi needs media that supports it as well). I uploaded an iso that works for me at this link Live DVDs newer then 12.1 should also have support, but the previous link has the stable version of zfsonlinux. If you need to install the bootloader via uefi, you can use one of the latest Fedora CDs, though the gentoo media should be getting support 'soon'. You can install your system normally up until the formatting begins.

Formatting

I will be assuming the following.

/boot on /dev/sda1
cryptroot on /dev/sda2
swap inside cryptroot OR not used.

When using GPT for partitioning, create the first partition at 1M, just to make sure you are on a sector boundry Most newer drives are 4k advanced format drives. Because of this you need ashift=12, some/most newer SSDs need ashift=13 compression set to lz4 will make your system incompatible with upstream (oracle) zfs, if you want to stay compatible then just set compression=on

General Setup

#setup encrypted partition
cryptsetup luksFormat -l 512 -c aes-xts-plain64 -h sha512 /dev/sda2
cryptsetup luksOpen /dev/sda2 cryptroot

#setup ZFS
zpool create -f -o ashift=12 -o cachefile=/tmp/zpool.cache -O normalization=formD -m none -R /mnt/gentoo rpool /dev/mapper/cryptroot
zfs create -o mountpoint=none -o compression=lz4 rpool/ROOT
#rootfs
zfs create -o mountpoint=/ rpool/ROOT/rootfs
zfs create -o mountpoint=/opt rpool/ROOT/rootfs/OPT
zfs create -o mountpoint=/usr rpool/ROOT/rootfs/USR
zfs create -o mountpoint=/var rpool/ROOT/rootfs/VAR
#portage
zfs create -o mountpoint=none rpool/GENTOO
zfs create -o mountpoint=/usr/portage rpool/GENTOO/portage
zfs create -o mountpoint=/usr/portage/distfiles -o compression=off rpool/GENTOO/distfiles
zfs create -o mountpoint=/usr/portage/packages -o compression=off rpool/GENTOO/packages
#homedirs
zfs create -o mountpoint=/home rpool/HOME
zfs create -o mountpoint=/root rpool/HOME/root

cd /mnt/gentoo

#Download the latest stage3 and extract it.
wget ftp://gentoo.osuosl.org/pub/gentoo/releases/amd64/autobuilds/current-stage3-amd64-hardened/stage3-amd64-hardened-*.tar.bz2
tar -xf /mnt/gentoo/stage3-amd64-hardened-*.tar.bz2 -C /mnt/gentoo

#get the latest portage tree
emerge --sync

#copy the zfs cache from the live system to the chroot
mkdir -p /mnt/gentoo/etc/zfs
cp /tmp/zpool.cache /mnt/gentoo/etc/zfs/zpool.cache

Kernel Config

If you are compiling the modules into the kernel staticly, then keep these things in mind.

When configuring the kernel, make sure that CONFIG_SPL and CONFIG_ZFS are set to 'Y'.
Portage will want to install sys-kernel/spl when emerge sys-fs/zfs is run because of dependencies. Also, sys-kernel/spl is still necessary to make the sys-fs/zfs configure script happy.
You do not need to run or install module-rebuild.
There have been some updates to the kernel/userspace ioctl since 0.6.0-rc9 was tagged.
- An issue occurs if newer userland utilities are used with older kernel modules.

Install as normal up until the kernel install.

echo "=sys-kernel/genkernel-3.4.40 ~amd64       #needed for zfs and encryption support" &gt;&gt; /etc/portage/package.accept_keywords
emerge sys-kernel/genkernel
emerge sys-kernel/gentoo-sources                #or hardned-sources

#patch the kernel

#If you want to build the modules into the kernel directly, you will need to patch the kernel directly.  Otherwise, skip the patch commands.
env EXTRA_ECONF='--enable-linux-builtin' ebuild /usr/portage/sys-kernel/spl/spl-0.6.1.ebuild clean configure
(cd /var/tmp/portage/sys-kernel/spl-0.6.1/work/spl-0.6.1 &amp;&amp; ./copy-builtin /usr/src/linux)
env EXTRA_ECONF='--with-spl=/usr/src/linux --enable-linux-builtin' ebuild /usr/portage/sys-fs/zfs-kmod/zfs-kmod-0.6.1.ebuild clean configure
(cd /var/tmp/portage/sys-fs/zfs-kmod-0.6.1/work/zfs-zfs-0.6.1/ &amp;&amp; ./copy-builtin /usr/src/linux)
mkdir -p /etc/portage/profile
echo 'sys-fs/zfs -kernel-builtin' &gt;&gt; /etc/portage/profile/package.use.mask
echo 'sys-fs/zfs kernel-builtin' &gt;&gt; /etc/portage/package.use

#finish configuring, building and installing the kernel making sure to enable dm-crypt support

#if not building zfs into the kernel, install module-rebuild
emerge module-rebuild

#install SPL and ZFS stuff zfs pulls in spl automatically
mkdir -p /etc/portage/profile                                                   
echo 'sys-fs/zfs -kernel-builtin' &gt;&gt; /etc/portage/profile/package.use.mask      
echo 'sys-fs/zfs kernel-builtin' &gt;&gt; /etc/portage/package.use                    
emerge sys-fs/zfs

# Add zfs to the correct runlevels
rc-update add zfs boot
rc-update add zfs-shutdown shutdown

#initrd creation, add '--callback="module-rebuild rebuild"' to the options if not building the modules into the kernel
genkernel --luks --zfs --disklabel initramfs

Finish installing as normal, your kernel line should look like this, and you should also have a the initrd defined.

#kernel line for grub2, libzfs support is not needed in grub2 because you are not mounting the filesystem directly.
linux  /kernel-3.5.0-gentoo real_root=ZFS=rpool/ROOT/rootfs crypt_root=/dev/sda2 dozfs=force ro
initrd /initramfs-genkernel-x86_64-3.5.0

In /etc/fstab, make sure BOOT, ROOT and SWAP lines are commented out and finish the install.

You should now have a working encryped zfs install.

Openstack on Gentoo

Matthew Thode (prometheanfire) — Mon, 28 Jan 2013 00:00:00 -0600

Just a simple announcement for now. It's a bit messy, but should work :D

I have packaged Openstack for Gentoo and it is now in tree, the most complete packaging is probably for Openstack Swift. Nova and some of the others are missing init scripts (being worked on). If you have problems or bugs, report as normal.

Gentoo Hardened ZFS rootfs with dm-crypt/luks updated 2012-12-12

Matthew Thode (prometheanfire) — Tue, 11 Dec 2012 00:00:00 -0600

Disclaimer

Keep in mind that ZFS on Linux is not fully supported, for differing values of support
I don't care much for hibernate, normal suspending works.
This is for a laptop/desktop, so I choose multilib.
If you patch the kernel to add in ZFS support directly, you cannot share the binary, the cddl and gpl2 are not compatible in that way.

Initialization

Make sure your installation media supports zfs on linux and installing whatever bootloader is required (uefi needs media that supports it as well). You can use the Gentoo LiveDVD, look for 12.1 or newer. If you need to install the bootloader via uefi, you can use one of the latest Fedora CDs, though the gentoo media should be getting support 'soon'. You can install your system normally up until the formatting begins.

Formatting

I will be assuming the following.

/boot on /dev/sda1
cryptroot on /dev/sda2
swap inside cryptroot OR not used.

When using GPT for partitioning, create the first partition at 1M, just to make sure you are on a sector boundry

General Setup

#setup encrypted partition
cryptsetup luksFormat -l 512 -c aes-xts-plain64 -h sha512 /dev/sda2
cryptsetup luksOpen /dev/sda2 cryptroot

#setup ZFS
zpool create -f -o ashift=12 -o cachefile= -O normalization=formD -m none -R /mnt/gentoo rpool /dev/mapper/cryptroot
zfs create -o mountpoint=none -o compression=on rpool/ROOT
#rootfs
zfs create -o mountpoint=/ rpool/ROOT/rootfs
zfs create -o mountpoint=/opt rpool/ROOT/rootfs/OPT
zfs create -o mountpoint=/usr rpool/ROOT/rootfs/USR
zfs create -o mountpoint=/var rpool/ROOT/rootfs/VAR
#portage
zfs create -o mountpoint=none rpool/GENTOO
zfs create -o mountpoint=/usr/portage rpool/GENTOO/portage
zfs create -o mountpoint=/usr/portage/distfiles -o compression=off rpool/GENTOO/distfiles
zfs create -o mountpoint=/usr/portage/packages -o compression=off rpool/GENTOO/packages
#homedirs
zfs create -o mountpoint=/home rpool/HOME
zfs create -o mountpoint=/root rpool/HOME/root

cd /mnt/gentoo

#Download the latest stage3 and extract it.
wget ftp://gentoo.osuosl.org/pub/gentoo/releases/amd64/autobuilds/current-stage3-amd64-hardened/stage3-amd64-hardened-*.tar.bz2
tar -xf /mnt/gentoo/stage3-amd64-hardened-*.tar.bz2 -C /mnt/gentoo

#get the latest portage tree
emerge --sync

#copy the zfs cache from the live system to the chroot
mkdir -p /mnt/gentoo/etc/zfs
cp /etc/zfs/zpool.cache /mnt/gentoo/etc/zfs/zpool.cache

Kernel Config

If you are compiling the modules into the kernel staticly, then keep these things in mind.

When configuring the kernel, make sure that CONFIG_SPL and CONFIG_ZFS are set to 'Y'.
Portage will want to install sys-kernel/spl when emerge sys-fs/zfs is run because of dependencies. Also, sys-kernel/spl is still necessary to make the sys-fs/zfs configure script happy.
You do not need to run or install module-rebuild.
There have been some updates to the kernel/userspace ioctl since 0.6.0-rc9 was tagged.
- An issue occurs if newer userland utilities are used with older kernel modules.

Install as normal up until the kernel install.

echo "=sys-kernel/genkernel-3.4.40 ~amd64       #needed for zfs and encryption support" &gt;&gt; /etc/portage/package.accept_keywords
emerge sys-kernel/genkernel
emerge sys-kernel/gentoo-sources

#patch the kernel

#If you want to build the modules into the kernel directly, you will need to patch the kernel directly.  Otherwise, skip the patch commands.
env EXTRA_ECONF='--enable-linux-builtin' ebuild /usr/portage/sys-kernel/spl/spl-9999.ebuild clean configure
(cd /var/tmp/portage/sys-kernel/spl-9999/work/spl-9999 &amp;&amp; ./copy-builtin /usr/src/linux)
env EXTRA_ECONF='--with-spl=/usr/src/linux --enable-linux-builtin' ebuild /usr/portage/sys-fs/zfs-kmod/zfs-kmod-9999.ebuild clean configure
(cd /var/tmp/portage/sys-fs/zfs-kmod-9999/work/zfs-/ &amp;&amp; ./copy-builtin /usr/src/linux)
mkdir -p /etc/portage/profile
echo 'sys-fs/zfs -kernel-builtin' &gt;&gt; /etc/portage/profile/package.use.mask
echo 'sys-fs/zfs kernel-builtin' &gt;&gt; /etc/portage/package.use

#finish configuring, building and installing the kernel making sure to enable dm-crypt support

#if not building zfs into the kernel, install module-rebuild
emerge module-rebuild

#install SPL and ZFS stuff zfs pulls in spl automatically
echo "=sys-kernel/spl-0.6.0_rc12 ~amd64       #needed for zfs support" &gt;&gt; /etc/portage/package.accept_keywords
echo "=sys-fs/zfs-0.6.0_rc12-r1 ~amd64           #needed for zfs support" &gt;&gt; /etc/portage/package.accept_keywords
emerge sys-fs/zfs

# Add zfs to the correct runlevels
rc-update add zfs boot
rc-update add zfs-shutdown shutdown

#initrd creation, add '--callback="module-rebuild rebuild"' to the options if not building the modules into the kernel
genkernel --luks --zfs --disklabel initramfs

Finish installing as normal, your kernel line should look like this, and you should also have a the initrd defined.

#kernel line for grub2, libzfs support is not needed in grub2 because you are not mounting the filesystem directly.
linux  /kernel-3.5.0-gentoo real_root=ZFS=rpool/ROOT/rootfs crypt_root=/dev/sda2 dozfs=force ro
initrd /initramfs-genkernel-x86_64-3.5.0

In /etc/fstab, make sure BOOT, ROOT and SWAP lines are commented out and finish the install.

You should now have a working encryped zfs install.

VLAN trunking to KVM VMs

Matthew Thode (prometheanfire) — Sun, 14 Oct 2012 00:00:00 -0500

Why this is needed

In testing linux bridging I noticed a problem that took me much longer then I feel comfortable admitting. You cannot break out the VLANs to from a physical device and also use that physical device (attached to a bridge) to forward forward the entire trunk to a set of VMs. The reason this occurs is that once linux starts inspecting for vlans on an interface to split them out it discards all those you do not have defined, so you have to trick it.

Setup

I had my Trunk on eth1. What you need to do is directly attach eth1 to a bridge (vmbr1). This bridge now has the entire trunk associated with it. Here's the fun part, you can break out vlans on the bridge, so you would have an interface for vlan 13 named vmbr1.13 and then attach that to a brige, allowing you to have a group of machines only exposed to vlan 13.

The networking goes like this.

               /-&gt; vmbr1.13 -&gt; vmbr13 -&gt; VM2
eth1 -&gt; vmbr1 ---&gt; VM1
               \-&gt; vmbr1.42 -&gt; vmbr42 -&gt; VM3

Example

Here is the script I used with proxmox (you can set up the bridge in proxmox, but not the source for the bridges data (the 'input'). This is for VLANs 1-13 and assumes you have vyatta set up the target bridges. I had this start at boot (via rc.local).

vconfig add vmbr1 2
vconfig add vmbr1 3
vconfig add vmbr1 4
vconfig add vmbr1 5
vconfig add vmbr1 6
vconfig add vmbr1 7
vconfig add vmbr1 9
vconfig add vmbr1 10
vconfig add vmbr1 11
vconfig add vmbr1 12
vconfig add vmbr1 13
ifconfig eth1 up
ifconfig vmbr1 up
ifconfig vmbr1.2 up
ifconfig vmbr1.3 up
ifconfig vmbr1.4 up
ifconfig vmbr1.5 up
ifconfig vmbr1.6 up
ifconfig vmbr1.7 up
ifconfig vmbr1.8 up
ifconfig vmbr1.9 up
ifconfig vmbr1.10 up
ifconfig vmbr1.11 up
ifconfig vmbr1.12 up
ifconfig vmbr1.13 up
brctl addif vmbr1 eth1
brctl addif vmbr2 vmbr1.2
brctl addif vmbr3 vmbr1.3
brctl addif vmbr4 vmbr1.4
brctl addif vmbr5 vmbr1.5
brctl addif vmbr6 vmbr1.6
brctl addif vmbr7 vmbr1.7
brctl addif vmbr8 vmbr1.8
brctl addif vmbr9 vmbr1.9
brctl addif vmbr10 vmbr1.10
brctl addif vmbr11 vmbr1.11
brctl addif vmbr12 vmbr1.12
brctl addif vmbr13 vmbr1.13

Gentoo Hardened ZFS rootfs with dm-crypt/luks

Matthew Thode (prometheanfire) — Tue, 31 Jul 2012 00:00:00 -0500

Disclaimer

Keep in mind that ZFS on Linux is not fully supported and stuff...
I don't care much for hibernate, normal suspending works.
This is for a laptop/desktop, so I choose multilib.
If you patch the kernel to add in ZFS support directly, you cannot share the binary, the cddl and gpl2 are not compatible in that way.

updated howto is here

Initialization

Make sure your installation media supports zfs on linux and installing whatever bootloader is required (uefi needs media that supports it as well). You can use the Gentoo LiveDVD, look for 12.1 or newer for the zfs portion of it, then, if you need to install the bootloader via uefi, you can use one of the latest fedora CDs. This is the method I used on my 2011 MacBook Pro, because Apple hardware is 'special'. You can install your system normally up until the formatting begins.

Formatting

I will be assuming the following.

/boot on /dev/sda1
cryptroot on /dev/sda2
swap inside cryptroot OR not used.

When using GPT for partitioning, create the first partition at 1M, just to make sure you are on a sector boundry

General Setup

#setup encrypted partition
cryptsetup luksFormat -l 512 -c aes-xts-plain64 -h sha512 /dev/sda2
cryptsetup luksOpen /dev/sda2 cryptroot

#setup ZFS
zpool create -f -o ashift=12 -o cachefile= -O normalization=formD -m none -R /mnt/gentoo rpool /dev/mapper/cryptroot
zfs create -o mountpoint=none -o compression=on rpool/ROOT
#rootfs
zfs create -o mountpoint=/ rpool/ROOT/rootfs
zfs create -o mountpoint=/opt rpool/ROOT/rootfs/OPT
zfs create -o mountpoint=/usr rpool/ROOT/rootfs/USR
zfs create -o mountpoint=/var rpool/ROOT/rootfs/VAR
#portage
zfs create -o mountpoint=none rpool/GENTOO
zfs create -o mountpoint=/usr/portage rpool/GENTOO/portage
zfs create -o mountpoint=/usr/portage/distfiles -o compression=off rpool/GENTOO/distfiles
zfs create -o mountpoint=/usr/portage/packages -o compression=off rpool/GENTOO/packages
#homedirs
zfs create -o mountpoint=/home rpool/HOME
zfs create -o mountpoint=/root rpool/HOME/root

cd /mnt/gentoo

#Download the latest stage3 and extract it.
wget ftp://gentoo.osuosl.org/pub/gentoo/releases/amd64/autobuilds/current-stage3-amd64-hardened/stage3-amd64-hardened-*.tar.bz2
tar -xf /mnt/gentoo/stage3-amd64-hardened-*.tar.bz2 -C /mnt/gentoo

#get the latest portage tree
emerge --sync

#copy the zfs cache from the live system to the chroot
mkdir -p /mnt/gentoo/etc/zfs
cp /etc/zfs/zpool.cache /mnt/gentoo/etc/zfs/zpool.cache

Kernel Config

If you are compiling the modules into the kernel staticly, then keep these things in mind.

You will need to use vanilla or gentoo sources because there is no 3.5 for hardened sources as of 2012-08-07.
When configuring the kernel, make sure that CONFIG_SPL and CONFIG_ZFS are set to 'Y'.
Portage will want to install sys-kernel/spl when emerge sys-fs/zfs is run because of dependencies. Also, sys-kernel/spl is still necessary to make the sys-fs/zfs configure script happy.
You do not need to run or install module-rebuild.
There have been some updates to the kernel/userspace ioctl since 0.6.0-rc9 was tagged.
- An issue occurs if newer userland utilities are used with older kernel modules.

Install as normal up until the kernel install.

echo "=sys-kernel/genkernel-3.4.40 ~amd64       #needed for zfs and encryption support" &gt;&gt; /etc/portage/package.accept_keywords
echo "=sys-apps/openrc-0.9.9.3 ~amd64           #needed for zfs support" &gt;&gt; /etc/portage/package.accept_keywords
echo "=sys-kernel/gentoo-sources-3.5.0 ~amd64   #needed for non-module zfs" &gt;&gt; /etc/portage/package.accept_keywords
emerge sys-kernel/genkernel
emerge sys-kernel/gentoo-sources

#patch the kernel
wget http://dev.gentoo.org/~prometheanfire/dist/kernel-patches/linux-3.5.0-gfp-vmalloc.patch -O - | patch -p1 -d /usr/src/linux

#If you want to build the modules into the kernel directly, you will need to patch the kernel directly.  Otherwise, skip the patch commands.
wget http://dev.gentoo.org/~ryao/dist/linux-3.5.0-zfs.patch -O - | patch -p1 -d /usr/src/linux
wget http://dev.gentoo.org/~prometheanfire/dist/kernel-patches/linux-3.5.0-zfs-builtin.patch -O - | patch -p1 -d /usr/src/linux

#finish configuring, building and installing the kernel making sure to enable dm-crypt support

#if not building zfs into the kernel, install module-rebuild
emerge module-rebuild

#install SPL (needed for ZFS) and ZFS (also aparently needed for zfs), zfs pulls in spl automatically
echo "=sys-kernel/spl-0.6.0_rc9-r2 ~amd64       #needed for zfs support" &gt;&gt; /etc/portage/package.accept_keywords
echo "=sys-fs/zfs-0.6.0_rc9-r6 ~amd64           #needed for zfs support" &gt;&gt; /etc/portage/package.accept_keywords
emerge sys-fs/zfs

# Add zfs to the correct runlevels
rc-update add zfs boot
rc-update add zfs-shutdown shutdown

#initrd creation, add '--callback="module-rebuild rebuild"' to the options if not building the modules into the kernel
genkernel --luks --zfs --disklabel initramfs

Finish installing as normal, your kernel line should look like this, and you should also have a the initrd defined.

#kernel line for grub2, libzfs support is not needed in grub2 because you are not mounting the filesystem directly.
linux  /kernel-3.5.0-gentoo real_root=ZFS=rpool/ROOT/rootfs crypt_root=/dev/sda2 dozfs=force ro
initrd /initramfs-genkernel-x86_64-3.5.0

In /etc/fstab, make sure BOOT, ROOT and SWAP lines are commented out and finish the install.

You should now have a working encryped zfs install.

Well, here we go again.

Matthew Thode (prometheanfire) — Fri, 20 Jul 2012 00:00:00 -0500

Well, here we go again, I think (hope) to actually keep this blog more up to date with the little projects I do. Going to be writing about selinux zfsonlinux the 'cloud' and whatever strikes my fancy.

I supose I should introduce myself, name's Matthew Thode, I currently work for Rackspace as a Linux Admin with a current focus on Big Data, I am also a Gentoo Developer on the hardened project, helping mostly with virtualization, selinux and testing. I know enough python to get by (it's in github actually) and that's about it for now.

Let's Play a Game

Building Gentoo disk images

Disclaimer

Why this is useful / needed

Pre-work

The actual setup

Generic images

Fin

Native ZFS encryption for your rootfs

Disclaimer

Why do this instead of luks

Prework

The actual setup

decrypting at boot

notes

Undervolting your CPU for fun and profit

Disclaimer

Why do this

Prework

Undervolting - The How

Undervolting - The Integration

Testing for stability

Gentoo Puppet Portage Package Provider

Why do this

How do I get this awesome thing

What it can do

fin

Gentoo portage templates

Why do this

Requirements

Hiera config

Basic Setup

Datastructures

Configuration Templates

package.keywords

Hiera data

Template

package.use

make.conf

The puppet module itself

fin

Gentoo at Fosdem

At the stand

Openstack Newton Update

The short of it

Gentoo, Openstack and OSIC

What to use it for

(pre)work

(actual)work

More to do

Of OpenStack and uwsgi

Why use uwsgi

Basic Setup

Configs and permissions

uwsgi configs

wsgi scripts

Creating Gentoo VM Images

Initial Setup and Info

Generating an Openstack image

Generating a Generic Image You Can Log Into

Stage4 tarballs, minimal and cloud

Where are they

Minimal

Cloud

Next

Of OpenStack and SSL

SSL in vanila OpenStack

Basic Setup

Configs and Tuning

General Config for All Services/Sites

Keystone and Uwsgi

Glance

Nova

Rabbitmq

Openstack Configs

General Config

Nova

Cinder

Glance

Glance