Remotely unlocking encrypted root file systems

In my latest Ubuntu server installation I was aiming for a fully encrypted file system. In order to unlock the file system from remote this requires some extra work – here’s what I found:

How does it work?

The first obvious question is: If everything is encrypted, how can we even boot up a system?

Well basically this is because not everything is encrypted: A few boot files need to be accessible to BIOS/UEFI to make a system bootable. But that’s only a very tiny part (mainly a kernel image and an initial RAM disk, short “initrd”).

The initial RAM disk contains a very compact Linux environment – just enough to boot up the target system.

Every modern Linux system comes with that and uses it to mount the file system and start the system from there.

What’s special?

While a normal boot up does not require user interaction, we’ll need someone to enter the password for the encrypted disk before starting the system.

As I usually/often do not have physical access to the system, we need to be able to do this from a remote location.

So we require our initial RAM disk to contain three things:

• tools to unlock the encrypted filesystem
• networking support
• SSH access

SecureShell via dropbear

As mentioned above the initrd system is a very small linux system, using busybox as replacement for most small unix tools and dropbear as a small OpenSSH replacement.

While the initrd with busybox is installed by default, we still need to install the dropbear component:

linux # apt install dropbear-initramfs

Depending on how it’s build dropbear may be missing some of OpenSSH’s features (such as certain key formats). That’s why we generate a RSA key for connecting to then initial ram disk:

linux # ssh-keygen -t rsa -f ~/.ssh/server-initrd -P ''

This will create both private and public key. Now we need to add the public key to dropbear’s authorized_keys file:

linux # cat ~/.ssh/server-initrd.pub >> /etc/dropbear/initramfs/authorized_keys
linux # chmod 600 /etc/dropbear/initramfs/authorized_keys

And make sure to store the private key on the machine you’ll be using to connect to the encrypted system.

After a reboot you can now connect to the system using:

linux # ssh -l root server.mydomain.de -i ~/.ssh/server-initrd

On successful login you’ll be located on a (busybox) shell where you can enter the unlock command. For dm-crypt encryption:

# cryptroot-unlock
Please unlock disk dm_crypt-0: 
cryptsetup: dm_crypt-0 set up successfully

And for direct ZFS encryption:

# zfsunlock
Unlocking encrypted ZFS filesystems...
Enter the password or press Ctrl-C to exit.

🔐 Encrypted ZFS password for rpool: ••••••••••••••••••••••••••••••••
Password for rpool accepted.
Unlocking complete.  Resuming boot sequence...
Please reconnect in a while.
Connection to srv.linux-ng.de closed.

As this is normally the only useful thing you can do at that point, you can also make this command the only one being executed as soon as you login:

linux # cat /etc/dropbear/initramfs/dropbear.conf
<...>
DROPBEAR_OPTIONS="-I 180 -j -k -p 22 -s -c cryptroot-unlock"

or

linux # cat /etc/dropbear/initramfs/dropbear.conf
<...>
DROPBEAR_OPTIONS="-I 180 -j -k -p 22 -s -c zfsunlock"

Don’t forget to run update-initramfs -u to apply changes.

Cleanups

On a standard setup dropbear creates its own host keys (independent from your usual OpenSSH host keys). As a result you will be prompted a warning “hostkey changed” every time you try to connect via ssh.

You could also use a different IP/hostname for the minimal unlock boot image, but I found that to be confusing.

I simply used the existing OpenSSH keys and converted them to dropbear format and used those converted keys on both: OpenSSH and (initial RAM disk) dropbear.

The conversion is as easy as this:

linux # dropbearconvert openssh dropbear /etc/ssh/ssh_host_ecdsa_key /etc/dropbear/initramfs/dropbear_ecdsa_host_key
Key is a ecdsa-sha2-nistp256 key
Wrote key to '/etc/dropbear/initramfs/dropbear_ecdsa_host_key'
linux # dropbearconvert openssh dropbear /etc/ssh/ssh_host_ed25519_key /etc/dropbear/initramfs/dropbear_ed25519_host_key
Key is a ssh-ed25519 key
Wrote key to '/etc/dropbear/initramfs/dropbear_ed25519_host_key'
linux # dropbearconvert openssh dropbear /etc/ssh/ssh_host_rsa_key /etc/dropbear/initramfs/dropbear_rsa_host_key
Key is a ssh-rsa key
Wrote key to '/etc/dropbear/initramfs/dropbear_rsa_host_key'

Don’t forget to rebuild your RAM disk afterwards:

linux # update-initramfs -u 

Other findings

While testing I encountered several crashes shortly after entering the ZFS password over SSH connection. The culprit here was plymouth (or one of its components) – which is not required for ZFS encryption. Removing those packages and re-building the RAM disks helped:

linux # apt remove plymouth plymouth-theme-ubuntu-text
linux # update-initramfs -u