In order to investigate a possible Ubuntu server installation on top of (encrypted) ZFS, I decided to first do this with the desktop version. Reason for that is simple: The desktop version of the installer contains an option to install on a root ZFS volume (guess what: the server version does not).
So I started up virt-manager and installed Ubuntu 24.04 desktop on a virtual machine (patitioning scheme: ZFS encrypted).
Here’s what I got after successful installation:
linux # fdisk -l /dev/vda
Disk /dev/vda: 25 GiB, 26843545600 bytes, 52428800 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: F99672A2-F78E-4A41-9DA4-8E224A8A89B2
Device Start End Sectors Size Type
/dev/vda1 2048 4095 2048 1M BIOS boot
/dev/vda2 4096 4198399 4194304 2G Linux filesystem
/dev/vda3 4198400 12218367 8019968 3.8G Linux filesystem
/dev/vda4 12218368 52426751 40208384 19.2G Linux filesystem
linux # zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
bpool 1.88G 98.6M 1.78G - - 0% 5% 1.00x ONLINE -
rpool 19G 3.94G 15.1G - - 2% 20% 1.00x ONLINE -
linux # zfs list
NAME USED AVAIL REFER MOUNTPOINT
bpool 98.5M 1.65G 96K /boot
bpool/BOOT 98.0M 1.65G 96K none
bpool/BOOT/ubuntu_p5sqwo 97.9M 1.65G 97.9M /boot
rpool 3.95G 14.5G 192K /
rpool/ROOT 3.92G 14.5G 192K none
rpool/ROOT/ubuntu_p5sqwo 3.92G 14.5G 2.84G /
rpool/ROOT/ubuntu_p5sqwo/srv 192K 14.5G 192K /srv
rpool/ROOT/ubuntu_p5sqwo/usr 608K 14.5G 192K /usr
rpool/ROOT/ubuntu_p5sqwo/usr/local 416K 14.5G 416K /usr/local
rpool/ROOT/ubuntu_p5sqwo/var 1.08G 14.5G 192K /var
rpool/ROOT/ubuntu_p5sqwo/var/games 192K 14.5G 192K /var/games
rpool/ROOT/ubuntu_p5sqwo/var/lib 1.08G 14.5G 970M /var/lib
rpool/ROOT/ubuntu_p5sqwo/var/lib/AccountsService 212K 14.5G 212K /var/lib/AccountsService
rpool/ROOT/ubuntu_p5sqwo/var/lib/NetworkManager 256K 14.5G 256K /var/lib/NetworkManager
rpool/ROOT/ubuntu_p5sqwo/var/lib/apt 82.5M 14.5G 82.5M /var/lib/apt
rpool/ROOT/ubuntu_p5sqwo/var/lib/dpkg 49.5M 14.5G 49.5M /var/lib/dpkg
rpool/ROOT/ubuntu_p5sqwo/var/log 2.27M 14.5G 2.27M /var/log
rpool/ROOT/ubuntu_p5sqwo/var/mail 192K 14.5G 192K /var/mail
rpool/ROOT/ubuntu_p5sqwo/var/snap 2.16M 14.5G 2.16M /var/snap
rpool/ROOT/ubuntu_p5sqwo/var/spool 244K 14.5G 244K /var/spool
rpool/ROOT/ubuntu_p5sqwo/var/www 192K 14.5G 192K /var/www
rpool/USERDATA 5.46M 14.5G 192K none
rpool/USERDATA/home_tibfof 4.93M 14.5G 4.93M /home
rpool/USERDATA/root_tibfof 352K 14.5G 352K /root
rpool/keystore 22.5M 14.5G 16.5M -
linux # mount | grep zfs
rpool/ROOT/ubuntu_p5sqwo on / type zfs (rw,relatime,xattr,posixacl,casesensitive)
rpool/ROOT/ubuntu_p5sqwo/srv on /srv type zfs (rw,relatime,xattr,posixacl,casesensitive)
rpool/ROOT/ubuntu_p5sqwo/usr/local on /usr/local type zfs (rw,relatime,xattr,posixacl,casesensitive)
rpool/ROOT/ubuntu_p5sqwo/var/games on /var/games type zfs (rw,relatime,xattr,posixacl,casesensitive)
rpool/ROOT/ubuntu_p5sqwo/var/lib on /var/lib type zfs (rw,relatime,xattr,posixacl,casesensitive)
rpool/ROOT/ubuntu_p5sqwo/var/lib/AccountsService on /var/lib/AccountsService type zfs (rw,relatime,xattr,posixacl,casesensitive)
rpool/ROOT/ubuntu_p5sqwo/var/lib/NetworkManager on /var/lib/NetworkManager type zfs (rw,relatime,xattr,posixacl,casesensitive)
rpool/ROOT/ubuntu_p5sqwo/var/lib/apt on /var/lib/apt type zfs (rw,relatime,xattr,posixacl,casesensitive)
rpool/ROOT/ubuntu_p5sqwo/var/lib/dpkg on /var/lib/dpkg type zfs (rw,relatime,xattr,posixacl,casesensitive)
rpool/ROOT/ubuntu_p5sqwo/var/log on /var/log type zfs (rw,relatime,xattr,posixacl,casesensitive)
rpool/ROOT/ubuntu_p5sqwo/var/mail on /var/mail type zfs (rw,relatime,xattr,posixacl,casesensitive)
rpool/ROOT/ubuntu_p5sqwo/var/snap on /var/snap type zfs (rw,relatime,xattr,posixacl,casesensitive)
rpool/ROOT/ubuntu_p5sqwo/var/spool on /var/spool type zfs (rw,relatime,xattr,posixacl,casesensitive)
rpool/ROOT/ubuntu_p5sqwo/var/www on /var/www type zfs (rw,relatime,xattr,posixacl,casesensitive)
rpool/USERDATA/home_tibfof on /home type zfs (rw,nodev,relatime,xattr,posixacl,casesensitive)
rpool/USERDATA/root_tibfof on /root type zfs (rw,nodev,relatime,xattr,posixacl,casesensitive)
bpool/BOOT/ubuntu_p5sqwo on /boot type zfs (rw,nodev,relatime,xattr,posixacl,casesensitive)That looks exactly like the ZFS filesystem names in the documentation. So to achieve maximum compatibility I’ll follow that scheme.
linux # mount | grep keystore
/dev/mapper/keystore-rpool on /run/keystore/rpool type ext4 (rw,relatime,stripe=4)
linux # # ls -l1 /run/keystore/rpool/
total 20
drwx------ 2 root root 16384 Feb 21 06:12 lost+found
-r-------- 1 root root 32 Feb 21 06:12 system.key
linux # dmsetup ls
dm_crypt-0 (252:1)
keystore-rpool (252:0)
linux # dmsetup info dm_crypt-0
Name: dm_crypt-0
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 1
Event number: 0
Major, minor: 252, 1
Number of targets: 1
UUID: CRYPT-PLAIN-dm_crypt-0
linux # dmsetup info keystore-rpool
Name: keystore-rpool
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 1
Event number: 0
Major, minor: 252, 0
Number of targets: 1
UUID: CRYPT-LUKS2-2f097f7ea262446d9da51d4dcd9fb99c-keystore-rpoolNice thing about ZFS is: it logs every critical management activity. So it’s quite easy to see what the installer did to create the filesystems:
linux # zpool history rpool
History for 'rpool':
2025-02-21.06:12:12 zpool create -o ashift=12 -o autotrim=on -O canmount=off -O normalization=formD -O acltype=posixacl -O compression=lz4 -O devices=off -O dnodesize=auto -O relatime=on -O sync=standard -O xattr=sa -O encryption=on -O keylocation=file:///tmp/tmp8fdjx_yo -O keyformat=raw -O mountpoint=/ -R /target rpool /dev/vda4
2025-02-21.06:12:12 zpool set cachefile=/etc/zfs/zpool.cache rpool
2025-02-21.06:12:12 zfs create -o encryption=off -V 20971520 rpool/keystore
2025-02-21.06:12:22 zfs set keylocation=file:///run/keystore/rpool/system.key rpool
2025-02-21.06:12:22 zfs create -o canmount=off -o mountpoint=none rpool/ROOT
2025-02-21.06:12:22 zfs create -o canmount=on -o mountpoint=/ rpool/ROOT/ubuntu_p5sqwo
2025-02-21.06:12:22 zfs create -o canmount=off rpool/ROOT/ubuntu_p5sqwo/var
2025-02-21.06:12:22 zfs create -o canmount=on rpool/ROOT/ubuntu_p5sqwo/var/lib
2025-02-21.06:12:22 zfs create -o canmount=on rpool/ROOT/ubuntu_p5sqwo/var/lib/AccountsService
2025-02-21.06:12:22 zfs create -o canmount=on rpool/ROOT/ubuntu_p5sqwo/var/lib/apt
2025-02-21.06:12:22 zfs create -o canmount=on rpool/ROOT/ubuntu_p5sqwo/var/lib/dpkg
2025-02-21.06:12:22 zfs create -o canmount=on rpool/ROOT/ubuntu_p5sqwo/var/lib/NetworkManager
2025-02-21.06:12:22 zfs create -o canmount=on rpool/ROOT/ubuntu_p5sqwo/srv
2025-02-21.06:12:22 zfs create -o canmount=off rpool/ROOT/ubuntu_p5sqwo/usr
2025-02-21.06:12:22 zfs create -o canmount=on rpool/ROOT/ubuntu_p5sqwo/usr/local
2025-02-21.06:12:22 zfs create -o canmount=on rpool/ROOT/ubuntu_p5sqwo/var/games
2025-02-21.06:12:22 zfs create -o canmount=on rpool/ROOT/ubuntu_p5sqwo/var/log
2025-02-21.06:12:23 zfs create -o canmount=on rpool/ROOT/ubuntu_p5sqwo/var/mail
2025-02-21.06:12:23 zfs create -o canmount=on rpool/ROOT/ubuntu_p5sqwo/var/snap
2025-02-21.06:12:23 zfs create -o canmount=on rpool/ROOT/ubuntu_p5sqwo/var/spool
2025-02-21.06:12:23 zfs create -o canmount=on rpool/ROOT/ubuntu_p5sqwo/var/www
2025-02-21.06:12:23 zfs create -o canmount=off -o mountpoint=none rpool/USERDATA
2025-02-21.06:12:23 zfs create -o canmount=on -o mountpoint=/root rpool/USERDATA/root_tibfof
2025-02-21.06:12:23 zfs create -o canmount=on -o mountpoint=/home rpool/USERDATA/home_tibfof
2025-02-21.06:18:10 zpool set cachefile= rpool
2025-02-21.06:18:16 zpool export -a
2025-02-21.06:20:02 zpool import -N rpool
2025-02-21.06:20:08 zfs load-key rpool
linux # zpool history bpool
History for 'bpool':
2025-02-21.06:12:12 zpool create -o ashift=12 -o autotrim=on -o feature@async_destroy=enabled -o feature@bookmarks=enabled -o feature@embedded_data=enabled -o feature@empty_bpobj=enabled -o feature@enabled_txg=enabled -o feature@extensible_dataset=enabled -o feature@filesystem_limits=enabled -o feature@hole_birth=enabled -o feature@large_blocks=enabled -o feature@lz4_compress=enabled -o feature@spacemap_histogram=enabled -O canmount=off -O normalization=formD -O acltype=posixacl -O compression=lz4 -O devices=off -O relatime=on -O sync=standard -O xattr=sa -O mountpoint=/boot -R /target -d bpool /dev/vda2
2025-02-21.06:12:12 zpool set cachefile=/etc/zfs/zpool.cache bpool
2025-02-21.06:12:12 zfs create -o canmount=off -o mountpoint=none bpool/BOOT
2025-02-21.06:12:23 zfs create -o canmount=on -o mountpoint=/boot bpool/BOOT/ubuntu_p5sqwo
2025-02-21.06:18:10 zpool set cachefile= bpool
2025-02-21.06:18:15 zpool export -a
2025-02-21.06:20:11 zpool import -c /etc/zfs/zpool.cache -aNAnd about the encryption:
linux # zfs get keylocation,encryption,keyformat rpool/ROOT/ubuntu_p5sqwo
NAME PROPERTY VALUE SOURCE
rpool/ROOT/ubuntu_p5sqwo keylocation none default
rpool/ROOT/ubuntu_p5sqwo encryption aes-256-gcm -
rpool/ROOT/ubuntu_p5sqwo keyformat raw -