ZFS snapshot problems

While experimenting with encrypted ZFS root file systems, I encountered some odd behavior concerning snapshots: I had created a separate /etc file system with snapshots. The listing of the snapshots shows up, however all snapshot directories were empty:

linux # ls -1 /etc/.zfs/snapshot/
<...>
zfs-auto-snap_hourly-2025-02-28-1000
zfs-auto-snap_hourly-2025-02-28-1100
zfs-auto-snap_hourly-2025-02-28-1200
linux # ls -l /etc/.zfs/snapshot/zfs-auto-snap_hourly-2025-02-28-1200
total 0

However zfs reported, that the snapshots contain data, so I guess the snapshots are created, they’re just not accessible/mounted:

linux # zfs list -t snapshot /etc
<...>
rpool/ROOT/ubuntu_i7qijx/etc@zfs-auto-snap_hourly-2025-02-28-1000     0B      -  51.7M  -
rpool/ROOT/ubuntu_i7qijx/etc@zfs-auto-snap_hourly-2025-02-28-1100     0B      -  51.7M  -
rpool/ROOT/ubuntu_i7qijx/etc@zfs-auto-snap_hourly-2025-02-28-1200   144K      -  51.7M  -

The ZFS kernel debug interface confirmed, that for some reason (auto)mounting the snapshots fails:

linux # cat /proc/spl/kstat/zfs/dbgmsg
<...>
1740744285   zfs_ctldir.c:1161:zfsctl_snapshot_mount(): Unable to automount /root/etc/.zfs/snapshot/zfs-auto-snap_hourly-2025-02-28-1200 error=512

What makes things worse (and investigating this bug nasty): Trying to access/mount the snapshots manually sooner or later crashes the whole system (that’s with ZFS 2.3, did not observe this with 2.2). Only the root ZFS volume (and its children) seem to be affected.

No solution yet

For now I’m lost – if you’ve got ideas: Please drop me a note.

Hints about the cause of this may be (source github issues):

• Booting from (encrypted) root ZFS via initramfs
  • Multiple problems about that were reported, mostly disappeared with some ZFS update
    • Possible causes: Bad initramfs, bad change of root file systems during boot
      • s. man 8 switch_root / pivot_root
      • Possible paths mentioned: /sysroot, /new_root, /root
    • mount.zfs instead of busybox mount in initramfs
• Maybe OpenZFS 2.3.0 related?
  • Most distributions come with version 2.2.x
  • I downgraded to version 2.2 – same behavior