Kerberos problems need a domain to reproduce them in: a KDC, service accounts, keytabs, and services that actually accept tickets. The usual answer is a few VMs or a docker-compose file. For the krb5-testsuite I wanted something the test runner can set up and throw away in seconds, on a developer laptop or in a CI job, as a normal user: no root, no containers, no VMs, and if necessary not even a real network port.
The result is one shell script, tests/lab/samba_lab.sh. It provisions a Samba AD domain and starts ten “machines” around it: a member file server, IMAP and SMTP with Kerberos, OpenLDAP, Apache with mod_auth_gssapi, sshd, an NFS server and a KDC proxy. Setup takes 13 seconds, start 4. The trick is cwrap, a set of preload libraries the Samba team wrote for its own test suite. This post explains how they work and which traps each daemon had in store; most of them show up in the same way when you run these daemons in containers with reduced privileges.
All output below is real (Samba 4.23, MIT krb5 1.22, Dovecot 2.4, Postfix 3.10, OpenLDAP 2.6, Ubuntu). The lab directory is /tmp/adlab; why it has to be that short comes further down.
The lab at a glance

Each service has its own address in 127.0.0.0/8. That matters for Kerberos: the client builds the SPN from the host name (HTTP/www1.ad.test, imap/mail1.ad.test), some libraries canonicalise it through reverse DNS, and every service needs its own account and keytab. With everything on localhost you would test none of that.
Using it takes three commands. env prints the client environment; every program started with it sees the lab:
$ tests/lab/samba_lab.sh /tmp/adlab setup
provisioning AD.TEST in /tmp/adlab/samba (mode: socket) ...
creating users, service account, computer account and keytabs ...
adding SSH server ssh1 (OpenSSH sshd with GSSAPI) ...
adding OpenLDAP server ldap1 (slapd with GSSAPI) ...
adding member file server fs1 (joined at the first start) ...
adding KDC proxy kkdcp (python-kdcproxy over HTTPS) ...
adding web server www1 (Apache with mod_auth_gssapi) ...
adding mail server mail1 (Dovecot IMAP, Postfix submission) ...
note: no NFS server in socket mode (nfs-ganesha listens on IPv6 sockets, which socket_wrapper keeps apart from the IPv4 clients)
Samba AD lab ready in /tmp/adlab (mode: socket, ...)
$ tests/lab/samba_lab.sh /tmp/adlab start
$ eval "$(tests/lab/samba_lab.sh /tmp/adlab env)"
$ kinit alice
$ smbclient //fs1.ad.test/data --use-kerberos=required -c ls
setup also writes a lab.yaml for the test suite, so python3 -m krb5test -c /tmp/adlab/lab.yaml runs all checks against the lab. Services whose software is not installed are simply left out.
Three libraries that lie to the daemons
cwrap consists of preload libraries: loaded with LD_PRELOAD, they replace a group of libc functions and answer them from their own configuration. The daemon itself is unmodified.

uid_wrapper: root for the daemon, not for the kernel
Samba’s file server insists on running as root: it switches to the user’s uid for every file access and back again. uid_wrapper (UID_WRAPPER=1 UID_WRAPPER_ROOT=1) makes getuid() return 0 and lets setuid(), seteuid() and setgroups() “succeed” by remembering the values per thread. The kernel never sees any of it:
$ id -u
1000
$ LD_PRELOAD=libuid_wrapper.so:libnss_wrapper.so UID_WRAPPER=1 UID_WRAPPER_ROOT=1 ... id
uid=0(root) gid=0(0) groups=0(0)
Without it, smbd does not even get as far as an error message about permissions:
INTERNAL ERROR: Security context active token stack underflow! in () () pid 1107409 (4.23.6-Ubuntu-4.23.6+dfsg-1ubuntu2.2)
PANIC (pid 1107409): Security context active token stack underflow!
nss_wrapper: private passwd, group and hosts files
nss_wrapper answers getpwnam(), getgrnam(), gethostbyname(), getaddrinfo() and gethostname() from files named in NSS_WRAPPER_PASSWD, NSS_WRAPPER_GROUP, NSS_WRAPPER_HOSTS and NSS_WRAPPER_HOSTNAME. That replaces both /etc/hosts and DNS for the lab names, and it lets every daemon believe it runs on its own host:
$ cat /tmp/adlab/nss/hosts
127.0.0.2 dc1.ad.test dc1
127.0.0.3 web.ad.test web
...
127.0.0.9 www1.ad.test www1 intranet.ad.test
127.0.0.10 fs1.ad.test fs1
127.0.0.11 kkdcp.ad.test kkdcp
$ getent hosts 127.0.0.8
127.0.0.8 ldap1.ad.test ldap1
The passwd and group files are the second half of the root illusion. uid_wrapper says uid 0, but every file the daemon creates still belongs to uid 1000. When Samba’s provisioning then tries to chown() a file to “root”, the kernel refuses. The fix is a passwd file in which root has the real primary group of the lab user, so the ownership Samba asks for is the one the file already has:
$ cat /tmp/adlab/nss/passwd /tmp/adlab/nss/group
root:x:0:1000:root:/:/bin/false
nobody:x:65534:65534:nobody:/:/bin/false
root:x:1000:
wheel:x:1000:
users:x:1000:
nogroup:x:65534:
One trap cost me an evening: one line per address. An extra line 127.0.0.8 ldap.ad.test next to the existing one for ldap1 made getaddrinfo() return 127.0.0.8 twice. slapd binds to every address it gets back, so its second bind() failed with “Address already in use” and slapd exited. Aliases belong on the same line, as with intranet.ad.test above.
socket_wrapper: ports without ports
The KDC wants port 88, SMB 445, LDAP 389. On a normal desktop a user may not bind ports below 1024. socket_wrapper intercepts socket(), bind(), connect() and friends and turns every IPv4 connection into a unix domain socket in SOCKET_WRAPPER_DIR. Each file name encodes the protocol, the last octet of the address (the “interface”) and the port, in hex:
$ ls /tmp/adlab/sw
T020058 # TCP, 127.0.0.2, port 0x58 = 88 KDC
T020185 # port 0x185 = 389 LDAP
T0201BD # port 0x1BD = 445 SMB
T0201D0 # port 0x1D0 = 464 kpasswd
T02027C # port 0x27C = 636 LDAPS
T0346A1 # TCP, 127.0.0.3, port 18081 test web server
T0608AE # TCP, 127.0.0.6, port 2222 sshd
... (24 sockets in total)
$ ss -ltn | grep -c '127\.0\.0\.'
1
The one real listener on 127.0.0.x is Docker’s embedded DNS resolver, not the lab. The price: only processes with socket_wrapper preloaded can reach the lab. A browser or a tool from another shell sees nothing, which is why the client environment from samba_lab.sh env loads socket_wrapper as well, and gives the client its own interface number (9).
Three modes
socket_wrapper is only needed when low ports are forbidden. Many containers and CI runners set net.ipv4.ip_unprivileged_port_start=0, and then the lab can use real ports, with uid_wrapper only for the daemons that need root. The script detects which case applies:

Two services lose out in socket mode. nfs-ganesha opens IPv6 sockets even for an IPv4 bind address, and socket_wrapper keeps IPv4 and IPv6 strictly apart, so the NFS server is skipped. And DNS SRV lookups are switched off in the generated lab.yaml: socket_wrapper also captures the UDP packets to the real resolver, so every lookup would just run into its timeout.
Samba AD DC: four detours
Samba’s own selftest runs the DC under cwrap, so most of it just works. Four things did not:
- sysvol ACLs.
samba-tool domain provisionsets NT ACLs on sysvol, which needs real root privileges for the underlying xattrs. Only Group Policy needs them, so the script calls the provisioning code from Python withsamba.provision.setsysvolaclreplaced by a no-op. ACLs and DOS attributes go into TDB files instead of xattrs:vfs objects = dfs_samba4 acl_tdb xattr_tdb. - The winbindd socket. Samba’s DC and its client libraries find winbindd through a socket directory that is compiled in. libwbclient accepts another one in
SELFTEST_WINBINDD_SOCKET_DIR, but only while nss_wrapper is active. That variable is Samba’s own selftest hook, not a documented interface. - Starting the daemons.
sambastarts smbd and winbindd itself, from its compiled-in paths. The script starts them separately and adds--option="server role check:inhibit=yes", otherwise smbd and winbindd refuse to run in a DC configuration. - The DC’s own SPNs.
ldap/dc1.ad.testand friends are normally added bysamba_spnupdateat runtime. Without them an LDAP GSSAPI bind fails with “Server not found in Kerberos database”. The script runssamba_spnupdateonce after provisioning.
And one rule for the lab directory: keep its path short. Unix socket paths are limited to 108 bytes, and Samba, Dovecot and socket_wrapper all create sockets several levels deep below it. A lab in a deep CI workspace failed with errors that mention neither the path nor its length. /tmp/adlab is safe.
The member file server: a real net ads join
fs1 is a second Samba instance with its own smb.conf, its own winbindd and its own nss_wrapper host name. It joins the domain with net ads join at the first start, exactly as a real server would. Three settings were needed:
[global]
security = ads
realm = AD.TEST
create krb5 conf = no
sync machine password to keytab = /tmp/adlab/fs1.keytab:account_name:sync_spns:spn_prefixes=cifs:sync_kvno:machine_password
idmap config AD : backend = rid
idmap config AD : range = 10000-999999
vfs objects = acl_tdb xattr_tdb
create krb5 conf = no: otherwise winbindd writes its own krb5.conf, which ignores the lab’s.sync machine password to keytab(Samba 4.21 and later) keeps a keytab in sync with the machine password, so the keytab follows every password change. The value is a list, and smb.conf splits lists at commas as well as at spaces:spn_prefixes=cifs,hostturns into two broken entries. Either use one prefix, orsync_spns, which takes the SPNs from the account.- AD users on the file server come from
libnss_winbind. Without root it cannot be added to/etc/nsswitch.conf, but nss_wrapper can load an NSS module directly:NSS_WRAPPER_MODULE_SO_PATH=.../libnss_winbind.so.2andNSS_WRAPPER_MODULE_FN_PREFIX=winbind. With that,aliceresolves to uid 11103 on fs1.
Dovecot and Postfix
Mail servers are built to start as root and drop privileges to dedicated users. Dovecot can run entirely as a normal user if every user in its configuration is that user:
default_internal_user = labuser
default_login_user = labuser
base_dir = /tmp/adlab/mail/run
auth_mechanisms = gssapi
auth_krb5_keytab = /tmp/adlab/mail1.keytab
auth_gssapi_hostname = mail1.ad.test
passdb static {
fields {
nopassword = yes # GSSAPI is the only mechanism
}
}
service imap-login {
chroot =
}
service anvil {
chroot =
}
# Dovecot cleans the environment of its child processes
import_environment {
KRB5_CONFIG = %{env:KRB5_CONFIG}
LD_PRELOAD = %{env:LD_PRELOAD}
NSS_WRAPPER_HOSTS = %{env:NSS_WRAPPER_HOSTS}
SOCKET_WRAPPER_DIR = %{env:SOCKET_WRAPPER_DIR}
...
}
The login processes chroot by default, which needs root, hence the empty chroot. The import_environment block is the most important part: Dovecot starts its children with a clean environment, so without it the auth process loses both KRB5_CONFIG and the wrappers and looks for the KDC in the system’s krb5.conf.
Postfix is stricter: it wants a dedicated mail_owner that is not root, owning the queue, and a setgid_group that differs from the mail owner’s group. Postfix runs under uid_wrapper with its own nss_wrapper passwd and group:
# passwd: "postfix" is the lab user's real uid, so the queue files belong to it
root:x:0:0::/:/bin/false
postfix:x:1000:1000::/tmp/adlab/mail/postfix:/bin/false
# group: postdrop must differ from postfix's group; it is never used
postdrop:x:65533:
# main.cf
mail_owner = postfix
setgid_group = postdrop
inet_interfaces = loopback-only
inet_protocols = ipv4
maillog_file = /tmp/adlab/mail/postfix/maillog
maillog_file_prefixes = /tmp/adlab/mail/postfix
import_environment = MAIL_CONFIG ... LD_PRELOAD UID_WRAPPER UID_WRAPPER_ROOT NSS_WRAPPER_PASSWD NSS_WRAPPER_GROUP NSS_WRAPPER_HOSTS SOCKET_WRAPPER_DIR SOCKET_WRAPPER_DEFAULT_IFACE
smtpd_sasl_type = dovecot
smtpd_sasl_path = inet:127.0.0.7:12345
Three details: there is no syslog to write to, so Postfix logs to a file, and maillog_file_prefixes must allow that file’s location. import_environment is Postfix’s version of the same environment trap as in Dovecot. And the script creates the queue directories itself, so they belong to the lab user, which is postfix in the wrapped passwd file. Kerberos itself is handled by Dovecot: Postfix passes SASL to Dovecot’s auth service over TCP.
The rest: sshd, OpenLDAP, Apache, NFS, KDC proxy
- sshd runs on port 2222 with its own host key,
UsePAM noandStrictModes no; nss_wrapper makesgethostname()returnssh1.ad.test, so the defaultGSSAPIStrictAcceptorCheck yesworks. Without root it can only log in its own user, so the lab’s krb5.conf maps the principal withauth_to_local = RULE:[1:$1@$0](alice@AD.TEST)s/.*/labuser/. - OpenLDAP needs nothing special apart from a keytab with
ldap/ldap1.ad.testand its own schema and module directories. - Apache runs with
-f,-DFOREGROUNDand a minimal config that loads only the modules it needs;mod_auth_gssapiuses its own keytab viaGssapiCredStore keytab:. - nfs-ganesha (direct and root mode only) serves an in-memory export with
sec=krb5:krb5i:krb5p. Without root it needsallow_set_io_flusher_fail = true, otherwise it exits becauseprctl(PR_SET_IO_FLUSHER)fails. - The KDC proxy is python-kdcproxy behind a small TLS server with a certificate from a lab CA; the details are in the KKDCP post.
Two shell traps
Starting and stopping ten daemons from a script produced two problems that have nothing to do with Kerberos. First, daemons that stay in the foreground are started with exec setsid in a subshell, and stop kills the whole process group with kill -- -$pid. In an interactive shell with job control, setsid forks instead of exec’ing, the recorded PID is the wrong one, and old daemons survive; running the script with bash avoids that. Second, pkill -f with a pattern that also appears in the calling command line kills the calling shell itself. The script never uses it.
The result
The full test suite against the lab in socket mode, as a normal user without a single real listening port:
$ python3 -m krb5test -c /tmp/adlab/lab.yaml
...
== Summary ==
[OK ] Local environment
[OK ] Realm AD.TEST
[OK ] Client alice@AD.TEST
[WARN] Keytab /tmp/adlab/client1.keytab (nfs-client) [client1]
warning in GSS daemon - rpc.gssd (rpc.gssd not running)
[OK ] Keytab /tmp/adlab/web.keytab (generic) [web]
[OK ] Keytab /tmp/adlab/ssh1.keytab (sshd) [ssh1]
[OK ] Keytab /tmp/adlab/mail1.keytab (generic) [mail1]
[OK ] dc-smb (smb, dc1.ad.test:445)
[OK ] dc-ldap (ldap, dc1.ad.test:389)
[OK ] dc-ldaps (ldap, dc1.ad.test:636)
[OK ] dc-ldap-starttls (ldap, dc1.ad.test:389)
[OK ] ad-web (http, web.ad.test:18081)
[OK ] ad-ssh (ssh, ssh1.ad.test:2222)
[OK ] ad-openldap (ldap, ldap1.ad.test:389)
[OK ] fs1-smb (smb, fs1.ad.test:445)
[OK ] ad-apache (http, www1.ad.test:80)
[OK ] ad-imap (imap, mail1.ad.test:143)
[OK ] ad-smtp (smtp, mail1.ad.test:587)
The one WARN is correct: no rpc.gssd runs on this machine. The integration tests break the lab on purpose (password reset behind the keytab’s back, RC4-only accounts, removed SPNs, disabled users) and check that the suite names the cause.
The lab paid for itself quickly. Against real services it found bugs that unit tests with mocked tools had missed: Heimdal’s klist has no -V and was detected as MIT; module = lines in krb5.conf’s [plugins] section were dropped by the parser; Python 3.13’s stricter certificate checks rejected the lab CA; and an error hint claimed that strict acceptors such as sshd reject tickets for alias SPNs, which a test against the real sshd disproved. Several of the traps in the posts on keytabs and LDAP over TLS were first reproduced here.
Limits
- It is a lab, not a model of production: no GPOs, no sysvol ACLs, no replication, one DC.
- Kernel clients are out of reach: the NFS server works, but
mount -t nfsneeds root, so the suite tests the RPCSEC_GSS handshake with its own probe. - Machine password rotation from the member server (
net ads changetrustpw) does not work in the lab yet. - cwrap is a test tool. The Samba variables used here are internal hooks and can change between releases.
The script is in the repository at tests/lab/samba_lab.sh. On Debian and Ubuntu, the wrappers are in the packages libuid-wrapper, libnss-wrapper and libsocket-wrapper.
