Categories
Directory Service Kerberos Linux Samba

A complete Active Directory lab without root: Samba, NFS, mail, LDAP and Apache with cwrap

A Samba AD domain with a member file server, NFS, IMAP/SMTP, OpenLDAP, Apache and a KDC proxy, set up by one script, as a normal user, without containers or VMs, if needed even without real network ports. How cwrap’s uid_wrapper, nss_wrapper and socket_wrapper make that possible, and the traps each daemon had in store.

Kerberos problems need a domain to reproduce them in: a KDC, service accounts, keytabs, and services that actually accept tickets. The usual answer is a few VMs or a docker-compose file. For the krb5-testsuite I wanted something the test runner can set up and throw away in seconds, on a developer laptop or in a CI job, as a normal user: no root, no containers, no VMs, and if necessary not even a real network port.

The result is one shell script, tests/lab/samba_lab.sh. It provisions a Samba AD domain and starts ten “machines” around it: a member file server, IMAP and SMTP with Kerberos, OpenLDAP, Apache with mod_auth_gssapi, sshd, an NFS server and a KDC proxy. Setup takes 13 seconds, start 4. The trick is cwrap, a set of preload libraries the Samba team wrote for its own test suite. This post explains how they work and which traps each daemon had in store; most of them show up in the same way when you run these daemons in containers with reduced privileges.

All output below is real (Samba 4.23, MIT krb5 1.22, Dovecot 2.4, Postfix 3.10, OpenLDAP 2.6, Ubuntu). The lab directory is /tmp/adlab; why it has to be that short comes further down.

The lab at a glance

Topology of the lab on one host as a normal user: the Samba AD DC dc1.ad.test on 127.0.0.2 with KDC, kpasswd, LDAP, LDAPS and SMB; services on their own 127.0.0.x addresses: member file server fs1, Dovecot and Postfix on mail1, OpenLDAP on ldap1, Apache with mod_auth_gssapi on www1, sshd on ssh1, nfs-ganesha on nfs1, KDC proxy on kkdcp, a test web server and the client1 computer account. The test suite resolves all names through the nss_wrapper hosts file.
Every service gets its own loopback address and host name, so SPNs, reverse lookups and keytabs behave as on separate machines.

Each service has its own address in 127.0.0.0/8. That matters for Kerberos: the client builds the SPN from the host name (HTTP/www1.ad.test, imap/mail1.ad.test), some libraries canonicalise it through reverse DNS, and every service needs its own account and keytab. With everything on localhost you would test none of that.

Using it takes three commands. env prints the client environment; every program started with it sees the lab:

$ tests/lab/samba_lab.sh /tmp/adlab setup
provisioning AD.TEST in /tmp/adlab/samba (mode: socket) ...
creating users, service account, computer account and keytabs ...
adding SSH server ssh1 (OpenSSH sshd with GSSAPI) ...
adding OpenLDAP server ldap1 (slapd with GSSAPI) ...
adding member file server fs1 (joined at the first start) ...
adding KDC proxy kkdcp (python-kdcproxy over HTTPS) ...
adding web server www1 (Apache with mod_auth_gssapi) ...
adding mail server mail1 (Dovecot IMAP, Postfix submission) ...
note: no NFS server in socket mode (nfs-ganesha listens on IPv6 sockets, which socket_wrapper keeps apart from the IPv4 clients)
Samba AD lab ready in /tmp/adlab (mode: socket, ...)

$ tests/lab/samba_lab.sh /tmp/adlab start
$ eval "$(tests/lab/samba_lab.sh /tmp/adlab env)"
$ kinit alice
$ smbclient //fs1.ad.test/data --use-kerberos=required -c ls

setup also writes a lab.yaml for the test suite, so python3 -m krb5test -c /tmp/adlab/lab.yaml runs all checks against the lab. Services whose software is not installed are simply left out.

Three libraries that lie to the daemons

cwrap consists of preload libraries: loaded with LD_PRELOAD, they replace a group of libc functions and answer them from their own configuration. The daemon itself is unmodified.

Diagram: the three cwrap preload libraries sit between the daemon and libc. uid_wrapper intercepts getuid and setuid and reports uid 0; nss_wrapper answers getpwnam, getgrnam and getaddrinfo from the lab's own files; socket_wrapper turns 127.0.0.x traffic into unix sockets in SOCKET_WRAPPER_DIR. Below: where the illusion leaks, such as daemons that clean their environment, chroot, the winbindd socket directory, IPv6 listeners and the 108-byte socket path limit.
uid_wrapper, nss_wrapper and socket_wrapper, and where each one leaks.

uid_wrapper: root for the daemon, not for the kernel

Samba’s file server insists on running as root: it switches to the user’s uid for every file access and back again. uid_wrapper (UID_WRAPPER=1 UID_WRAPPER_ROOT=1) makes getuid() return 0 and lets setuid(), seteuid() and setgroups() “succeed” by remembering the values per thread. The kernel never sees any of it:

$ id -u
1000
$ LD_PRELOAD=libuid_wrapper.so:libnss_wrapper.so UID_WRAPPER=1 UID_WRAPPER_ROOT=1 ... id
uid=0(root) gid=0(0) groups=0(0)

Without it, smbd does not even get as far as an error message about permissions:

INTERNAL ERROR: Security context active token stack underflow! in () () pid 1107409 (4.23.6-Ubuntu-4.23.6+dfsg-1ubuntu2.2)
PANIC (pid 1107409): Security context active token stack underflow!

nss_wrapper: private passwd, group and hosts files

nss_wrapper answers getpwnam(), getgrnam(), gethostbyname(), getaddrinfo() and gethostname() from files named in NSS_WRAPPER_PASSWD, NSS_WRAPPER_GROUP, NSS_WRAPPER_HOSTS and NSS_WRAPPER_HOSTNAME. That replaces both /etc/hosts and DNS for the lab names, and it lets every daemon believe it runs on its own host:

$ cat /tmp/adlab/nss/hosts
127.0.0.2 dc1.ad.test dc1
127.0.0.3 web.ad.test web
...
127.0.0.9 www1.ad.test www1 intranet.ad.test
127.0.0.10 fs1.ad.test fs1
127.0.0.11 kkdcp.ad.test kkdcp

$ getent hosts 127.0.0.8
127.0.0.8       ldap1.ad.test ldap1

The passwd and group files are the second half of the root illusion. uid_wrapper says uid 0, but every file the daemon creates still belongs to uid 1000. When Samba’s provisioning then tries to chown() a file to “root”, the kernel refuses. The fix is a passwd file in which root has the real primary group of the lab user, so the ownership Samba asks for is the one the file already has:

$ cat /tmp/adlab/nss/passwd /tmp/adlab/nss/group
root:x:0:1000:root:/:/bin/false
nobody:x:65534:65534:nobody:/:/bin/false
root:x:1000:
wheel:x:1000:
users:x:1000:
nogroup:x:65534:

One trap cost me an evening: one line per address. An extra line 127.0.0.8 ldap.ad.test next to the existing one for ldap1 made getaddrinfo() return 127.0.0.8 twice. slapd binds to every address it gets back, so its second bind() failed with “Address already in use” and slapd exited. Aliases belong on the same line, as with intranet.ad.test above.

socket_wrapper: ports without ports

The KDC wants port 88, SMB 445, LDAP 389. On a normal desktop a user may not bind ports below 1024. socket_wrapper intercepts socket(), bind(), connect() and friends and turns every IPv4 connection into a unix domain socket in SOCKET_WRAPPER_DIR. Each file name encodes the protocol, the last octet of the address (the “interface”) and the port, in hex:

$ ls /tmp/adlab/sw
T020058     # TCP, 127.0.0.2, port 0x58  = 88    KDC
T020185     #                 port 0x185 = 389   LDAP
T0201BD     #                 port 0x1BD = 445   SMB
T0201D0     #                 port 0x1D0 = 464   kpasswd
T02027C     #                 port 0x27C = 636   LDAPS
T0346A1     # TCP, 127.0.0.3, port 18081         test web server
T0608AE     # TCP, 127.0.0.6, port 2222          sshd
...         (24 sockets in total)

$ ss -ltn | grep -c '127\.0\.0\.'
1

The one real listener on 127.0.0.x is Docker’s embedded DNS resolver, not the lab. The price: only processes with socket_wrapper preloaded can reach the lab. A browser or a tool from another shell sees nothing, which is why the client environment from samba_lab.sh env loads socket_wrapper as well, and gives the client its own interface number (9).

Three modes

socket_wrapper is only needed when low ports are forbidden. Many containers and CI runners set net.ipv4.ip_unprivileged_port_start=0, and then the lab can use real ports, with uid_wrapper only for the daemons that need root. The script detects which case applies:

Flowchart: running as root leads to root mode with the standard Samba setup; otherwise, if the user can bind 127.0.0.2 port 88, direct mode uses uid_wrapper and real ports; otherwise socket mode uses socket_wrapper with unix sockets only, without the NFS server and without DNS SRV lookups.
root, direct or socket: detected automatically, or forced with SAMBA_LAB_MODE.

Two services lose out in socket mode. nfs-ganesha opens IPv6 sockets even for an IPv4 bind address, and socket_wrapper keeps IPv4 and IPv6 strictly apart, so the NFS server is skipped. And DNS SRV lookups are switched off in the generated lab.yaml: socket_wrapper also captures the UDP packets to the real resolver, so every lookup would just run into its timeout.

Samba AD DC: four detours

Samba’s own selftest runs the DC under cwrap, so most of it just works. Four things did not:

  • sysvol ACLs. samba-tool domain provision sets NT ACLs on sysvol, which needs real root privileges for the underlying xattrs. Only Group Policy needs them, so the script calls the provisioning code from Python with samba.provision.setsysvolacl replaced by a no-op. ACLs and DOS attributes go into TDB files instead of xattrs: vfs objects = dfs_samba4 acl_tdb xattr_tdb.
  • The winbindd socket. Samba’s DC and its client libraries find winbindd through a socket directory that is compiled in. libwbclient accepts another one in SELFTEST_WINBINDD_SOCKET_DIR, but only while nss_wrapper is active. That variable is Samba’s own selftest hook, not a documented interface.
  • Starting the daemons. samba starts smbd and winbindd itself, from its compiled-in paths. The script starts them separately and adds --option="server role check:inhibit=yes", otherwise smbd and winbindd refuse to run in a DC configuration.
  • The DC’s own SPNs. ldap/dc1.ad.test and friends are normally added by samba_spnupdate at runtime. Without them an LDAP GSSAPI bind fails with “Server not found in Kerberos database”. The script runs samba_spnupdate once after provisioning.

And one rule for the lab directory: keep its path short. Unix socket paths are limited to 108 bytes, and Samba, Dovecot and socket_wrapper all create sockets several levels deep below it. A lab in a deep CI workspace failed with errors that mention neither the path nor its length. /tmp/adlab is safe.

The member file server: a real net ads join

fs1 is a second Samba instance with its own smb.conf, its own winbindd and its own nss_wrapper host name. It joins the domain with net ads join at the first start, exactly as a real server would. Three settings were needed:

[global]
    security = ads
    realm = AD.TEST
    create krb5 conf = no
    sync machine password to keytab = /tmp/adlab/fs1.keytab:account_name:sync_spns:spn_prefixes=cifs:sync_kvno:machine_password
    idmap config AD : backend = rid
    idmap config AD : range = 10000-999999
    vfs objects = acl_tdb xattr_tdb
  • create krb5 conf = no: otherwise winbindd writes its own krb5.conf, which ignores the lab’s.
  • sync machine password to keytab (Samba 4.21 and later) keeps a keytab in sync with the machine password, so the keytab follows every password change. The value is a list, and smb.conf splits lists at commas as well as at spaces: spn_prefixes=cifs,host turns into two broken entries. Either use one prefix, or sync_spns, which takes the SPNs from the account.
  • AD users on the file server come from libnss_winbind. Without root it cannot be added to /etc/nsswitch.conf, but nss_wrapper can load an NSS module directly: NSS_WRAPPER_MODULE_SO_PATH=.../libnss_winbind.so.2 and NSS_WRAPPER_MODULE_FN_PREFIX=winbind. With that, alice resolves to uid 11103 on fs1.

Dovecot and Postfix

Mail servers are built to start as root and drop privileges to dedicated users. Dovecot can run entirely as a normal user if every user in its configuration is that user:

default_internal_user = labuser
default_login_user = labuser
base_dir = /tmp/adlab/mail/run
auth_mechanisms = gssapi
auth_krb5_keytab = /tmp/adlab/mail1.keytab
auth_gssapi_hostname = mail1.ad.test
passdb static {
  fields {
    nopassword = yes   # GSSAPI is the only mechanism
  }
}
service imap-login {
  chroot =
}
service anvil {
  chroot =
}
# Dovecot cleans the environment of its child processes
import_environment {
  KRB5_CONFIG = %{env:KRB5_CONFIG}
  LD_PRELOAD = %{env:LD_PRELOAD}
  NSS_WRAPPER_HOSTS = %{env:NSS_WRAPPER_HOSTS}
  SOCKET_WRAPPER_DIR = %{env:SOCKET_WRAPPER_DIR}
  ...
}

The login processes chroot by default, which needs root, hence the empty chroot. The import_environment block is the most important part: Dovecot starts its children with a clean environment, so without it the auth process loses both KRB5_CONFIG and the wrappers and looks for the KDC in the system’s krb5.conf.

Postfix is stricter: it wants a dedicated mail_owner that is not root, owning the queue, and a setgid_group that differs from the mail owner’s group. Postfix runs under uid_wrapper with its own nss_wrapper passwd and group:

# passwd: "postfix" is the lab user's real uid, so the queue files belong to it
root:x:0:0::/:/bin/false
postfix:x:1000:1000::/tmp/adlab/mail/postfix:/bin/false
# group: postdrop must differ from postfix's group; it is never used
postdrop:x:65533:

# main.cf
mail_owner = postfix
setgid_group = postdrop
inet_interfaces = loopback-only
inet_protocols = ipv4
maillog_file = /tmp/adlab/mail/postfix/maillog
maillog_file_prefixes = /tmp/adlab/mail/postfix
import_environment = MAIL_CONFIG ... LD_PRELOAD UID_WRAPPER UID_WRAPPER_ROOT NSS_WRAPPER_PASSWD NSS_WRAPPER_GROUP NSS_WRAPPER_HOSTS SOCKET_WRAPPER_DIR SOCKET_WRAPPER_DEFAULT_IFACE
smtpd_sasl_type = dovecot
smtpd_sasl_path = inet:127.0.0.7:12345

Three details: there is no syslog to write to, so Postfix logs to a file, and maillog_file_prefixes must allow that file’s location. import_environment is Postfix’s version of the same environment trap as in Dovecot. And the script creates the queue directories itself, so they belong to the lab user, which is postfix in the wrapped passwd file. Kerberos itself is handled by Dovecot: Postfix passes SASL to Dovecot’s auth service over TCP.

The rest: sshd, OpenLDAP, Apache, NFS, KDC proxy

  • sshd runs on port 2222 with its own host key, UsePAM no and StrictModes no; nss_wrapper makes gethostname() return ssh1.ad.test, so the default GSSAPIStrictAcceptorCheck yes works. Without root it can only log in its own user, so the lab’s krb5.conf maps the principal with auth_to_local = RULE:[1:$1@$0](alice@AD.TEST)s/.*/labuser/.
  • OpenLDAP needs nothing special apart from a keytab with ldap/ldap1.ad.test and its own schema and module directories.
  • Apache runs with -f, -DFOREGROUND and a minimal config that loads only the modules it needs; mod_auth_gssapi uses its own keytab via GssapiCredStore keytab:.
  • nfs-ganesha (direct and root mode only) serves an in-memory export with sec=krb5:krb5i:krb5p. Without root it needs allow_set_io_flusher_fail = true, otherwise it exits because prctl(PR_SET_IO_FLUSHER) fails.
  • The KDC proxy is python-kdcproxy behind a small TLS server with a certificate from a lab CA; the details are in the KKDCP post.

Two shell traps

Starting and stopping ten daemons from a script produced two problems that have nothing to do with Kerberos. First, daemons that stay in the foreground are started with exec setsid in a subshell, and stop kills the whole process group with kill -- -$pid. In an interactive shell with job control, setsid forks instead of exec’ing, the recorded PID is the wrong one, and old daemons survive; running the script with bash avoids that. Second, pkill -f with a pattern that also appears in the calling command line kills the calling shell itself. The script never uses it.

The result

The full test suite against the lab in socket mode, as a normal user without a single real listening port:

$ python3 -m krb5test -c /tmp/adlab/lab.yaml
...
== Summary ==
 [OK  ] Local environment
 [OK  ] Realm AD.TEST
 [OK  ] Client alice@AD.TEST
 [WARN] Keytab /tmp/adlab/client1.keytab (nfs-client) [client1]
         warning in GSS daemon - rpc.gssd (rpc.gssd not running)
 [OK  ] Keytab /tmp/adlab/web.keytab (generic) [web]
 [OK  ] Keytab /tmp/adlab/ssh1.keytab (sshd) [ssh1]
 [OK  ] Keytab /tmp/adlab/mail1.keytab (generic) [mail1]
 [OK  ] dc-smb (smb, dc1.ad.test:445)
 [OK  ] dc-ldap (ldap, dc1.ad.test:389)
 [OK  ] dc-ldaps (ldap, dc1.ad.test:636)
 [OK  ] dc-ldap-starttls (ldap, dc1.ad.test:389)
 [OK  ] ad-web (http, web.ad.test:18081)
 [OK  ] ad-ssh (ssh, ssh1.ad.test:2222)
 [OK  ] ad-openldap (ldap, ldap1.ad.test:389)
 [OK  ] fs1-smb (smb, fs1.ad.test:445)
 [OK  ] ad-apache (http, www1.ad.test:80)
 [OK  ] ad-imap (imap, mail1.ad.test:143)
 [OK  ] ad-smtp (smtp, mail1.ad.test:587)

The one WARN is correct: no rpc.gssd runs on this machine. The integration tests break the lab on purpose (password reset behind the keytab’s back, RC4-only accounts, removed SPNs, disabled users) and check that the suite names the cause.

The lab paid for itself quickly. Against real services it found bugs that unit tests with mocked tools had missed: Heimdal’s klist has no -V and was detected as MIT; module = lines in krb5.conf’s [plugins] section were dropped by the parser; Python 3.13’s stricter certificate checks rejected the lab CA; and an error hint claimed that strict acceptors such as sshd reject tickets for alias SPNs, which a test against the real sshd disproved. Several of the traps in the posts on keytabs and LDAP over TLS were first reproduced here.

Limits

  • It is a lab, not a model of production: no GPOs, no sysvol ACLs, no replication, one DC.
  • Kernel clients are out of reach: the NFS server works, but mount -t nfs needs root, so the suite tests the RPCSEC_GSS handshake with its own probe.
  • Machine password rotation from the member server (net ads changetrustpw) does not work in the lab yet.
  • cwrap is a test tool. The Samba variables used here are internal hooks and can change between releases.

The script is in the repository at tests/lab/samba_lab.sh. On Debian and Ubuntu, the wrappers are in the packages libuid-wrapper, libnss-wrapper and libsocket-wrapper.

Leave a Reply

Your email address will not be published. Required fields are marked *

Are you human? Please solve:Captcha