Diagnosing DNS resolver problems with systemd

As usual the first step is to find the right tools to start the debugging. For systemd systems (like my Ubuntu LTS 24.04) on helper is resolvectl:

linux # resolvectl status
Global
           Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
    resolv.conf mode: stub
  Current DNS Server: 127.0.0.1
         DNS Servers: 127.0.0.1 192.168.1.34
Fallback DNS Servers: 208.67.222.222 208.67.220.220 2620:119:35::35 2620:119:53::53

Link 2 (ens3)
    Current Scopes: DNS
         Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 46.38.225.230
       DNS Servers: 46.38.225.230 46.38.252.230

Hint: older systems may use systemd-resolve --status instead.

Some words about my environment:

• 127.0.0.1: local unbound DNS server
• 192.168.1.34: internal primary DNS server running bind

Some of the settings listed above can be modified here:

linux # cat /etc/systemd/resolved.conf 
[Resolve]
DNS=127.0.0.1 192.168.1.34
FallbackDNS=208.67.222.222 208.67.220.220 2620:119:35::35 2620:119:53::53
CacheFromLocalhost=yes

Quite naively I was expecting local DNS queries to use the default DNS servers. And both of them were configured to respond with then internal DNS view.

So when querying those DNS servers (127.0.0.1 / 192.168.1.34) directly I always got the correct (internal) IP address.

To my big surprise that was not the case when using the systemd resolver in between (listening on IP 127.0.0.53 and used by default on Ubuntu systems):

linux # dig +short mail.mydomain.de
192.168.1.10
<... repeat often enough ...>
linux # dig +short mail.mydomain.de
12.34.56.78

So sometimes I get the internal IP, sometimes the public one of the local machine.

Explanation / solution

Using tcpdump I found, that (for some reason) some of the queries were issued using the DNS server set for the public network interface (ens3).

After applying the same DNS servers to that interface things started to work as expected.