I have tested version 2.86-1.1ubuntu0.2 and that seems to have fixed it. The bug only appears when dnsmasq is forwarding responses in --strict- order mode and there's a duplicate request packet received before dnsmasq has replied to the original request packet. This is a procedure I came up with to reproduce and test that specific circumstance:
I am using a network namespace to simplify networking, avoid port conflicts, and maintain tcpdump's protocol decoding. This test process requires running and interacting with multiple processes at once, and so it's easiest to just open multiple terminal windows. I create the net namespace with the first terminal, and then join the created namespace with additional terminals. To create the NS in the first terminal: $ sudo unshare --net # echo $$ > 12123 # ip addr add 127.0.0.1/8 dev lo # ip link set up dev lo # The `echo $$` gives the shell PID (12123 in this example), which is also the namespace PID. This can optionally be verified with lsns: # lsns -t net > NS TYPE NPROCS PID USER NETNSID NSFS COMMAND > 1234567890 net 2 12123 root unassigned -bash To join the NS from subsequent terminals: $ sudo nsenter -n -t 12123 # Test process: The first process to run is a fake upstream server for dnsmasq to forward responses from. We will run this on an alternate port (5353) and have it respond to requests for a test domain. # dnsmasq -d -p 5353 --no-resolv -A /example.com/192.0.2.1 The second process is the main dnsmasq process we want to actually test. It uses --strict-order and uses our fake upstream to answer all requests. We will also use systemd-run to limit dnsmasq's cpu resources to slow it down enough to more easily queue up duplicate requests. # systemd-run --scope -p CPUQuota=1% -- dnsmasq -d --strict-order --no- resolv -S 127.0.0.1#5353 The third process is just tcpdump to monitor requests to and responses from dnsmasq. I have piped the output through grep to make it easier to analyze the results for refused packets, but you can also leave that off and search the output manually. # tcpdump -ln udp port 53 | grep --line-buffered -iC 5 refused The final process is using `scapy` to generate dns packets with a duplicated request id. Scapy can be installed from the default repositories as `python3-scapy`. Scapy has to be switched to raw sockets mode to be able to work over the loopback interface before we can start sending dns packets. # scapy >>> conf.L3socket = L3RawSocket Now we can send some dns packets with scapy. All the packets should have the same request id in them, so we set it to a static number (we picked 47 in this example). We have also set it to send 50 packets, which is usually enough to get a few packets queued before dnsmasq can process the previous ones. >>> send(IP(dst='127.0.0.1')/UDP()/DNS(id=47,qd=DNSQR(qname='example.com')),count=50) You can also choose to send packets continuously until cancelled with Ctrl-C. >>> send(IP(dst='127.0.0.1')/UDP()/DNS(id=47,qd=DNSQR(qname='example.com')),loop=True) We can now check tcpdump for refused packets, which should show up anytime after there are two or more requests in a row without a response in between them. If you don't see any send another batch or two of packets. Here's some example output: 22:10:39.518439 IP 127.0.0.1.53 > 127.0.0.1.53: 47+ A? example.com. (29) 22:10:39.518896 IP 127.0.0.1.53 > 127.0.0.1.53: 47+ A? example.com. (29) 22:10:39.519366 IP 127.0.0.1.53 > 127.0.0.1.53: 47+ A? example.com. (29) 22:10:39.519824 IP 127.0.0.1.53 > 127.0.0.1.53: 47+ A? example.com. (29) 22:10:39.520290 IP 127.0.0.1.53 > 127.0.0.1.53: 47+ A? example.com. (29) 22:10:39.524495 IP 127.0.0.1.53 > 127.0.0.1.53: 47 Refused 0/0/0 (29) 22:10:39.524504 IP 127.0.0.1.53 > 127.0.0.1.53: 47 Refused 0/0/0 (29) 22:10:39.524512 IP 127.0.0.1.53 > 127.0.0.1.53: 47 Refused 0/0/0 (29) 22:10:39.524534 IP 127.0.0.1.53 > 127.0.0.1.53: 47 Refused 0/0/0 (29) 22:10:39.524547 IP 127.0.0.1.53 > 127.0.0.1.53: 47* 1/0/0 A 192.0.2.1 (45) 22:10:39.524575 IP 127.0.0.1.53 > 127.0.0.1.53: 47 Refused 0/0/0 (29) 22:10:39.524583 IP 127.0.0.1.53 > 127.0.0.1.53: 47 Refused 0/0/0 (29) 22:10:39.524590 IP 127.0.0.1.53 > 127.0.0.1.53: 47* 1/0/0 A 192.0.2.1 (45) 22:10:39.524612 IP 127.0.0.1.53 > 127.0.0.1.53: 47 Refused 0/0/0 (29) 22:10:39.524620 IP 127.0.0.1.53 > 127.0.0.1.53: 47 Refused 0/0/0 (29) 22:10:39.524628 IP 127.0.0.1.53 > 127.0.0.1.53: 47* 1/0/0 A 192.0.2.1 (45) 22:10:39.524648 IP 127.0.0.1.53 > 127.0.0.1.53: 47 Refused 0/0/0 (29) 22:10:39.524657 IP 127.0.0.1.53 > 127.0.0.1.53: 47 Refused 0/0/0 (29) 22:10:39.524664 IP 127.0.0.1.53 > 127.0.0.1.53: 47* 1/0/0 A 192.0.2.1 (45) 22:10:40.537105 IP 127.0.0.1.53 > 127.0.0.1.53: 47+ A? example.com. (29) 22:10:40.537236 IP 127.0.0.1.53 > 127.0.0.1.53: 47* 1/0/0 A 192.0.2.1 (45) 22:10:40.537591 IP 127.0.0.1.53 > 127.0.0.1.53: 47+ A? example.com. (29) 22:10:40.537636 IP 127.0.0.1.53 > 127.0.0.1.53: 47* 1/0/0 A 192.0.2.1 (45) If you then install the patched package and restart the dnsmasq process doing the forwarding you should no longer be seeing any refused packets. ** Tags removed: verification-needed-jammy ** Tags added: verification-done-jammy -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to dnsmasq in Ubuntu. https://bugs.launchpad.net/bugs/1981794 Title: Duplicate/retried DNS queries fail with REFUSED (Fixed in upstream) Status in dnsmasq package in Ubuntu: Fix Released Status in dnsmasq source package in Jammy: Fix Committed Status in dnsmasq source package in Kinetic: Fix Released Bug description: [Impact] When a DNS query fails to complete and the system retries it, subsequent copies of the query will be refused by dnsmasq. The client will automatically receive the REFUSED return value without a retry attempt. Adding this fix will stop dnsmasq from unnecessarily breaking connections, especially for situations where an internet connection is flaky. This bug is fixed by patching in an upstream commit - https://thekelleys.org.uk/gitweb/?p=dnsmasq.git;a=commit;h=2561f9fe0eb9c0be - which allows retried DNS requests through rather than refusing them. [Test Plan] The fix can be tested using 2 lxd containers, 1 for running the fix, and 1 for acting as a dns server. Start by setting up the dns server container: # lxc launch images:ubuntu/jammy dns-resolver # lxc exec dns-resolver bash # apt update && apt dist-upgrade -y # systemctl disable systemd-resolved # systemctl stop systemd-resolved # unlink /etc/resolv.conf # echo nameserver 8.8.8.8 | tee /etc/resolv.conf # apt install net-tools dnsmasq -y # systemctl enable dnsmasq Get the container's ip on lxd's network, in this case ifconfig is used, showing 10.62.42.157: # ifconfig eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.62.42.157 netmask 255.255.255.0 broadcast 10.62.42.255 ... Now set up the test container: # lxc launch images:ubuntu/jammy test-dnsmasq # lxc exec test-dnsmasq bash # apt update && apt dist-upgrade -y # systemctl disable systemd-resolved # systemctl stop systemd-resolved # unlink /etc/resolv.conf Use other container's ip here, along with an ip that does not resolve # echo "nameserver 10.62.42.157 nameserver 192.0.2.1" | tee /etc/resolv.conf # apt install dnsmasq -y # systemctl enable dnsmasq On the dns server side, set the nameserver to 127.0.0.1 to cause denials on the test server: # echo nameserver 127.0.0.1 | tee /etc/resolv.conf; systemctl restart dnsmasq Now ping a known domain on the test container, and while it runs set the dns server side nameserver back to 8.8.8.8: # ping ubuntu.com > swap containers # echo nameserver 8.8.8.8 | tee /etc/resolv.conf; systemctl restart dnsmasq Ping will continue to not pick up the domain and fails with: ping: ubuntu.com: Temporary failure in name resolution With the fix, ping should now pick up the new successful responses: PING ubuntu.com (185.125.190.29) 56(84) bytes of data. 64 bytes from website-content-cache-3.ps5.canonical.com (185.125.190.29): icmp_seq=1 ttl=48 time=165 ms 64 bytes from website-content-cache-3.ps5.canonical.com (185.125.190.29): icmp_seq=2 ttl=48 time=162 ms 64 bytes from website-content-cache-3.ps5.canonical.com (185.125.190.29): icmp_seq=3 ttl=48 time=166 ms 64 bytes from website-content-cache-3.ps5.canonical.com (185.125.190.29): icmp_seq=4 ttl=48 time=164 ms 64 bytes from website-content-cache-3.ps5.canonical.com (185.125.190.29): icmp_seq=5 ttl=48 time=163 ms 64 bytes from website-content-cache-3.ps5.canonical.com (185.125.190.29): icmp_seq=6 ttl=48 time=163 ms ^C --- ubuntu.com ping statistics --- 6 packets transmitted, 6 received, 0% packet loss, time 5005ms Note: ping is used here instead of another dns tester such as dig because it continues its dns resolution attempts with retry packets even after receiving a REFUSED error. Other programs fail immediately on REFUSED and are unable to send duplicate packets to reproduce the issue. [Where problems could occur] This change was added upstream in version 2.87, which means it has not been tested in many situations alongside 2.86. Allowing the retries could lead to a flood of requests to remote DNS servers if the replies are unable to make it back through dnsmasq to the user. [Other Info] This bug was fixed in Kinetic in version 2.86-1.1ubuntu2. [Original Description] Duplicate or retried DNS queries will return REFUSED for one of the queries causing intermittent failures in clients. This probably breaks lots of things, but for me is causing 22.04's internet connection sharing to be unstable. It's particularly bad for my Xbox which seems to like sending duplicate queries. Here's an example capture: 22:37:25.308212 IP 10.42.0.16.54248 > 10.42.0.1.53: 22442+ A? title.auth.xboxlive.com. (41) 22:37:25.332711 IP 10.42.0.16.54248 > 10.42.0.1.53: 22442+ A? title.auth.xboxlive.com. (41) 22:37:25.332740 IP 10.42.0.1.53 > 10.42.0.16.54248: 22442 Refused 0/0/0 (41) 22:37:25.353003 IP 10.42.0.1.53 > 10.42.0.16.54248: 22442 2/0/0 CNAME title.auth.xboxlive.com.akadns.net., A 40.64.90.82 (105) This has been fixed in upstream as of Sept 2021 in the unreleased 2.87 version. It's apparently a regression in version 2.86 (also released in Sept 2021). Ubuntu 22.04 and later all use the broken 2.86 version. Upstream fix: https://thekelleys.org.uk/gitweb/?p=dnsmasq.git;a=commit;h=2561f9fe0eb9c0be1df48da1e2bd3d3feaa138c2 Upstream bug thread: https://www.mail-archive.com/search?l=dnsmasq-discuss%40lists.thekelleys.org.uk&q=subject:%22%5C%5BDnsmasq%5C-discuss%5C%5D+REFUSED+after+dropped+packets%22&o=oldest&f=1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/1981794/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
