Does it only happen for IPv6 DNS requests? Maybe it is related to this: https://open.nlnetlabs.nl/pipermail/nsd-users/2014-January/001783.html
klaus On 05.03.2014 14:16, Kostas Zorbadelos wrote:
Greetings to all, we operate an anycast caching resolving farm for our customer base, based on CentOS (6.4 or 6.5), BIND (9.9.2, 9.9.5 or the stock CentOS package BIND 9.8.2rc1-RedHat-9.8.2-0.23.rc1.el6_5.1) and quagga (the stock CentOS package). The problem is that we have noticed sporadic but noticable SERVFAILs in 3 out of 10 total machines. Cacti measurements obtained via the BIND XML interface show traffic from 1.5K queries/sec (lowest loaded machines) to 15K queries/sec (highest). The problem is that in 3 specific machines in a geolocation with a BIND restart we notice after a period of time that can range between half an hour and several hours SERVFAILs in resolutions. The 3 machines do not have the highest load in the farm (6-8K q/sec). The resolution problems are noticable in the customers ending up in these machines but do not show up as high numbers in the BIND XML Resolver statistics (ServFail number). We reproduce the problem, by querying for a specific domain name using a loop of the form while [ 1 ]; do clear; rndc flushname www.linux-tutorial.info; sleep 1; dig www.linux-tutorial.info @localhost; sleep 2; done | grep SERVFAIL The www.linux-tutorial.info is not the only domain experiencing resolution problems of course. The above loop can run for hours even without issues on low-traffic hours (night, after a clean BIND restart) but during the day it shows quite a few SERVFAILs, which affect other domains as well. During the problem we notice with tcpdump, that when SERVFAIL is produced, no query packet exits the server for resolution. We have noticed nothing in BIND logs (we even tried to raise debugging levels and log all relevant categories). An example capture running the above loop: # tcpdump -nnn -i any -p dst port 53 or src port 53 | grep 'linux-tutorial' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes 14:33:03.590908 IP6 ::1.53059 > ::1.53: 15773+ A? www.linux-tutorial.info. (41) 14:33:03.591292 IP 83.235.72.238.45157 > 213.133.105.6.53: 19156% [1au] A? www.linux-tutorial.info. (52) ^^^^ Success 14:33:06.664411 IP6 ::1.45090 > ::1.53: 48526+ A? www.linux-tutorial.info. (41) 14:33:06.664719 IP6 2a02:587:50da:b::1.23404 > 2a00:1158:4::add:a3.53: 30244% [1au] A? www.linux-tutorial.info. (52) ^^^^ Success 14:33:31.434209 IP6 ::1.43397 > ::1.53: 26607+ A? www.linux-tutorial.info. (41) ^^^^ SERVFAIL 14:33:43.672405 IP6 ::1.58282 > ::1.53: 27125+ A? www.linux-tutorial.info. (41) ^^^^ SERVFAIL 14:33:49.706645 IP6 ::1.54936 > ::1.53: 40435+ A? www.linux-tutorial.info. (41) 14:33:49.706976 IP6 2a02:587:50da:b::1.48961 > 2a00:1158:4::add:a3.53: 4287% [1au] A? www.linux-tutorial.info. (52) ^^^^ Success The main actions we have done on the problem machines are - change the BIND version (we initially used a custom compiled 9.9.2, we moved to 9.9.5 and finally switched over to the CentOS stock package 9.8.2rc1). We noticed the problem in all versions - disable IPtables (we use a ruleset with connection tracking in all of our machines with no problems on the other machines in the farm). Again no solution - introduce query-source-v6 address in named.conf (we already had query-source). Each machine has a single physical interface and 3 loopbacks with the anycast IPs, announced via Quagga ospfd to the rest of the network. No solution. The main difference in the 3 machines from the rest is the IPv6 operation. Those machines are dual stack, having /30 (v4) and /127 (v6) on the physical interface. Needless to say that the next trial is to remove the relevant IPv6 configuration. I understand that there are many parameters to the problem, we try and debug the issue several days now. Any suggestion, suspicion or hint is highly welcome. I can provide all sorts of traces from the machines (I already have pcap files at the moment of the problem, plus pstack, rndc status, OS process limits, rndc recursing, rndc dumpdb -all, according to https://kb.isc.org/article/AA-00341/0/What-to-do-with-a-misbehaving-BIND-server.html) Thanks in advance, Kostas
_______________________________________________ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
