Three words: tcpdump and wireshark It is like peanut and jelly .. hall and oates .. salt and pepper .. ebb and flow .. pen and paper .. I could go on but …
Know them. Love them. They are your newest best friends. <grin> Using tcpdump IMHO should be the first tool anyone uses when troubleshooting seemly unexplainable DNS weirdness. Knowing what is being put on the wire (or lack thereof) is critical since it provides key factual data points that decisions can be made on. When running tcpdump on the DNS server I personally prefer this command: tcpdump -n -i <interface eg eth0> -s 65535 -w <filename.pcap> dash n is telling tcpdump that you do not want it to resolve hostnames. This is an important option when doing DNS troubleshooting because you do not want extra resolutions taking place. dash s is saying gimme the full packet. dash w is the name of the file you want the output saved in. After starting the command, run several queries from a host and ctrl+c to exit. Once you get your file into wireshark now you can start slicing n dicing on the data! Here is handy wireshark filter: dns.qry.name == internet-dns1.state.ma.us By using a filter of dns.flags.rcode == (number here) you can drive off into the weeds and get super granular with sorting the data. For example “dns.flags.rcode == 2” will show you all of the server failures for queries. It is hard to provide further guidance on what to do since what you find in the pcap is only a starting point. Good hunting! As an aside I would like to mention that you do not need to travel home to get situational awareness when the diggui.com website can be used instead. Also. For the people running .us tld .. SHA1 for DNSSEC .. really? https://dnsviz.net/d/state.ma.us/dnssec/ John From: bind-users [mailto:bind-users-boun...@lists.isc.org] On Behalf Of sami's strat Sent: Wednesday, February 10, 2021 11:54 AM To: Mark Andrews Cc: bind-users Subject: Re: Bind 9.11 serving up false answers for a single domain. Thank you all for responding. One final query about this. I'm seeing this issue on my production servers at work. Yet, when I run the same queries at home, I don't see those failed queries. I actually flushed DNS cache, cleared Linux O/S cache, and even bounced my personal DNS server trying to reproduce the issue. But I could not. TIA On Wed, Feb 10, 2021 at 12:09 AM Mark Andrews <ma...@isc.org<mailto:ma...@isc.org>> wrote: Run ‘dig +trace +all internet-dns1.state.ma.us<http://internet-dns1.state.ma.us>’ which will show you the glue records then try ‘dig +dnssec +norec internet-dns1.state.ma.us<http://internet-dns1.state.ma.us> @<address>’ for all the addresses in the glue records. e.g. dig +dnssec +norec internet-dns1.state.ma.us<http://internet-dns1.state.ma.us> @146.243.122.17<http://146.243.122.17> Mark > On 10 Feb 2021, at 14:50, sami's strat > <sami.st...@gmail.com<mailto:sami.st...@gmail.com>> wrote: > > Thanks Mark. > > However, the traceroute to the hostnamed failed for the same reason. Please > note: > > [root@myhost data]# dig > internet-dns1.state.ma.us<http://internet-dns1.state.ma.us> > > ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.el7 <<>> > internet-dns1.state.ma.us<http://internet-dns1.state.ma.us> > ;; global options: +cmd > ;; Got answer: > ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 61641 > ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 > > ;; OPT PSEUDOSECTION: > ; EDNS: version: 0, flags:; udp: 4096 > ;; QUESTION SECTION: > ;internet-dns1.state.ma.us<http://internet-dns1.state.ma.us>. IN A > > ;; Query time: 1263 msec > ;; SERVER: 192.168.33.12#53(192.168.33.12) > ;; WHEN: Tue Feb 09 22:34:15 EST 2021 > ;; MSG SIZE rcvd: 54 > > [root@myhost data]# dig > internet-dns1.state.ma.us<http://internet-dns1.state.ma.us> +trace > > ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.el7 <<>> > internet-dns1.state.ma.us<http://internet-dns1.state.ma.us> +trace > ;; global options: +cmd > . 516485 IN NS > c.root-servers.net<http://c.root-servers.net>. > . 516485 IN NS > e.root-servers.net<http://e.root-servers.net>. > . 516485 IN NS > f.root-servers.net<http://f.root-servers.net>. > . 516485 IN NS > l.root-servers.net<http://l.root-servers.net>. > . 516485 IN NS > m.root-servers.net<http://m.root-servers.net>. > . 516485 IN NS > d.root-servers.net<http://d.root-servers.net>. > . 516485 IN NS > g.root-servers.net<http://g.root-servers.net>. > . 516485 IN NS > k.root-servers.net<http://k.root-servers.net>. > . 516485 IN NS > b.root-servers.net<http://b.root-servers.net>. > . 516485 IN NS > h.root-servers.net<http://h.root-servers.net>. > . 516485 IN NS > a.root-servers.net<http://a.root-servers.net>. > . 516485 IN NS > i.root-servers.net<http://i.root-servers.net>. > . 516485 IN NS > j.root-servers.net<http://j.root-servers.net>. > . 516485 IN RRSIG NS 8 0 518400 20210222230000 > 20210209220000 42351 . > QCzDH8eHlHVbx4SxIIwk8xnk6ky/q+zRh8KAUfI98lqHcIP4NLxzCe6f > mC2sNX1VcthEy6Lwnobm8OyJCRpNEHedYrS01aMhAVzUfM+/PJ9MWn0w > SkmXxyZMJZXF/kl4GDNX0x/GW3+DkeTeZI9+B540Yvj47qJv2bD9nIQG > NtE7bDze7bgMJkIuBlEzPfwp7YW5ud8qdC6HdUoEMqygwZcWAiQu8gpb > q21z8W5hcdci1OouDFytNWrXAvfSsuR635+GzSj+RZjYo+447uP7lKsK > N5aeVQ/BPh5jM32xVO+zwyp7v9Nky1vSP/BchMQ/3cqg3Ee7zobl8OQd CSd/SA== > ;; Received 1097 bytes from 192.168.33.12#53(192.168.33.12) in 0 ms > > us. 172800 IN NS a.cctld.us<http://a.cctld.us>. > us. 172800 IN NS b.cctld.us<http://b.cctld.us>. > us. 172800 IN NS c.cctld.us<http://c.cctld.us>. > us. 172800 IN NS e.cctld.us<http://e.cctld.us>. > us. 172800 IN NS f.cctld.us<http://f.cctld.us>. > us. 172800 IN NS k.cctld.us<http://k.cctld.us>. > us. 86400 IN DS 21364 8 1 > 260D0461242BCF8F05473A08B05ED01E6FA59B9C > us. 86400 IN DS 21364 8 2 > B499CFA7B54D25FDE1E6FE93076FB013DAA664DA1F26585324740A1E 6EBDAB26 > us. 86400 IN RRSIG DS 8 1 86400 20210222230000 > 20210209220000 42351 . > rujvGB0s2bsqzBuzRliH6QK9vH84ETZV7gZMEhJyzMFofWhj9ZZaNWE/ > VvdA9rC16IOEocvARv2rOqk7G3KTzdkHHZcwcZSQyVqsOIaIywGFuEgd > viSXF6+M5MocUgEMp5dtt6SBLHG+lE/FV/3HylKSHsxdO/F6PeWKgcBZ > D4lZQ6w5asmlbdKJKMhlWPp6UaxBE7ACaxndBQixoNqXQuPrXpXi1Fnj > ntFtTfn57hMyrdTojIJ8X7/HKjCrbm3CL/WJ+VZR051OGCdZVjpUaDXR > x7G9lDhu3K5clar9PGYyUJM7+RBKzrQJep7HrjL2nZdoTyfY4i33S+EZ sTlTOA== > ;; Received 707 bytes from > 199.7.91.13#53(d.root-servers.net<http://d.root-servers.net>) in 4 ms > > state.ma.us<http://state.ma.us>. 7200 IN NS > internet-dns3.state.ma.us<http://internet-dns3.state.ma.us>. > state.ma.us<http://state.ma.us>. 7200 IN NS > internet-dns1.state.ma.us<http://internet-dns1.state.ma.us>. > state.ma.us<http://state.ma.us>. 7200 IN NS > internet-dns2.state.ma.us<http://internet-dns2.state.ma.us>. > state.ma.us<http://state.ma.us>. 3600 IN DS 47628 7 2 > 5379F9F747214E5A63416775396BCFF98FA4867AE66E09BCBEBE0DCC 1682C369 > state.ma.us<http://state.ma.us>. 3600 IN DS 41388 7 1 > 36D899932AF794EADD671161515E48FE829BB7FE > state.ma.us<http://state.ma.us>. 3600 IN DS 41388 7 2 > BBAB433D3853571F42516E70659AF1F85FA4FBA0FDFCEAD4D092592A 00C78769 > state.ma.us<http://state.ma.us>. 3600 IN DS 47628 7 1 > 485E0EE2F7C08FCE51D1E284321242930274833A > state.ma.us<http://state.ma.us>. 3600 IN RRSIG DS 8 3 > 3600 20210307200856 20210205191212 53985 us. > O8KqBHzlZsDqrZi0NQO4JEiN0b8j04/Lb8W2uVz5PyrAat1VgZKQ3Ws6 > 6PNtbZDMv6YX6QA8fWFLxNmeJ1/4L3wLu8EKYXaThA9Zxll7mKFj1iPf > nqiVq5hOo8Ul3inmfM/tjCQ21IHc/v0JZygZNd/h0SxXWlQXi+W3G9LN > +4z/qxtl9dGD1ka54Ln3MAVxB1Tp4pt0ri4qPLmfGKf/HA== > couldn't get address for > 'internet-dns3.state.ma.us<http://internet-dns3.state.ma.us>': not found > couldn't get address for > 'internet-dns1.state.ma.us<http://internet-dns1.state.ma.us>': not found > couldn't get address for > 'internet-dns2.state.ma.us<http://internet-dns2.state.ma.us>': not found > dig: couldn't get address for > 'internet-dns3.state.ma.us<http://internet-dns3.state.ma.us>': no more > [root@myhost data]# > > On Tue, Feb 9, 2021 at 10:10 PM Mark Andrews > <ma...@isc.org<mailto:ma...@isc.org>> wrote: > Well you could try tracing the addresses of the nameservers for which > there where errors reported. It could be as simple as a routing issue > between you and these servers. > > > On 10 Feb 2021, at 13:25, sami's strat > > <sami.st...@gmail.com<mailto:sami.st...@gmail.com>> wrote: > > > > couldn't get address for > > 'internet-dns1.state.ma.us<http://internet-dns1.state.ma.us>': not found > > couldn't get address for > > 'internet-dns3.state.ma.us<http://internet-dns3.state.ma.us>': not found > > couldn't get address for > > 'internet-dns2.state.ma.us<http://internet-dns2.state.ma.us>': not found > > dig: couldn't get address for > > 'internet-dns1.state.ma.us<http://internet-dns1.state.ma.us>': no more > > Yet, I do this on my personal computer at home, and it works without an issue. > > Any other thoughts? TIA -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org<mailto:ma...@isc.org>
_______________________________________________ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
