Subject says it all, and I’m in 500-mile email mystery here. Anybody want to
take a guess what it could be?
Synopsis: Queries to .ca zones randomly fail with a DNSSEC validation error but
it appears to be region dependent and zone dependent. Anycast verifying
resolvers seem most prone to trigger the failure mode. I can’t trigger it
running a local verifying resolver (unbound).
I tried raising this with CIRA on Friday morning, and have observed it since
Wednesday, but nothing back from CIRA yet. Being geo-dependent, I’m guessing
the resolver that’s shortest path to me might have an issue, but I can trigger
it on multiple services usually (Google + CloudFlare at least) so I can’t see
that being the issue.
I can trigger this from the Google DNS Web page as well, but not reliably. I
can trigger this on domains I am not authoritive for (random domains I found
while browsing) but I’ll use two authoritive domains here.
$ dig seattle.mediadrive.ca @8.8.8.8
; <<>> DiG 9.10.6 <<>> seattle.mediadrive.ca
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 56149
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
; OPT=15: 00 07 45 78 70 69 72 65 64 20 52 52 53 49 47 20 66 6f 75 6e 64 20 66
6f 72 20 70 75 66 35 32 6b 70 68 36 75 30 71 35 67 68 73 69 6c 72 33 68 63 31
64 37 65 6c 62 61 68 67 33 2e 63 61 2f 6e 73 65 63 33 20 28 6b 65 79 74 61 67
3d 35 36 38 31 36 29 ("..Expired RRSIG found for
puf52kph6u0q5ghsilr3hc1d7elbahg3.ca/nsec3 (keytag=56816)")
;; QUESTION SECTION:
;seattle.mediadrive.ca. IN A
;; Query time: 67 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Sat Jul 27 16:26:37 EDT 2024
;; MSG SIZE rcvd: 136
$ dig adamdaniels.ca @1.1.1.1
; <<>> DiG 9.10.6 <<>> adamdaniels.ca @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 4638
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; OPT=15: 00 0a 66 61 69 6c 65 64 20 74 6f 20 76 65 72 69 66 79 20 73 69 67 6e
61 74 75 72 65 73 20 66 6f 72 20 61 64 61 6d 64 61 6e 69 65 6c 73 2e 63 61 2e
20 6f 70 74 2d 6f 75 74 20 70 72 6f 6f 66 ("..failed to verify signatures for
adamdaniels.ca. opt-out proof")
;; QUESTION SECTION:
;adamdaniels.ca. IN A
;; Query time: 50 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Sat Jul 27 17:27:28 EDT 2024
;; MSG SIZE rcvd: 110
If I let it run long enough, I can trigger it on canada.ca but not with any
frequency.
Performing the queries from my home is incredibly reliable for reproducing
this, but I can trigger it from a facility I colocate with in Toronto as well.
== MTR from Toronto, Canada
Start: 2024-07-27T17:30:07-0400
HOST: manager Loss% Snt Last Avg Best
Wrst StDev
1.|-- _gateway 0.0% 10 1.2 1.0 0.8
1.2 0.1
2.|-- 198.55.53.14 90.0% 10 0.5 0.5 0.5
0.5 0.0
3.|-- i.cr003.ca1-01.yyz.as1100.net 0.0% 10 0.3 0.4 0.2
0.8 0.2
4.|-- i.rogers.ca1-01.yyz.as1100.net 0.0% 10 0.4 0.5 0.3
0.7 0.2
5.|-- 99.209.203.17 0.0% 10 0.6 0.6 0.5
0.9 0.1
6.|-- 24.153.31.130 0.0% 10 1.4 1.6 1.3
1.9 0.2
7.|-- 3021-cgw01.mtnk.asr9k.rmgt.net.rogers.com 0.0% 10 1.5 1.8 1.5
2.1 0.2
8.|-- 209.148.235.222 0.0% 10 2.9 4.7 2.6
14.6 3.9
9.|-- ??? 100.0 10 0.0 0.0 0.0
0.0 0.0
10.|-- 192.178.99.39 0.0% 10 2.4 2.2 2.0
2.4 0.1
11.|-- 216.239.50.119 0.0% 10 3.2 3.1 2.9
3.3 0.1
12.|-- dns.google 0.0% 10 2.0 2.0 1.9
2.2 0.1
== MTR from my home (Niagara region, Canada)
Start: 2024-07-27T17:30:21-0400
HOST: Adams-MacBook-Air.local Loss% Snt Last Avg Best Wrst StDev
1.|-- 192.168.1.1 0.0% 10 2.7 3.1 2.7 3.4 0.2
2.|-- 10.202.100.1 0.0% 10 74.9 20.2 9.0 74.9 21.7
3.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
4.|-- c8.tpia.start.ca 0.0% 10 43.5 27.1 14.8 58.8 16.8
5.|-- 72.14.198.214 0.0% 10 41.7 22.9 16.4 41.7 7.6
6.|-- 192.178.99.31 0.0% 10 37.8 20.0 14.2 37.8 7.6
7.|-- 216.239.41.175 0.0% 10 41.2 19.3 15.3 41.2 7.8
8.|-- dns.google 0.0% 10 19.2 17.8 14.7 21.5 2.2
I’ve tried the same queries from NYC and Seattle but do not trigger any
failures.
Thoughts?
Adam
