Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

On 18. 07. 23 23:53, Viktor Dukhovni wrote: Currently, it’s 7 days for .com which almost exactly matches the RRSIG expiry-inception difference and that doesn’t give any wiggle room if things go wrong. Expiry in the SOA applies to AXFR, but may deployments are not AXFR-based. And Verisign appare

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

> On 20 Jul 2023, at 01:23, Shumon Huque wrote: > > On Wed, Jul 19, 2023 at 3:28 AM Shane Kerr wrote: > Shumon and all, > > On 18/07/2023 21.41, Shumon Huque wrote: > > On Tue, Jul 18, 2023 at 3:29 PM Viktor Dukhovni > > wrote: > > > > Yes, I agree. A resolve

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

On Wed, Jul 19, 2023 at 3:28 AM Shane Kerr wrote: > Shumon and all, > > On 18/07/2023 21.41, Shumon Huque wrote: > > On Tue, Jul 18, 2023 at 3:29 PM Viktor Dukhovni > > wrote: > > > > Yes, I agree. A resolver can't really tell that a response with an > > expired si

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

Shumon and all, On 18/07/2023 21.41, Shumon Huque wrote: On Tue, Jul 18, 2023 at 3:29 PM Viktor Dukhovni > wrote: Yes, I agree. A resolver can't really tell that a response with an expired signature wasn't an attacker trying to replay old data. For robustness ag

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

--- Begin Message --- On 18/07/2023 23.53, Viktor Dukhovni wrote: On Tue, Jul 18, 2023 at 10:25:01PM +0200, Ondřej Surý wrote: It’s exactly like the serve-stale. The inception of the protocol change is driven by this isolated incident. That’s not a proper design, that’s slapping more bandaids on

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

--- Begin Message --- Ondřej Surý wrote on 2023-07-18 13:25: ... There’s already mechanism for not serving a stale RRSIGs. The EXPIRE field in the SOA record should be set to a value that’s lower than the RRSIG resigning interval (the minimal interval between now and shortest RRSIG expiry in

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

Thanks Mark for the clarification. I just hate adding new knobs and exceptions in a scramble mode. If the knob is already there then it’s already there. There’s already too many knobs in DNS and we all know that. -- Ondřej Surý (He/Him) > On 19. 7. 2023, at 0:43, Mark Andrews wrote: > > Ex

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

On Tue, Jul 18, 2023, 3:47 PM Mark Andrews wrote: > > > If you have stale DS’s then you will get validation failures if the child > zone had already remove the DNSKEYs those DS refer to. > The second level domain in question didn't have a DS at all. The problem, as far as I could tell, was tha

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

> On 19 Jul 2023, at 05:51, Gavin McCullagh wrote: > > > > On Tue, Jul 18, 2023 at 12:45 PM Shumon Huque wrote: > Yes, I agree. A resolver can't really tell that a response with an expired > signature wasn't an attacker trying to replay old data. For robustness > against attacks, it must r

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

Except BIND does exactly this. It retries and if all the servers for the zone fail the is flagged as bad for 10 minutes and any validation that depends on that lookup fails with DNS_R_BROKENCHAIN which results in SERVFAIL rather than a retry. This was how we dealt with the so called “rollover

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

On Tue, Jul 18, 2023 at 10:25:01PM +0200, Ondřej Surý wrote: > It’s exactly like the serve-stale. The inception of the protocol > change is driven by this isolated incident. That’s not a proper > design, that’s slapping more bandaids on the camel. I don't even see a "protocol change" here. A bog

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

It’s exactly like the serve-stale. The inception of the protocol change is driven by this isolated incident. That’s not a proper design, that’s slapping more bandaids on the camel. There’s already mechanism for not serving a stale RRSIGs. The EXPIRE field in the SOA record should be set to a va

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

--- Begin Message --- On Tue, Jul 18, 2023 at 7:53 PM Gavin McCullagh wrote: > On Tue, Jul 18, 2023 at 12:45 PM Shumon Huque wrote: >> Yes, I agree. A resolver can't really tell that a response with an expired >> signature wasn't an attacker trying to replay old data. For robustness >> against

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

On Tue, Jul 18, 2023 at 12:51:39PM -0700, Gavin McCullagh wrote: > We definitely saw Unbound returning SERVFAIL for unsigned com domains > though. Failures for even for some "unsigned" domains were to be expected if retries were either not happening or the retry count was at times exceeded. The

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

On Tue, Jul 18, 2023 at 07:38:51PM +, Matt Nordhoff via dns-operations wrote: > ³ Actually the xn--wgbh1c TLD is broken right now but I haven't told anyone. Well, at least that IDN ccTLD (Egypt, "dotmasr") is broken for all servers. Nice to see a rollover in progress to algorithm 13, but of

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

On Tue, Jul 18, 2023 at 12:45 PM Shumon Huque wrote: > Yes, I agree. A resolver can't really tell that a response with an expired > signature wasn't an attacker trying to replay old data. For robustness > against attacks, it must re-query other available other servers if they > exist. > > Also, I

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

On Tue, Jul 18, 2023 at 3:29 PM Viktor Dukhovni wrote: > On Tue, Jul 18, 2023 at 08:54:04PM +0200, Ondřej Surý wrote: > > > With my implementor’s hat on, I think this is wrong approach. It > > (again) adds a complexity to the resolvers and yet again based > > (mostly) on isolated incident. I real

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

--- Begin Message --- On Tue, Jul 18, 2023 at 6:21 PM Gavin McCullagh wrote: > Hi, > > sorry to dredge this back up, but I just want to give anyone the chance to > object. > > My read of what Viktor and others have indicated here is that, when a > validating resolver receives a response with exp

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

--- Begin Message --- i have two comments here. Ondřej Surý wrote on 2023-07-18 11:54: With my implementor’s hat on, I think this is wrong approach. It (again) adds a complexity to the resolvers and yet again based (mostly) on isolated incident. I really don’t want yet another “serve-stale” in

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

On Tue, Jul 18, 2023 at 08:54:04PM +0200, Ondřej Surý wrote: > With my implementor’s hat on, I think this is wrong approach. It > (again) adds a complexity to the resolvers and yet again based > (mostly) on isolated incident. I really don’t want yet another > “serve-stale” in the resolvers. I have

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

With my implementor’s hat on, I think this is wrong approach. It (again) adds a complexity to the resolvers and yet again based (mostly) on isolated incident. I really don’t want yet another “serve-stale” in the resolvers. I have to yet see an evidence that serve-stale has helped anything since

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

Hi, sorry to dredge this back up, but I just want to give anyone the chance to object. My read of what Viktor and others have indicated here is that, when a validating resolver receives a response with expired rrsigs, it's okay (and encouraged?) for that resolver to treat that as an invalid respo

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

--- Begin Message --- On Thu Jul 13, 2023 at 7:16 PM UTC, Gavin McCullagh wrote: > ... > > I assume lots of us on this mailing list operate authoritative dns > servers. When one of our PoPs or nameservers is unresponsive, most of us > rely on retries against other nameservers (aka PoPs) to ensure

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

Hi, On Thu, Jul 13, 2023 at 1:18 PM Viktor Dukhovni wrote: > On Thu, Jul 13, 2023 at 12:16:37PM -0700, Gavin McCullagh wrote: > > > When faced with ~4x obviously bogus, broken nameservers (the stale pop) > and > > ~9x fresh working nameservers with valid signatures, the DNSSEC RFCs > appear > >

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

On Thu, Jul 13, 2023 at 12:16:37PM -0700, Gavin McCullagh wrote: > When faced with ~4x obviously bogus, broken nameservers (the stale pop) and > ~9x fresh working nameservers with valid signatures, the DNSSEC RFCs appear > to specify (and Unbound appears to implement) that resolvers must accept >

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

On Wed, Jul 12, 2023, 5:28 PM Olafur Gudmundsson wrote: > > > On Jul 11, 2023, at 8:24 PM, Gavin McCullagh wrote: > > That is true of course, but the magnitude of this event was made much > worse by dnssec. The entire COM and NET zones being bogus (including the > unsigned delegations) is very

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

> On Jul 11, 2023, at 8:24 PM, Gavin McCullagh wrote: > > That is true of course, but the magnitude of this event was made much worse > by dnssec. The entire COM and NET zones being bogus (including the unsigned > delegations) is very different to the few that saw record changes in the > pr

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

On 2023-07-12 05:50, Viktor Dukhovni wrote: On Tue, Jul 11, 2023 at 10:51:47PM -0400, Viktor Dukhovni wrote: In .COM CZDS zone file snapshot of .COM from ~midnight UTC 2023-07-11 the range of non-apex RRSIG inception times was: 20230707025004 – 20230710225021 With corresponding expirat

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

On Tue, Jul 11, 2023 at 10:51:47PM -0400, Viktor Dukhovni wrote: > In .COM CZDS zone file snapshot of .COM from ~midnight UTC 2023-07-11 > the range of non-apex RRSIG inception times was: > > 20230707025004 – 20230710225021 > > With corresponding expiration times: > > 20230714040004 – 2

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

On Tue, Jul 11, 2023 at 05:24:57PM -0700, Gavin McCullagh wrote: > > Consequently, some users unlucky enough to have switched providers or > > moved to new NS hosts at the same a provider after the site was cut off > > from updates also observed some issues, whether or not DNSSEC happened > > to b

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

More of a routing thing than DNS - but this type of view from the outside in is really helpful to detect by providers feeding RIPE RIS or route views so there are better external views into networks. This is an area where I want to expand and improve coverage after things like the silent and h

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

> On Tue, Jul 11, 2023, 4:30 PM Viktor Dukhovni wrote: > On Tue, Jul 11, 2023 at 10:24:21PM +, Wessels, Duane wrote: > > > Over the weekend, this caused an issue that may have affected the > > ability of some internet users in the region to reach some .com and > > .net domains, as DNSSEC

Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

On Tue, Jul 11, 2023 at 10:24:21PM +, Wessels, Duane wrote: > Last week, during a migration of one of our DNS resolution sites in > Singapore, from one provider to another, we unexpectedly lost > management access and the ability to deliver changes and DNS updates > to the site. Following our

[dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

--- Begin Message --- All, Last week, during a migration of one of our DNS resolution sites in Singapore, from one provider to another, we unexpectedly lost management access and the ability to deliver changes and DNS updates to the site. Following our standard procedure, we disabled all transi