Re: Trying to get in touch: ntp.org sites broken
Thanks Marco, I'll check with the crew. On 8/2/2021 11:59 PM, Marco Davids (Private) via NANOG wrote: > Hello, > > I'm trying to get in touch with webmas...@ntp.org. But so far without > luck. Maybe this route will help. > > I noticed that quit a number of pages you mention on > https://www.ntp.org/ are no longer functioning. > > Like http://lists.ntp.org/ and http://support.ntp.org/. > > If anyone knows a way to get this fixed, please help. > > Thank you. > -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: CISCO 0-day exploits
On 2/11/2020 2:04 AM, Saku Ytti wrote: > On Tue, 11 Feb 2020 at 09:09, Ahmed Borno wrote: > >> So yeah iACLs, CoPP and all sorts of basic precautions are needed, but I'm >> thinking something more needs to be done, specially if these ancient code >> stacks are being imported into new age 'IoT' devices, multiplying the attack >> vector by a factor of too many. > > I can't see situation getting better. Why should vendor invest in high > quality code, certainly the cultural shift will cost something, it's > not 0 cost and what is the upside? If IOS and JunOS realistically were > significantly less buggy many of us would stop buying support, because > we either know how to configure these or can get help faster free from > the community, we largely need the support because the software > quality is so bad _everyone_ finds new bugs all the time and we don't > have the source code to fix it as a community. > So I suspect significantly better quality software would at least > initially cost more to produce and it would reduce revenue in loss of > support. Yeah, things need to get better, and soon. At least, some things... Was I too subtle just now? -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: UDP/123 policers & status
On 3/18/2020 4:46 PM, Damian Menscher via NANOG wrote: > On Wed, Mar 18, 2020 at 8:45 AM Steven Sommars > mailto:stevesommars...@gmail.com>> wrote: > > The various NTP filters (rate limits, packet size limits) are > negatively affecting the NTP Pool, the new secure NTP protocol > (Network Time Security) and other clients. NTP filters were > deployed several years ago to solve serious DDoS issues, I'm not > second guessing those decisions. Changing the filters to instead > block NTP mode 7, which cover monlist and other diagnostics, would > improve NTP usability. > > > http://www.leapsecond.com/ntp/NTP_Suitability_PTTI2020_Revised_Sommars.pdf > > > I've advocated a throttle (not a hard block) on udp/123 packets with 468 > Bytes/packet (the size of a full monlist response). In your paper you > mention NTS extensions can be 200+ bytes. How large do those packets > typically get, in practice? And how significant is packet loss for them > (if there's high packet loss during the occasional attack, does that > pose a problem)? I expect to see NTP UDP packets that would approach the MTU limit, in some cases. If a packet is "too big" for some pathway, then are we talking about a fractional packet loss or are we talking about 100% packet loss (dropped mid-way due to size)? > Damian -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: UDP/123 policers & status
On 3/28/2020 3:29 PM, Bottiger wrote: > but why isn't BCP 38 widely deployed? > > > Because it costs time and money. People have been asking for it to be > implemented for decades. It is never going to be deployed on every network. So you are claiming BCP 38 has to be all or nothing? That there is *no* benefit to incremental deployment? > What fraction of the > world does implement BCP 38? > > > Not enough. Everyone has to use it for it to work. Otherwise the > hackers will still find a network that doesn't have it. I disagree. Enough people have to use it for it to work. And as more folks use it, there is increasing motivation for more folks to use it. As the number of deployments increases, one can assume (perhaps correctly) that it will become less expensive to deploy, and that additional measures will be found to help accomplish the same thing. > I'd also be interested in general background info on DDoS. Who is > DDoS-ing > whom and/or why? Is this gamers trying to get an advantage on a > competitor? > Bad guys making a test run to see if the server can be used for a > real run? > > > Most motivations for attacks can't be traced. But this is not just a > gaming problem. It is used to extort businesses for money, destroy > competitors, shutdown government critics, fame. > > Is DDoS software widely available on the dark web? > > > You don't need the dark web. It is widely available on Github like most > other attack types. > > https://github.com/search?q=ntp+ddos > > Broken protocols need to be removed and blacklisted at every edge. > Pushing the responsibility to BCP38 is unrealistic. The monlist attack was mitigated many years' ago. The problem is that too many folks don't upgrade their software. > On Mon, Mar 23, 2020 at 7:43 AM Hal Murray > <mailto:hgm%2bna...@ip-64-139-1-69.sjc.megapath.net>> wrote: > > Steven Sommars said: > > The secure time transfer of NTS was designed to avoid > amplification attacks. Uh, no. If you understand what's going on from the perspective of both the client and the server and think about the various cases, I think you'll see what I mean. NTS is a task-specific hammer. -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: UDP/123 policers & status
Ragnar, On 3/28/2020 4:09 PM, Ragnar Sundblad wrote: > >> On 28 Mar 2020, at 23:58, Harlan Stenn wrote: >> >>> Steven Sommars said: >>>> The secure time transfer of NTS was designed to avoid >>>amplification attacks. >> >> Uh, no. > > Yes, it was. > > As Steven said, “The secure time transfer of NTS was designed to > avoid amplification attacks”. I would even say - to make it > impossible to use for amplification attacks. Please tell me how. I've been part of this specific topic since the original NTS spec. For what y'all are saying to be true, there are some underlying assumptions that would need to be in place, and they are clearly not in place now and won't be until people update their software, and even better, tweak their configs. >> If you understand what's going on from the perspective of both the >> client and the server and think about the various cases, I think you'll >> see what I mean. > > Hopefully, no-one exposes mode 6 or mode 7 on the internet anymore > at least not unauthenticated, and at least not the commands that are > not safe from amplification attacks. Those just can not be allowed > to be used anonymously. But mode 6/7 is completely independent of NTS. It's disingenuous for people to imply otherwise. >> NTS is a task-specific hammer. > > Yes. > > Ragnar -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: UDP/123 policers & status
Ragnar, On 3/28/2020 4:59 PM, Ragnar Sundblad wrote: > > >> On 29 Mar 2020, at 00:35, Harlan Stenn wrote: >> >> Ragnar, >> >> On 3/28/2020 4:09 PM, Ragnar Sundblad wrote: >>> >>>> On 28 Mar 2020, at 23:58, Harlan Stenn wrote: >>>> >>>>> Steven Sommars said: >>>>>> The secure time transfer of NTS was designed to avoid >>>>> amplification attacks. >>>> >>>> Uh, no. >>> >>> Yes, it was. >>> >>> As Steven said, “The secure time transfer of NTS was designed to >>> avoid amplification attacks”. I would even say - to make it >>> impossible to use for amplification attacks. >> >> Please tell me how. I've been part of this specific topic since the >> original NTS spec. For what y'all are saying to be true, there are some >> underlying assumptions that would need to be in place, and they are >> clearly not in place now and won't be until people update their >> software, and even better, tweak their configs. > > The NTS protected NTP request is always of the same size, or in some > cases larger, than the NTS protected NTP response. It is carefully > designed to work that way. So what? The use of NTS is completely independent of whether or not a server will respond to a packet. And an unauthenticated NTP request that generates an unauthenticated response is *always* smaller than an authenticated request, regardless of whether or not the server responds. Claiming that amplification is a significant issue in the case where there's no amplification but the base packet size is bigger is ignoring a key piece of information, and is disingenuous in my book. > Hence, [what Steven said]. > >>>> If you understand what's going on from the perspective of both the >>>> client and the server and think about the various cases, I think you'll >>>> see what I mean. >>> >>> Hopefully, no-one exposes mode 6 or mode 7 on the internet anymore >>> at least not unauthenticated, and at least not the commands that are >>> not safe from amplification attacks. Those just can not be allowed >>> to be used anonymously. >> >> But mode 6/7 is completely independent of NTS. > > Exactly. No one needs to, or should, expose mode6/7 at all. They were > designed at a time when the internet was thought to be nice place were > people behaved, decades ago, today they are just huge pains in the > rear. Sadly allowing anonymous mode 6/7 was left in there far to long > (admittedly being wise in hindsight is so much easier than in advance). > And here we are, with UDP port 123 still being abused by the bad > guys, and still being filtered by the networks. Your statement about "exposing" is imprecise and bordering on incorrect, at least in some cases. But again, the use of mode 6/7 is completely independent of NTS. You are trying to tie them together. >> It's disingenuous for people to imply otherwise. > > I couldn’t say, I don’t even know of an example of someone who does. You've done it in two cases here, from everything I have seen. > Ragnar -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: UDP/123 policers & status
I think I see the disconnect. One of the design goals of NTS was to prevent NTS-protected time requests from being used in amplification attacks. Yes, that's true. I've been interpreting this thread as people claiming that NTS will solve a wider class of amplification vectors, and that's simply not true. "Universal affirmatives can only be partially converted: all of Alma Cogen is dead, but only some of the class of dead people are Alma Cogen." It's also true, and generally not stated, that to the extent that NTS-protected packets are exchanged, they will require increased network capacity. A cynic could argue that requiring additional internet bandwidth is a profitable goal, and the drama about requiring that extra protection is the distraction that normalizes that cost. H On 3/28/2020 5:18 PM, Harlan Stenn wrote: > Ragnar, > > On 3/28/2020 4:59 PM, Ragnar Sundblad wrote: >> >> >>> On 29 Mar 2020, at 00:35, Harlan Stenn wrote: >>> >>> Ragnar, >>> >>> On 3/28/2020 4:09 PM, Ragnar Sundblad wrote: >>>> >>>>> On 28 Mar 2020, at 23:58, Harlan Stenn wrote: >>>>> >>>>>> Steven Sommars said: >>>>>>> The secure time transfer of NTS was designed to avoid >>>>>> amplification attacks. >>>>> >>>>> Uh, no. >>>> >>>> Yes, it was. >>>> >>>> As Steven said, “The secure time transfer of NTS was designed to >>>> avoid amplification attacks”. I would even say - to make it >>>> impossible to use for amplification attacks. >>> >>> Please tell me how. I've been part of this specific topic since the >>> original NTS spec. For what y'all are saying to be true, there are some >>> underlying assumptions that would need to be in place, and they are >>> clearly not in place now and won't be until people update their >>> software, and even better, tweak their configs. >> >> The NTS protected NTP request is always of the same size, or in some >> cases larger, than the NTS protected NTP response. It is carefully >> designed to work that way. > > So what? > > The use of NTS is completely independent of whether or not a server will > respond to a packet. > > And an unauthenticated NTP request that generates an unauthenticated > response is *always* smaller than an authenticated request, regardless > of whether or not the server responds. > > Claiming that amplification is a significant issue in the case where > there's no amplification but the base packet size is bigger is ignoring > a key piece of information, and is disingenuous in my book. > >> Hence, [what Steven said]. >> >>>>> If you understand what's going on from the perspective of both the >>>>> client and the server and think about the various cases, I think you'll >>>>> see what I mean. >>>> >>>> Hopefully, no-one exposes mode 6 or mode 7 on the internet anymore >>>> at least not unauthenticated, and at least not the commands that are >>>> not safe from amplification attacks. Those just can not be allowed >>>> to be used anonymously. >>> >>> But mode 6/7 is completely independent of NTS. >> >> Exactly. No one needs to, or should, expose mode6/7 at all. They were >> designed at a time when the internet was thought to be nice place were >> people behaved, decades ago, today they are just huge pains in the >> rear. Sadly allowing anonymous mode 6/7 was left in there far to long >> (admittedly being wise in hindsight is so much easier than in advance). >> And here we are, with UDP port 123 still being abused by the bad >> guys, and still being filtered by the networks. > > Your statement about "exposing" is imprecise and bordering on incorrect, > at least in some cases. > > But again, the use of mode 6/7 is completely independent of NTS. You > are trying to tie them together. > > >>> It's disingenuous for people to imply otherwise. >> >> I couldn’t say, I don’t even know of an example of someone who does. > > You've done it in two cases here, from everything I have seen. > >> Ragnar > -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: UDP/123 policers & status
On 3/28/2020 5:35 PM, Ragnar Sundblad wrote: > > >> On 29 Mar 2020, at 01:18, Harlan Stenn wrote: >> >> Ragnar, >> >> On 3/28/2020 4:59 PM, Ragnar Sundblad wrote: >>> >>> >>>> On 29 Mar 2020, at 00:35, Harlan Stenn wrote: >>>> >>>> Ragnar, >>>> >>>> On 3/28/2020 4:09 PM, Ragnar Sundblad wrote: >>>>> >>>>>> On 28 Mar 2020, at 23:58, Harlan Stenn wrote: >>>>>> >>>>>>> Steven Sommars said: >>>>>>>> The secure time transfer of NTS was designed to avoid >>>>>>> amplification attacks. >>>>>> >>>>>> Uh, no. >>>>> >>>>> Yes, it was. >>>>> >>>>> As Steven said, “The secure time transfer of NTS was designed to >>>>> avoid amplification attacks”. I would even say - to make it >>>>> impossible to use for amplification attacks. >>>> >>>> Please tell me how. I've been part of this specific topic since the >>>> original NTS spec. For what y'all are saying to be true, there are some >>>> underlying assumptions that would need to be in place, and they are >>>> clearly not in place now and won't be until people update their >>>> software, and even better, tweak their configs. >>> >>> The NTS protected NTP request is always of the same size, or in some >>> cases larger, than the NTS protected NTP response. It is carefully >>> designed to work that way. >> >> So what? >> >> The use of NTS is completely independent of whether or not a server will >> respond to a packet. >> >> And an unauthenticated NTP request that generates an unauthenticated >> response is *always* smaller than an authenticated request, regardless >> of whether or not the server responds. >> >> Claiming that amplification is a significant issue in the case where >> there's no amplification but the base packet size is bigger is ignoring >> a key piece of information, and is disingenuous in my book. > > You are now comparing unauthenticated mode 3 and mode 4 packets to > cryptographically secured ones, which is a completely different thing. > > Disingenuous? I made no such claim. I was talking about: >>>>> As Steven said, “The secure time transfer of NTS was designed to >>>>> avoid amplification attacks”. I would even say - to make it >>>>> impossible to use for amplification attacks. and that statement is not as clear as it could be. Specifically: NTS was designed to avoid amplification attacks is vague. You have just now written: > You are now comparing unauthenticated mode 3 and mode 4 packets to > cryptographically secured ones, which is a completely different thing. Completely different? How? Where is the amplification in an unauthenticated mode 3 request and an unauthenticated response? How does cryptographically securing these packets make any difference here? > A protocol with varying packet size, as the NTS protected NTP is, > can easily have the bad property of having responses larger than the > requests if not taken care. Don’t you see that? I sure think I understand these points. Who here has said that there was any problem with there being an amplification issue with properly-authenticated NTS packets? The current NTS spec is *only* written for mode 3/4 exchanges. While it might be applicable for mode 6/7, I haven't seen any specs for this usage. In the NTP Project's Reference implementation there are extra implementation-specific protections built in to mode 6/7 exchanges. While some of this might be addressed in the NTS spec, I don't recall ever seeing this. So why are you talking about mode 6/7 in this context? >>> Hence, [what Steven said]. >>> >>>>>> If you understand what's going on from the perspective of both the >>>>>> client and the server and think about the various cases, I think you'll >>>>>> see what I mean. >>>>> >>>>> Hopefully, no-one exposes mode 6 or mode 7 on the internet anymore >>>>> at least not unauthenticated, and at least not the commands that are >>>>> not safe from amplification attacks. Those just can not be allowed >>>>> to be used anonymously. >>>> >>>> But mode 6/7 is completely independent of NTS. >>> >>> Exactly. No one needs to, or should, expose mode6/7 at all. They were >>> designed at a time when the internet was thought to be
Re: UDP/123 policers & status
On 3/29/2020 11:18 PM, Saku Ytti wrote: > On Mon, 30 Mar 2020 at 01:58, Ragnar Sundblad wrote: > >> A protocol with varying packet size, as the NTS protected NTP is, >> can easily have the bad property of having responses larger than the >> requests if not taken care. Don’t you see that? > > Why? Why not pad requests to guarantee attenuation vector until > authenticity of packets can be verified? > > MinimaLT does this. I think all UDP based and initial TCP should do > it, doing it for existing protocols may not be possible, but why not > for new? > > I proposed similar method for proxy-trace (bidir tracerouting) - > https://github.com/ytti/proxy-trace/blob/master/draft-ytti-intarea-proxy-trace.xml#L169 Please help me understand this. Exactly how bad is it if the query and response packets are of a different size? Does it matter at 4 bytes? 32? what are the expected costs of different %s of response packets being sent back? I think this needs to be understood for a variety of different sized query/response packets. Then factor in the costs of padding the shorter packets. Include the cost of the actual bump in network bandwidth, which I suspect is negligible, and at the same time we can do a reality check on the folks who claim that NTP packets are a significant cause of the bufferbloat problem. Then we need to thoroughly study how the various size differences between the client request and server response packets affect the quality of time synchronization, in various network scenarios. Some have claimed this is clearly noticeable and significant. I'd like to see the experiments and the data. NTF is very happy to do this work, incrementally if needed, if we can get the necessary support and cooperation/collaboration. -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: UDP/123 policers & status
On 3/30/2020 1:27 AM, Saku Ytti wrote: > On Mon, 30 Mar 2020 at 11:15, Harlan Stenn wrote: > >> Please help me understand this. >> >> Exactly how bad is it if the query and response packets are of a >> different size? Does it matter at 4 bytes? 32? > > Presumably, if it's attenuation vector (1byte or more), presumably > attacker will use any of the other many vectors which are > amplification vectors or will directly attack from the zombie machines > they pwn. Since NST would have negative ROI on attack if there is > _any_ attenuation. OK, and exactly how bad is a single byte attenuation, when compared against the cost of 100% of all of the 1-byte shorter NTP packets being made bigger to make the attenuation vector 0? -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: UDP/123 policers & status
On 3/30/2020 2:01 AM, Ragnar Sundblad wrote: > > >> On 30 Mar 2020, at 08:18, Saku Ytti wrote: >> >> On Mon, 30 Mar 2020 at 01:58, Ragnar Sundblad wrote: >> >>> A protocol with varying packet size, as the NTS protected NTP is, >>> can easily have the bad property of having responses larger than the >>> requests if not taken care. Don’t you see that? >> >> Why? Why not pad requests to guarantee attenuation vector until >> authenticity of packets can be verified? > > Right, and NTS does that. There is more to NTP than NTS. Are y'all seriously recommending that NTP always sends a max-sized packet as a client request so the client/server can send back an identical response? The alternative seems to be that the client sends a smaller request and is ready when the response from the server is "Send your request again, but this time pad it to NNN bytes so I can respond with the same sized packet"? > Ragnar -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: UDP/123 policers & status
I found this as an unsent draft - I hope I didn't send it before. On 3/30/2020 2:01 AM, Ragnar Sundblad wrote: > > >> On 30 Mar 2020, at 08:18, Saku Ytti wrote: >> >> On Mon, 30 Mar 2020 at 01:58, Ragnar Sundblad wrote: >> >>> A protocol with varying packet size, as the NTS protected NTP is, >>> can easily have the bad property of having responses larger than the >>> requests if not taken care. Don’t you see that? >> >> Why? Why not pad requests to guarantee attenuation vector until >> authenticity of packets can be verified? > > Right, and NTS does that. There is more to NTP than NTS. Are y'all seriously recommending that NTP always sends a max-sized packet as a client request so the client/server can send back an identical response? That's just wasting huge amounts of bandwidth to save the possibility of a possibly larger response. And just becase a responbse may be larger, that doesn't necessarily translate into an amplification vector. The alternative seems to be that the client sends a smaller request and is ready when the response from the server is "Send your request again, but this time pad it to NNN bytes so I can respond with the same sized packet"? > Ragnar -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: UDP/123 policers & status
NTP uses UDP for time. I'm not sure what you're talking about. H On 4/17/20 1:32 AM, Ragnar Sundblad wrote: > > >> On 17 Apr 2020, at 01:28, Harlan Stenn wrote: >> >> I found this as an unsent draft - I hope I didn't send it before. >> >> On 3/30/2020 2:01 AM, Ragnar Sundblad wrote: >>> >>> >>>> On 30 Mar 2020, at 08:18, Saku Ytti wrote: >>>> >>>> On Mon, 30 Mar 2020 at 01:58, Ragnar Sundblad wrote: >>>> >>>>> A protocol with varying packet size, as the NTS protected NTP is, >>>>> can easily have the bad property of having responses larger than the >>>>> requests if not taken care. Don’t you see that? >>>> >>>> Why? Why not pad requests to guarantee attenuation vector until >>>> authenticity of packets can be verified? >>> >>> Right, and NTS does that. >> >> There is more to NTP than NTS. >> >> Are y'all seriously recommending that NTP always sends a max-sized >> packet as a client request so the client/server can send back an >> identical response? That's just wasting huge amounts of bandwidth to >> save the possibility of a possibly larger response. > > If there is no sender verification, yes. It doesn’t have to be larger > than the maximum response size. > > Another option is to use TCP - the handshake solves the problem. > >> And just becase a responbse may be larger, that doesn't necessarily >> translate into an amplification vector. > > If there is no sender verification, it generally does. > In what case does it not? > >> The alternative seems to be that the client sends a smaller request and >> is ready when the response from the server is "Send your request again, >> but this time pad it to NNN bytes so I can respond with the same sized >> packet"? > > Sure, that is one way! > Or - Here are the first N entries, send another request for the > next batch, optionally: there are M entries in total. > Or - TCP. > There likely are many other options. > > Ragnar > > -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: UDP/123 policers & status
On 4/17/2020 2:01 AM, Ragnar Sundblad wrote: > > I thought we were talking about control traffic. I expect there will be a TCP control traffic option. I expect there will continue to be a UDP control traffic option. These are "mechanisms", there will be a reasonable default policy (that will change and evolve over time), and there will be a mechanism to allow the local admins to implement their local policy choices. > If you want to do some NTP time comparison mode with larger responses > than requests, I agree that TCP is likely not a good option. I still don't see why, in the general case, some here think we must force a smaller request packet to be padded just so a possibly larger response packet will provide no amplification. Basic packets are of a known and identical size. Before the 7822 braindamage (my opinion of 7822), EFs *required* MACs. Post 7822 they don't, but even so, if people implement EFs and don't include (disable?) "protection" they've pretty much made their choice. But we're speaking in generalities here. What new or proposed packet content would trigger a larger response that would not require an authenticated request in the first place? And I just realized this is the NANOG list and not the NTP list, so I'm happy to stop. H -- > Ragnar > >> On 17 Apr 2020, at 10:44, Harlan Stenn wrote: >> >> NTP uses UDP for time. >> >> I'm not sure what you're talking about. >> >> H >> >> On 4/17/20 1:32 AM, Ragnar Sundblad wrote: >>> >>> >>>> On 17 Apr 2020, at 01:28, Harlan Stenn wrote: >>>> >>>> I found this as an unsent draft - I hope I didn't send it before. >>>> >>>> On 3/30/2020 2:01 AM, Ragnar Sundblad wrote: >>>>> >>>>> >>>>>> On 30 Mar 2020, at 08:18, Saku Ytti wrote: >>>>>> >>>>>> On Mon, 30 Mar 2020 at 01:58, Ragnar Sundblad wrote: >>>>>> >>>>>>> A protocol with varying packet size, as the NTS protected NTP is, >>>>>>> can easily have the bad property of having responses larger than the >>>>>>> requests if not taken care. Don’t you see that? >>>>>> >>>>>> Why? Why not pad requests to guarantee attenuation vector until >>>>>> authenticity of packets can be verified? >>>>> >>>>> Right, and NTS does that. >>>> >>>> There is more to NTP than NTS. >>>> >>>> Are y'all seriously recommending that NTP always sends a max-sized >>>> packet as a client request so the client/server can send back an >>>> identical response? That's just wasting huge amounts of bandwidth to >>>> save the possibility of a possibly larger response. >>> >>> If there is no sender verification, yes. It doesn’t have to be larger >>> than the maximum response size. >>> >>> Another option is to use TCP - the handshake solves the problem. >>> >>>> And just becase a responbse may be larger, that doesn't necessarily >>>> translate into an amplification vector. >>> >>> If there is no sender verification, it generally does. >>> In what case does it not? >>> >>>> The alternative seems to be that the client sends a smaller request and >>>> is ready when the response from the server is "Send your request again, >>>> but this time pad it to NNN bytes so I can respond with the same sized >>>> packet"? >>> >>> Sure, that is one way! >>> Or - Here are the first N entries, send another request for the >>> next batch, optionally: there are M entries in total. >>> Or - TCP. >>> There likely are many other options. >>> >>> Ragnar >>> >>> >> >> -- >> Harlan Stenn >> http://networktimefoundation.org - be a member! > > -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: IERS ponders reverse leapsecond...
On 8/3/2022 8:09 AM, Jay Ashworth wrote: General press loses its *mind*: https://www.cbsnews.com/news/earth-spinning-faster-than-usual-shortest-day-ever/#app <https://www.cbsnews.com/news/earth-spinning-faster-than-usual-shortest-day-ever/#app> Have you tested leap second handling, especially in reverse? How do you simulate it? Are there existing test harnesses for simulating it? It's somewhere between trivial and straightforward to test forward/reverse leap seconds with an NTP server and a (dedicated) client. Cheers, -- jra -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: IERS ponders reverse leapsecond...
I've been thinking about this problem for several decades. I believe I have a Good solution to it. It's the General Timestamp API effort at Network Time Foundation. If you have been paying any attention, you will immediately realize that the effort is stalled because of a lack of funding. I'm hesitant to say more, as there are some other groups with partial solutions who think they have the complete solution; and since these groups are much better funded, they are happy to forge ahead based on their view of the world. I will say that the GTSAPI is designed around a "robust mechanism" that can implement every "local policy choice" I have seen around "how do we want to deal with 'time'?" It addresses using timestamps beyond the execution of the current boot of the local system, and includes conversions between different versions of different timescales, and a library for arithmetic and comparison functions. H On 8/3/2022 8:33 AM, Matthew Huff wrote: True, But it's hard enough to get developers to understand the need to code for 61 seconds in a minute, and now they would need to code for 59 seconds as well. If time systems simply skewed the time so that 60 seconds actually just took 61 seconds or 59 seconds, there would be other issues, but coders wouldn't be involved. -Original Message- From: NANOG On Behalf Of Stephane Bortzmeyer Sent: Wednesday, August 3, 2022 11:19 AM To: Jay Ashworth Cc: nanog@nanog.org Subject: Re: IERS ponders reverse leapsecond... On Wed, Aug 03, 2022 at 11:09:25AM -0400, Jay Ashworth wrote a message of 32 lines which said: General press loses its *mind*: Indeed, they seem not to know what they write about. "atomic time – the universal way time is measured on Earth – may have to change" They don't even know the difference between TAI and UTC. -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: NTP question
So I gotta ask, just as a reality check: - Why do folks want to have one or more NTP server masters that have at least 1 refclock on them in a data center, instead of having their data center NTP server masters that only get time over the internet? - What % of data center operators provide time servers in their data centers for their tenants (or for the general public)? -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: NTP question
On 5/1/19 2:59 PM, Andreas Ott wrote: > On Wed, May 01, 2019 at 02:35:58PM -0700, Harlan Stenn wrote: >> - Why do folks want to have one or more NTP server masters that have at >> least 1 refclock on them in a data center, instead of having their data >> center NTP server masters that only get time over the internet? > > I had that discussion before with the QSA for a compliance audit, pointing > to requirement "10.4.3 Time settings are received from industry-accepted > time sources" and "verify that the time server(s) accept time updates from > specific, industry-accepted external sources (to prevent a malicious > individual from changing the clock)" in the PCI-DSS document. He > non-jokingly suggested "why don't you use pool.ntp.org?", not really > realizing how many servers are in fact just someone's PC behind a cable > modem in their home, which negated the "do I trust the time I am > receiving?". My immediate answer was "we could use NIST servers", > but the easiest way out of this is "we operate our own NTP appliance > with a GPS receiver" and provide that as evidence. > > Don't get me wrong, I support pool.ntp.org by operating and contributing > servers to it, but it is not deemed good enough if you need traceability > of your NTP time source(s), even though the pool will only admit members > above a certain quality threshold. I have no immediate agenda here. My sole purpose is to get information about this, as I mostly work with people who a) believe accurate time is important, and b) at least have an appreciation for how unexpectedly difficult it is to synchronize time in a predictable and stable way across a large population of systems in a diverse set of environments. In my experience, people who don't fall in to either of those categories are pretty well invested in their opinions. >> - What % of data center operators provide time servers in their data >> centers for their tenants (or for the general public)? > > My $employer does that in our datacenters and points of presence for > our customers. Glad to hear it! > -andreas > -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: NTP question
On 5/1/2019 4:17 PM, Brandon Martin wrote: > On 5/1/19 7:03 PM, Harald Koch wrote: >> Properly deployed NTP should calibrate the local hardware clocks to >> prevent drift even during connectivity outages. (I'm talking both the >> low resolution hardware clocks used for timing across power cycles and >> reboots, and the oscillators used while the OS is running). While most >> computer hardware is temperature sensitive, if your datacenter is >> suddenly changing temperature enough to cause clock drift, well, you >> have bigger problems.:) > > For sure, sudden loss of time "shouldn't" happen, but having a local > refclock is comparatively cheap insurance against it in many deployments. BCP these days is "orphan mode", not "local refclock". > I've seen things like this when there's a sudden power loss across a > small site e.g. a remote PoP. Think a loss of utility power and UPS > fails to transfer for some unanticipated reason. Everything will come > back up when either the utility power comes back or generator spins up, > but it will all be hard reset. Depending on your NTP implementation, > the local hardware clock may not be particularly accurate. Even good > implementations often lack the necessary hardware capabilities to trim > the low-resolution hardware reference and have to resort to simply > flushing the time to hardware every so often. > > Relative inaccuracies of a few seconds are pretty normal in that kind of > situation in my experience. Putting everything together from logs where > there's an unknown time offset of a few seconds after the fact can be > tough. Then again, maybe you don't care in this example case since the > cause of the problem is proximate - the frigging UPS didn't do its job. > More complex scenarios might be easily envisioned, though. > > Now, obviously you've still got an issue of the fact that the GPS refclk > will take a while to lock and start serving time, but at least you've > potentially got known-good time info before you start bringing > higher-level network protocols up (and can purposely delay until you do, > if desired) which is potentially impossible if your only source of time > is the network itself. Ah, this is the dance with "have enough sources of time"... -- Harlan Stenn, Network Time Foundation http://nwtime.org - be a Member!
Re: NTP question
On 5/1/19 5:39 PM, William Herrin wrote: > On Wed, May 1, 2019 at 12:23 PM Mehmet Akcin wrote: > >> I am trying to buy a GPS based NTP server like this one >> >> https://timemachinescorp.com/product/gps-time-server-tm1000a/ >> >> but I will be placing this inside a data center, do these need an actual >> view of a sky to be able to get signal or will they work fine inside a data >> center building? if you have any other hardware requirements to be able to >> provide stable time service for hundreds of customers, please let me know. >> > > You buy a powered GPS antenna for it. Which antenna depends on the cable > length and type. The amplifier in the antenna amplifies the signal just > enough to overcome the cable loss between the antenna and the receiver. > Nice thick cables lose less signal. Dinky thin ones are easier to work with. > > You sure you need a GPS NTP server? You understand that if you do, you need > two for reliability right, and probably at geographically diverse > locations? If you're not on an air-gapped network, consider syncing a > couple head-end NTP servers against tick and tock (.usno.navy.mil, the > naval observatory) and not worrying about it. One less piece of equipment > to manage, update, secure, etc. Two is not a great number. If they disagree, there is no majority clique to be found. Also, there is something to be said for using different models/vendors for the time sources. If you only have the same model from one vendor and there is a bug, you can lose all your time sources at once. The GPS week rollover happens every ~19.7 years, and when that problem hits is a function of the firmware and a manufacturing date put in the firmware. These problems can be mitigated if you have "enough" time sources for your internal NTP servers and you peer with enough other, possibly your, servers. > Regards, > Bill Herrin -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: NTP question
On 5/1/19 4:53 PM, Mel Beckman wrote: > Ask, > > But with a small compact server like the DC-powered TimeMachines Inc unit, > which costs something like $300, you simply put the server where the > visibility is and connect back to the nearest Ethernet port in your network, > up to 300’ away, or virtually any distance with fiber transceivers. We’ve > installed these in Cantex boxes on a windy, rainy tenth-story rooftop in > upstate NY and it runs flawlessly, warmed by its own internal heat at > sub-zero temps, and perfectly happy at ambient temps of 110F. > > It’s hard to consider messing with signal converters and pricey > remotely-powered active antennas when you can solve the problem for $300. :) I sure hope you have ntpd set up to peer or get time with enough other servers. H -- > -mel > >> On May 1, 2019, at 4:44 PM, Ask Bjørn Hansen wrote: >> >> >> >>> On May 1, 2019, at 12:22, Mehmet Akcin wrote: >>> >>> I am trying to buy a GPS based NTP server like this one >>> >>> https://timemachinescorp.com/product/gps-time-server-tm1000a/ >>> >>> but I will be placing this inside a data center, do these need an actual >>> view of a sky to be able to get signal or will they work fine inside a data >>> center building? if you have any other hardware requirements to be able to >>> provide stable time service for hundreds of customers, please let me know. >> >> [ with my hobby-hat on … ] >> >> tl;dr: if any of the below is too much work, just run reasonably well >> monitored NTP server syncing from other NTP servers. If you want more than >> that, you need to see the sky. Don’t do the CDMA thing. >> >> Depending on your requirements having the antenna in the window may or may >> not be satisfactory. If it’s fine you probably could just have done a >> regular NTP server in the first place. For long swaths of the day you might >> not see too many satellites which will add to the uncertainty of the signal. >> >> Meinberg’s GPS antenna has a bit more smarts which helps it work on up to >> 300 meters on RG58 or 700 meters on RG213. (They also have products that >> use regular L1 antennas with the limitations Bryan mentioned). >> >> https://www.meinbergglobal.com/english/products/gps-antenna-converter.htm >> >> They also have a multi-mode fiber box to have the antenna be up to 2km from >> the box or 20km with their single mode fiber box, if you have fiber to >> somewhere else where you can see the sky and place an antenna. >> >> It will be more than the one you linked to, but their systems are very >> reasonably priced, too. For “hundreds of customers” whatever is the >> smallest/cheapest box they have will work fine. Even their smallest models >> have decent oscillators (for keeping the ticks accurate between GPS signals). >> >> The Meinberg time server products (I am guessing all of them, but I’m not >> sure) also have a mode where they poll an upstream NTP server aggressively >> and then steer the oscillator after it. I haven’t used it in production, but >> it worked a lot better than it sounded like it would. (In other words, even >> without GPS it’s a better time server than most systems). >> >> >> Ask -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: NTP Question
On 5/1/19 4:28 PM, Mel Beckman wrote: > Harlan and Mehmet, > > I can expand on one important reason that James only alluded to with his > “Kepping the Auditors happy” comment. > > Passing NTP through a firewall and then using that as a critical time > reference source represents a huge security risk. Here’s one detailed > explanation of that risk: > > https://insights.sei.cmu.edu/sei_blog/2017/04/best-practices-for-ntp-services.html I have some significant disagreements with some of the assumptions and positions in that posting, for whatever that's worth. And there are some good points in there, too. H -- > -mel > > On May 1, 2019, at 3:48 PM, James R Cutler > mailto:james.cut...@consultant.com>> wrote: > > On Wed, May 01, 2019 at 02:35:58PM -0700, Harlan Stenn wrote: > - Why do folks want to have one or more NTP server masters that have at > least 1 refclock on them in a data center, instead of having their data > center NTP server masters that only get time over the internet? > > Answers to that include: > > * Keeping the Auditors happy > * Knowing that “everyone does it” - the vendor told them so > * Bragging rights (expensive hardware) > * Being unbothered by fighting with facilities for building penetrations > and antenna mounts > * Misunderstanding the beauty and economy Dave Mills marvelous algorithms > for consistent time based on multiple sources, even those connected via > internet > * Unwillingness or inability to leverage other local resources capacity > to run ntpd with minimal impact in order to have a good constellation of > local NTP servers > * Willingness to farm out time service without doing a deep dive into why > and how, just leaving the design to the appliance vendors > > This covers most of what I have encountered in providing enterprise time > services for $dayjob+clients. I probably left out some significant points, > but it has been a few years... > > > > -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: NTP question
On 5/1/19 5:55 PM, William Herrin wrote: > On Wed, May 1, 2019 at 5:48 PM Keith Medcalf wrote: > >> If you have one such installation, then you really do not care about the >> "accuracy" of the time. However if you have multiple such installations >> then you want them all to have the same time (if you will be comparing logs >> between them, for example). At some point it becomes "cheaper" to spend >> thousands of dollars per site to have a single Stratum 0 timesource (for >> example, the GPS system) at each site (and thus comparable time stamps) >> than it is to pay someone to go though the rigamarole of computing offsets >> and slew rates between sites to be able to do accurate comparison. And if >> you communicate any of that info to outsiders then being able to say "my >> log timestamps are accurate to +/- 10 nanoseconds so it must be you who is >> farked up" (and be able to prove it) has immense value. >> > > If your network is air gapped from the Internet then sure. If it's not, you > can run NTP against a reasonably reliable set of time sources (not random > picks from Pool) and be able to say, "my log timestamps are accurate to +/- > 10 milliseconds so it must be you who is farked up." While my milliseconds > loses the pecking order contest, it's just as good for practical purposes > and a whole lot less expensive. It's not clear to me that there's anything *wrong* with using the pool, especially if you're using our 'pool' directive in your config file. That directive will bring up ~10 associations and continuously evaluate their quality, throwing out the poor performers and soliciting new servers of currently-good quality to replace them. This goes to "have _enough_ good-quality servers, and monitor your ntpd". > If your system is Internet-connected. If you run an air gapped network then > yeah, get your time out of band. > > Regards, > Bill Herrin > > -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: NTP question
Hi Keith, On 5/1/19 6:17 PM, Keith Medcalf wrote: > >> If your network is air gapped from the Internet then sure. If it's >> not, you can run NTP against a reasonably reliable set of time >> sources (not random picks from Pool) and be able to say, "my log >> timestamps are accurate to +/- 10 milliseconds so it must be you who >> is farked up." While my milliseconds loses the pecking order contest, >> it's just as good for practical purposes and a whole lot less >> expensive. > > You mean something like this, which is relatively easy to achieve: > > == > offset -0.09, frequency -0.823, time_const 30, watchdog 238 > synchronised to NTP server (192.5.41.40) at stratum 2 >time correct to within 12 ms >polling server every 1024 s > == > remote refid st t when poll reach delay offset jitter > == > +clock.sjc.he.ne .CDMA. 1 u 287 1024 377 64.3130.337 0.867 > -tock.usnogps.na .IRIG. 1 u5 1024 377 103.080 -2.097 0.316 > -tick.usnogps.na .IRIG. 1 u 806 1024 377 103.053 -2.328 0.363 > +india.colorado. .NIST. 1 u 270 1024 377 41.214 -0.159 0.113 > +time-b-b.nist.g .NIST. 1 u 984 1024 377 42.6090.200 0.045 > +time-c-b.nist.g .NIST. 1 u 180 1024 377 42.5630.201 0.064 > +time-a-b.nist.g .NIST. 1 u 163 1024 377 42.6390.137 0.032 > *192.5.41.40 .PTP.1 u 235 1024 377 12.756 -0.388 12.479 > -192.5.41.41 .IRIG. 1 u 312 1024 377 13.575 -1.172 2.425 > LOCAL(0).LOCL. 10 l- 6400.0000.000 0.000 > -- > pll offset: -8.474e-06 s > pll frequency:-0.823 ppm > maximum error:0.123149 s > estimated error: 0.000122 s > status: 2001 pll nano > pll time constant:10 > precision:1e-09 s > frequency tolerance: 500 ppm > == That all looks great except for the LOCAL clock at S10. In the event you lose connectivity to the outside, this system will jump from S2 to S10. Depending on the setup of your other systems, groups of them will go sailing off in their own directions. http://support.ntp.org/bin/view/Support/OrphanMode is the better solution. If you cannot do that for some reason, please see the "Dual Time Servers" case at http://support.ntp.org/bin/view/Support/UndisciplinedLocalClock . -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: NTP question
On 5/1/19 7:54 PM, Mel Beckman wrote: > Harlan, > > Why? The GPS NTP Server is Stratum-1. If it fails computer clocks will > freewheel for hours or days before losing significant time, during which > period you can simply order a replacement unit. If that isn’t fast enough, > buy two $300 boxes. The “consensus” issue is moot, since a GPS server gets a > consensus of clock time from the GPS satellite constellation. > > The “enough NTP peers” you speak of are simply not necessary. You might be right about the GPS server. It depends on how your $300 box behaves if it loses the GPS signal. The consensus issue isn't about the number of satellites the GPS receiver sees, it's about the number of time sources your NTP servers see. H -- > -mel via cell > >> On May 1, 2019, at 6:49 PM, Harlan Stenn wrote: >> >> >> >>> On 5/1/19 4:53 PM, Mel Beckman wrote: >>> Ask, >>> >>> But with a small compact server like the DC-powered TimeMachines Inc unit, >>> which costs something like $300, you simply put the server where the >>> visibility is and connect back to the nearest Ethernet port in your >>> network, up to 300’ away, or virtually any distance with fiber >>> transceivers. We’ve installed these in Cantex boxes on a windy, rainy >>> tenth-story rooftop in upstate NY and it runs flawlessly, warmed by its own >>> internal heat at sub-zero temps, and perfectly happy at ambient temps of >>> 110F. >>> >>> It’s hard to consider messing with signal converters and pricey >>> remotely-powered active antennas when you can solve the problem for $300. :) >> >> I sure hope you have ntpd set up to peer or get time with enough other >> servers. >> >> H >> -- >>> -mel >>> >>>> On May 1, 2019, at 4:44 PM, Ask Bjørn Hansen wrote: >>>> >>>> >>>> >>>>> On May 1, 2019, at 12:22, Mehmet Akcin wrote: >>>>> >>>>> I am trying to buy a GPS based NTP server like this one >>>>> >>>>> https://timemachinescorp.com/product/gps-time-server-tm1000a/ >>>>> >>>>> but I will be placing this inside a data center, do these need an actual >>>>> view of a sky to be able to get signal or will they work fine inside a >>>>> data center building? if you have any other hardware requirements to be >>>>> able to provide stable time service for hundreds of customers, please let >>>>> me know. >>>> >>>> [ with my hobby-hat on … ] >>>> >>>> tl;dr: if any of the below is too much work, just run reasonably well >>>> monitored NTP server syncing from other NTP servers. If you want more than >>>> that, you need to see the sky. Don’t do the CDMA thing. >>>> >>>> Depending on your requirements having the antenna in the window may or may >>>> not be satisfactory. If it’s fine you probably could just have done a >>>> regular NTP server in the first place. For long swaths of the day you >>>> might not see too many satellites which will add to the uncertainty of the >>>> signal. >>>> >>>> Meinberg’s GPS antenna has a bit more smarts which helps it work on up to >>>> 300 meters on RG58 or 700 meters on RG213. (They also have products that >>>> use regular L1 antennas with the limitations Bryan mentioned). >>>> >>>> https://www.meinbergglobal.com/english/products/gps-antenna-converter.htm >>>> >>>> They also have a multi-mode fiber box to have the antenna be up to 2km >>>> from the box or 20km with their single mode fiber box, if you have fiber >>>> to somewhere else where you can see the sky and place an antenna. >>>> >>>> It will be more than the one you linked to, but their systems are very >>>> reasonably priced, too. For “hundreds of customers” whatever is the >>>> smallest/cheapest box they have will work fine. Even their smallest models >>>> have decent oscillators (for keeping the ticks accurate between GPS >>>> signals). >>>> >>>> The Meinberg time server products (I am guessing all of them, but I’m not >>>> sure) also have a mode where they poll an upstream NTP server aggressively >>>> and then steer the oscillator after it. I haven’t used it in production, >>>> but it worked a lot better than it sounded like it would. (In other >>>> words, even without GPS it’s a better time server than most systems). >>>> >>>> >>>> Ask >> >> -- >> Harlan Stenn >> http://networktimefoundation.org - be a member! -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: NTP question
On 5/2/2019 9:13 AM, James R Cutler wrote: >> On May 2, 2019, at 10:59 AM, William Herrin > <mailto:b...@herrin.us>> wrote: >> >> On Wed, May 1, 2019 at 7:03 PM Harlan Stenn > <mailto:st...@nwtime.org>> wrote: >> >> It's not clear to me that there's anything *wrong* with using the >> pool, >> especially if you're using our 'pool' directive in your config file. >> >> >> The one time I relied on the pool I lost sync a year later when all >> three servers the configuration picked withdrew time services and the >> still-running ntp client didn't return to the names to find new ones. >> Wonderful if that's fixed now but the pool folks argued just as >> strongly for using it back then. >> >> Also, telling the security auditor that you have no idea who supplies >> your time source is pretty much a non-starter. You can convince them >> of a lot of things but you can't convince them it's OK to have no idea >> where critical services come from. >> >> That's what's wrong with the pool. >> >> Regards, >> Bill Herrin >> >> >> -- >> William Herrin her...@dirtside.com >> <mailto:her...@dirtside.com> b...@herrin.us <mailto:b...@herrin.us> >> Dirtside Systems . Web: <http://www.dirtside.com/> > > I have only ever used the pool as a supplement to other servers. Here is > a snippet from ntp.conf that was found in the bottom of a locked filing > cabinet stuck in a disused lavatory with a sign on the door saying > 'Beware of the Leopard.’ * > > #External Time Synchronization Source Servers > # > servertick.usno.navy.mil# open access > servertime.apple.com <http://time.apple.com># open access > serverTime1.Stupi.SE# open access > serverntps1-0.uni-erlangen.de <http://ntps1-0.uni-erlangen.de># open > access > server0.pool.ntp.org <http://0.pool.ntp.org># open access > server1.pool.ntp.org <http://1.pool.ntp.org># open access > server2.pool.ntp.org <http://2.pool.ntp.org># open access I recommend you replace the above 3 lines with: pool CC.pool.ntp.org where CC is an appropriate country code or region. H -- > servernist1-nj2-ustiming.org <http://nist1-nj2-ustiming.org># open > access > servernist1-chi-ustiming.org <http://nist1-chi-ustiming.org># open > access > servernist1-pa-ustiming.org <http://nist1-pa-ustiming.org># open access > # > > > I have not kept up with pool changes since then. > > *Apologies to Douglas Adams -- Harlan Stenn, Network Time Foundation http://nwtime.org - be a Member!
Re: NTP question
On 5/2/2019 7:59 AM, William Herrin wrote: > On Wed, May 1, 2019 at 7:03 PM Harlan Stenn <mailto:st...@nwtime.org>> wrote: > > It's not clear to me that there's anything *wrong* with using the pool, > especially if you're using our 'pool' directive in your config file. > > > The one time I relied on the pool I lost sync a year later when all > three servers the configuration picked withdrew time services and the > still-running ntp client didn't return to the names to find new ones. > Wonderful if that's fixed now but the pool folks argued just as strongly > for using it back then. Were you using 'server' entries in your ntp.conf file or a 'pool' directive? > Also, telling the security auditor that you have no idea who supplies > your time source is pretty much a non-starter. You can convince them of > a lot of things but you can't convince them it's OK to have no idea > where critical services come from. I'm not saying you *should* use the pool, or that you should *only* use the pool. The pool *can* be used responsibly. And I suspect Ask and his crew have documented things well enough that you could point an auditor at the docs for the 'pool' directive and the monitoring efforts that the Pool does, and between that and peering with your other internal S2 sites and some well-chosen external site and perhaps some local refclocks you would be in fine shape. > That's what's wrong with the pool. > > Regards, > Bill Herrin > > > -- > William Herrin ........ her...@dirtside.com > <mailto:her...@dirtside.com> b...@herrin.us <mailto:b...@herrin.us> > Dirtside Systems . Web: <http://www.dirtside.com/> -- Harlan Stenn, Network Time Foundation http://nwtime.org - be a Member!
Re: Paging anyone from ntpd.org
On 12/30/2019 8:32 PM, Seth Mattinen wrote: > On 12/30/19 8:22 PM, Seth Mattinen wrote: >> Is anyone from ntpd.org on here? You're pointing DNS at me for some >> reason. That zone (ntpd.org) isn't in our system. Your NS looks odd >> too, *.darkness-reigns.net and .nl? Is that legit? I don't know what >> it was before because I've never looked, but that seems off. >> >> > > nevermind, I'm tired and confused ntpd.org with ntp.org. Just going to > wildcard *.ntpd.org to 127.0.0.1 and go back to sleep. I did think about replying, saying "Just to be clear, this isn't about ntp.org." -- Harlan Stenn, Network Time Foundation http://nwtime.org - be a Member!
Re: Paging anyone from ntpd.org
On 12/31/2019 7:21 AM, Seth Mattinen wrote: > On 12/31/19 1:32 AM, Harlan Stenn wrote: >> On 12/30/2019 8:32 PM, Seth Mattinen wrote: >>> On 12/30/19 8:22 PM, Seth Mattinen wrote: >>>> Is anyone from ntpd.org on here? You're pointing DNS at me for some >>>> reason. That zone (ntpd.org) isn't in our system. Your NS looks odd >>>> too, *.darkness-reigns.net and .nl? Is that legit? I don't know what >>>> it was before because I've never looked, but that seems off. >>>> >>>> >>> >>> nevermind, I'm tired and confused ntpd.org with ntp.org. Just going to >>> wildcard *.ntpd.org to 127.0.0.1 and go back to sleep. >> >> I did think about replying, saying "Just to be clear, this isn't about >> ntp.org." >> > > > What I did learn though there are a lot of people configuring their NTP > with servers that are identical to the legitimate *.ntp.org names, > except they're mistyping ntpd instead of ntp. Enough to generate >2Gbps > worth of query traffic (pointed at a DNS server with a 1gbps interface). > > I have to admit I'm kind of curious how many unique clients that would > be if I answered back with a working IP address instead of localhost. If the folks who registered ntpd.org are OK with it, we'd be happy to take that domain over and do right by it. -- Harlan Stenn, Network Time Foundation http://nwtime.org - be a Member!
Re: NIST NTP servers
Leo Bicknell writes: > ... > > The correct answer here is to run multiple NTP servers in your > network. And by servers I mean real servers, with good quality > oscellators on the motherboard. Then configure them to talk to > _many_ sources. You need 4 sources of time minimum to redundantly > detect false tickers. If you're serious about it then find ~10 > Stratum 1 sources (ideally authenticated and from trusted entities), Byzantine General's problem. With 3 sources you can detect *1* false ticker. But if one of those becomes unreachable you only have 2 time sources. Dilemma. With 4 sources you run the risk of 2 going one way, and 2 going another way. This happened to several folks recently, when some sites put NTP servers on the 'net that offered leap-smeared time. That's really a different problem where one of the effects is that it causes "time islands". > one of which could be GPS as several have suggested. You'll then > have high quality false ticker rejection. For extra points, use GPS receivers from different manufacturers, using whatever "variety" you can get for all of the components involved. Are you mounting each GPS receiver inside a coffee can to prevent drive-by jamming? Are the cables properly grounded? Using gas discharge tubes? Periodically tested/inspected? How much fun do you want to have thinking about all of these cases? > Configure all of your devices to get NTP from the servers you run > using authentication. Yes, and properly monitor your ntpd instances. -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: NIST NTP servers
Sharon Goldberg writes: > Well, if you really want to learn about the NTP servers a target is using > you can always just sent them a regular NTP timing query (mode 3) and just > read off the IP address in the reference ID field of the response (mode 4). Unless the server is an IPv6 server. This trick only works for IPv4. And we have a fix for all of this that will be out soon. -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: NIST NTP servers
Harlan Stenn writes: > Sharon Goldberg writes: > > Well, if you really want to learn about the NTP servers a target is using > > you can always just sent them a regular NTP timing query (mode 3) and just > > read off the IP address in the reference ID field of the response (mode 4). > > Unless the server is an IPv6 server. This trick only works for IPv4. > > And we have a fix for all of this that will be out soon. Also, the attacker will need to know the correct origin timestamp for the brief window where that attack will work, and even if this happens either the client or the server will see syslog entries alerting to the abuse (if folks are running new enough versions of ntpd). H
Re: Leap Second planned for 2016
On 7/8/16 4:47 PM, Saku Ytti wrote: > On 9 July 2016 at 02:27, Jared Mauch wrote: >> Time is actually harder than it seems. Many bits of software break in >> unexpected ways. Expect the unexpected. > > Aye. How many have written code like this: > > start = time(); > do_something(); > elapsed = time() - start; > > Virtually all code dealing with passage of time assumes time moves > only forward, I'm amazed we don't see more issues during leap seconds. > Portable monotonic time isn't even available in many languages > standard libraries. > > Hopefully they'll decide in 2023 finally to get rid of leap seconds > from UTC. Then GPS_TIME, TAI and UTC are all same with different > static offset. How about you run your systems on TAI or satellite time? -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: Yet another NTP security bug we fixed before the CVE issued
"Eric S. Raymond" writes: > ... Yawn. We disabled interleave a while ago. Interleave is the best way to get the next major step in accurate time using the NTP Protocol. Yes, it needs work. A reference implementation is where this work happens. Yes, we have another release about to happen. Mostly "security" bugs that folks will not see, if they're being at all responsible. Eric, you are loved and appreciated, and respected and admired. -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: Recent NTP pool traffic increase
On 12/20/16 7:27 PM, Laurent Dumont wrote: > I do think that the point of the Pool network is to be used by both > consumers and vendors. And as mentioned before, there is a process if > you are a vendor and want to use the pool within a commercial product. I > have 3 NTP servers running and I don't really care who is using it. > > That said, setting up your own infrastructure is also worth considering > if it's a business critical feature. I assume that a Snapchat app that > fails to have accurate time or correct itself could be abused. > > To be honest, the fact that NTP is still something managed by volunteers > and not a regulated entity (a bit like DNS) is mind boggling. Time *is* managed by regulated entities - the National Time Labs. And Network Time Foundation's NTP Project (the reference implementation for NTP) could do lots more if we had a useful budget. Folks pay money for DNS registrations. There's no revenue stream around "time". Help us get enough support to NTF, and we'll have the staff and infrastructure to do more for folks. -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: Recent NTP pool traffic increase
On 12/20/16 9:21 PM, Royce Williams wrote: > On Tue, Dec 20, 2016 at 8:19 PM, Royce Williams > wrote: >> On Tue, Dec 20, 2016 at 8:04 PM, Yury Shefer wrote: >>> >>> Google announced public NTP service some time ago: >>> https://developers.google.com/time/ >> >> Leap smearing does look interesting as way to sidestep the >> potentially-jarring leap-second problem ... but a note of caution. >> >> I've had multiple time geeks tell me that leap-smearing is pretty >> different from strict-RFC NTP, and Google themselves say on that page: >> >> "We recommend that you don’t configure Google Public NTP together with >> non-leap-smearing NTP servers." >> >> So it looks like we shouldn't mix and match. And since most folks >> should probably want some heterogeneity in their NTP, it may be a >> little premature to jump on the leap-smear bandwagon just yet. >> >> I'm vague on the details, so I could be wrong. > > This is informative: > > https://docs.ntpsec.org/latest/leapsmear.html > >> Does anyone know of any other (non Google) leap-smearing NTP implementations? The NTP Project has had a leap-smear implementation for a while. We also have a proposal for a REFID that indicates the provided time is a leap-smear time, and Network Time Foundation is working on a new timestamp format and API that will easily allow time exchange between systems using different timescales. -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: Recent NTP pool traffic increase
On 12/22/16 4:11 PM, Ask Bjørn Hansen wrote: >> On Dec 20, 2016, at 8:02 PM, Harlan Stenn >> wrote: >> >>> On 12/20/16 7:27 PM, Laurent Dumont wrote: To be honest, the fact >>> that NTP is still something managed by volunteers and not a >>> regulated entity (a bit like DNS) is mind boggling. >> >> Time *is* managed by regulated entities - the National Time Labs. > > That was pretty clearly not what Laurent was talking about. Not all that clearly, at least to me. >> And Network Time Foundation's NTP Project (the reference >> implementation for NTP) could do lots more if we had a useful >> budget. >> >> Folks pay money for DNS registrations. There's no revenue stream >> around "time". >> >> Help us get enough support to NTF, and we'll have the staff and >> infrastructure to do more for folks. > > What does the NTF have to do with the NTP Pool (or the “recent NTP > pool traffic increase”)? Nothing. But it has to do with companies putting NTP into their products. When they do this with problematic configurations, it causes trouble. In this case, it was trouble for the Pool project. The NTP Pool project certainly isn't the place to solve this issue. > The NTP Pool is run by volunteers, as you very well know. Both the > management and DNS system and the thousands of people who contribute > their NTP service to the system. (And we manage on a pretty scarce > budget). What's your point? This sort of misconfiguration will happen and the NTP Pool Project clearly isn't the place to solve this problem overall. It *is* something NTF is in a position to address. And we're almost exclusively an overstretched volunteer organization, too, as you very well know. Do we have different goals around this issue? -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: Recent NTP pool traffic increase
On 12/22/16 5:25 PM, Royce Williams wrote: > On Thu, Dec 22, 2016 at 4:05 PM, Harlan Stenn wrote: > >> This sort of misconfiguration will happen and the NTP Pool Project >> clearly isn't the place to solve this problem overall. It *is* >> something NTF is in a position to address. > > Harlan, could you be more specific about how NTF can address this? > Would it require modification of the NTP protocol, or something else? Network Time Foundation has two projects to directly help here. One is a Certification and Compliance program. The other is a project to analyze/produce BCP-compliant ntp.conf files. Due to limited resources we're making slow progress on each of these. That's the backwards way of saying we can make much more useful progress on these and other issues (like the development of NTPv5 and the General Timestamp API) if we can get useful support. With more support we can also make more pool servers available. -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: Recent NTP pool traffic increase (update)
Hi Fujimura-san, On 12/24/16 6:11 AM, FUJIMURA Sho wrote: > Hello, > > I know 133.100.9.2 and 133.100.11.8 are listed. > The Server Contact is old information. > So, I sent e-mail to webmas...@ntp.org a few times. > But, I have't received e-mail from them. > I'd like them to change the information. > Is there the person knowing the contact information to ntp.org? I don't recall seeing the emails you sent to webmaster, but we do have a new group of folks watching the Servers web. We would be happy to work with you to give you access to those entries so you can update them. -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: Recent NTP pool traffic increase
On 12/30/16 11:26 AM, Emille Blanc wrote: > Ah, but who do you trust? Trump, Putin, or Xi's clock? > > That said, we use a Stratum2 clock for our AS, which syncs using GPS > at $dayjob. So... I guess we trust Trump's clock. > > Perhaps there's a market for a device that takes GPS, GLONASS, and > Beidou, and references the three for sanity checks in the event of > $unforseen_circumstance. Assuming such a thing were possible - > admittedly I know little about GLONASS, and even less about Beidou. Why are you trying to re-invent a properly configured S2 NTP server? H -- > "Perhaps some kind of death clock?" > > -Original Message- From: NANOG > [mailto:nanog-bounces+emille=abccommunications@nanog.org] On > Behalf Of Allan Liska Sent: December-30-16 11:09 AM To: Majdi S. > Abbas; Laurent Dumont Cc: nanog@nanog.org Subject: Re: Recent NTP > pool traffic increase > > > On 12/30/2016 at 1:20 PM, "Majdi S. Abbas" wrote:On Thu, Dec 22, > 2016 at 11:31:08PM -0500, Laurent Dumont wrote: >> What I mostly meant is that there should be a regulated, > industry-wide >> effort in order to provide a stable and active pool program. With > the >> current models, a protocol that is widely used by commercial >> devices > is >> being supported by the time and effort of volunteers around the > world. > > Who's authoritative for time? Even the national labs aren't -- UTC is > figured well after the fact. > > In the United States that would the United States Naval Observatory > (USNO) Master Clock (http://tycho.usno.navy.mil/). You can read > more about it here: > http://motherboard.vice.com/read/demetrios-matsakis-and-the-master-clock > > allan > > -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: Temperature monitoring
On 7/13/17 7:33 PM, Dovid Bender wrote: > All, > > We had an issue with a DC where temps were elevated. The one bit of > hardware that wasn't watched much was the one that sent out the initial > alert. Looking for recommendations on hardware that I can mount/hang in > each cabinet that is easy to set up and will alert us if temps go beyond a > certain point. > > TIA. > > Dovid Are you running ntpd on your boxes? See what happens when you plot its drift against other temperature sensors, the closer to the clock chip the better. If you do this on enough boxes, you should have an easy time seeing what happens on boxes where you have an easier time watching ntpd's drift value than you have watching a nearby dedicated temperature sensor. -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: OpenNTPProject.org
Kate Gerry writes: > Just add these to your ntp.conf configuration then restart the service: (Wo= > rks with all default installations that I've found) > > restrict default kod nomodify notrap nopeer noquery > restrict -6 default kod nomodify notrap nopeer noquery KOD only works with "limited" in the release of NTP you are using. H
Re: OpenNTPProject.org
If somebody has contacts at Juniper who is involved in this, I'd like to get their contact information. -- Harlan Stenn http://networktimefoundation.org - be a member!
NTP DRDos Blog post
Folks, I just posted http://nwtime.org/ntp-winter-2013-network-drdos-attacks/ . In general we've never allowed comments to blog posts on that site; we're currently discussing if we should allow them for this post. I'd love to hear any feedback about the post. Thanks... -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: Filter NTP traffic by packet size?
"Dobbins, Roland" writes: > Operators are using this size-based filtering to effect without > breaking the world. As a reality check, with this filtering in place does "ntptrace" still work? H
Re: Net Neutrality...
Brett Glass writes: > At 12:19 PM 7/15/2014, Barry Shein wrote: > > >There exists a low and high (practical) bandwidth range within which > >it simply doesn't make any difference to a given business model. > > Very true. And there's another factor to consider. > > Estimates of the maximum bandwidths of all the human senses, combined, > range between the capacity of a T1 line (at the low end) and > about 4 Mbps (at the high end). A human being simply is not wired to > accept more input. (Yes, machines could digest more... which means that > additional bandwidth to and from the home might be useful for the purpose > of spying on us.) What does this imply about the FCC's proposal to > redefine "broadband" as a symmetrical 10 Mbps? For single-person households, nefarious things. For households (or small businesses) things change. And while most folks will not need those uplink speeds, for others it can be real useful. And yes, there is room for abuse. H
Re: Muni Fiber and Politics
Greg Walden (R-OR) is similarly funded by the cable and telecom folks, and is also loud and clear that he thinks we should forget about net neutrality and let the companies do what is best. H
Security release scheduling
I'm looking for some general "calendar" help to use for our security release scheduling process. Something that usefully accounts for clients all over the world. By "usefully accounts" I mean that we want to be able to have reasonable confidence that we're not going to pick a public release date that is a national holiday for one country, and we're also not going to avoid a date because it's "let's order pizza" day somewhere else, but the purpose of the holiday is obscured because of language translation issues. I figure this is something others must have solved, and I'm hoping some folks on this list might be able to offer me some pointers. -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: Security release scheduling
On 9/28/15 11:08 PM, Mark Andrews wrote: > In message <560a13e6.7060...@nwtime.org>, Harlan Stenn writes: >> I'm looking for some general "calendar" help to use for our security >> release scheduling process. Something that usefully accounts for >> clients all over the world. >> >> By "usefully accounts" I mean that we want to be able to have reasonable >> confidence that we're not going to pick a public release date that is a >> national holiday for one country, and we're also not going to avoid a >> date because it's "let's order pizza" day somewhere else, but the >> purpose of the holiday is obscured because of language translation issues. >> >> I figure this is something others must have solved, and I'm hoping some >> folks on this list might be able to offer me some pointers. > > There's a national holiday basically everyday of the year. > > http://www.timeanddate.com/holidays/ Thanks, Mark! So much for that idea... I guess the best we can hope for is to minimize the impact. -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: Security release scheduling
Good info, Barry - thanks! I appreciate your offer, too! H -- On 9/29/15 12:39 AM, Barry Greene wrote: >> >> Hi Harlan, > > The general principle is look out for the major network lock downs. Some > times that is overlap with holidays. Other times it is over financial close > months. > > My personal $.02 is to avoid major vulnerability disclosures in December, > during Lunar New Year weeks, during Ramadan, and June. Some would also > include August (Euro holidays). > > But these days there are timers given by the vulnerability finder (or CERT > Team) and conference disclosures (security rock stars) that drive the > disclosure to a time which is not optimal to the people who have to roll out > the remediation. > > In essence, write a disclose policy, put it on your website, and be open for > improvements based on input from your constituents. Do your best. That is all > your can do. > > Barry > > PS - Let me know if you need help writing the disclosure policy. > > > -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: Advance notice - H-root address change on December 1, 2015
On 11/16/15 4:55 PM, Jared Mauch wrote: > This action by red hat is nice from a stability perspective but > infuriates many standards derived folks like ISC/BIND and NTP amongst > others as a version number means something to them. > > This dialogue is typically broken from both sides as expectations are > different and bug reports get lost between the OS packaging and the > supplier packaging. Sometimes this is good other times it can be > quite bad. > > I would prefer to see fewer variants and better bug fixes across the > board but the enterprise side often want a specific version number > and expect fixes that the upstream maintainers don't want to keep the > same version numbering for "that fix" or add a stealth feature and > red hat may not want to pick it up... > > Not saying who is right or wrong but these views sometimes drive the > intractable situation where 4.2.6 is shipped for NTP from red hat (as > example) but it is EOL from the NTP.org folks. Red Hat has known for YEARS that we only have the resources to support one stable code branch, and that if they want support for older versions we can offer that at cost. Nobody has ever approached us about support for older releases of NTP. The folks who scream the loudest about our not supporting older releases are the folks who charge their customers money for support. These same companies do nothing to support us. Anybody can see the list of companies that support NTP at: http://nwtime.org/current-members-donors/ The free software companies listed there do not give us any money. >> On Nov 16, 2015, at 7:16 PM, Mark Andrews wrote: >> >> The point of putting out maintainence releases is to fix bugs in >> the existing code not to introduce features. We leave features to >> the .0 releases. The [func] below are bug fixes / security fixes. >> >> Even with 60% of the changes one is missing a huge number of bug >> fixes. In NTP's case, we fixed over 1100 things between 4.2.6 and 4.2.8. Skippy (Harlan'e evil twin brother)
Re: BCOP appeals numbering scheme -- feedback requested
Rob Seastrom writes: > The wiki/living document approach others have suggested seems like a > poor one to me, for the same reason that I dislike the current trend > of "there's no release tarball, major release, point release, or > regression testing - just git clone the repository" in free software > development. And I like wikis for some things, here it might work (I haven't looked) and I still do release tarballs with version numbers and some (we're actively adding more) regression/unit/functional testing. > Releng is hard and thankless :) > but adds enormous value and > serves as a forcing function for some level of review, cursory though > it may be. I think so too. Hey everybody, please support Network Time. Spread the word. OK, I said it. -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: Supporting network time software development/maintenance (was: Re: BCOP appeals numbering scheme -- feedback requested)
Rob Seastrom writes: > New subject so as to minimize threadjacking, not the least because > this is important stuff. > > Harlan Stenn writes: > >>> Releng is hard and thankless but adds enormous value and >>> serves as a forcing function for some level of review, cursory though >>> it may be. >> >> I think so too. >> >> Hey everybody, please support Network Time. Spread the word. OK, I >> said it. > > The check, as we say, is in the mail (literally). Very much appreciated! > I wish I'd known about Network Time Foundation before you and I > started corresponding a little over a week ago about GPS cores. > > > Harlan Stenn > > http://networktimefoundation.org - be a member! > > NANOG colleagues, I know it can be hard getting the employer to pony > up for organizations like this, but if correct timestamps in your logs > and being able to correlate your packet captures are something you > find personally valuable and sanity-preserving, you might consider an > individual membership. I know it's pretty important to me... > > http://nwtime.org/individual-membership-application/ And Sue Graves is working with NTF to help with all of this, so if there is anything we can do to help show value to your employers please let her know. The individual donations are great - they've quickly added a bit over 2 weeks' time to the "financial clock" regarding my "full time" efforts (apparently my definition of full time is a bit unusual). Please help keep those coming in. it's going to take Institutional members to provide the more significant resources NTF needs to bring the bigger goals on-line, things like even faster NTP development, completing Ntimed, getting the General Timestamp API spec'd and implemented, and our Certification and Compliance programs operational: http://nwtime.org/membership http://nwtime.org/join http://nwtime.org/projects We've heard from some institutions that they'll be joining us, but nothing has happened there yet. Thanks a bunch, folks! Please keep spreading the word. -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: [SECURITY] Application layer attacks/DDoS attacks
Just to ask, what is the expected effect on DDoS attacks if folks implemented BCP38? How does the cost of implementing BCP38 compare to the cost of other solution attempts? H
Re: REMINDER: LEAP SECOND
Saku Ytti writes: > Hopefully this is last leap second we'll ever see. Non-monotonic time > is an abomination and very very few programs measuring passage of time > are correct. Even those which are, usually are not portable, most > languages do not even offer monotonic time in standard libraries. > Canada, China, England and Germany, shame on you for opposing > leapsecondless UTC. It's a problem with POSIX, not UTC. UTC is monotonic. > Next year hopefully GPSTIME. TAI and UTC are the same thing, with different > static offset. The General Timestamp API that Network Time Foundation is working on can solve this problem. People use different timescales for different reasons. The Agile folks like the "pigs and chickens" analogy: in a bacon and egg breakfast, the chicken is invested while the pig is committed. It's lame for a chicken to dictate to a pig. It's lame to change an existing Standard. Leave that one alone and choose to follow a new/different Standard. If you don't have a system that can properly handle leapseconds, there are several solutions to this, including: - implement DLM's leap second process in the kernel, described over 20 years ago - use the posix-right timezone files - help Network Time Foundation get the General Timestamp API implemented and deployed, which will let folks use whatever timescale they want. -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: REMINDER: LEAP SECOND
Bad idea. When restarting ntpd your clocks will likely be off by a second, which will cause a backward step, which will force the problem you claim to be avoiding. There are plenty of ways to solve this problem, and you just get to choose what you want to risk/pay. -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: REMINDER: LEAP SECOND
Baldur Norddahl writes: > On 19 June 2015 at 23:58, Harlan Stenn wrote: > > > Bad idea. > > > > When restarting ntpd your clocks will likely be off by a second, which > > will cause a backward step, which will force the problem you claim to be > > avoiding. > > If you are afraid that your routers will crash due to the leapsecond, > then it would help to disable the thing that you think will crash > them. Even if the router crashes when you enable it later on. Because > then you can have one router crash at a time and have it happen in a > service window where you are ready for it. Instead of having all > routers in your whole network crash at exactly the same time. That' seems fair, as long as you turn off the time stuff only on your routers, and I'm assuming this is on routers that don't have supported software. H
Re: REMINDER: LEAP SECOND
Mel Beckman writes: > Harlan, > > This is cisco's recommended workaround, the ultimate conclusion of an exhau= > stive study of all Cisco firmware and after detailed post mortem analysis o= > f two previous Leap seconds: > > https://tools.cisco.com/bugsearch/bug/CSCut33302 Fair enough. And I've been trying to get Cisco to work with us for very many years. They have yet to show any interest. But they'd be paying us for that. We have no leverage with them. But folks who are paying Cisco for support? For the number of years Cisco has been using NTP and for the number of product lines that use it, they could certainly do better. I know they were current when I did the port for the MDS switch line, years ago. -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: REMINDER: LEAP SECOND
shawn wilson writes: > ... I mean letting computers figure out slower earth rotation on the > fly would seem more accurate than leap seconds anyway. And then all of > us who do earthly things and would like simpler libraries could live > in peace. Really? Have you looked in to those calculations, and I'm only talking about the allegedly predictable parts of those calculations, not things like the jetstream, the circumpolar currents, or earthquakes. H
Re: REMINDER: LEAP SECOND
Tony Finch writes: > Harlan Stenn wrote: > > > It's a problem with POSIX, not UTC. > > > > UTC is monotonic. > > The problems are that UTC is unpredictable, and it breaks the standard > labelling of points in time that was used for hundreds (arguably > thousands) of years before 1972. You mean back when seconds were rubbery, and before the earth's rotational speed could be easily and accurately measured, or at least when the wobbles at that level of accuracy became so noticeable that they could no longer be ignored? H
Re: REMINDER: LEAP SECOND
Doug Barton writes: > This is an OpenPGP/MIME signed message (RFC 4880 and 3156) > On 6/19/15 2:58 PM, Harlan Stenn wrote: >> Bad idea. >> >> When restarting ntpd your clocks will likely be off by a second, >> which will cause a backward step, which will force the problem you >> claim to be avoiding. >> >> There are plenty of ways to solve this problem, and you just get to >> choose what you want to risk/pay. > > You misunderstand the problem. :) The problem is not "clock skips > backward one second," because most of the time that's not what > happens. The problem is that most software does not handle it well > when the clock ticks ... :59 :60 :00 instead of ticking directly from > :59 to :00. POSIX NEVER shows :60. > THAT problem is avoided by temporarily turning off NTP and then > turning it back on again when "the coast is clear." Most software can > handle the "clock skips forward or backwards one second" problem > fairly robustly,= and as Baldur pointed out by doing the reset in a > controlled manner you greatly reduce your overall risk. Time going backwards is deadly to a number of applications. But apparently not to applications you care about. You're also not doing anything where somebody is going to get sued because a timestamp is off by a second. There are people for whom this is a very real risk. H
Re: REMINDER: LEAP SECOND
This stuff can make my head explode. When a leap second is added, like on 30 June 2015 at the last second of the day, POSIX insists that the day still have 86400 seconds in it. This makes the day longer by one second, so time has to either slow down or move backwards. The "dumb" way to do this is to step the clock back by 1 second at the instant before the stroke of midnight. The allegedly better way to do this would be to stop the clock a bit before midnight, and hold the time for 1 second. To continue providing monotonic time, every time somebody says "what time is it" during that holding period one would want to bump the time by the smallest amount possible, usually 1 nanosecond (assuming the kernel keeps time in nanoseconds). Ideally you wouldn't want to add enough nanoseconds to cause the clock to roll over into the next day "too early". But apparently nobody has implemented this, even though Prof. Mills described it in RFC-i-forget about 20 years ago. This is mostly because POSIX deals with absolute time and not relative time. In the unlikely event of a leap second deletion, there would be no 23:59:59, so when the clock is about to strike 23:59:59 it's OK to add an extra second to the clock to effectively have the time "jump" from 23:59:58 to 00:00:00. This is still a monotonic increment in time. Whatever you decide to do, I recommend you actually test it out to see if it behaves the way you think it will. You have a whole week still. H
Re: NANOG Digest, Vol 89, Issue 24
Alex Hardie writes: > Not to inject more confusion - but GPS and NTP are noted in the > thread... but not PTP (IEEE1588)? I don't belive PTP generally uses UTC as a timescale. H
Re: REMINDER: LEAP SECOND
shawn wilson writes: > On Jun 23, 2015 6:26 AM, "Nick Hilliard" wrote: > > > > > > > Blocking NTP at the NTP edge will probably work fine for most situations. > > Bear in mind that your NTP edge is not necessarily the same as your > network > > edge. E.g. you might have internal GPS / radio sources which could > > unexpectedly inject the leap second. The larger the network, the more > > likely this is to happen. Most organisations have network fossils and ntp > > is an excellent source of these. I.e. systems which work away for years > > without any problems before one day accidentally triggering meltdown > > because some developer didn't understand the subtleties of clock > monotonicity. > > > > NTP causes jumps - not skews, right? Left to its default condition, ntp will step/jump a change in excess of 128msec. If you want to slew the clock instead, a 1 second correction will take a little over 33 minutes' time to apply. I don't understand why people believe that stopping ntpd for a few minutes while the leap second is applied will help. If the system clock keeps good time, it will *still* be about 1 second ahead when ntpd is restarted, and that will trigger a backward step which is fatal to a number of applications. H
Re: REMINDER: LEAP SECOND
Matthew Huff writes: > A backward step is a known issue and something that people are more > comfortable dealing with as it can happen on any machine with a noisy > clock crystal. A clock crystal has to be REALLY bad for ntpd to need to step the clock. > Having 61 seconds in a minute or 86401 seconds in a day is a different > story. Yeah, leap years suck too. And those jumps around daylight savings time. H
Re: leap second outage
Joe writes: > A leap sec causing issues. For about 40 years now, there have been > these leap seconds to no real issue. All of these are "go-forwards" No, they're all "go-backwards" events. That's no big deal to things that don't care about monotonic time, or to folks who aren't in violation of something if their timestamps are off by a second. What I'm about to say may not be as stupid as it sounds: The problems here aren't problems for cases where it's not a problem. It is a problem where it *is* a problem. It's a case where one person's signal is another person's noise. H
Re: leap second outage
Mikael Abrahamsson writes: > This is similar to the jiffycounter wrapping, since this doesn't happen > that often, it's not commonly tested for. Good way is to start the jiffy > counter so it wraps after 10 minutes of uptime. That way you'll run into > any bugs quickly. Either we should abolish the leap second or we should > make leap second adjustments (back and forth) on a monthly basis to > exercise the code. > > This is a hard sell though... and it's perversely interesting. It would even be tolerable when the difference between UTC and UT1 is such that the insertions and deletions maintain the +/- .9 s difference. There would even be enough time to warn folks about this. H
Re: leap second outage
Jimmy Hess writes: > On Wed, Jul 1, 2015 at 12:38 AM, Mikael Abrahamsson wrote: > > quickly. Either we should abolish the leap second or we should make leap > > second adjustments (back and forth) on a monthly basis to exercise the code > . > > See maybe there should some day be building codes for > commercially marketed software that provide minimum independent > formal testing to be done by licensed independent testers, including > leap seconds and such. ^_^ And NTF's Certification and Compliance programs are going to do this. At least as soon as NTF has the resources to get this moving. > The leap second issues are possibly rare and intermittent, therefore, > having a few per month is not necessarily giving adequate exposure > to code paths that may go wrong during an insert/del event. If they happened every 6 month's time that would be often enough, but the earth hasn't slowed down that much yet. There will be enough times that we could insert or delete one every month and still have |UT-UT1| be under .9 seconds. If it was announced that "starting in 6 months' time we'll be inserting or deleting a leap second every month or so that would give folks enough time to prep for it, and I'm pretty confident that the leap-second would soon become a non-event. > There's never been a negative leap second, only insertions, but how > deletions are implemented might expose new bugs, since there hasn't > been one before, And you can only have one leap per 24 hours, > positive or minus, pick one. Yup. > & Shouldn't this kind of 'exercise' be done during the QA process > before releasing new system software, rather than mucking with clock > accuracy? leap second handling is a "mechanism" question. Which one to choose is a "policy" question. IMO, a vendor should provide adequate mechanism. The customer should get to choose policy. > There is a recent article with some Leap Second 'stress testing' code: > https://access.redhat.com/articles/199563 > > > Readily available test methods are available, there ought to be > little legitimate excuse for anyone writing serious software that has > long-running processes or threads not to include evaluation for > possible leap second issues and other possible clock-related issues > such as clock stepping, DST, and Year 2038 in their standard smoke > tests Yes. And even so, testing these things takes time and equipment. -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: REMINDER: LEAP SECOND
Mike Hammett writes: > It looks to have only affected the CCR line and only those running the > NTP and not the SNTP package. Any idea what version of NTP or what their configuration looked like? H
Re: NTP versions in production use?
Resending... On 7/10/15 12:29 PM, Harlan Stenn wrote: > I'm trying to build a list of the versions of NTP that are in active use > on various active pieces of network gear. > > I know that Cisco, for example, uses NTP in around 10 different product > lines, but I don't know what versions of NTP are in current use. > > I'm also curious about the answers here for Juniper and other network > gear providers. That would include routers, switches, and other types > of gear. > > If you have information about this I'd appreciate your letting me know. > -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: NTP versions in production use?
Dovid, Thanks, and I'm kinda stunned that folks are running such ancient versions of NTP. https://support.ntp.org/bin/view/Dev/ReleaseTimeline 4.2.0 was EOL'd in June of 2006, and we've fixed about 3,000 issues in the codebase since then. H On 7/11/15 7:58 PM, Dovid Bender wrote: > Juniper MX5 > r...@yyy.xx.net> show ntp status > status=06a4 leap_none, sync_ntp, 10 events, event_peer/strat_chg, > version="ntpd 4.2.0-a Thu Mar 13 08:29:55 UTC 2014 (1)", > processor="powerpc", system="JUNOS12.3R6.6", leap=00, stratum=3, > precision=-18, rootdelay=90.375, rootdispersion=20.620, peer=29748, > refid=208.75.88.4, > reftime=d94c5338.ac6565a8 Sat, Jul 11 2015 22:45:12.673, poll=7, > clock=d94c55ad.b634aa52 Sat, Jul 11 2015 22:55:41.711, state=4, > offset=-0.428, frequency=2.394, jitter=3.505, stability=0.004 > > Juniper EX4200: > root@> show ntp status > status=c011 sync_alarm, sync_unspec, 1 event, event_restart, > version="ntpd 4.2.0-a Sat Jan 5 18:41:34 UTC 2013 (1)", > processor="powerpc", system="JUNOS11.4R6.6", leap=11, stratum=16, > precision=-18, rootdelay=0.000, rootdispersion=656381.655, peer=0, > refid=INIT, reftime=. Thu, Feb 7 2036 1:28:16.000, > poll=4, clock=d94c5a40.fa58e5f0 Sat, Jul 11 2015 23:15:12.977, state=0, > offset=0.000, frequency=0.000, jitter=0.004, stability=0.000 > > > On Fri, Jul 10, 2015 at 4:30 PM, Harlan Stenn wrote: > >> Resending... >> >> On 7/10/15 12:29 PM, Harlan Stenn wrote: >>> I'm trying to build a list of the versions of NTP that are in active use >>> on various active pieces of network gear. >>> >>> I know that Cisco, for example, uses NTP in around 10 different product >>> lines, but I don't know what versions of NTP are in current use. >>> >>> I'm also curious about the answers here for Juniper and other network >>> gear providers. That would include routers, switches, and other types >>> of gear. >>> >>> If you have information about this I'd appreciate your letting me know. >>> >> >> -- >> Harlan Stenn >> http://networktimefoundation.org - be a member! >> >> > -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: NTP versions in production use?
We will. But we're going to be asking them for support for network time. Folks like you are probably paying them for support. They'll listen more to people like you. This goes to *all* vendors who embed NTP in their products, we're not interested in in picking on anybody here. H -- On 7/11/15 8:21 PM, Dovid Bender wrote: > You would need to ask Juniper that > > > On Sat, Jul 11, 2015 at 11:17 PM, Harlan Stenn wrote: > >> Dovid, >> >> Thanks, and I'm kinda stunned that folks are running such ancient >> versions of NTP. >> >> https://support.ntp.org/bin/view/Dev/ReleaseTimeline >> >> 4.2.0 was EOL'd in June of 2006, and we've fixed about 3,000 issues in >> the codebase since then. >> >> H >> >> On 7/11/15 7:58 PM, Dovid Bender wrote: >>> Juniper MX5 >>> r...@yyy.xx.net> show ntp status >>> status=06a4 leap_none, sync_ntp, 10 events, event_peer/strat_chg, >>> version="ntpd 4.2.0-a Thu Mar 13 08:29:55 UTC 2014 (1)", >>> processor="powerpc", system="JUNOS12.3R6.6", leap=00, stratum=3, >>> precision=-18, rootdelay=90.375, rootdispersion=20.620, peer=29748, >>> refid=208.75.88.4, >>> reftime=d94c5338.ac6565a8 Sat, Jul 11 2015 22:45:12.673, poll=7, >>> clock=d94c55ad.b634aa52 Sat, Jul 11 2015 22:55:41.711, state=4, >>> offset=-0.428, frequency=2.394, jitter=3.505, stability=0.004 >>> >>> Juniper EX4200: >>> root@> show ntp status >>> status=c011 sync_alarm, sync_unspec, 1 event, event_restart, >>> version="ntpd 4.2.0-a Sat Jan 5 18:41:34 UTC 2013 (1)", >>> processor="powerpc", system="JUNOS11.4R6.6", leap=11, stratum=16, >>> precision=-18, rootdelay=0.000, rootdispersion=656381.655, peer=0, >>> refid=INIT, reftime=. Thu, Feb 7 2036 1:28:16.000, >>> poll=4, clock=d94c5a40.fa58e5f0 Sat, Jul 11 2015 23:15:12.977, state=0, >>> offset=0.000, frequency=0.000, jitter=0.004, stability=0.000 >>> >>> >>> On Fri, Jul 10, 2015 at 4:30 PM, Harlan Stenn wrote: >>> >>>> Resending... >>>> >>>> On 7/10/15 12:29 PM, Harlan Stenn wrote: >>>>> I'm trying to build a list of the versions of NTP that are in active >> use >>>>> on various active pieces of network gear. >>>>> >>>>> I know that Cisco, for example, uses NTP in around 10 different product >>>>> lines, but I don't know what versions of NTP are in current use. >>>>> >>>>> I'm also curious about the answers here for Juniper and other network >>>>> gear providers. That would include routers, switches, and other types >>>>> of gear. >>>>> >>>>> If you have information about this I'd appreciate your letting me know. >>>>> >>>> >>>> -- >>>> Harlan Stenn >>>> http://networktimefoundation.org - be a member! >>>> >>>> >>> >> >> -- >> Harlan Stenn >> http://networktimefoundation.org - be a member! >> >> > -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: NTP versions in production use?
Harlan Stenn writes: > We will. But we're going to be asking them for support for network > time. Folks like you are probably paying them for support. They'll > listen more to people like you. > > This goes to *all* vendors who embed NTP in their products, we're not > interested in in picking on anybody here. Network Time doesn't *only* need support from network equipment providers. If accurate time is important to you, or you and your customers, please pitch in. I've probably strayed offtopic here. Sorry about that. But help us anyway. H -- > -- > On 7/11/15 8:21 PM, Dovid Bender wrote: > > You would need to ask Juniper that > > > > > > On Sat, Jul 11, 2015 at 11:17 PM, Harlan Stenn wrote: > > > >> Dovid, > >> > >> Thanks, and I'm kinda stunned that folks are running such ancient > >> versions of NTP. > >> > >> https://support.ntp.org/bin/view/Dev/ReleaseTimeline > >> > >> 4.2.0 was EOL'd in June of 2006, and we've fixed about 3,000 issues in > >> the codebase since then. > >> > >> H > >> > >> On 7/11/15 7:58 PM, Dovid Bender wrote: > >>> Juniper MX5 > >>> r...@yyy.xx.net> show ntp status > >>> status=06a4 leap_none, sync_ntp, 10 events, event_peer/strat_chg, > >>> version="ntpd 4.2.0-a Thu Mar 13 08:29:55 UTC 2014 (1)", > >>> processor="powerpc", system="JUNOS12.3R6.6", leap=00, stratum=3, > >>> precision=-18, rootdelay=90.375, rootdispersion=20.620, peer=29748, > >>> refid=208.75.88.4, > >>> reftime=d94c5338.ac6565a8 Sat, Jul 11 2015 22:45:12.673, poll=7, > >>> clock=d94c55ad.b634aa52 Sat, Jul 11 2015 22:55:41.711, state=4, > >>> offset=-0.428, frequency=2.394, jitter=3.505, stability=0.004 > >>> > >>> Juniper EX4200: > >>> root@> show ntp status > >>> status=c011 sync_alarm, sync_unspec, 1 event, event_restart, > >>> version="ntpd 4.2.0-a Sat Jan 5 18:41:34 UTC 2013 (1)", > >>> processor="powerpc", system="JUNOS11.4R6.6", leap=11, stratum=16, > >>> precision=-18, rootdelay=0.000, rootdispersion=656381.655, peer=0, > >>> refid=INIT, reftime=. Thu, Feb 7 2036 1:28:16.000, > >>> poll=4, clock=d94c5a40.fa58e5f0 Sat, Jul 11 2015 23:15:12.977, state=0, > >>> offset=0.000, frequency=0.000, jitter=0.004, stability=0.000 > >>> > >>> > >>> On Fri, Jul 10, 2015 at 4:30 PM, Harlan Stenn wrote: > >>> > >>>> Resending... > >>>> > >>>> On 7/10/15 12:29 PM, Harlan Stenn wrote: > >>>>> I'm trying to build a list of the versions of NTP that are in active > >> use > >>>>> on various active pieces of network gear. > >>>>> > >>>>> I know that Cisco, for example, uses NTP in around 10 different product > >>>>> lines, but I don't know what versions of NTP are in current use. > >>>>> > >>>>> I'm also curious about the answers here for Juniper and other network > >>>>> gear providers. That would include routers, switches, and other types > >>>>> of gear. > >>>>> > >>>>> If you have information about this I'd appreciate your letting me know. > >>>>> > >>>> > >>>> -- > >>>> Harlan Stenn > >>>> http://networktimefoundation.org - be a member! > >>>> > >>>> > >>> > >> > >> -- > >> Harlan Stenn > >> http://networktimefoundation.org - be a member! > >> > >> > > > > -- > Harlan Stenn > http://networktimefoundation.org - be a member! > >
Re: NTP versions in production use?
On 7/12/15 11:31 AM, valdis.kletni...@vt.edu wrote: > On Sun, 12 Jul 2015 10:15:20 -0400, "Mike O'Connor" said: > >> :Thanks, and I'm kinda stunned that folks are running such ancient >> :versions of NTP. >> >> I suggest you get accustomed to being stunned. > > He obviously didn't see my post a few weeks back about hosts that were > looking for an NTP server that went out of service back in 1999. And yes, > some were still using NTP v1 and v2. > > There's a *lot* of stuff on very serious autopilot out there I did see it, and I was assuming it was a "local" configuration problem. This is "death by 1,000 cuts" and when I wrote my recent query I was looking for the big offenders. To me this situation goes hand-in-hand with the problems getting bcp38 deployed, and what Dan Geer talked about in his keynote speech at Black Hat 2014: http://www.youtube.com/watch?v=nT-TGvYOBpI I get that some folks have real problems with their build systems and it's hard to upgrade software tools in that environment. I know it's can be expensive to solve that problem. I'd love to find a way to have the "versioned tool chain" stuff that I implemented at Cisco/Andiamo be generally available, but I haven't found that many folks willing to support it yet and I just don't have the spare cycles to add that to my "do it for free" pile. I do know that if more companies were to use this sort of tool that the argument of "we can't patch older releases because we don't have those tools anymore and the Q/A process becomes horribly expensive" goes away. And that also means that it's far less expensive and therefore far more profitable to offer maintenance support on older software releases for much longer periods of time. But I must be missing something here as well, as I was never able to make headway with this idea when I was at Cisco. -- Harlan Stenn http://networktimefoundation.org - be a member! signature.asc Description: OpenPGP digital signature
Re: Did *bufferbloat* cause the 2010 flashcrash?
In 8/6/15 10:44 AM, William Herrin wrote: > The intermediate cause of the problem was propagation delay (including > buffer bloat) which induced an oscillating set of states in the > trading software. > > The root cause was a flipping jassack trying to out-time his > competitors by assuming a degree of instantaneity which proved untrue. > Don't do that. Don't make assumptions about network timing. You can > count on being wrong. If timing matters to your application, find a > way to continuously measure. Similar things happen when folks decide they are going to twiddle the knobs of NTP's behavior. NTP works locally, and gets/provides information globally. More or less. When folks decide to make a change in its core behavior, the usually don't consider how those changes will affect anybody else. I know enough about this to know I don't know anywhere near enough about it, so I leave the knobs alone. -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: Small Internet border router options?
On 5/13/2024 12:45 PM, Jean Franco wrote: Hi Tom. Ubiquiti EdgeRouter Infinity. What version of firmware are you using? We have never seen our box (an ER-8-XG) route packets for more than about an hour before it falls over. If you have having better luck with this and would be open to helping us, please let me know - we'd greatly appreciate it. Best regards, On Mon, May 13, 2024 at 3:54 PM Tom Samplonius <mailto:t...@samplonius.org>> wrote: What are using for small campus border routers? So four to eight 10G ports with a FIB for full scale L3? Tom -- Harlan Stenn https://www.nwtime.org/ - be a member!
