Re: Is python2 dead?
Gary said: > Please, no. Go is a garbage collected language. Just what NTPsec does not > need, random, unpredictable delays. I was thinking of the Python code in ntpclients/ and pylib/ Is there anything in there that is time sensitive? There are lots of ways to inject timing bumps before we get to garbage collecting. cache, scheduler, interrupts, CPU speed, ... Do you have any data on Go GC times? -- These are my opinions. I hate spam. ___ devel mailing list devel@ntpsec.org https://lists.ntpsec.org/mailman/listinfo/devel
Re: Is python2 dead?
> On 09/12/2023 12:00 AM PDT Hal Murray via devel wrote: > > > Gary said: > > Please, no. Go is a garbage collected language. Just what NTPsec does not > > need, random, unpredictable delays. It would appear there is a way to turn off GC under runtime/, Given this, it might be reasonable to schedule run a light GC at an odd corner of every minute and a heavier one less often. > I was thinking of the Python code in ntpclients/ and pylib/ > Is there anything in there that is time sensitive? Maybe ntpdig with time sitting or the SNMP agent? I think the premise there is everything is human speed. > There are lots of ways to inject timing bumps before we get to garbage > collecting. cache, scheduler, interrupts, CPU speed, ... > > Do you have any data on Go GC times? It is probably somewhere in the previous considerable discussion back in June/July 2021. ___ devel mailing list devel@ntpsec.org https://lists.ntpsec.org/mailman/listinfo/devel
Re: Is python2 dead?
On 2023-09-12 00:03, Hal Murray via devel wrote: Maybe it's time to switch to Go? The opportunity for that may have passed. There's a new ntpd-rs project writing an NTP daemon in Rust: https://github.com/pendulum-project/ntpd-rs It's certainly not a full ntpd replacement yet (e.g. no local refclock support), but they are still doing active development. -- Richard ___ devel mailing list devel@ntpsec.org https://lists.ntpsec.org/mailman/listinfo/devel
Re: Is python2 dead?
Yo James! On Tue, 12 Sep 2023 06:46:38 -0700 (PDT) James Browning via devel wrote: > > On 09/12/2023 12:00 AM PDT Hal Murray via devel > > wrote: > > > > > > Gary said: > > > Please, no. Go is a garbage collected language. Just what > > > NTPsec does not need, random, unpredictable delays. > > It would appear there is a way to turn off GC under runtime/, How? Link? RGDS GARY --- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 g...@rellim.com Tel:+1 541 382 8588 Veritas liberabit vos. -- Quid est veritas? "If you can't measure it, you can't improve it." - Lord Kelvin pgpdclVCKifCv.pgp Description: OpenPGP digital signature ___ devel mailing list devel@ntpsec.org https://lists.ntpsec.org/mailman/listinfo/devel
Re: Is python2 dead?
Yo Hal! On Tue, 12 Sep 2023 00:00:47 -0700 Hal Murray via devel wrote: > Gary said: > > Please, no. Go is a garbage collected language. Just what NTPsec > > does not need, random, unpredictable delays. > > I was thinking of the Python code in ntpclients/ and pylib/ > Is there anything in there that is time sensitive? That could work. I wrote a Go client for gpsd from scratch. Not hard. > There are lots of ways to inject timing bumps before we get to > garbage collecting. cache, scheduler, interrupts, CPU speed, ... Any that work? > Do you have any data on Go GC times? Just what is in the doc: https://go.dev/doc/gc-guide The programmer has very little control over the GC. RGDS GARY --- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 g...@rellim.com Tel:+1 541 382 8588 Veritas liberabit vos. -- Quid est veritas? "If you can't measure it, you can't improve it." - Lord Kelvin pgp4XtybeBuJP.pgp Description: OpenPGP digital signature ___ devel mailing list devel@ntpsec.org https://lists.ntpsec.org/mailman/listinfo/devel
Go GC
Gary said: >James Browning via devel wrote: >> It would appear there is a way to turn off GC under runtime/, > How? Link? https://pkg.go.dev/runtime/debug#SetGCPercent It's not clear to me how to take advantage of that. You still have to turn it on occasionally or your world will fill up with garbage. I poked around a bit. I'm pretty sure that we can write a server that doesn't generate any garbage when processing a normal client request. The APIs for recvmsg/sendto don't allocate anything. If we split ntpd into client side and server side, I think we can write the server code such that the GC never runs. Or maybe never needs to run and we have to explicitly tell it not to bother trying. Logging stuff would probably generate garbage. The server side doesn't need to do that. Gary said: > Hal said: >> There are lots of ways to inject timing bumps before we get to >> garbage collecting. cache, scheduler, interrupts, CPU speed, ... > Any that work? What do you mean by "work"? I don't know how to avoid any of the above. Note that there are 2 levels of interrupt. The firmware steals a few cycles every now and then for things that it doesn't trust the OS to get right. The main example is checking the temperature and turning the CPU clock down if things are too hot. Then there are interrupts that get passed to the OS. You can fight that somewhat by manually assigning work to CPUs. But the scheduler still has to run occasionally and if your workload doesn't use the whole CPU, that CPU is likely to slow down when you are waiting for work. I did a bit of hacking with attic/clocks.c On this machine, the average time to read the clock is 13 ns. Within a burst of a million samples, there is usually a few in the 10-15 microsecond range. Occasinally, there is something in the 60-70 microseconds range. They are rare enough that it's easy to miss one in a million sample pairs of reading the clock. Slowest from each batch of 100... 11331 18540 11282 11341 11306 11311 11307 11316 11307 11322 16188 14920 11322 11293 13337 13025 32270 11352 21706 11313 32463 22764 11812 11308 11319 60664 11301 14530 20428 11319 14973 11308 11287 14181 13127 11320 11298 11312 12053 15081 17762 17329 11279 12430 11299 16946 14470 14745 13816 11323 Slowest was 60664 Histogram: CLOCK_REALTIME, 1 ns per bucket, 100 samples. ns hits 10 6646 11124028 12410522 13229036 14177996 15 48724 16 259 17 535 18 1430 19 585 2070 2124 2214 2313 2410 59 samples were bigger than 24. Histogram: CLOCK_REALTIME, 250 ns per bucket, 100 samples. ns hits 049 2250 2 3250 1 3500 3 3750 3 4000 1 8250 1 8500 1 875020 9000 1 9250 1 10250 1 11000 9 11250 4 13250 1 2 samples were bigger than 13250. Slowest was 14424. -- These are my opinions. I hate spam. ___ devel mailing list devel@ntpsec.org https://lists.ntpsec.org/mailman/listinfo/devel
Re: Go GC
Yo Hal! On Tue, 12 Sep 2023 16:55:12 -0700 Hal Murray via devel wrote: > Gary said: > >James Browning via devel wrote: > >> It would appear there is a way to turn off GC under runtime/, > > How? Link? > > https://pkg.go.dev/runtime/debug#SetGCPercent > > It's not clear to me how to take advantage of that. You still have > to turn it on occasionally or your world will fill up with garbage. Assuming you create garbage. Avoiding creating garbage is hard. > I poked around a bit. I'm pretty sure that we can write a server > that doesn't generate any garbage when processing a normal client > request. The problem is not when you generate garbage, but when the garbage collector wakes up. > Occasinally, there is something in the 60-70 microseconds range. > They are rare enough that it's easy to miss one in a million sample > pairs of reading the clock. Which is why NTP slowly adjusts PLL's instead of jumping around. RGDS GARY --- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 g...@rellim.com Tel:+1 541 382 8588 Veritas liberabit vos. -- Quid est veritas? "If you can't measure it, you can't improve it." - Lord Kelvin pgpkyZe9wWCRx.pgp Description: OpenPGP digital signature ___ devel mailing list devel@ntpsec.org https://lists.ntpsec.org/mailman/listinfo/devel
Re: Go GC
Gary said: > Avoiding creating garbage is hard. In general, yes. But the inner loop of the server side is not very complicated. The APIs that I'm looking at are read-into-my-buffer rather than return a new buffer that needs to be GCed. I think it's worth some effort to investigate this area. I'm prepared to give up if we find a fatal problem. Again, I'm assuming that we split ntpd into client and server parts so all we have to work on is the server half. -- These are my opinions. I hate spam. ___ devel mailing list devel@ntpsec.org https://lists.ntpsec.org/mailman/listinfo/devel
Re: Go GC
Yo Hal! On Tue, 12 Sep 2023 18:49:35 -0700 Hal Murray via devel wrote: > I think it's worth some effort to investigate this area. I'm > prepared to give up if we find a fatal problem. Again, I'm assuming > that we split ntpd into client and server parts so all we have to > work on is the server half. I look forward to your results. I found working with Go was nice. RGDS GARY --- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 g...@rellim.com Tel:+1 541 382 8588 Veritas liberabit vos. -- Quid est veritas? "If you can't measure it, you can't improve it." - Lord Kelvin pgpHZrLIVFtSe.pgp Description: OpenPGP digital signature ___ devel mailing list devel@ntpsec.org https://lists.ntpsec.org/mailman/listinfo/devel
