@Rustam Abdullaev you're a lifesaver!! I was having this same issue with rising CPU and was going crazy trying to find the resource leak until I saw your comment - pulled up a heap dump and saw 75% time.NewTicker, found a dangling ticker I was instantiating in a goroutine, added a defer ticker.Stop() and voila, problem solved.
For my own edification and that of future debuggers, as far as I understand it the issue with leaked tickers is that the time package uses the netpoller (interaction there is beyond me) to continuously loop over all timers on the heap, checking if each is ready to fire and running the associated function if so. With respect to tickers, that means that every new ticker will produce a new call to time.sendTime() on the configured interval until its timer is removed from the heap via Ticker.Stop() - that call to time.sendTime() runs a non-blocking send of time.Now() to the ticker channel (which explicitly never gets closed as per the ticker docs), which altogether comprises that 'tight loop' that Dave suggested earlier in this thread. Is it just me or could this be made a little clearer in the docs? I suppose you're meant to implicitly understand that any allocated resources need to be cleaned up, but this rather mystifying rising CPU issue seems like it could be worth an explicit mention. On Monday, March 8, 2021 at 12:25:11 PM UTC-5 Rustam Abdullaev wrote: > You might have a ticker leak. > Check that you're not accidentally calling time.NewTicker in a loop. > Get a pprof heap dump and look for time.NewTicker objects. The total size > of them should not be large. If it is, then you have a ticker leak. > > On Tuesday, 13 November 2018 at 15:26:29 UTC+1 Anon wrote: > >> Hi Ian, >> >> I am facing the same issue. The CPU usage on my servers increases from >> very low to 100% of CPU. Did you find the solution to your problem? >> >> go version : 1.10.1 >> OS : centOS >> >> Thanks, >> >> >> On Thursday, January 31, 2013 at 7:27:33 AM UTC+5:30, Ian Ragsdale wrote: >>> >>> Thanks Dave, those look like great suggestions. I'm running Go 1.0.3 on >>> Ubuntu: >>> >>> # go version >>> >>> >>> go version >>> go1.0.3 >>> >>> # uname -a >>> Linux 4bf343d4-786f-4fa6-b37c-8bdd018fc63b 2.6.32-350-ec2 #57-Ubuntu SMP >>> Thu Nov 15 15:59:03 UTC 2012 x86_64 GNU/Linux >>> >>> I'll work on gathering the SIGQUIT & GOGCTRACE output - didn't know >>> about those techniques. I can't really share the source, and I'm not sure >>> if there's a good way to cut it down to a sharable chunk, but I'll give it >>> a shot. Strace was going to be my next move. >>> >>> I think your theory in general makes sense, as I'm familiar with that >>> kind of failure, but if that was the case, would it not immediately shoot >>> up to 100% cpu usage as soon as the problem occurred? In this situation, >>> the cpu usage creeps up quite slowly, in a nearly perfect linear fashion >>> over the course of many hours. >>> >>> - Ian >>> >>> On Jan 30, 2013, at 4:00 PM, Dave Cheney <da...@cheney.net> wrote: >>> >>> >>> Can you please provide the following details >>> >>> * your Go version and operating system details >>> * the full output from the process when sent a SIGQUIT, once it entered >>> the high CPU usage scenario. >>> * the full output from running the process with GOGCTRACE=1 as above. >>> * the source, I possible, or at least a sample that reproduces the >>> issue. >>> * if you are using Linux, try running the process under the perf(1) >>> tool. >>> * if you able, try running the process under strace(1). >>> >>> My gut feeling is your code, or a library is not checking an error code >>> from a socket operation and entering into a tight loop. If this is the case >>> temporarily breaking the networking on this host may induce the failure. >>> >>> >>> >>> >> ::DISCLAIMER:: >> >> >> ---------------------------------------------------------------------------------------------------------------------------------------------------- >> >> >> This message is intended only for the use of the addressee and may >> contain information that is privileged, confidential and exempt from >> disclosure under applicable law. If the reader of this message is not the >> intended recipient, or the employee or agent responsible for delivering the >> message to the intended recipient, you are hereby notified that any >> dissemination, distribution or copying of this communication is strictly >> prohibited. If you have received this e-mail in error, please notify us >> immediately by return e-mail and delete this e-mail and all attachments >> from your system. >> > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/1aa24a3b-07bc-4c39-bbf1-0396394131a3n%40googlegroups.com.
