Ok, so your data collectors never complete. A simple change to make this easier to diagnose is to not spin up another collector controller at the 2 min mark if the previous has not completed.
I would determine if the stuck collector is BQ or Elastic and check the server side logs. > On Aug 26, 2020, at 11:29 PM, Kurtis Rader <kra...@skepticism.us> wrote: > > >> On Wed, Aug 26, 2020 at 8:51 PM Siddhesh Divekar >> <siddhesh.dive...@gmail.com> wrote: > >> Right, then it looks less likely that we are blocked on a mutex. >> >> Every 2 minutes we spin up a go routine which then in turn spins up a bunch >> of go routines to collect data from big query & elastic (data collector >> routines). >> The 2 minutes go routine then in turn waits on waitgroup for data collector >> routines to come back. So its not single go routine from our side at least. >> From backtrace we have from sigabort we see only one such data collector go >> routine blocked and other 2 2 mins thread waiting on waitgroup. > > Are you spinning up a Go routine every 2 minutes regardless of whether the > previous instance had completed? Your comment is not clear whether you have > one, or more than one, goroutine issuing a BigQuery query at the time you saw > the problem and captured the backtraces. If you are, in fact, starting those > Go routines without regard to whether the prior instance had completed that > seems like a "yellow flag", "here be dragons", situation. > > -- > Kurtis Rader > Caretaker of the exceptional canines Junior and Hank > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/CABx2%3DD-y%3DQg_dx5W0qDWiMV9sB0eFToTEoiD83_mbn7RHC%3DQ%3Dg%40mail.gmail.com. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/6E5112C8-0C08-4F37-A98C-38C9D6BB56A5%40ix.netcom.com.
