Any updates, I am also facing this issue in my case Golang server just stuck and openFD gets spiked. -Rahul
On Thursday, August 27, 2020 at 8:40:19 PM UTC+5:30 Siddhesh Divekar wrote: > Ok, so your data collectors never complete. A simple change to make this >> easier to diagnose is to not spin up another collector controller at the 2 >> min mark if the previous has not completed. >> >> I would determine if the stuck collector is BQ or Elastic and check the >> server side logs. >> > We start a data collector (DC) every 2 mins irrespective of whether the > previous DC has completed or not, so we would have a few DCs stuck but not > all. (2 from backtrace) > > I would determine if the stuck collector is BQ or Elastic and check the >> server side logs. >> > In server side logs we don't see any activity at all. That is the > confusing part why regular user requests are not reaching the go server > from the users. > Will check BQ and elastic logs around that time. But after restart > everything was working fine. > > Could be a bug in the http/2 implementation. I would disable http/2 and >> see if you encounter the problem. >> > If we hit it again we will collect github.com/robaho/goanalyzer & see > what is happening. > Between is there a compile time flag to disable http2 while building ? > > Thanks, > > On Thu, Aug 27, 2020 at 4:05 AM Robert Engels <ren...@ix.netcom.com> > wrote: > >> Could be a bug in the http/2 implementation. I would disable http/2 and >> see if you encounter the problem. >> >> On Aug 27, 2020, at 6:02 AM, Robert Engels <ren...@ix.netcom.com> wrote: >> >> >> Ok, so your data collectors never complete. A simple change to make this >> easier to diagnose is to not spin up another collector controller at the 2 >> min mark if the previous has not completed. >> >> I would determine if the stuck collector is BQ or Elastic and check the >> server side logs. >> >> On Aug 26, 2020, at 11:29 PM, Kurtis Rader <kra...@skepticism.us> wrote: >> >> >> On Wed, Aug 26, 2020 at 8:51 PM Siddhesh Divekar <siddhesh...@gmail.com> >> wrote: >> >>> Right, then it looks less likely that we are blocked on a mutex. >>> >>> Every 2 minutes we spin up a go routine which then in turn spins up a >>> bunch of go routines to collect data from big query & elastic (data >>> collector routines). >>> The 2 minutes go routine then in turn waits on waitgroup for data >>> collector routines to come back. So its not single go routine from our side >>> at least. >>> From backtrace we have from sigabort we see only one such data collector >>> go routine blocked and other 2 2 mins thread waiting on waitgroup. >>> >> >> Are you spinning up a Go routine every 2 minutes regardless of whether >> the previous instance had completed? Your comment is not clear whether you >> have one, or more than one, goroutine issuing a BigQuery query at the time >> you saw the problem and captured the backtraces. If you are, in fact, >> starting those Go routines without regard to whether the prior instance had >> completed that seems like a "yellow flag", "here be dragons", situation. >> >> -- >> Kurtis Rader >> Caretaker of the exceptional canines Junior and Hank >> >> -- >> You received this message because you are subscribed to the Google Groups >> "golang-nuts" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to golang-nuts...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/golang-nuts/CABx2%3DD-y%3DQg_dx5W0qDWiMV9sB0eFToTEoiD83_mbn7RHC%3DQ%3Dg%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/golang-nuts/CABx2%3DD-y%3DQg_dx5W0qDWiMV9sB0eFToTEoiD83_mbn7RHC%3DQ%3Dg%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> > > -- > -Siddhesh. > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/8a8a4445-7b57-410a-bb33-fa5d603b5f9fn%40googlegroups.com.
