Just a random thought, but have you tried 1.16 with GODEBUG=asyncpreemptoff=1 ? The preemptive scheduling stuff was introduced in 1.14 I believe.
On Thursday, 9 September 2021 at 16:28:38 UTC+1 varun...@gmail.com wrote: > @Kurtis, Thanks for the reply. > > Yes. We upgraded to 1.16.5 due to this issue and had to revert back to > 1.13.7 (over earlier working version) as we are frequently seeing these > errors in 1.16.5. > > I looked into all the related issues on golang forums. We are not on > NetBSD (We are on CentOS7.8) > > We also suspect it is a bug in CGO code (specifically some memory > violation) but finding the bug is like finding needle in a haystack. The > SIGSEGV CGO signal handler did not catch anything at the time of this > error. The error message by itself does not mention any thing debuggable. > There is no stack or core dump. We tried working with GOTRACEBACK but that > did not generate any core dump as well. > > Thanks, > Varun > > > On Thursday, September 9, 2021 at 8:07:00 PM UTC+5:30 Kurtis Rader wrote: > >> Googling "go morestack on gsignal" turns up quite a few reports. Some of >> which involved kernel bugs (e.g., >> https://github.com/golang/go/issues/19652) but most seem to involve >> SIGSEGV errors in non-kernel code; sometimes the Go runtime or stdlib >> (e.g., https://github.com/golang/go/issues/35235) and sometimes user >> code. Given that you're using CGO a likely explanation is a bug in your C >> code. However, you say you're using Go 1.13.7 which is getting long in the >> tooth. Are you sure you're seeing the same failure with Go 1.16.5? If yes >> then I would bet the bug is in your C code. If no then it could be a Go bug >> that has likely already been fixed. >> >> On Thu, Sep 9, 2021 at 4:15 AM varun...@gmail.com <varun...@gmail.com> >> wrote: >> >>> Minor update: >>> >>> a. The crash with "morestack on gsignal" happens to be independent of >>> the kernel versions mentioned earlier >>> b. We implemented a signal handler to catch SIGSEGV from CGO. That is >>> not helping either >>> c. From system audit logs, we can see the following message: >>> >>> ./audit/audit.log.2:22944:type=ANOM_ABEND >>> msg=audit(1630376735.155:5486532): auid=4294967295 uid=996 gid=994 >>> ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 pid=26422 >>> comm="indexer" reason="memory violation" sig=5 >>> >>> Sig=5 is SIGTRAP. At this point, we suspect there is a SIGSEGV in CGO >>> layer and that is being returned as SIGTRAP by runtime. >>> >>> I am still clueless as to why a process can crash with "morestack on >>> gsignal". Any pointers to debug this further would be of great help. >>> >>> Thanks, >>> Varun >>> On Wednesday, September 1, 2021 at 2:44:52 PM UTC+5:30 >>> varun...@gmail.com wrote: >>> >>>> Hello, >>>> >>>> Recently we have been seeing multiple occurrences of "fatal: morestack >>>> on gsignal" error in our system tests. The issue happens with golang >>>> 1.13.7 >>>> and golang 1.16.5 runtimes. The process restarts after this error and no >>>> stack trace is dumped. Our process uses CGO calls, I/O ops etc. >>>> >>>> So far, the issue seem to happen only on CentOS7 with linux kernel >>>> version: 3.10.0-1127.19.1.el7.x86_64. The issue is not seen on: >>>> 3.10.0-693.5.2.el7.x86_64 >>>> >>>> At this point, we have no clue as to why this is happening and no stack >>>> trace is making it difficult to debug. Any pointers on debugging the issue >>>> would be of great help. >>>> >>>> Thanks, >>>> Varun >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "golang-nuts" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to golang-nuts...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/golang-nuts/9f36ab30-3222-4eed-8388-93ecd259bf55n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/golang-nuts/9f36ab30-3222-4eed-8388-93ecd259bf55n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> >> >> -- >> Kurtis Rader >> Caretaker of the exceptional canines Junior and Hank >> > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/4e6c8f7d-86be-4ecf-b4f5-19554f3d1379n%40googlegroups.com.
