On Thu, Sep 9, 2021 at 10:29 AM Varun V <varun.1...@gmail.com> wrote: > > @Brian, We did not try that as the issue is happening with go 1.13.7 as well
The "fatal: morestack on gsignal" error is more or less impossible. Some things that can cause it to happen are 1) C code calls sigaltstack, but the alternate signal stack is too small. 2) C code trashes the TLS slot that stores the value of G during a cgo call. 3) C code fails to correctly preserve registers across a cgo call. None of these are at all likely. Ian > On Thu, 9 Sep, 2021, 10:48 pm Brian Candler, <b.cand...@pobox.com> wrote: >> >> Just a random thought, but have you tried 1.16 with >> GODEBUG=asyncpreemptoff=1 ? The preemptive scheduling stuff was introduced >> in 1.14 I believe. >> >> On Thursday, 9 September 2021 at 16:28:38 UTC+1 varun...@gmail.com wrote: >>> >>> @Kurtis, Thanks for the reply. >>> >>> Yes. We upgraded to 1.16.5 due to this issue and had to revert back to >>> 1.13.7 (over earlier working version) as we are frequently seeing these >>> errors in 1.16.5. >>> >>> I looked into all the related issues on golang forums. We are not on NetBSD >>> (We are on CentOS7.8) >>> >>> We also suspect it is a bug in CGO code (specifically some memory >>> violation) but finding the bug is like finding needle in a haystack. The >>> SIGSEGV CGO signal handler did not catch anything at the time of this >>> error. The error message by itself does not mention any thing debuggable. >>> There is no stack or core dump. We tried working with GOTRACEBACK but that >>> did not generate any core dump as well. >>> >>> Thanks, >>> Varun >>> >>> >>> On Thursday, September 9, 2021 at 8:07:00 PM UTC+5:30 Kurtis Rader wrote: >>>> >>>> Googling "go morestack on gsignal" turns up quite a few reports. Some of >>>> which involved kernel bugs (e.g., >>>> https://github.com/golang/go/issues/19652) but most seem to involve >>>> SIGSEGV errors in non-kernel code; sometimes the Go runtime or stdlib >>>> (e.g., https://github.com/golang/go/issues/35235) and sometimes user code. >>>> Given that you're using CGO a likely explanation is a bug in your C code. >>>> However, you say you're using Go 1.13.7 which is getting long in the >>>> tooth. Are you sure you're seeing the same failure with Go 1.16.5? If yes >>>> then I would bet the bug is in your C code. If no then it could be a Go >>>> bug that has likely already been fixed. >>>> >>>> On Thu, Sep 9, 2021 at 4:15 AM varun...@gmail.com <varun...@gmail.com> >>>> wrote: >>>>> >>>>> Minor update: >>>>> >>>>> a. The crash with "morestack on gsignal" happens to be independent of the >>>>> kernel versions mentioned earlier >>>>> b. We implemented a signal handler to catch SIGSEGV from CGO. That is not >>>>> helping either >>>>> c. From system audit logs, we can see the following message: >>>>> >>>>> ./audit/audit.log.2:22944:type=ANOM_ABEND >>>>> msg=audit(1630376735.155:5486532): auid=4294967295 uid=996 gid=994 >>>>> ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 pid=26422 >>>>> comm="indexer" reason="memory violation" sig=5 >>>>> >>>>> Sig=5 is SIGTRAP. At this point, we suspect there is a SIGSEGV in CGO >>>>> layer and that is being returned as SIGTRAP by runtime. >>>>> >>>>> I am still clueless as to why a process can crash with "morestack on >>>>> gsignal". Any pointers to debug this further would be of great help. >>>>> >>>>> Thanks, >>>>> Varun >>>>> On Wednesday, September 1, 2021 at 2:44:52 PM UTC+5:30 varun...@gmail.com >>>>> wrote: >>>>>> >>>>>> Hello, >>>>>> >>>>>> Recently we have been seeing multiple occurrences of "fatal: morestack >>>>>> on gsignal" error in our system tests. The issue happens with golang >>>>>> 1.13.7 and golang 1.16.5 runtimes. The process restarts after this error >>>>>> and no stack trace is dumped. Our process uses CGO calls, I/O ops etc. >>>>>> >>>>>> So far, the issue seem to happen only on CentOS7 with linux kernel >>>>>> version: 3.10.0-1127.19.1.el7.x86_64. The issue is not seen on: >>>>>> 3.10.0-693.5.2.el7.x86_64 >>>>>> >>>>>> At this point, we have no clue as to why this is happening and no stack >>>>>> trace is making it difficult to debug. Any pointers on debugging the >>>>>> issue would be of great help. >>>>>> >>>>>> Thanks, >>>>>> Varun >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "golang-nuts" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to golang-nuts...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/golang-nuts/9f36ab30-3222-4eed-8388-93ecd259bf55n%40googlegroups.com. >>>> >>>> >>>> >>>> -- >>>> Kurtis Rader >>>> Caretaker of the exceptional canines Junior and Hank >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "golang-nuts" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/golang-nuts/msVmzxWjIOI/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> golang-nuts+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/golang-nuts/4e6c8f7d-86be-4ecf-b4f5-19554f3d1379n%40googlegroups.com. > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/CAMzZVzBrwcw%2BMJOjvXgg_YhFA0ZgYD%3DiLhYJyYHCxzXFKYD%2BeQ%40mail.gmail.com. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAOyqgcX%3DDSaZBZVa69xM4RNxVhfV2KianmVOfmXm2Bs_3bUcxA%40mail.gmail.com.
