On Mon, Sep 13, 2021 at 5:34 AM Robert Engels <reng...@ix.netcom.com> wrote: > > If the C code overruns a stack allocated variable couldn’t it easily corrupt > the saved registers ?
Yes, that is another possible cause. Thanks. Ian > > On Sep 9, 2021, at 2:14 PM, Ian Lance Taylor <i...@golang.org> wrote: > > > > On Thu, Sep 9, 2021 at 10:29 AM Varun V <varun.1...@gmail.com> wrote: > >> > >> @Brian, We did not try that as the issue is happening with go 1.13.7 as > >> well > > > > The "fatal: morestack on gsignal" error is more or less impossible. > > Some things that can cause it to happen are > > > > 1) C code calls sigaltstack, but the alternate signal stack is too small. > > 2) C code trashes the TLS slot that stores the value of G during a cgo call. > > 3) C code fails to correctly preserve registers across a cgo call. > > > > None of these are at all likely. > > > > Ian > > > > > >>> On Thu, 9 Sep, 2021, 10:48 pm Brian Candler, <b.cand...@pobox.com> wrote: > >>> > >>> Just a random thought, but have you tried 1.16 with > >>> GODEBUG=asyncpreemptoff=1 ? The preemptive scheduling stuff was > >>> introduced in 1.14 I believe. > >>> > >>> On Thursday, 9 September 2021 at 16:28:38 UTC+1 varun...@gmail.com wrote: > >>>> > >>>> @Kurtis, Thanks for the reply. > >>>> > >>>> Yes. We upgraded to 1.16.5 due to this issue and had to revert back to > >>>> 1.13.7 (over earlier working version) as we are frequently seeing these > >>>> errors in 1.16.5. > >>>> > >>>> I looked into all the related issues on golang forums. We are not on > >>>> NetBSD (We are on CentOS7.8) > >>>> > >>>> We also suspect it is a bug in CGO code (specifically some memory > >>>> violation) but finding the bug is like finding needle in a haystack. The > >>>> SIGSEGV CGO signal handler did not catch anything at the time of this > >>>> error. The error message by itself does not mention any thing > >>>> debuggable. There is no stack or core dump. We tried working with > >>>> GOTRACEBACK but that did not generate any core dump as well. > >>>> > >>>> Thanks, > >>>> Varun > >>>> > >>>> > >>>> On Thursday, September 9, 2021 at 8:07:00 PM UTC+5:30 Kurtis Rader wrote: > >>>>> > >>>>> Googling "go morestack on gsignal" turns up quite a few reports. Some > >>>>> of which involved kernel bugs (e.g., > >>>>> https://github.com/golang/go/issues/19652) but most seem to involve > >>>>> SIGSEGV errors in non-kernel code; sometimes the Go runtime or stdlib > >>>>> (e.g., https://github.com/golang/go/issues/35235) and sometimes user > >>>>> code. Given that you're using CGO a likely explanation is a bug in your > >>>>> C code. However, you say you're using Go 1.13.7 which is getting long > >>>>> in the tooth. Are you sure you're seeing the same failure with Go > >>>>> 1.16.5? If yes then I would bet the bug is in your C code. If no then > >>>>> it could be a Go bug that has likely already been fixed. > >>>>> > >>>>> On Thu, Sep 9, 2021 at 4:15 AM varun...@gmail.com <varun...@gmail.com> > >>>>> wrote: > >>>>>> > >>>>>> Minor update: > >>>>>> > >>>>>> a. The crash with "morestack on gsignal" happens to be independent of > >>>>>> the kernel versions mentioned earlier > >>>>>> b. We implemented a signal handler to catch SIGSEGV from CGO. That is > >>>>>> not helping either > >>>>>> c. From system audit logs, we can see the following message: > >>>>>> > >>>>>> ./audit/audit.log.2:22944:type=ANOM_ABEND > >>>>>> msg=audit(1630376735.155:5486532): auid=4294967295 uid=996 gid=994 > >>>>>> ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 > >>>>>> pid=26422 comm="indexer" reason="memory violation" sig=5 > >>>>>> > >>>>>> Sig=5 is SIGTRAP. At this point, we suspect there is a SIGSEGV in CGO > >>>>>> layer and that is being returned as SIGTRAP by runtime. > >>>>>> > >>>>>> I am still clueless as to why a process can crash with "morestack on > >>>>>> gsignal". Any pointers to debug this further would be of great help. > >>>>>> > >>>>>> Thanks, > >>>>>> Varun > >>>>>> On Wednesday, September 1, 2021 at 2:44:52 PM UTC+5:30 > >>>>>> varun...@gmail.com wrote: > >>>>>>> > >>>>>>> Hello, > >>>>>>> > >>>>>>> Recently we have been seeing multiple occurrences of "fatal: > >>>>>>> morestack on gsignal" error in our system tests. The issue happens > >>>>>>> with golang 1.13.7 and golang 1.16.5 runtimes. The process restarts > >>>>>>> after this error and no stack trace is dumped. Our process uses CGO > >>>>>>> calls, I/O ops etc. > >>>>>>> > >>>>>>> So far, the issue seem to happen only on CentOS7 with linux kernel > >>>>>>> version: 3.10.0-1127.19.1.el7.x86_64. The issue is not seen on: > >>>>>>> 3.10.0-693.5.2.el7.x86_64 > >>>>>>> > >>>>>>> At this point, we have no clue as to why this is happening and no > >>>>>>> stack trace is making it difficult to debug. Any pointers on > >>>>>>> debugging the issue would be of great help. > >>>>>>> > >>>>>>> Thanks, > >>>>>>> Varun > >>>>>> > >>>>>> -- > >>>>>> You received this message because you are subscribed to the Google > >>>>>> Groups "golang-nuts" group. > >>>>>> To unsubscribe from this group and stop receiving emails from it, send > >>>>>> an email to golang-nuts...@googlegroups.com. > >>>>>> To view this discussion on the web visit > >>>>>> https://groups.google.com/d/msgid/golang-nuts/9f36ab30-3222-4eed-8388-93ecd259bf55n%40googlegroups.com. > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Kurtis Rader > >>>>> Caretaker of the exceptional canines Junior and Hank > >>> > >>> -- > >>> You received this message because you are subscribed to a topic in the > >>> Google Groups "golang-nuts" group. > >>> To unsubscribe from this topic, visit > >>> https://groups.google.com/d/topic/golang-nuts/msVmzxWjIOI/unsubscribe. > >>> To unsubscribe from this group and all its topics, send an email to > >>> golang-nuts+unsubscr...@googlegroups.com. > >>> To view this discussion on the web visit > >>> https://groups.google.com/d/msgid/golang-nuts/4e6c8f7d-86be-4ecf-b4f5-19554f3d1379n%40googlegroups.com. > >> > >> -- > >> You received this message because you are subscribed to the Google Groups > >> "golang-nuts" group. > >> To unsubscribe from this group and stop receiving emails from it, send an > >> email to golang-nuts+unsubscr...@googlegroups.com. > >> To view this discussion on the web visit > >> https://groups.google.com/d/msgid/golang-nuts/CAMzZVzBrwcw%2BMJOjvXgg_YhFA0ZgYD%3DiLhYJyYHCxzXFKYD%2BeQ%40mail.gmail.com. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "golang-nuts" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to golang-nuts+unsubscr...@googlegroups.com. > > To view this discussion on the web visit > > https://groups.google.com/d/msgid/golang-nuts/CAOyqgcX%3DDSaZBZVa69xM4RNxVhfV2KianmVOfmXm2Bs_3bUcxA%40mail.gmail.com. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAOyqgcXs2_B%2BsYz0RS1BxSdfTMo%3DSFq9nryqLUYjU%2BW6Sn1crg%40mail.gmail.com.
