Hi,
On an embedded 16 core arm device we have been observing infrequent runtime
crashes, with different signatures every time.
One examples is this:
unexpected fault address 0x6b6fc6da
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x6b6fc6da pc=0x7f72c]
goroutine 1 [running, locked to thread]:
runtime.throw(0xc3074, 0x5)
/usr/local/go/src/runtime/panic.go:1116 +0x54 fp=0x4000072c20
sp=0x4000072bf0 pc=0x3dcc4
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:749 +0x3b8 fp=0x4000072c50
sp=0x4000072c20 pc=0x51b48
reflect.(*rtype).Method(0xc55fe, 0x14, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0, ...)
/usr/local/go/src/reflect/type.go:809 +0xcc fp=0x4000072de0
sp=0x4000072c60 pc=0x7f72c
runtime: unexpected return pc for runtime.doInit called from 0x4000098060
stack: frame={sp:0x4000072de0, fp:0x4000072e20}
stack=[0x4000072000,0x4000073000)
0000004000072ce0: 000000400009a4e0 000000400009c340
0000004000072cf0: 0000000000000006 000000400009c3a8
0000004000072d00: 000000400009c3f8 000000400009c410
0000004000072d10: 0000000000000002 000000400009c438
0000004000072d20: 000000400009c4a8 0000004000072d58
0000004000072d30: 000000000001fb14 <runtime.mapassign_faststr+804>
00000000000ae3e0
0000004000072d40: 0000004000098060 0000000000000000
0000004000072d50: 00000040000a4d00 0000004000072dc8
0000004000072d60: 000000000007d790 <unicode.init+16592> 00000000000ae3e0
0000004000072d70: 0000004000098060 0000000000000001
0000004000072d80: 000000400009a4e0 0000000000002001
0000004000072d90: 000000400009806e 0000004000072dc8
0000004000072da0: 000000000007d4c0 <unicode.init+15872> 6e97779bde52831f
0000004000072db0: 0000000000000001 0000000000000000
0000004000072dc0: 000000400009c410 0000004000072e28
0000004000072dd0: 000000000004d02c <runtime.doInit+156> 00000000000ae3e0
0000004000072de0: <0000004000098060 00000000000c55fe
0000004000072df0: 0000000000000014 0000000000000000
0000004000072e00: 0000000000000000 0000000000000000
0000004000072e10: 0000000000000000 0000000000000000
0000004000072e20: >0000000000000000 0000000000000000
0000004000072e30: 0000000000000000 0000000000000000
0000004000072e40: 0000000000000000 0000004000072e78
0000004000072e50: 0000000000000000 0000000000000000
0000004000072e60: 0000000000150e98 0000004000072ea8
0000004000072e70: 000000000004cfe8 <runtime.doInit+88> 0000000000150e80
0000004000072e80: 000000000004d02c <runtime.doInit+156> 00000000000c45db
0000004000072e90: 000000000000000e 0000000000000002
0000004000072ea0: 0000004000096010 0000004000072ee8
0000004000072eb0: 000000000004cfe8 <runtime.doInit+88> 0000000000151ec0
0000004000072ec0: 0000000000000060 0000007f9bd197d0
0000004000072ed0: 0000000000000000 0000000000000000
0000004000072ee0: 00000000001518e8 0000004000072f28
0000004000072ef0: 000000000004cfe8 <runtime.doInit+88> 0000000000151500
0000004000072f00: 0000000000000000 000000400004c738
0000004000072f10: 0000000000025004 <runtime.gcenable+148> 0000000000000002
runtime.doInit(0x0)
/usr/local/go/src/runtime/proc.go:5652 +0x9c fp=0x4000072e20
sp=0x4000072de0 pc=0x4d02c
goroutine 2 [force gc (idle)]:
runtime.gopark(0xcaff8, 0x165dd0, 0x1411, 0x1)
/usr/local/go/src/runtime/proc.go:306 +0xd0 fp=0x400004cfa0
sp=0x400004cf80 pc=0x40540
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:312
runtime.forcegchelper()
/usr/local/go/src/runtime/proc.go:255 +0xc4 fp=0x400004cfd0
sp=0x400004cfa0 pc=0x403f4
runtime.goexit()
/usr/local/go/src/runtime/asm_arm64.s:1136 +0x4 fp=0x400004cfd0
sp=0x400004cfd0 pc=0x6b8f4
created by runtime.init.5
/usr/local/go/src/runtime/proc.go:243 +0x30
goroutine 3 [GC sweep wait]:
runtime.gopark(0xcaff8, 0x165ee0, 0x140c, 0x1)
/usr/local/go/src/runtime/proc.go:306 +0xd0 fp=0x400004d7a0
sp=0x400004d780 pc=0x40540
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:312
runtime.bgsweep(0x400006a000)
/usr/local/go/src/runtime/mgcsweep.go:163 +0xb0 fp=0x400004d7d0
sp=0x400004d7a0 pc=0x2e880
runtime.goexit()
/usr/local/go/src/runtime/asm_arm64.s:1136 +0x4 fp=0x400004d7d0
sp=0x400004d7d0 pc=0x6b8f4
created by runtime.gcenable
/usr/local/go/src/runtime/mgc.go:217 +0x54
goroutine 4 [GC scavenge wait]:
runtime.gopark(0xcaff8, 0x165f80, 0x140d, 0x1)
/usr/local/go/src/runtime/proc.go:306 +0xd0 fp=0x400004df70
sp=0x400004df50 pc=0x40540
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:312
runtime.bgscavenge(0x400006a000)
/usr/local/go/src/runtime/mgcscavenge.go:265 +0xec fp=0x400004dfd0
sp=0x400004df70 pc=0x2cd4c
runtime.goexit()
/usr/local/go/src/runtime/asm_arm64.s:1136 +0x4 fp=0x400004dfd0
sp=0x400004dfd0 pc=0x6b8f4
created by runtime.gcenable
/usr/local/go/src/runtime/mgc.go:218 +0x74
The source code for the app is at: https://github.com/stavrospen/gomemverify
This is with go version go1.15.11 linux/amd64, cross compiling for arm64.
CGO disabled.
We also use upx version 3.95
The background is that we have been seeing our go application crash, and
were suspecting some external entity corrupting the memory, so created this
small utility to try to catch memory corruption. We never managed to catch
any corruption in the heap, but we keep seeing these random crashes with
what seems to be stack corruption.
I should note here, that we also have C++ application running side by side,
but we have never seen any similar crash there.
Was wondering if someone has any idea on how to debug this further, or what
Go tools we can use to try to capture the possible corruption. We have
tried 'GODEBUG="efence=1"' but didn't seem to catch anything.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/11628ac8-eb65-452a-8845-941806e4b38cn%40googlegroups.com.
