I suggest using pstack to capture the stacks of all threads of you can. Because it is external ut is not subject to the stopping bias of the internal go facilities.
> On Aug 27, 2019, at 9:05 AM, Ian Lance Taylor <i...@golang.org> wrote: > >> On Mon, Aug 26, 2019 at 2:12 PM Michael Andersen <mich...@steelcode.com> >> wrote: >> >> I was concerned that perhaps the stacks that I dumped after the scheduling >> stall did not reflect the actual PC's of the goroutines during the stall, so >> I modified schedtrace to dump out g.sched.pc and g.m.vdsoPC for each G >> currently on the P at the time schedtrace prints. >> >> It just occurred again a few minutes ago. Previously, all the P's were >> running the problematic code, so other things weren't scheduled. This time >> it appears that only a few P's (4 of 8) were running the problematic code >> but the GC was trying to stop the world so even though there are P's with >> nothing on them, nothing else ran. All the PC's on the problematic >> goroutines (those with status=_Prunning) were static for the 3 seconds they >> stalled. They all had the same PC's: >> >> g.sched.pc was 0x43389f which maps to: Line 302 of >> "/usr/local/go/src/runtime/proc.go" starts at address 0x43389f >> <runtime.gopark+239> and ends at 0x4338b7 <runtime.gopark+263>. >> g.m.vdsoPC was 0x450c68 which maps to: Line 17 of >> "/usr/local/go/src/runtime/timestub.go" starts at address 0x450c63 >> <time.now+51> and ends at 0x450c8d <time.now+93>. >> >> In case there is more useful information that I missed, I attached the log >> output. > > g.sched.pc doesn't tell us much about the current state. It tells us > the PC value of the goroutine the last time the goroutine blocked. > What you are seeing is that goroutine's block in gopark. That is not > surprising. What we don't know is what the goroutine is currently > doing. Similarly, vdsoPC tells us the PC value the last time the > goroutine went into a VDSO call. Having that be time.now is not > surprising. It doesn't tell us anything about what the goroutine is > doing now. Also note that vdsoPC is only meaningful if vdsoSP != 0. > > >> I'm trying to make a simpler program that reproduces this, but thus far >> running time.Now in a tight loop across multiple goroutines isn't causing >> it. Any ideas what I could try next? > > If it's true that some goroutines are simply looping, then the only > way to get their current PC is to crash the program with SIGQUIT and > look at the stack trace. The runtime doesn't have any way to ask a > goroutine what the currently executing PC is. > > Ian > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/CAOyqgcU1pvzbun3488exV%3D%2BhTauSxd8JpBh0Oo76inVafHu-rA%40mail.gmail.com. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/75A93102-CECD-467E-A092-0E9FF780696A%40ix.netcom.com.
