Dear Ian,
Thank you so much for your hint, I think I've figured it out.
The root cause seems similar to the "uncertainty principle".
As an observer, by adding a `println` call in `asyncPreempt`
as well as `asyncPreempt2` influences the actual behavior
after signal handling. The `println` involves stack split check,
which calls the `morestack`.
It took me a while to realize that `morestack` stores its caller
pc in `g.m.morebuf.pc` since `getcallerpc` in `newstack`
always returns the pc from `morestack`, which doesn't tell
too much information.
//go:nosplit
func asyncPreempt2() {
// println("asyncPreempt2 is called") // comment here omits calling
morestack.
gp := getg()
gp.asyncSafePoint = true
if gp.preemptStop {
mcall(preemptPark)
} else {
mcall(gopreempt_m)
}
println("asyncPreempt2 finished")
gp.asyncSafePoint = false
}
On Sunday, December 8, 2019 at 7:37:12 PM UTC+1, Ian Lance Taylor wrote:
>
> On Sun, Dec 8, 2019 at 7:02 AM changkun <euryu...@gmail.com <javascript:>>
> wrote:
> >
> > As we all know that a program is interrupted by a signal and entering
> kernel space then switch to a userspace signal handler. After the signal
> handler is accomplished, it will be reentering the kernel space then switch
> back to where was interrupted.
> >
> > I am recently reading the newly implemented async preemption in go 1.14,
> which uses the OS signal to interrupt a "non-preemptive" user goroutine. I
> am debugging very simple program:
> >
> > package main
> >
> >
> > import (
> > "runtime"
> > "time"
> > )
> >
> >
> > func tightloop() {
> > for {
> > }
> > }
> >
> >
> > func main() {
> > runtime.GOMAXPROCS(1)
> > go tightloop()
> >
> >
> > time.Sleep(time.Millisecond)
> > println("OK")
> > runtime.Gosched()
> > }
> >
> >
> > In Go 1.14, when a preempt signal arrives, the `tightloop` will be
> interrupted by the OS and entering the pre-configured signal handler
> `runtime·sigtramp`:
> >
> > TEXT runtime·sigtramp(SB),NOSPLIT,$72
> > MOVQ DX, ctx-56(SP)
> > MOVQ SI, info-64(SP)
> > MOVQ DI, signum-72(SP)
> > MOVQ $runtime·sigtrampgo(SB), AX
> > CALL AX
> > RET
> >
> >
> > which `sigtrampgo` eventually calls the `sighandler`.
> >
> > //go:nosplit
> > //go:nowritebarrierrec
> > func sigtrampgo(sig uint32, info *siginfo, ctx unsafe.Pointer) {
> > (...)
> > setg(g.m.gsignal)
> > (...)
> > sighandler(sig, info, ctx, g)
> > setg(g)
> > (...)
> > }
> >
> >
> > As far as I read the `sighandler` function, it calls `doSigPreempt` and
> modifies the `ctx` that passed from system kernel, and sets the `rip` to
> the prologue of `runtime.asyncPreempt`.
> >
> > //go:nowritebarrierrec
> > func sighandler(sig uint32, info *siginfo, ctxt unsafe.Pointer, gp
> *g) {
> > _g_ := getg()
> > c := &sigctxt{info, ctxt}
> >
> >
> > (...)
> > if sig == sigPreempt {
> > doSigPreempt(gp, c)
> > }
> > }
> > func doSigPreempt(gp *g, ctxt *sigctxt) {
> > if canPreempt {
> > // here modifies the rip and rsp
> > ctxt.pushCall(funcPC(asyncPreempt))
> > }
> >
> >
> > (...)
> > }
> >
> >
> > However, I noticed that the asyncPreempt is not immediately executed
> when
> > the signal handler is complete, instead:
> >
> > 1. `morestack` or `morestack_noctxt` is called after `sighandler` is
> **returned** (not entering either the epilogue or prologue), which calls
> `newstack` and check checks the preempt flag and entering schedule loop and
> therefore schedules the main goroutine to finish the async preemption.
> >
> > 2. the `OK` outputs before executing `asyncPreempt`
> >
> > Here are my inserted print logs in runtime:
> >
> > mstart1 call schedule()
> > enter schedule()
> > park_m call schedule()
> > enter schedule()
> > mstart1 call schedule()
> > enter schedule()
> > mstart1 call schedule()
> > enter schedule()
> > park_m call schedule()
> > enter schedule()
> > park_m call schedule()
> > enter schedule()
> > park_m call schedule()
> > enter schedule()
> > mstart1 call schedule()
> > enter schedule()
> > park_m call schedule()
> > enter schedule()
> > rip: 17149264 eip: 824634034136
> > before pushCall asyncPreempt
> > after pushCall asyncPreempt
> > rip: 17124704 eip: 824634034128 // rip points to asyncPreempt
> > calling newstack: m0, g0 // how could newstack is
> called?
> > newstack call gopreempt_m
> > gopreempt_m call goschedImpl
> > goschedImpl call schedule()
> > enter schedule()
> > OK
> > gosched_m call goschedImpl
> > goschedImpl call schedule()
> > enter schedule()
> > asyncPreempt2
> > asyncPreempt2
> > asyncPreempt2
> > asyncPreempt2
> > preemptPark
> > gopreempt_m call goschedImpl
> > goschedImpl call schedule()
> > enter schedule()
> >
> >
> > while I checked the dumped assembly code, there is no stack split check
> > in neither `asyncPreempt` or `sigtramp`.
> >
> > Sorry for the long story, my questions are:
> >
> > - When, who, and how runtime calls the `morestack` after `sighandler`?
> What did I miss?
> > - Does modifying `ctx` changes program jumps to the modified `rip`
> instruction after finishing the signal handler?
> >
> > Thank you very much for reading the question and thanks to the go team
> building such a brilliant feature.
>
>
> It sounds like you are doing a good job of investigating. I don't
> know the answer to your question, but it sounds worth figuring out. I
> would encourage you to keep looking into this. Be sure to track which
> goroutine is seeing the preemption call; perhaps there is some
> confusion between different goroutines. Or, in newstack you could
> perhaps call getcallerpc to find out exactly what is calling the
> function.
>
> Ian
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/72b64657-43dc-484f-8f05-1c75bac7190d%40googlegroups.com.
