Hi, I ran into a smart/fast shutdown hang on master while looking at crash restart. It happens in the window after we relaunch startup, but before WAL redo starts. FatalError is still set, so we wait for the new checkpointer and I/O workers after sending SIGTERM, which they ignore. AbortStartTime has already been reset too, so nothing escalates to SIGKILL.
I first tried routing that shutdown through HandleFatalError(PMQUIT_FOR_STOP, false), i.e. sending SIGQUIT to the children. That fixed the hang, but still went down the abnormal-shutdown path and left the stale flag in place. Then I remembered Noah's suggestion [1] in Justin's older thread on this same hang [2]: clear FatalError when we relaunch startup. The attached v1 tries that instead, with a small TAP test. AFAICS, the old children are gone and shmem is rebuilt by then. FWIW, if the new checkpointer crashes in that window, we now do a full cleanup instead of silently respawning it. Does this look like the right point to clear the flag? One wrinkle is #19623 [3]. The special case from ead8f696b7c becomes unreachable with this, so the patch removes it and puts back the Assert(!FatalError) in HandleFatalError(). IIUC, if every new child fails, we could now reap another child's exit before the startup process's and go for another restart instead of giving up. Michael, do you think that case still needs special handling? I haven't tested that scenario. I've added a WIP TAP test too which may need modifications. (I haven't tried the patch on the back branches yet.) Regards, Ayush [1] https://www.postgresql.org/message-id/[email protected] [2] https://www.postgresql.org/message-id/flat/ZWlrdQarrZvLsgIk@pryzbyj2023 (Thomas's patch there is CF 4884) [3] https://www.postgresql.org/message-id/flat/[email protected]
v1-0001-Clear-FatalError-when-relaunching-startup.patch
Description: Binary data
