Lars Bjørndal writes: > [Gerald] > > What is in the logs about why the locks remain? There should be something > > in > > the system log about it if systemd kills a service process because it > > doesn't stop in time during shutdown... -Gerald > > I got this:
It's hard to say what's happening because your traceback(s) are truncated. The fact that the timestamps are all the same suggests that the problem isn't that systemd gets tired of waiting and eventually force-kills (signal 9) Mailman. Rather, systemd times out because Mailman aborts from an internal Exception (not in the log you reported) and never removes the pidfile, so systemd doesn't know that Mailman exited. The locks remain because Mailman aborted. Normally you don't clean up locks because you don't want whatever happened to destroy even more data, but experience has shown that these events "never" (as far as we know) destroy data. The store-and-forward discipline for the message-handling queues is very robust. I would say it's nearly mathematically certain that Mailman exceptions cannot destroy messages. Apparently we get the granularity of database transactions right, but I'm not as sure of that. I'm guessing that all of the runners eventually raise the same exception, but the first mention is likely to be quite a way down in the log file. We need to see the tracebacks all the way to the end or we can't help. It's likely to be *very* long if all 15 or so runners raise the exception. For the terminally curious, a few comments are below the lines being analyzed. Since the timestamps are all the same, I'm going to collect the logs for each process for clarity. > juni 12 08:13:28 dalen.lamasti.net systemd[1]: Stopping mailman3.service - > GNU Mailing List Manager... > juni 12 08:13:49 dalen.lamasti.net mailman[497907]: Shutting down Mailman's > master runner This process is the `mailman` command line utility to shut down the Mailman system. (Mailman appears to have run quite a while before being stopped.) > juni 12 08:13:51 dalen.lamasti.net mailman[1851]: Traceback (most recent > call last): This appears to be a runner process (queue handling worker), forked from the master process mentioned above. But it might be the master itself. This is probably in the boot process for the OS, invoked automatically by init (systemd). > juni 12 08:13:51 dalen.lamasti.net mailman[1851]: File > "/opt/mailman3/venv/lib64/python3.12/site-packages/click/core.py", line > 1363, in main > juni 12 08:13:52 dalen.lamasti.net mailman[1851]: rv = self.invoke(ctx) > juni 12 08:13:52 dalen.lamasti.net mailman[1851]: ^^^^^^^^^^^^^^^^ > juni 12 08:13:52 dalen.lamasti.net mailman[1851]: File > "/opt/mailman3/venv/lib64/python3.12/site-packages/click/core.py", line > 1226, in invoke > juni 12 08:13:52 dalen.lamasti.net mailman[1851]: return > ctx.invoke(self.callback, **ctx.params) > juni 12 08:13:52 dalen.lamasti.net mailman[1851]: > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > juni 12 08:13:52 dalen.lamasti.net mailman[1851]: File > "/opt/mailman3/venv/lib64/python3.12/site-packages/click/core.py", line 794, > in invoke > juni 12 08:13:52 dalen.lamasti.net mailman[1851]: return callback(*args, > **kwargs) > juni 12 08:13:52 dalen.lamasti.net mailman[1851]: > ^^^^^^^^^^^^^^^^^^^^^^^^^ > juni 12 08:13:52 dalen.lamasti.net mailman[1851]: File > "/opt/mailman3/venv/lib64/python3.12/site-packages/click/decorators.py", > line 34, in new_func > juni 12 08:13:52 dalen.lamasti.net mailman[1851]: return > f(get_current_context(), *args, **kwargs) > juni 12 08:13:52 dalen.lamasti.net mailman[1851]: > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > juni 12 08:13:52 dalen.lamasti.net mailman[1851]: File > "/opt/mailman3/venv/lib64/python3.12/site-packages/mailman/bin/runner.py", > line 198, in main > juni 12 08:13:52 dalen.lamasti.net mailman[1851]: sys.exit(runner.status) > juni 12 08:13:52 dalen.lamasti.net mailman[1851]: SystemExit: 15 This is a "shut down gracefully" signal received presumably from the master runner process. This looks normal. > juni 12 08:13:52 dalen.lamasti.net mailman[1851]: During handling of the > above exception, another exception occurred: "Above exception" == SystemExit, raised when receiving the "shut down" signal. > juni 12 08:13:52 dalen.lamasti.net mailman[1851]: Traceback (most recent > call last): > juni 12 08:13:52 dalen.lamasti.net mailman[1851]: File > "/opt/mailman3/venv/bin/runner", line 8, in <module> > juni 12 08:13:52 dalen.lamasti.net mailman[1851]: sys.exit(main()) > juni 12 08:13:52 dalen.lamasti.net mailman[1851]: ^^^^^^ > juni 12 08:13:52 dalen.lamasti.net mailman[1851]: File > "/opt/mailman3/venv/lib64/python3.12/site-packages/click/core.py", line > 1442, in __call__ > juni 12 08:13:52 dalen.lamasti.net mailman[1851]: return > self.main(*args, **kwargs) > juni 12 08:13:52 dalen.lamasti.net mailman[1851]: > ^^^^^^^^^^^^^^^^^^^^^^^^^^ > juni 12 08:13:52 dalen.lamasti.net mailman[1851]: File > "/opt/mailman3/venv/lib64/python3.12/site-packages/click/core.py", line > 1362, in main > juni 12 08:13:52 dalen.lamasti.net mailman[1851]: with > self.make_context(prog_name, args, **extra) as ctx: > juni 12 08:13:52 dalen.lamasti.net mailman[1851]: > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > juni 12 08:13:52 dalen.lamasti.net mailman[1851]: File > "/opt/mailman3/venv/lib64/python3.12/site-packages/click/core.py", line 480, > in __exit__ > juni 12 08:13:52 dalen.lamasti.net mailman[1851]: def __exit__( > juni 12 08:13:52 dalen.lamasti.net mailman[1851]: File > "/opt/mailman3/venv/lib64/python3.12/site-packages/mailman/core/runner.py", > line 117, in signal_handler This is not the end of the log for this process, the line of code should be quoted later in the log, and probably the log continues. > juni 12 08:13:52 dalen.lamasti.net mailman[1855]: Traceback (most recent > call last): Another runner process. Note that the traceback is the same as the traceback for process 1851, as far as it goes. > juni 12 08:13:52 dalen.lamasti.net mailman[1855]: File > "/opt/mailman3/venv/lib64/python3.12/site-packages/click/core.py", line 794, > in invoke > juni 12 08:13:52 dalen.lamasti.net mailman[1855]: return callback(*args, > **kwargs) > juni 12 08:13:52 dalen.lamasti.net mailman[1855]: > ^^^^^^^^^^^^^^^^^^^^^^^^^ > juni 12 08:13:52 dalen.lamasti.net mailman[1855]: File > "/opt/mailman3/venv/lib64/python3.12/site-packages/click/decorators.py", > line 34, in new_func > juni 12 08:13:52 dalen.lamasti.net mailman[1855]: return > f(get_current_context(), *args, **kwargs) > juni 12 08:13:52 dalen.lamasti.net mailman[1855]: > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > juni 12 08:13:52 dalen.lamasti.net mailman[1855]: File > "/opt/mailman3/venv/lib64/python3.12/site-packages/mailman/bin/runner.py", > line 198, in main > juni 12 08:13:52 dalen.lamasti.net mailman[1855]: sys.exit(runner.status) > juni 12 08:13:52 dalen.lamasti.net mailman[1855]: SystemExit: 15 > juni 12 08:13:52 dalen.lamasti.net mailman[1855]: During handling of the > above exception, another exception occurred: Again, the log is truncated for this process. > juni 12 08:13:52 dalen.lamasti.net mailman[1860]: Traceback (most recent > call last): Another runner process. Note that the traceback is the same as the traceback for process 1851, as far as it goes. > juni 12 08:13:52 dalen.lamasti.net mailman[1860]: File > "/opt/mailman3/venv/lib64/python3.12/site-packages/click/core.py", line 794, > in invoke > juni 12 08:13:52 dalen.lamasti.net mailman[1860]: return callback(*args, > **kwargs) > juni 12 08:13:52 dalen.lamasti.net mailman[1860]: > ^^^^^^^^^^^^^^^^^^^^^^^^^ > juni 12 08:13:52 dalen.lamasti.net mailman[1860]: File > "/opt/mailman3/venv/lib64/python3.12/site-packages/click/decorators.py", > line 34, in new_func > juni 12 08:13:52 dalen.lamasti.net mailman[1860]: return > f(get_current_context(), *args, **kwargs) > juni 12 08:13:52 dalen.lamasti.net mailman[1860]: > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > juni 12 08:13:52 dalen.lamasti.net mailman[1860]: File > "/opt/mailman3/venv/lib64/python3.12/site-packages/mailman/bin/runner.py", > line 198, in main > juni 12 08:13:52 dalen.lamasti.net mailman[1860]: sys.exit(runner.status) The exception is almost certainly another SystemExit. It looks like all of the runners are going to have the same logs, interleaved. Steve -- GNU Mailman consultant (installation, migration, customization) Sirius Open Source https://www.siriusopensource.com/ Software systems consulting in Europe, North America, and Japan _______________________________________________ Mailman-users mailing list -- [email protected] To unsubscribe send an email to [email protected] https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ Archived at: https://lists.mailman3.org/archives/list/[email protected]/message/R52VEGZEOW3QGIMIVTZ6U4N552ZQBPUU/ This message sent to [email protected]
