Actually, I realized that dependency on the network.target is not the sole problem why shutdown logs are not persisted. Upon shutdown, there's a delay in rsyslog fetching new logs from the journal via the journal API. By the time it can access the journal, a SIGTERM signal has already been sent to the main rsyslog process. While this signal doesn't immediately terminate rsyslog, it does halt the processing of incoming data instantly.
On Wed, Mar 13, 2024 at 7:32 PM David Lang <da...@lang.hm> wrote: > you could put the remote sender things in a seprate ruleset with a queue > on that > ruleset, that would let the rest of the config run without the network > (accumulating early logs and gathering shutdown logs up to the point that > rsyslog gets shut down) > > In my use case, messages enter rsyslog via the imjournal module. By default, I have only one ruleset (the default). I want to store logs locally and also forward them to a remote rsyslog server. So do I understand correctly, that I need to create a ruleset that will have a queue and will contain an omfwd action and I need to call it in the default ruleset? The default ruleset would also handle writing logs to local files. > you can configure rsyslog to save the queue to disk at shutdown (but this > can > take time, so you may need to increase the systemd timeout for letting > rsyslog > do a clean shutdown) > > David Lang > > On Wed, 13 Mar 2024, Attila Lakatos via rsyslog wrote: > > > Date: Wed, 13 Mar 2024 13:49:19 +0100 > > From: Attila Lakatos via rsyslog <rsyslog@lists.adiscon.com> > > To: rsyslog-users <rsyslog@lists.adiscon.com> > > Cc: Attila Lakatos <alaka...@redhat.com> > > Subject: [rsyslog] Capturing shutdown logs > > > > Recently I came across an observation where we are not able to capture > > normal reboot/shutdown logs on Fedora/RHEL distributions. In these > > environments, systemd is responsible for starting the rsyslog service. A > > service can have multiple dependencies, which influence how early or how > > late rsyslog is started or stopped. Many years ago, we added dependency > for > > the network.target and network-online.target into the service file [1]. > If > > rsyslog started before establishing network access, it would be unable to > > transmit messages to remote destinations during that period, resulting in > > the generation of misleading information about the unavailability of > > certain remote targets (e.g. not able to resolve hostnames). > > However, this approach results in a significant tradeoff. While it > prevents > > misleading unavailability messages during network setup and shutdown, it > > also causes rsyslog to *exit* *early* during shutdown, leading to missed > > logs regarding the graceful termination of other programs. This > limitation > > extends to system reboots as well. Thus, while addressing one issue, the > > current service configuration introduces another. > > By default, we retrieve shutdown events from the journal using the > > imjournal module. Journal log data is stored in memory so after shutdown, > > logs are not preserved. > > > > Has someone faced this problem? Are there any known workarounds? > > > > [1] > > > https://github.com/deoren/rsyslog-examples/blob/master/etc/systemd/system/rsyslog.service.d/10-wait-on-network.conf > > _______________________________________________ > > rsyslog mailing list > > https://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > > > > _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
