we'd need more details about exactly what is happening, but I'll point out that
when the system crashes, all bets are off, anything that any application tries
to do to save it's state may fail because of the crash. Power failures are even
worse (unless you have a UPS with enough notification to do a clean shutdown)
rsyslog keeps logs in ram unless you configure i to use disk-only queues (not
disk assisted queues), and if you do use disk-only queues, you will slow your
performance by a factor of 1000x or more, as well as doing a hube amount of disk
I/O (with the performance impact that causes). you are probably better in
practice to send your logs out over the network and let them get picked up by
another system than thinking you will avoid loosing logs in a crash/power outage
So let's backup and have you explain the failure scenario you are looking at and
what you expect rsyslog to do under those conditions.
David Lang
On Mon, 27 Jan 2020,
Roman Shaposhnik via rsyslog wrote:
Date: Mon, 27 Jan 2020 00:10:59 -0800
From: Roman Shaposhnik via rsyslog <[email protected]>
To: [email protected]
Cc: Roman Shaposhnik <[email protected]>
Subject: [rsyslog] rsyslog 8.1904.0 (aka 2019.04) (Alpine build) fails to
recover
Hi!
over at Linux Foundation's Project EVE
https://github.com/lf-edge/eve/blob/master/docs/README.md
https://www.lfedge.org/projects/eve/
we've recently considered moving to rsyslog
as our nexus for all the logging needs. Our
use case is pretty close to what you would find
in an embedded linux (EVE is meant to run on
IoT/Edge like devices) and as such power crashes
and intermittent connectivity are the norm. Looking
at rsyslog's persistent queue architecture gave us
a nice feeling that rsyslog project may actually be
a great fit for our needs.
However, recently, we've seen quite a few crashes
related to rsyslog trying to recover from a previous
power failure. You can see the log at the bottom
of this email.
With that in mind, I'm wondering if a use case like
that would still be considered a priority for the project
and what would be the best way to make sure that
we iron out the kinks together.
Btw, here's our rsyslog.conf:
https://github.com/lf-edge/eve/blob/master/pkg/rsyslog/rsyslog.conf
Please let me know if you need any more details and
otherwise thanks for an amazing project so far!
Thanks,
Roman.
rsyslogd: file '/persist/rsyslog/main_edge_node_log_queue.00000980':
open error: No such file or directory [v8.1904.0 try
https://www.rsyslog.com/e/2040 ]
rsyslogd: main Q: qDeqDisk error happened at around offset 0
[v8.1904.0 try https://www.rsyslog.com/e/2040 ]
rsyslogd: main Q: error dequeueing element - ignoring, but strange
things may happen [v8.1904.0 try https://www.rsyslog.com/e/2040 ]
rsyslogd: file /persist/rsyslog/main_edge_node_log_queue.00000979: fd
8 no longer valid, recovery by reopen; if you see this, consider
reporting at
https://github.com/rsyslog/rsyslog/issues/3404 so that we know when it
happens. Include output of uname -a. OS error reason: Bad file
descriptor [v8.1904.0 try https://www.rsyslog.com/e/2027 ]
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
