Thanks for the replies.
My requirement is that I have a daemon that may generate a burst of
syslogs, say, every minute (when a certain subsystem is overloaded).
We do not want to write all these syslogs to /var/log/messages. We
also do not want to forward so many of those syslogs to a remote
syslog server. We wanted to rate limit them, maybe 1 per hour or so.
For now, I have tried 'omprog' module and sending such syslogs to an
external program.
module( load="omprog" )
if ($syslogfacility-text == 'daemon' and $msg contains "Out of memory") then {
action(type="omprog"
binary="/usr/bin/rate_limit_daemon_syslogs.sh"
template="RSYSLOG_FileFormat")
stop
}
/usr/bin/rate_limit_daemon_syslogs.sh maintains state on when was the
last syslog forwarded or written locally. If this script decides its
ok to syslog, then it will call 'logger' with differently worded
message and different facility.
In our case we'd like to limit these logs at the source and not at an
intermediate syslog collector. I took a quick look at the sampling
docs and that may not apply to our scenario.
Thanks.
On Wed, Apr 3, 2024 at 12:07 AM Mariusz Kruk via rsyslog
<rsyslog@lists.adiscon.com> wrote:
>
> A piece of feedback against
> https://www.rsyslog.com/doc/tutorials/random_sampling.html
>
> I know it's obvious for some people but for some it might not be - it
> will collect _about_ 20% of logs provided the overall number of events
> is big enough. Since it's a random-based mechanism, it's... random.
>
> Additionally, I had a similar thing implemented for load-balancing to
> different "backends" and for some reason it didn't work uniformly (I
> think I already wrote about this) - one of four backends was never
> chosen. The thing is I was choosing not based directly on random(4) but
> on (random(some_bigger_integer) % 4) which I thought would yield more
> uniform randomness. It appears I was wrong.
>
> MK
>
>
> On 3.04.2024 08:51, Rainer Gerhards via rsyslog wrote:
> > This sounds a bit like you are looking for this:
> >
> > https://www.rsyslog.com/doc/tutorials/log_sampling.html
> >
> > HTH
> > Rainer
> >
> > El mié, 3 abr 2024 a las 3:25, Prasad Koya via rsyslog
> > (<rsyslog@lists.adiscon.com>) escribió:
> >> Hi
> >>
> >>
> >> module( load="imuxsock" )
> >> module( load="imklog" )
> >>
> >> if ($syslogfacility-text == 'kern' and $msg contains "Out of memory") then
> >> {
> >> action(type="omfile" queue.size="1000" queue.type="LinkedList"
> >> queue.dequeueSlowDown="3600000000" file="/var/log/oom")
> >> }
> >>
> >>
> >> If we have a flurry of, for example, "Out of memory" messages from the
> >> facility 'kern', we'd like to keep only, say, one per hour in a
> >> separate file.
> >>
> >> https://rsyslog.readthedocs.io/en/latest/rainerscript/queue_parameters.html
> >> says do not set low value for queue.size.
> >>
> >> Above page also says queue.dequeueslowdown can be used as
> >> rate-limiting. Maybe that doesn't apply for "omfile" as I don't see
> >> above rsyslog.conf working. high and low watermarks are not applicable
> >> here as it's not a disk based queue. Appreciate any pointers.
> >>
> >> Thank you.
> >> _______________________________________________
> >> rsyslog mailing list
> >> https://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >> DON'T LIKE THAT.
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> > LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
