On Mon, 25 Apr 2022, Steven D wrote:
David,
Thanks for all your help today, I committed a few changed to our config today
and i'll keep an eye out for changes next day or so.
Mind if I drop another pstats here after some bake in for a re-review?
not a problem
Here's what I updated it too based on your pointers.
module(load="imudp")
module(load="imtcp" MaxListeners="100" AddtlFrameDelimiter="000" KeepAlive="on"
KeepAlive.Probes="1" KeepAlive.Time="10")
module(load="impstats" interval="30" ruleset="pstats_rule")
ruleset(name="pstats_rule") {
action(name="pstats_rule"
type="omfile"
File="/var/log/rsyslog_pstats.log"
FileCreateMode="0744"
FileOwner="loguser"
FileGroup="loguser")
}
input(type="imudp" port="10514" ruleset="firewall_rule")
input(type="imtcp" port="10514" ruleset="firewall_rule")
template(name="firewall_logs" type="string"
string="/data/logs/pan/10514/%fromhost-ip%/syslog.log")
ruleset(name="firewall_rule") {
action(name="firewall_rule"
type="omfile"
FileCreateMode="0744"
DirCreateMode="0755"
FileOwner="loguser"
FileGroup="loguser"
DirOwner="loguser"
DirGroup="loguser"
DynaFile="firewall_logs"
DynaFileCacheSize = "50")
hopefully this was 500 not 50 based on your prior comment (but the pstats will
be clear once you get this large enough)
David Lang
}
________________________________
From: David Lang <da...@lang.hm>
Sent: Sunday, April 24, 2022 12:32 PM
To: Steven D <pheerl...@hotmail.com>
Cc: David Lang <da...@lang.hm>; Steven D via rsyslog <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
On Sun, 24 Apr 2022, Steven D wrote:
* dynafilecachsize is a global setting, I don't need to specify it per
ruleset/action?
I believe that it's per action (with the action() syntax, there are no global
settings except encryption)
* Assuming so and I have ~300 unique hosts writing to files, would
"dynafilecachsize = 500" be too much?
no, that sounds very reasonable. The current pstats output shows hundreds of
thousands of cache evictions, that should drop to near zero (pretty much only
showing up if you have date as part of it and the date changes). A small number
is fine, thousands is bad, hundreds of thousands very bad
David Lang
Regards,
Steven
________________________________
From: David Lang <da...@lang.hm>
Sent: Sunday, April 24, 2022 11:37 AM
To: Steven D <pheerl...@hotmail.com>
Cc: David Lang <da...@lang.hm>; Steven D via rsyslog <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
you definantly need to increase the dynacachesize for the firewall logs
also, if you add name= to the action, the pstats lines will be named by that
rather than action #
bump up the cache size so that it can keep track of all the files that will be
getting logs at the same time (plus a bit to be on the safe side, it REALLY
hurts to have it below the working set size) and see what that does to things.
I'll bet that cpu utilization increases and you have less problems with losing
logs.
if you continue to have problems, try to get a pstats dump of the period where
you lose some logs so we can see what it looks like.
having the max main queue size hit almost 4k seems likely to be an indication of
a problem as well, but that may go away once we get the cache size reasonable
David Lang
On Sun, 24 Apr 2022, Steven D wrote:
Date: Sun, 24 Apr 2022 15:27:47 +0000
From: Steven D <pheerl...@hotmail.com>
To: David Lang <da...@lang.hm>
Cc: Steven D via rsyslog <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
Great, I'll eyeball the impstats module options a lil more closely.
Attached is a few cycles with the current settings, sanitized some of the rule
names.
________________________________
From: David Lang <da...@lang.hm>
Sent: Sunday, April 24, 2022 11:06 AM
To: David Lang <da...@lang.hm>
Cc: Steven D <pheerl...@hotmail.com>; Steven D via rsyslog
<rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
and if you can post a couple cycles of the pstats output I can help explain
what's what there and see if there's anything obvious.
David Lang
On Sun, 24 Apr 2022, David Lang wrote:
Date: Sun, 24 Apr 2022 08:05:22 -0700 (PDT)
From: David Lang <da...@lang.hm>
To: Steven D <pheerl...@hotmail.com>
Cc: David Lang <da...@lang.hm>,
Steven D via rsyslog <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
On Sun, 24 Apr 2022, Steven D wrote:
Re: Load balancer - that makes sense to me as well.
I've added this line to our config, does it seem appropriate for pstats?
Our Linux team keeps a tight grip on rights, so i'm pretty limited in what
I can do/access outside of rsyslog and the SIEM agent configs... I'll have
to write the file out where I can actually access it (rolleyes)
module(load="impstats" interval="30" ruleset="pstats_rule")
when things are working normally this is good, when they aren't, it's best to
have the module write to a file directly (see the module options)
ruleset(name="pstats_rule") {
action(type="omfile"
File="/var/log/rsyslog_pstats.log"
FileCreateMode="0744"
FileOwner="loguser"
FileGroup="loguser")
}
Running Top + H now to get a feel on resource usage, but at first glance
nothing is really about 1~2%
what does wait time look like?
David Lang
________________________________
From: David Lang <da...@lang.hm>
Sent: Sunday, April 24, 2022 10:39 AM
To: Steven D <pheerl...@hotmail.com>
Cc: David Lang <da...@lang.hm>; Steven D via rsyslog
<rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
On Sun, 24 Apr 2022, Steven D wrote:
Would setting the KeepAlives in the rsyslog config on the server-side help
to manage the (zombie?) TCP connections.?
* The load balancer being in the middle feels like it's the cause of
repeated ESTABLISHED connections, but to keep HA/redundancy it's kind of a
necessary evil.
by the way, I think the fact that the load balancer cuts the connection and
the
server doesn't know it's cut and has to wait for it to time out (a very
long
time) is the cause of the large number of ESTABLISHED connections
David Lang
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
