enqueued is a running total of how many messages have been put in teh queue
since you restarted (unless you configure impstats to reset it's counters each
run, but that can lose some data due to race conditions)
it's sad but true that most attempts to optimize rsyslog actually end up hurting
performance mroe than they help, and rsyslog with simple configs is frequently
fast enough to not need any optimization.
having too many threads and too many queues can actually slow you down.
with omfile for example, the overhead of locking the queue with one thread,
inserting the message, unlocking the queue and
then locking the queue with a different thread, marking that you are starting
to work on the message, unlocking the queue, locking the queue, marking that you
processed the message and unlocking the queue absolutly dwarf the cost of just
writing the log to disk
multiple threads can also cause more locking overhead. you should only increase
threads if your measurements show that you have a thread maxing out a core (top,
then hit H to show threads, see if any thread is hitting 100% cpu)
multiple thread when you are using omfile is even worse, as the omfile then has
to do locking itself to prevent the multiple threads from writing at the same
time.
you only want to use threads when you have expensive processing (which can be a
bad template, but there are ways to improve that)
now, a queue on a ruleset that is being tied to an input is a bit different,
that queue then replaces the use (and locking) of the main queue and can be a
win.
the bigger win is usually just increasing the batch size, but increasing the
size produces diminishing returns, above a few hundred to a few thousand is
seldom useful
What is the volume of logs you are trying to process? what is making you think
you need to change things to improve performance?
please show a couple rounds of impstats output under load, and ideally a
smapshot of top (with H to show the threads), and iostat -cdtyz 10 or something
similar to show the disk activity during this time.
David Lang
Ubuntu 22.04LTS
Rsyslog 8.2112.0
This server is setup to receive Syslog data from up to 13 sources, mostly
networking like Cisco and Meraki. Recently had been troubleshooting an issue
where the Rsyslog daemon will quit after 4-5 days. Have not been able to
determine an actual cause.. but when looking at the server I began to tail the
rsyslog_stat.log file. Today over the course of ~7 hours the enqueued value for
the `firewall` log for example, rose from 0 to 3.8M. There were no signs of it
ever emptying. The same goes with Meraki and VSCA.
So I found the document here:
https://www.rsyslog.com/doc/master/examples/high_performance.html Made some
changes to what I think might help but so far it’s been ~4 hours and the stats
log is exhibiting the same things as before.
Is this a valid way to determining the performance of Rsyslog? If not, is
there a better way?
Am I understanding queues correctly in that they should not just increase in
count forever?
Yesterday at around 3pm EST I restarted rsyslog, checking this morning the
`enequeue` values for meraki is 37.5M, vcsa 7.4M and firewalls 3.4M. I feel
like I’m doing something wrong here..
Below are the related conf files:
Rsyslog.conf:
$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog # provides kernel logging support (previously done by rklogd)
$ModLoad immark # provides --MARK-- message capability
module(load="imudp" threads="2" timeRequery="8" batchSize="128")
input(type="imudp"
port=["514","20514","20515","20516","20517","20518","20519","20520","20525","20526","20527","20528","20529","20530"]
name="" name.appendPort="on")
module(load="impstats" interval="10" log.file="/var/log/rsyslog_stats.log"
log.syslog="off")
module(load="imtcp" MaxSessions="500")
input(type="imtcp" port="514")
/etc/rsyslog.d/05-remote-syslog.conf:
ruleset(name="switches20514" queue.type="linkedlist" queue.workerThreads="4"
queue.workerThreadMinimumMessages="3000"){
action(type="omfile" file="/var/log/remote-syslog/switches.log")
}
ruleset(name="routers20515" queue.type="linkedlist" queue.workerThreads="2"
queue.workerThreadMinimumMessages="3000"){
action(type="omfile" file="/var/log/remote-syslog/routers.log")
}
ruleset(name="wlan20516" queue.type="linkedlist" queue.workerThreads="1"
queue.workerThreadMinimumMessages="5000"){
action(type="omfile" file="/var/log/remote-syslog/wlan.log")
}
ruleset(name="firewalls20517" queue.type="fixedArray" queue.size="250000"
queue.dequeueBatchSize="4096" queue.workerThreads="6"
queue.workerThreadMinimumMessages="60000"){
action(type="omfile" file="/var/log/remote-syslog/firewalls.log"
ioBufferSize="64K" flushOnTXEnd="off")
}
ruleset(name="stealth20518" queue.type="linkedlist" queue.workerThreads="2"
queue.workerThreadMinimumMessages="5000"){
action(type="omfile" file="/var/log/remote-syslog/stealth.log")
}
ruleset(name="nexus20519" queue.type="linkedlist" queue.workerThreads="2"
queue.workerThreadMinimumMessages="5000"){
action(type="omfile" file="/var/log/remote-syslog/nexus.log")
}
ruleset(name="lomsmx20521" queue.type="linkedlist" queue.workerThreads="1"
queue.workerThreadMinimumMessages="6000"){
action(type="omfile" file="/var/log/remote-syslog/lom_smx11.log")
}
ruleset(name="vcsa20525" queue.type="linkedlist" queue.workerThreads="4"
queue.workerThreadMinimumMessages="3000"){
action(type="omfile" file="/var/log/remote-syslog/vcsa.log")
}
ruleset(name="ciscoasa20526" queue.type="linkedlist" queue.workerThreads="2"
queue.workerThreadMinimumMessages="3000"){
action(type="omfile" file="/var/log/remote-syslog/asa.log")
}
ruleset(name="pwrapc20527" queue.type="linkedlist" queue.workerThreads="1"
queue.workerThreadMinimumMessages="3000"){
action(type="omfile" file="/var/log/remote-syslog/power_apc.log")
}
ruleset(name="pwrraritan20528" queue.type="linkedlist" queue.workerThreads="4"
queue.workerThreadMinimumMessages="6000"){
action(type="omfile" file="/var/log/remote-syslog/power_raritan.log")
}
ruleset(name="ise20529" queue.type="linkedlist" queue.workerThreads="4"
queue.workerThreadMinimumMessages="5000"){
action(type="omfile" file="/var/log/remote-syslog/ise.log")
}
ruleset(name="meraki20530" queue.type="fixedArray" queue.size="250000"
queue.dequeueBatchSize="4096" queue.workerThreads="4"
queue.workerThreadMinimumMessages="60000"){
action(type="omfile" file="/var/log/remote-syslog/meraki.log"
ioBufferSize="64K" flushOnTXEnd="off")
}
input(type="imudp" port="20514" ruleset="switches20514")
input(type="imudp" port="20515" ruleset="routers20515")
input(type="imudp" port="20516" ruleset="wlan20516")
input(type="imudp" port="20517" ruleset="firewalls20517")
input(type="imudp" port="20518" ruleset="stealth20518")
input(type="imudp" port="20519" ruleset="nexus20519")
input(type="imudp" port="20521" ruleset="lomsmx20521")
input(type="imudp" port="20525" ruleset="vcsa20525")
input(type="imudp" port="20526" ruleset="ciscoasa20526")
input(type="imudp" port="20527" ruleset="pwrapc20527")
input(type="imudp" port="20528" ruleset="pwrraritan20528")
input(type="imudp" port="20529" ruleset="ise20529")
input(type="imudp" port="20530" ruleset="meraki20530")
[Jamf]
Ben Hart
IT Systems Administrator II
100 Washington Ave S, Minneapolis, MN 55401
[Phone]
+00 1 989 424 0187
[Email]
ben.h...@jamf.com
[Web]
www.jamf.com<https://www.jamf.com>
[Facebook] [Twitter] [LinkedIn] [YouTube]
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
