Hi,
I'm hoping to get some help tuning my rsyslog configuration for
performance. I'm running rsyslog 8.2302.0 using the omkafka plugin on a set
of kubernetes pods.
However, just examining one pod, it feels like something is not configured
properly (see config below). What I'm trying to do is use disk-assisted
queues
so that if the queue is full or a kubernetes pod gets deleted, then the pod
should save the data to disk so that it can be processed when the next
kubernetes pod comes back up.
For one pod, I see it is writing about 150 msg/sec on average. However,
when a spike of incoming data came in, the rsyslog service started quickly
falling behind resulting in messages being written to disk.
I restarted the pod and we were no longer receiving as much input traffic
so when that pod came back up, it processed 2.6 million messages that were
on disk for about 15 minutes -- about 3000 msg/sec.
My question is this: Why was this faster to crank through all the messages
on disk than it would be when messages are just coming through? Is there
something I can do to tweak my config
to make this perform better?
Note: kafka cluster is in the US region and rsyslog-service is in the EU
region so I know there will be some latency there but not really sure how
to get that metric to see what that is.
Thanks,
-Ken
Rsyslog config:
module(load="omkafka")
module(load="imptcp" threads="2" ProcessOnPoller="off")
module(load="omprog")
module(
load="impstats"
interval="10"
format="json"
resetCounters="off"
ruleset="process_stats"
)
input(type="imptcp" port="1514" ruleset="writeToKafka" KeepAlive="on"
DiscardTruncatedMsg="on")
template(name="raw" type="string" string="%rawmsg%")
ruleset(name="writeToKafka") {
action(type="omkafka"
confParam=["socket.keepalive.enable=true",
"compression.codec=none",
"batch.num.messages=20000",
"statistics.interval.ms=15000",
"queue.buffering.max.ms=100",
"queue.buffering.max.messages=200000",
"security.protocol=SASL_SSL",
"sasl.mechanisms=PLAIN",
"sasl.username=<username>",
"sasl.password=<password>"]
topic="rsyslog-ingest"
broker="<kafkaBroker>"
template="raw"
partitions.auto="on"
resubmitonfailure="on"
keepfailedmessages="on"
failedmsgfile="/data/failed-messages.log"
action.resumeRetryCount="-1"
action.resumeInterval="10"
queue.type="LinkedList"
queue.filename="remote-kafka"
queue.spoolDirectory="/data"
queue.saveonshutdown="on"
queue.size="250000"
queue.highwatermark="225000"
queue.lowwatermark="200000"
queue.maxdiskspace="7g"
queue.maxfilesize="10m"
queue.discardseverity="8"
errorFile="/var/log/rsyslog-kafka-errors.log"
)
}
ruleset(name="process_stats") {
action(
type="omprog"
name="to_exporter"
binary="/usr/local/bin/rsyslog_exporter"
)
}
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
