read the documents I posted above and they will explain most of what you are
seeing.
the reason it doesn't fail instantly is that the OS has a buffer for the TCP
connection, and that buffer takes a little time to fill up.
retrycount tells rsyslog how many times to retry each message before dropping
it.
closing and re-opening the connection won't help you if the network is down, as
everything will block waiting for the connection to get re-established.
If you enable impstats and have it write directly to a file (not send it though
the normal queue) then you will see the failures and see the action being
suspended after some failures.
David Lang
On Tue, 6 Feb 2024, Prasad Koya wrote:
Alex, excuse me for using this thread for posting my question. But my
question is also on the same topic.
I have configured rsyslogd (v8.2102.0) to forward syslogs to two remote
servers, one using TCP protocol and second one using UDP. Besides
forwarding, I also have a rule to log to a local file. Here are the
relevant sections from my rsyslog.conf. I can post the full file if needed.
$MainMsgQueueTimeoutEnqueue 0
local4.* action(type="omfwd" target="10.240.219.64"
action.resumeRetryCount="10" protocol="tcp" port="36456"
template="Syslogs_ForwardFormat")
local4.* action(type="omfwd" target="10.16.39.124"
protocol="udp" port="36456" template="Syslogs_ForwardFormat")
*.* /var/log/syslogs
After the TCP connection is established and messages are getting forwarded,
say someone pulls out the ethernet cable at the remote syslog collector 1
that is using tcp protocol. Some number of syslogs from this point are
still forwarded to the server 2 using udp protocol and written to the local
file as well. However if the TCP connection to the first server does not
recover then after some time, we do not see any syslogs in the local file
or at the second syslog server. I straced rsyslogd and all I see is the
recvmsg() calls on the /dev/log socket.
How do we tell rsyslog to drop the TCP connection if the other end is no
longer receiving the syslogs. Perhaps it can drop the connection and retry
after say X minutes. I tried using resumeRetryCount as documented at
https://www.rsyslog.com/doc/configuration/actions.html, but that is not
helping in my situation. Perhaps I'm missing some other "action" setting?
Appreciate if you can give me a pointer to a sample configuration or point
me to relevant documentation.
Thank you.
On Mon, Feb 5, 2024 at 11:11 AM David Lang via rsyslog <
rsyslog@lists.adiscon.com> wrote:
you have a queue of 1024 for rabbitmq, if there are more messages than
that
pending, other processing will stop until the queue can accept more
messages.
Setup a larger queue (potentially a disk assisted queue) to handle longer
outages.
you may also want to consider configuring the queue to throw away messages
if it
gets too full.
David Lang
On Mon, 5 Feb 2024, Alex via rsyslog wrote:
I have rsyslog (8.2310.0) in docker container from latest alpine image
as a
syslog collector with forwarding to logstash.local and rabbitmq.local
Rsyslog stops working when rabbitmq is unavailable and work fine when
rabbitmq server is up
How to configure rsyslog to work when the destination is unavailable ?
module(load="omrabbitmq")
module(load="imptcp" threads="3")
input(type="imptcp" port="514" ruleset="syslogCollector")
template(
name="json_syslog"
type="list"
option.json="on"
) {
constant(value="{")
constant(value="\"@timestamp\":\"")
property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"type\":\"syslog_json")
constant(value="\",\"tag\":\"")
property(name="syslogtag" format="json")
constant(value="\",\"relayhost\":\"") property(name="fromhost")
constant(value="\",\"relayip\":\"") property(name="fromhost-ip")
constant(value="\",\"logsource\":\"") property(name="source")
constant(value="\",\"hostname\":\"") property(name="hostname"
caseconversion="lower")
constant(value="\",\"program\":\"") property(name="programname")
constant(value="\",\"source\":\"") property(name="app-name"
caseConversion="lower" onEmpty="null")
constant(value="\",\"priority\":\"") property(name="pri")
constant(value="\",\"severity\":\"")
property(name="syslogseverity" caseConversion="upper")
constant(value="\",\"facility\":\"")
property(name="syslogfacility")
constant(value="\",\"severity_label\":\"")
property(name="syslogseverity-text")
constant(value="\",\"facility_label\":\"")
property(name="syslogfacility-text")
constant(value="\",\"message\":\"") property(name="msg"
format="json")
constant(value="\",\"end_msg\":\"")
constant(value="\"}\n")
}
ruleset(
name="syslogCollector"
) {
@@elk.local:5000;json_syslog
call send2mqtt
}
ruleset (
name="send2mqtt"
queue.type="LinkedList"
queue.size="1024"
queue.dequeueBatchSize="512"
queue.filename="q_mqtt"
queue.saveonshutdown="off"
queue.highwatermark="450"
queue.lowwatermark="50"
) {
action(
type="omrabbitmq"
host="rabbitmq.local"
verify_peer="off"
verify_hostname="off"
virtual_host="/"
user="xxx"
password="xxx"
exchange="syslog"
routing_key="messages"
body_template="json_syslog"
action.resumeRetryCount="0"
action.reportSuspension="on"
action.reportSuspensionContinuation="on"
action.resumeInterval="10"
)
}
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
