Re: [rsyslog] Basic Rsyslog Troubleshooting

David Lang via rsyslog Sun, 24 Apr 2022 07:30:13 -0700

On Sun, 24 Apr 2022, Steven D wrote:

No sir, no encryption is in play. Just plain ol TCP syslog. We're running 
rsyslog v8.1911


I don't think we've enforced any additional dynafile settings, but i'll double 
check there's not an override I don't know about. Do youz have a rough order 
guidance for dynafilecachesize settings?

larger than the number of files that will be open at any time. If this issmaller than the number of files being actively written to, every time you get alog for a new file, rsyslog has to first close (and flush writes to) an oldfile. performance utterly collapses.

After getting thoroughly lost in the Google rabbit hole, I had other questions;

Would using imptcp over imtcp help?

 *   I saw mentions that imptcp handles connections better/performs better.

should not be a significant difference for the volume you describe, but won'thurt to try

Would setting the KeepAlives in the rsyslog config on the server-side help to 
manage the (zombie?) TCP connections.?

 *   The load balancer being in the middle feels like it's the cause of 
repeated ESTABLISHED connections, but to keep HA/redundancy it's kind of a 
necessary evil.

yes, that should help avoid the connection being idle long enough for the loadbalancer to break the connection (and each break probably causes a log loss), Iwould also enable rebind interval. I like to set it to reconnect every fewseconds under high log volume to give the load balancers the best chance to work

you can use a different method for failover. I like using pacemaker/corosync tomove an IP between the two systems (it has the added advantage that you can usethe CLUSTERIP feature to do rough load balancing between systems without anexternal loadbalancer)


David Lang

For reference, our main input config looks like this. There are a number of 
other input entries, but they're all variations on this for different log 
source types (servers, apps, etc).

module(load="imudp")
module(load="imtcp" MaxListeners="100" AddtlFrameDelimiter="000")

input(type="imudp" port="10514" ruleset="firewall_rule")
input(type="imtcp" port="10514" ruleset="firewall_rule")
template(name="firewall_logs" type="string" 
string="/data/logs/firewall/10514/%fromhost-ip%/syslog.log")
ruleset(name="firewall_rule") {
action(type="omfile"
     FileCreateMode="0744"
     DirCreateMode="0755"
     FileOwner="loguser"
     FileGroup="loguser"
     DirOwner="loguser"
     DirGroup="loguser"
     DynaFile="firewall_logs")
}
.
.
.
[snip]

Thank again, really appreciate the insight.
________________________________
From: rsyslog <rsyslog-boun...@lists.adiscon.com> on behalf of Steven D via rsyslog 
<rsyslog@lists.adiscon.com>
Sent: Sunday, April 24, 2022 8:57 AM
To: David Lang <da...@lang.hm>; Steven D via rsyslog <rsyslog@lists.adiscon.com>
Cc: Steven D <pheerl...@hotmail.com>
Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting

David

Thanks for the depth of this reply. Let me feed back in some additional info.

We've removed the load balancer from the syslog path as part of troubleshooting 
and the behavior didn't change. We continued to see log loss using TCP(most 
notably for our firewalls), even when directing to a single receiver server.

So far as the network goes there is a clean path from sending log source 
(firewall) to the rsyslog receiver. So I guess I'm looking for guidance on what 
knobs I should look at turning on the RHEL/rsyslog side.

As far as RELP, will that receive standard inbound TCP syslog on the receiver 
server? I was under the impression (possibly mistaken) both sender/receiver 
needed to use RELP. I'm happy to test using that module versus imtcp if I'm 
wrong.

Thanks again.

Regards,

Steven.



-------- Original message --------
From: David Lang <da...@lang.hm>
Date: 4/24/22 8:27 AM (GMT-05:00)
To: Steven D via rsyslog <rsyslog@lists.adiscon.com>
Cc: Steven D <pheerl...@hotmail.com>
Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting

One problem with TCP load balancing of syslog messages is that the load
balancers do not understand the syslog protocol, so they can't rebalance at a
message boundry.

A second problem is that when a firewall or load balancer drops a connection,
the sender doesn't know that it's dropped until the next time it tries to
deliver a message. Since TCP doesn't have any way for the OS TCP stack to tell
software "that message that you submitted to an open connection, and I accepted,
it can't be delivered" (once the OS accepts the message, the sender has to
assume that it will be delivered)

As a result, it's very easy for TCP syslog to be less reliable than UDP syslog.
The 'common sense wisdom' is that TCP is reliable because dropped packets inside
an ongoing connection will get retried, but dropped packets are actually very
uncommon inside a datacenter. They may happen when a firewall/router is
overloaded, but it's not very common. Back in 2006 or so I did testing and found
that within a local network, UDP was almost perfectly reliable (as long as the
receiver could keep up and not overflow the OS buffers)

Rsyslog has the rebindinterval feature, which tells the sender to disconnect and
reconnect periodically so that the load balancer has a chance to make a new
balancing decision.

you also want to make sure that the log stram is not idle for too long ('mark'
was the historical method of doing that, I prefer vmstat 60 |logger -t vmstat as
it's not much larger and an extremely dense set of information that can be very
useful when troubleshooting)

The other thing to look at is the RELP protocol, it was developed specifically
because TCP was designed to be reliable over an unreliable wire, but assumes
that both ends will remain up and the connection will not be cut by a middlebox.
RELP does full application level acks so that the sender knows that the receiver
rsyslog actually processed the message

with plain TCP, once the sending software submits data to the OS stack and the
OS stack says it's accepted the data, the data then sits in a buffer on the
sending machine, then gets sent over the wire (with retries), then sits in a
buffer on the receiving machine until the receiving software reads it. If
anything causes the connection to be terminated (firewall, load balancer, crash
on the receiving machine, etc) the data will be lost and the sending software
has no way of learning about it.

David Lang


 On Sun, 24 Apr 2022, Steven D via rsyslog wrote:

Date: Sun, 24 Apr 2022 12:14:35 +0000
From: Steven D via rsyslog <rsyslog@lists.adiscon.com>
To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
Cc: Steven D <pheerl...@hotmail.com>
Subject: [rsyslog] Basic Rsyslog Troubleshooting

Greetings list

New to rsyslog list, not new to logging. We're experiencing an odd issue where 
TCP syslog messages are being dropped at seemingly random intervals...hoping to 
get some input.

The TLDR on our architecture is we have set up a couple rsyslog receivers 
behind a Netscaler Load balancer. Multiple platforms/devices are configured to 
send syslog to the load balancer, which distributes to the receivers. Receivers 
are running RHEL v8 and rsyslog v8.1911. Receivers write files to disk, which 
we then read with a SIEM agent.

We've got a modestly sized environment with a syslog client base of 200-300 
servers, 30 networking devices (including firewalls) and some applications all 
directing logging to the load balancer.

Our config file is pretty vanilla, no cache, or advanced tweaks. Just using the "imtcp" 
and "imudp" modules and rulesets to write files to disk based on the sending host IP/port.

The first problem we're seeing is that hosts sending via TCP have log messages 
missed (never written to disk), where UDP seems more reliable. When switching 
the firewalls to UDP, throughput nearly doubles and message loss is less 
noticeable (yeah I know it's still UDP).

Possibly related is that we've noticed that each receiver also holds a lot of 
"Established" connections for back to the clients, but different ports. 
(Possible session/connection exhaustion?)

Any guidance on how we can approach and troubleshoot this issue would be 
appreciated. Commands, dummy guides, sarcasm all welcome.

Thanks much

Regards,

Steven.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Basic Rsyslog Troubleshooting

Reply via email to