yes, both sender and receiver need to use relp are you trying to use encryption on the connection? what version of rsyslog are you running?
you mention a couple hundred senders writing to files based on the sender, did you change the dynafilecachsize from the default? if not, this is something you need to do (it uses more memory if too large, but cripples performance if even slightly too small)
when troubleshooting, it's a good idea to setup impstats writing to a file directly, it gives a lot of info about what's going on, queue sizes, etc.
when you are bypassing the load balancer, do you still have the extra stuck connections? I think I saw some discussion on reducing those recently, so the just releases v8.2204 may include a fix to help that. (I'd have to go digging back through the mail archives), but if we fix the other things, I don't expect it to be an issue in any case.
David Lang On Sun, 24 Apr 2022, Steven D wrote:
Date: Sun, 24 Apr 2022 12:57:43 +0000 From: Steven D <pheerl...@hotmail.com> To: David Lang <da...@lang.hm>, Steven D via rsyslog <rsyslog@lists.adiscon.com> Subject: RE: [rsyslog] Basic Rsyslog Troubleshooting David Thanks for the depth of this reply. Let me feed back in some additional info. We've removed the load balancer from the syslog path as part of troubleshooting and the behavior didn't change. We continued to see log loss using TCP(most notably for our firewalls), even when directing to a single receiver server. So far as the network goes there is a clean path from sending log source (firewall) to the rsyslog receiver. So I guess I'm looking for guidance on what knobs I should look at turning on the RHEL/rsyslog side. As far as RELP, will that receive standard inbound TCP syslog on the receiver server? I was under the impression (possibly mistaken) both sender/receiver needed to use RELP. I'm happy to test using that module versus imtcp if I'm wrong. Thanks again. Regards, Steven. -------- Original message -------- From: David Lang <da...@lang.hm> Date: 4/24/22 8:27 AM (GMT-05:00) To: Steven D via rsyslog <rsyslog@lists.adiscon.com> Cc: Steven D <pheerl...@hotmail.com> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting One problem with TCP load balancing of syslog messages is that the load balancers do not understand the syslog protocol, so they can't rebalance at a message boundry. A second problem is that when a firewall or load balancer drops a connection, the sender doesn't know that it's dropped until the next time it tries to deliver a message. Since TCP doesn't have any way for the OS TCP stack to tell software "that message that you submitted to an open connection, and I accepted, it can't be delivered" (once the OS accepts the message, the sender has to assume that it will be delivered) As a result, it's very easy for TCP syslog to be less reliable than UDP syslog. The 'common sense wisdom' is that TCP is reliable because dropped packets inside an ongoing connection will get retried, but dropped packets are actually very uncommon inside a datacenter. They may happen when a firewall/router is overloaded, but it's not very common. Back in 2006 or so I did testing and found that within a local network, UDP was almost perfectly reliable (as long as the receiver could keep up and not overflow the OS buffers) Rsyslog has the rebindinterval feature, which tells the sender to disconnect and reconnect periodically so that the load balancer has a chance to make a new balancing decision. you also want to make sure that the log stram is not idle for too long ('mark' was the historical method of doing that, I prefer vmstat 60 |logger -t vmstat as it's not much larger and an extremely dense set of information that can be very useful when troubleshooting) The other thing to look at is the RELP protocol, it was developed specifically because TCP was designed to be reliable over an unreliable wire, but assumes that both ends will remain up and the connection will not be cut by a middlebox. RELP does full application level acks so that the sender knows that the receiver rsyslog actually processed the message with plain TCP, once the sending software submits data to the OS stack and the OS stack says it's accepted the data, the data then sits in a buffer on the sending machine, then gets sent over the wire (with retries), then sits in a buffer on the receiving machine until the receiving software reads it. If anything causes the connection to be terminated (firewall, load balancer, crash on the receiving machine, etc) the data will be lost and the sending software has no way of learning about it. David Lang On Sun, 24 Apr 2022, Steven D via rsyslog wrote:Date: Sun, 24 Apr 2022 12:14:35 +0000 From: Steven D via rsyslog <rsyslog@lists.adiscon.com> To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com> Cc: Steven D <pheerl...@hotmail.com> Subject: [rsyslog] Basic Rsyslog Troubleshooting Greetings list New to rsyslog list, not new to logging. We're experiencing an odd issue where TCP syslog messages are being dropped at seemingly random intervals...hoping to get some input. The TLDR on our architecture is we have set up a couple rsyslog receivers behind a Netscaler Load balancer. Multiple platforms/devices are configured to send syslog to the load balancer, which distributes to the receivers. Receivers are running RHEL v8 and rsyslog v8.1911. Receivers write files to disk, which we then read with a SIEM agent. We've got a modestly sized environment with a syslog client base of 200-300 servers, 30 networking devices (including firewalls) and some applications all directing logging to the load balancer. Our config file is pretty vanilla, no cache, or advanced tweaks. Just using the "imtcp" and "imudp" modules and rulesets to write files to disk based on the sending host IP/port. The first problem we're seeing is that hosts sending via TCP have log messages missed (never written to disk), where UDP seems more reliable. When switching the firewalls to UDP, throughput nearly doubles and message loss is less noticeable (yeah I know it's still UDP). Possibly related is that we've noticed that each receiver also holds a lot of "Established" connections for back to the clients, but different ports. (Possible session/connection exhaustion?) Any guidance on how we can approach and troubleshoot this issue would be appreciated. Commands, dummy guides, sarcasm all welcome. Thanks much Regards, Steven. _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
