Greetings list New to rsyslog list, not new to logging. We're experiencing an odd issue where TCP syslog messages are being dropped at seemingly random intervals...hoping to get some input.
The TLDR on our architecture is we have set up a couple rsyslog receivers behind a Netscaler Load balancer. Multiple platforms/devices are configured to send syslog to the load balancer, which distributes to the receivers. Receivers are running RHEL v8 and rsyslog v8.1911. Receivers write files to disk, which we then read with a SIEM agent. We've got a modestly sized environment with a syslog client base of 200-300 servers, 30 networking devices (including firewalls) and some applications all directing logging to the load balancer. Our config file is pretty vanilla, no cache, or advanced tweaks. Just using the "imtcp" and "imudp" modules and rulesets to write files to disk based on the sending host IP/port. The first problem we're seeing is that hosts sending via TCP have log messages missed (never written to disk), where UDP seems more reliable. When switching the firewalls to UDP, throughput nearly doubles and message loss is less noticeable (yeah I know it's still UDP). Possibly related is that we've noticed that each receiver also holds a lot of "Established" connections for back to the clients, but different ports. (Possible session/connection exhaustion?) Any guidance on how we can approach and troubleshoot this issue would be appreciated. Commands, dummy guides, sarcasm all welcome. Thanks much Regards, Steven. _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
