Hi Andre, Thank you for the additional feedback.
As you suggested, the problem is likely tied back to the TCP probe. We've not had it enabled since last Sunday and rsyslog has been running fine without making the changes we previously discussed (I'm still interested in making them, I've just been pulled in other directions). Do you happen to know of a safe way to check that the port is open remotely without triggering a failure from rsyslog's perspective? I'm guessing that a minimal RELP-compatible client would be the best approach. Is there such a tool that you're aware of that could be called periodically to confirm that a rsyslog receiver (RELP-enabled port) is functioning properly? Just thought I would ask. Thanks! -----Original Message----- From: Andre Lorbach <[email protected]> Sent: Monday, August 24, 2020 4:06 AM To: rsyslog-users <[email protected]> Cc: Adam Chalkley <[email protected]> Subject: AW: [rsyslog] Upgraded receiver from Ubuntu 16.04 to 18.04, connections from clients failing with a high number of CLOSE_WAIT connections on receiver I think those errors were there all the time but not reported in older librelp version. I reviewed the code and we added this error output about 2 years ago in librelp. Ubuntu 16.04 most likely is using an older librelp version, so you did not see the error there. The problem is caused by the TCP Probe, it may helps if you try to receive data before you drop the connection. Best regards, Andre Lorbach -- Adiscon GmbH Mozartstr. 21 97950 Großrinderfeld, Germany Ph. +49-9349-9298530 Geschäftsführer/President: Rainer Gerhards Reg.-Gericht Mannheim, HRB 560610 Ust.-IDNr.: DE 81 22 04 622 Web: www.adiscon.com - Mail: [email protected] Informations regarding your data privacy policy can be found here: https://www.adiscon.com/data-privacy-policy/ This e-mail may contain confidential and/or privileged information. If you are not the intended recipient or have received this e-mail in error please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren und die unbefugte Weitergabe dieser E-Mail sind nicht gestattet. > -----Ursprüngliche Nachricht----- > Von: rsyslog <[email protected]> Im Auftrag von Adam > Chalkley via rsyslog > Gesendet: Mittwoch, 19. August 2020 18:38 > An: rsyslog-users <[email protected]> > Cc: Adam Chalkley <[email protected]> > Betreff: [rsyslog] Upgraded receiver from Ubuntu 16.04 to 18.04, connections > from clients failing with a high number of CLOSE_WAIT connections on > receiver > > Hi, > > We upgraded the OS on our central receiver yesterday from Ubuntu 16.04 > (4.4 kernel) to 18.04 (4.15 kernel). > > We are using the upstream PPA, so running 8.2006.0 on receivers and > endpoints. > > When we started getting reports from our Nagios instance that the rsyslog > forward queues endpoints were beginning to fill we checked our receiver > (sawmill1) and saw 94 open TCP connections with 40 of them in CLOSE_WAIT > from our Nagios server, most of them I suspect from the TCP port connection > test performed every 5 minutes. > > Log samples from the receiver system (which are related to port probes from > our Nagios instance): > > 2020-08-19T10:05:01.279416-05:00 lincoln rsyslogd: -- MARK -- > 2020-08-19T10:05:08.249358-05:00 lincoln rsyslogd: imrelp[2514]: error 'server > closed relp session, session broken', object 'lstn 2514: conn to clt > 192.168.2.10/192.168.2.10' - input may not work as intended [v8.2006.0 try > https://www.rsyslog.com/e/2353 ] > 2020-08-19T10:05:08.249626-05:00 lincoln rsyslogd: imrelp[2514]: error 'error > sending relp: Bad file descriptor', object 'lstn 2514: conn to clt > 192.168.2.10/192.168.2.10' - input may not work as intended [v8.2006.0 try > https://www.rsyslog.com/e/2353 ] > 2020-08-19T10:08:08.020625-05:00 lincoln rsyslogd: imrelp[2514]: error 'server > closed relp session, session broken', object 'lstn 2514: conn to clt > 192.168.2.10/192.168.2.10' - input may not work as intended [v8.2006.0 try > https://www.rsyslog.com/e/2353 ] > 2020-08-19T10:08:08.021253-05:00 lincoln rsyslogd: imrelp[2514]: error 'error > sending relp: Bad file descriptor', object 'lstn 2514: conn to clt > 192.168.2.10/192.168.2.10' - input may not work as intended [v8.2006.0 try > https://www.rsyslog.com/e/2353 ] > 2020-08-19T10:11:08.074712-05:00 lincoln rsyslogd: imrelp[2514]: error 'server > closed relp session, session broken', object 'lstn 2514: conn to clt > 192.168.2.10/192.168.2.10' - input may not work as intended [v8.2006.0 try > https://www.rsyslog.com/e/2353 ] > > Log samples from the Nagios instance: > > 2020-08-19T11:19:53.444953-05:00 nagios rsyslogd: > omrelp[lincoln.lib.auburn.edu:2514]: error 'error waiting on required session > state, session broken', object 'conn to srvr lincoln.lib.auburn.edu:2514' - > action may not work as intended [v8.2006.0 try > https://www.rsyslog.com/e/2353 ] > 2020-08-19T11:19:53.445260-05:00 nagios rsyslogd: > omrelp[lincoln.lib.auburn.edu:2514]: error 'error opening connection to > remote peer', object 'conn to srvr lincoln.lib.auburn.edu:2514' - action may > not work as intended [v8.2006.0 try https://www.rsyslog.com/e/2353 ] > > Is there a setting I can apply to rsyslog to help resolve this? > > Is this a known bug? > > We didn't have the issue with v8.2006.0 on our receiver when it was running > Ubuntu 16.04 (the prior OS release), even though it made the same > complaints about the TCP port probes from Nagios. > > Thanks in advance. > > _______________________________________________ > rsyslog mailing list > https://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: > This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond > our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
