I was initially thinking it might be the certificate checks. But other then the openssl command I already used I don't see a way to verify if rsyslog is having issues with that or not.
Because doing this: openssl s_client -connect syslog.xx.nl:6518 Tells me the first certificate in the chain failed. I would expect that to work without the -CAfile tls-ca-bundle.pem parameter if the ca-cert is available on the system. Could it be possible that it's not working because I am providing a .pem instead of a .crt in the config? But I haven't had time to enable debugging and delve deeper into it. So I hope to get on that today. Also I found that CentOS7 doesn't include your latest version, so I also need to go check their policies or what else why they don't do that.. ________________________________ From: David Lang <[email protected]> Sent: Friday, February 21, 2020 6:54 PM To: Moll, Marco via rsyslog <[email protected]> Cc: Moll, Marco <[email protected]> Subject: Re: [rsyslog] Limitting rsyslog to just TLS v1.2 - broke tls connections after config change, rollback doesn't fix rsyslog restarting never requires a reboot. it looks like rsyslog is restarting, what is the problem you think is sticking around? the gnutls errors are indications that things are connecting to your encrypted port and not talking TLS properly, not something that rsyslog can fix. David Lang On Fri, 21 Feb 2020, Moll, Marco via rsyslog wrote: > Date: Fri, 21 Feb 2020 08:45:26 +0000 > From: "Moll, Marco via rsyslog" <[email protected]> > To: "Moll, Marco via rsyslog" <[email protected]> > Cc: "Moll, Marco" <[email protected]> > Subject: Re: [rsyslog] Limitting rsyslog to just TLS v1.2 - broke tls > connections after config change, rollback doesn't fix > > These are the steps I take any time I do a restart or stop/start for rsyslog, > and this is the current output: > sudo systemctl disable rsyslog > Removed symlink > /etc/systemd/system/multi-user.target.wants/rsyslog.service.sudo systemctl > stop rsyslogsudo systemctl enable rsyslog > Created symlink from > /etc/systemd/system/multi-user.target.wants/rsyslog.service > to > /usr/lib/systemd/system/rsyslog.service. > sudo systemctl start rsyslogsudo systemctl status rsyslog > ● rsyslog.service - System Logging Service > Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor > pres et: enabled) > Active: active (running) since Fri 2020-02-21 09:36:36 CET; 5s ago > Docs: man:rsyslogd(8) > http://www.rsyslog.com/doc/ > Main PID: 5372 (rsyslogd) > CGroup: /system.slice/rsyslog.service > └─5372 /usr/sbin/rsyslogd -n > > Feb 21 09:36:36 systemd[1]: Started System Loggi... > Feb 21 09:36:37 rsyslogd[5372]: gnutls returned error on handshake: The TLS > connection was non-properl ... > Feb 21 09:36:37 rsyslogd[5372]: gnutls returned error on handshake: The TLS > connection was non-properl ... > Feb 21 09:36:38 rsyslogd[5372]: peer did not provide a certificate, not > permitted to talk to it ... > Feb 21 09:36:38 rsyslogd[5372]: netstream session 0x7fa29004a7f0 from > 10.xx.xx.xx will be closed due... > Feb 21 09:36:38 rsyslogd[5372]: gnutls returned error on handshake: The TLS > connection was non-properl ... > Feb 21 09:36:38 rsyslogd[5372]: gnutls returned error on handshake: The TLS > connection was non-properl ... > Feb 21 09:36:38 rsyslogd[5372]: gnutls returned error on handshake: The TLS > connection was non-properl ... > Feb 21 09:36:40 rsyslogd[5372]: unexpected GnuTLS error -110 in > nsdsel_gtls.c:178: The TLS connection ... > Feb 21 09:36:40 rsyslogd[5372]: netstream session 0x7fa29004a7f0 from > 10.xx.xx.xx will be closed due... > Hint: Some lines were ellipsized, use -l to show in full. > > Im doing the disable/enable because I read in some other topic that a reboot > was needed, but rebooting means alerts get send out to our pagerduty. So im > circumventing that. > > When I try your command: > sudo rsyslogd -N 1 > rsyslogd: version 8.24.0-34.el7, config validation run (level 1), master > config /etc/rsyslog.conf > rsyslogd: End of config validation run. Bye. > > ________________________________ > From: David Lang <[email protected]> > Sent: Friday, February 21, 2020 9:31 AM > To: Moll, Marco via rsyslog <[email protected]> > Cc: Moll, Marco <[email protected]> > Subject: Re: [rsyslog] Limitting rsyslog to just TLS v1.2 - broke tls > connections after config change, rollback doesn't fix > > the only thing that rsyslog persists between runs is state files for queues, > do > you have any of those configured in your system?, if so you can try to remove > them > > do you get a syntax error if you do 'rsyslog -N 1' > > what error do you get at startup? > > > On Fri, 21 Feb 2020, Moll, Marco via rsyslog wrote: > >> Date: Fri, 21 Feb 2020 08:27:24 +0000 >> From: "Moll, Marco via rsyslog" <[email protected]> >> To: "Moll, Marco via rsyslog" <[email protected]> >> Cc: "Moll, Marco" <[email protected]> >> Subject: Re: [rsyslog] Limitting rsyslog to just TLS v1.2 - broke tls >> connections after config change, rollback doesn't fix >> >> I did try sudo yum update rsyslog, but all I get is that the latest packages >> are already installed. After checking, it seems we are using private >> repositories from one of our partners. But they say their repo is synced >> with the public repositories. >> I'm going to take a guess it's this one: >> http://mirror.centos.org/centos/7/os/x86_64/Packages/ >> >> And as far as I can see that one doesn't go beyond 8.24 .. >> >> And the entire system is ansible-managed, so whatever change I make: any >> rollback is basically just a revert on code. It's impossible for me to >> forget any part that got changed. >> >> So in order to upgrade I guess that means I need to add the repo via >> ansible, install the latest version and test. >> Still leaves me hanging with a non-working rollbacked scenario where rsyslog >> should not be persistent. >> What's the best way for me to figure out what's not working correct, and to >> fix it? >> turn on rsyslog with debugging on 2? (Guess i'll go do that anyways, see how >> far that gets me) >> Pointers for me to look at specifically maybe? >> ________________________________ >> From: David Lang <[email protected]> >> Sent: Thursday, February 20, 2020 8:09 PM >> To: Moll, Marco via rsyslog <[email protected]> >> Cc: Moll, Marco <[email protected]> >> Subject: Re: [rsyslog] Limitting rsyslog to just TLS v1.2 - broke tls >> connections after config change, rollback doesn't fix >> >> 8.24 is over 3 years old now, after 8.39 (over a year old) we switched to a >> date >> based version, and the current version is 8.2001 (8.YYMM) >> >> GNUTLS has some problems that will cause it to segfault under some >> conditions. >> You should switch to openssl (requires a newer version of rsyslog) >> >> The PriorityString requires a newer version of gnutls and openssl libraries, >> I'm >> not sure if centos 7 includes it, but centos6 does not. >> >> David Lang >> >> On Thu, 20 Feb 2020, Moll, Marco via rsyslog wrote: >> >>> Date: Thu, 20 Feb 2020 13:03:52 +0000 >>> From: "Moll, Marco via rsyslog" <[email protected]> >>> To: "[email protected]" <[email protected]> >>> Cc: "Moll, Marco" <[email protected]> >>> Subject: [rsyslog] Limitting rsyslog to just TLS v1.2 - broke tls >>> connections >>> after config change, rollback doesn't fix >>> >>> Hi, >>> >>> I'm new to rsyslog and I'm trying to move forward on rsyslog due to a >>> security advisory. >>> I'm trying to achieve that rsyslog will no long accept ssl2, ssl3 or tls1 >>> and only accepts tls 1.2 >>> >>> I figured this would work: >>> module( >>> load="imtcp" >>> # see https://www.rsyslog.com/doc/v8-stable/concepts/ns_gtls.html >>> StreamDriver.Name="gtls" >>> StreamDriver.Mode="1" >>> StreamDriver.Authmode="x509/name" >>> # See >>> https://www.rsyslog.com/doc/v8-stable/configuration/modules/imtcp.html#permittedpeer >>> # PermittedPeer=["*.enexis.nl"] >>> PermittedPeer=["*.xx.nl", "*.yy.com", "*.yy.com.crt"] >>> gnutlsPriorityString="Protocol=ALL,-SSLv2,-SSLv3,-TLSv1 >>> MinProtocol=TLSv1.2" >>> ) >>> input( >>> type="imtcp" >>> port="6518" >>> ) >>> >>> It did not, and I started seeing these errors in journal: >>> peer did not provide a certificate, not permitted to talk to it >>> [v8.24.0-34.el7 try http://www.rsyslog.com/e/2085 ] >>> netstream session 0x7f10e40e8d80 from 10.xx.xx.xx will be closed due to >>> error [v8.24.0-34.el7 try http://www.rsyslog.com/e/2089 ] >>> >>> >>> >>> I figured, well there is no change on the other end, so a rollback will fix >>> it. That would give me re-assurance that I can rollback any time. Since we >>> haven't tested that before. >>> It did not, and now I have a broken rsyslog server 🙂 >>> >>> Old config: >>> module( >>> load="imtcp" >>> # see https://www.rsyslog.com/doc/v8-stable/concepts/ns_gtls.html >>> StreamDriver.Name="gtls" >>> StreamDriver.Mode="1" >>> StreamDriver.Authmode="x509/name" >>> # See >>> https://www.rsyslog.com/doc/v8-stable/configuration/modules/imtcp.html#permittedpeer >>> # PermittedPeer=["*.enexis.nl"] >>> PermittedPeer=["*.xx.nl", "*.yy.com", "*.yy.com.crt"] >>> ) >>> input( >>> type="imtcp" >>> port="6518" >>> ) >>> >>> >>> I don't have many things I can check, as there is only 1 service relying on >>> this. But that service is actually the only thing that's providing us >>> insight into network logs. Incidentally our SIEM uses that as input. >>> So I did do a small check on the certificates: >>> openssl s_client -connect syslog.xx.nl:6518 -CAfile tls-ca-bundle.pem >>> >>> SSL handshake has read 17073 bytes and written 338 bytes >>> --- >>> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384 >>> Server public key is 2048 bit >>> Secure Renegotiation IS supported >>> Compression: NONE >>> Expansion: NONE >>> No ALPN negotiated >>> SSL-Session: >>> Protocol : TLSv1.2 >>> Cipher : ECDHE-RSA-AES256-GCM-SHA384 >>> Session-ID: >>> 966666C3F3245570181E11CB525457BAD70D3F0E208D0CED75361980B7935465 >>> Session-ID-ctx: >>> Master-Key: >>> 2570E3FEC0FEBCB661A213FC38EF685FBACEA796E31CDSADH78HDFG6ADDFD0FDE9788CFD4C43B101A307345EC66C5C5BC0 >>> Start Time: 1562755867 >>> Timeout : 7200 (sec) >>> Verify return code: 0 (ok) >>> >>> >>> Trying the endpoint of this server always fails on the first certificate in >>> the chain, hence why I'm adding it into that command. >>> >>> And in the config it does specify certs that are needed: >>> global( >>> DefaultNetstreamDriver="gtls" >>> DefaultNetstreamDriverCAFile="/etc/rsyslog.d/tls-ca-bundle.pem" >>> DefaultNetstreamDriverCertFile="/etc/rsyslog.d/server.crt" >>> DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/server.key" >>> ) >>> >>> I believe the use of gnutlsprioritystring setting requires version 8.29 >>> We are running 8.24, however I do not see version 8.29 in the public yum >>> repos for centos7? I might be looking in the wrong place though. >>> >>> Since it is broken I would rather move forward in limiting connections to >>> just tls 1.2 instead of investing time into getting it rolled back to a >>> working state. But I guess that all depends on how easy moving forward goes. >>> Is there anyone who has tips on moving forward? >>> >>> Best regards, >>> >>> Marco Moll >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >>> LIKE THAT. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
