Kees Bakker wrote: > On 29-10-18 19:30, Rob Crittenden wrote: >> Kees Bakker via FreeIPA-users wrote: >>> On 29-10-18 11:56, Kees Bakker via FreeIPA-users wrote: >>>> On 26-10-18 18:20, Florence Blanc-Renaud wrote: >>>>> On 10/26/18 6:09 PM, Kees Bakker via FreeIPA-users wrote: >>>>>> On 26-10-18 18:00, Timo Aaltonen wrote: >>>>>>> On 26.10.2018 18.59, Kees Bakker wrote: >>>>>>>> On 26-10-18 14:55, Timo Aaltonen wrote: >>>>>>>>> On 26.10.2018 09:59, Kees Bakker via FreeIPA-users wrote: >>>>>>>>>> On 25-10-18 20:46, Timo Aaltonen wrote: >>>>>>>>>>> On 25.10.2018 21.44, Rob Crittenden wrote: >>>>>>>>>>>> Kees Bakker wrote: >>>>>>>>>>>>> On 25-10-18 16:11, Rob Crittenden wrote: >>>>>>>>>>>>>> Kees Bakker via FreeIPA-users wrote: >>>>>>>>>>>>>>> On 25-10-18 14:18, Rob Crittenden wrote: >>>>>>>>>>>>>>>> Kees Bakker via FreeIPA-users wrote: >>>>>>>>>>>>>>>>> Could it be that this error already existed since we started? >>>>>>>>>>>>>>>>> Notice >>>>>>>>>>>>>>>>> the Request ID of 2016..., and the expires: 2018-10-24. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> # getcert list -n ipaCert | sed blabla >>>>>>>>>>>>>>>>> Number of certificates and requests being tracked: 8. >>>>>>>>>>>>>>>>> Request ID '20161103094546': >>>>>>>>>>>>>>>>> status: CA_UNREACHABLE >>>>>>>>>>>>>>>>> ca-error: Error 77 connecting to >>>>>>>>>>>>>>>>> https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: >>>>>>>>>>>>>>>>> Problem with the SSL CA cert (path? access rights?). >>>>>>>>>>>>>>>>> stuck: no >>>>>>>>>>>>>>>>> key pair storage: >>>>>>>>>>>>>>>>> type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS >>>>>>>>>>>>>>>>> Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt' >>>>>>>>>>>>>>>>> certificate: >>>>>>>>>>>>>>>>> type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS >>>>>>>>>>>>>>>>> Certificate DB' >>>>>>>>>>>>>>>>> CA: dogtag-ipa-ca-renew-agent >>>>>>>>>>>>>>>>> issuer: CN=Certificate Authority,O=MYDOMAIN >>>>>>>>>>>>>>>>> subject: CN=IPA RA,O=MYDOMAIN >>>>>>>>>>>>>>>>> expires: 2018-10-24 08:45:40 UTC >>>>>>>>>>>>>>>>> key usage: >>>>>>>>>>>>>>>>> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >>>>>>>>>>>>>>>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>>>>>>>>>>>>>>> pre-save command: >>>>>>>>>>>>>>>>> /usr/lib/ipa/certmonger/renew_ra_cert_pre >>>>>>>>>>>>>>>>> post-save command: /usr/lib/ipa/certmonger/renew_ra_cert >>>>>>>>>>>>>>>>> track: yes >>>>>>>>>>>>>>>>> auto-renew: yes >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> In other words, is this the same issue as >>>>>>>>>>>>>>>>> https://pagure.io/freeipa/issue/7422 ? >>>>>>>>>>>>>>>> The problem is your certs expired yesterday so connections >>>>>>>>>>>>>>>> won't work >>>>>>>>>>>>>>>> (the code and message don't come from within certmonger). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> certmonger _should_ have renewed them. Try killing ntpd, going >>>>>>>>>>>>>>>> back a >>>>>>>>>>>>>>>> few days, restart krb5kdc, dirsrv, httpd and the CA then >>>>>>>>>>>>>>>> certmonger and >>>>>>>>>>>>>>>> see what happens. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Easy for you to say. You know what you're doing :-) >>>>>>>>>>>>>>> For me it's all magic. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Anyway, I'll try it. I'm just scared to set the clock back, >>>>>>>>>>>>>>> because there may >>>>>>>>>>>>>>> be clients in the network that use this server as a NTP server. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Another thing I want to mention is that the error started >>>>>>>>>>>>>>> showing up two days >>>>>>>>>>>>>>> ago, on Oct 22, while the expiration is today, Oct 24. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> It shouldn't take more than a few minutes to roll back time, >>>>>>>>>>>>>> restart >>>>>>>>>>>>>> services and see what happens. I think your NTP clients will be >>>>>>>>>>>>>> able to >>>>>>>>>>>>>> recover ok if the server is not available for a few minutes. >>>>>>>>>>>>>> >>>>>>>>>>>>>> certmonger logs to syslog so you probably want to look at that >>>>>>>>>>>>>> to see if >>>>>>>>>>>>>> you can find a reason the certs weren't renewed automatically. >>>>>>>>>>>>>> >>>>>>>>>>>>> No, that didn't help. >>>>>>>>>>>>> And in the syslog there was nothing more than this. (I had to >>>>>>>>>>>>> stop the >>>>>>>>>>>>> nameserver because it was spitting out lots of messages.) >>>>>>>>>>>>> >>>>>>>>>>>>> Oct 11 06:00:00 ipasrv systemd[1]: Time has been changed >>>>>>>>>>>>> Oct 11 06:00:00 ipasrv systemd[52167]: Time has been changed >>>>>>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Stopping Certificate >>>>>>>>>>>>> monitoring and PKI enrollment... >>>>>>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Stopped Certificate monitoring >>>>>>>>>>>>> and PKI enrollment. >>>>>>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Starting Certificate >>>>>>>>>>>>> monitoring and PKI enrollment... >>>>>>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Started Certificate monitoring >>>>>>>>>>>>> and PKI enrollment. >>>>>>>>>>>>> Oct 11 06:00:05 ipasrv certmonger[131018]: 2018-10-11 06:00:05 >>>>>>>>>>>>> [131018] Error 77 connecting to >>>>>>>>>>>>> https://ipasrv.mydomain:8443/ca/agent/ca/profile >>>>>>>>>>>>> Review: Problem with the SSL CA cert (path? access rights?). >>>>>>>>>>>>> Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: >>>>>>>>>>>>> Forwarding request to dogtag-ipa-renew-agent >>>>>>>>>>>>> Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: >>>>>>>>>>>>> dogtag-ipa-renew-agent returned 3 >>>>>>>>>>>>> Oct 11 06:00:07 ipasrv certmonger[131018]: 2018-10-11 06:00:07 >>>>>>>>>>>>> [131018] Error 77 connecting to >>>>>>>>>>>>> https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem >>>>>>>>>>>>> with the SSL CA cert (path? access rights?). >>>>>>>>>>>>> Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: >>>>>>>>>>>>> Forwarding request to dogtag-ipa-renew-agent >>>>>>>>>>>>> Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: >>>>>>>>>>>>> dogtag-ipa-renew-agent returned 3 >>>>>>>>>>>>> Oct 11 06:00:17 ipasrv certmonger[131018]: 2018-10-11 06:00:17 >>>>>>>>>>>>> [131018] Error 77 connecting to >>>>>>>>>>>>> https://ipasrv:8443/ca/agent/ca/profileReview: Problem with the >>>>>>>>>>>>> SSL CA cert (path? access rights?). >>>>>>>>>>>>> >>>>>>>>>>>> Ok, I think I know what is going on. This is Ubuntu which AFAIK >>>>>>>>>>>> still >>>>>>>>>>>> lacks nss-pem. That is probably why it can't connect to renew the >>>>>>>>>>>> certs. >>>>>>>>>>>> >>>>>>>>>>>> I don't know if there is a workaround. Timo, do you know? >>>>>>>>>>> Ubuntu 18.04 and up have libnsspem, and certmonger depends on it. >>>>>>>>>>> I've >>>>>>>>>>> never tested cert renewal though. >>>>>>>>>>> >>>>>>>>>> Does that mean, I'm screwed? What options do I have? >>>>>>>>>> Live with it? >>>>>>>>>> Migrate to, say Centos? >>>>>>>>>> Try to upgrade the server to Ubuntu 18.04 (with uncertainty whether >>>>>>>>>> it will work)? >>>>>>>>>> Something else? >>>>>>>>> Stock 18.04 has other issues, there's an updated version on >>>>>>>>> ppa:freeipa/staging which is backported from 18.10 and should be fine >>>>>>>>> and hopefully provided as a stable update on 18.04 later on. >>>>>>>>> >>>>>>>>> But you could try pulling libnsspem from 18.04, and *then* roll back >>>>>>>>> time? >>>>>>>>> >>>>>>>> I installed libnsspem_1.0.3-0ubuntu2_amd64.deb >>>>>>>> >>>>>>>> Then I stopped ntp (and bind). >>>>>>>> Set the time back to Oct 11 >>>>>>>> Restarted krb5-kdc, dirsrv@MYDOMAIN, apache2, pki-tomcatd, certmonger >>>>>>>> (in that order). >>>>>>>> >>>>>>>> Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding >>>>>>>> request to dogtag-ipa-renew-agent >>>>>>>> Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: >>>>>>>> dogtag-ipa-renew-agent returned 3 >>>>>>>> Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 >>>>>>>> [168327] Error 60 connecting to >>>>>>>> https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer >>>>>>>> certificate cannot be authenticated with given CA certificates. >>>>>>>> Oct 11 06:08:12 ipasrv certmonger[168327]: 2018-10-11 06:08:12 >>>>>>>> [168327] Error 60 connecting to >>>>>>>> https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer >>>>>>>> certificate cannot be authenticated with given CA certificates. >>>>>>>> >>>>>>>> :-( >>>>>>>> >>>>>>>> Rob said also to restart CA. >>>>>>>> "restart krb5kdc, dirsrv, httpd and the CA then certmonger" >>>>>>>> I don't know which service that is. Does that matter? >>>>>>> systemctl restart ipa? >>>>>>> >>>>>>> >>>>>> I'm a bit scared to restart service ipa, because it also restarts >>>>>> several other services, >>>>>> link bind, and perhaps ntp. The latter is the one that I want to be >>>>>> absolutely in control >>>>>> of not starting. >>>>> And you're right! The CA is pki-tomcatd, so you already restarted it. >>>>> >>>>>> It's getting too late now, time for weekend. I'll give it another try on >>>>>> Monday. >>>>>> Meanwhile I want to point at the changed message. In case that rings a >>>>>> bell for >>>>>> someone. >>>>>> >>>>>> Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327] >>>>>> Error 60 connecting to >>>>>> https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate >>>>>> cannot be authenticated with given CA certificates. >>>>>> >>>>> You can have a look at Rob's blog for additional items to check: >>>>> https://rcritten.wordpress.com/2017/09/20/peer-certificate-cannot-be-authenticated-with-given-ca-certificates/ >>>> Thanks, I just stumbled on it myself. Interesting read, although I don't >>>> quite >>>> understand all details. >>>> >>>> I really need some guidance what to do next. I tried the date trick, I >>>> installed >>>> libnsspem (from Ubunu 18.04). The certmonger error message changed from >>>> Error 77 into Error 60, but the problem remained. >>>> >>>> Futhermore I noticed that pki-tomcat spits out a warning every 10 seconds >>>> >>>> Oct 29, 2018 11:47:05 AM org.apache.catalina.core.ContainerBase >>>> backgroundProcess >>>> WARNING: Exception processing realm >>>> com.netscape.cms.tomcat.ProxyRealm@5417a64d background process >>>> java.lang.NullPointerException >>>> at >>>> com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:113) >>>> at >>>> org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1357) >>>> at >>>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1543) >>>> at >>>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553) >>>> at >>>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553) >>>> at >>>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1521) >>>> at java.lang.Thread.run(Thread.java:748) >>>> >>>> I could do the date trick again, but then the question is, why didn't it >>>> work last time? >>> I tried it once more to set the date back and to restart the renewal >>> process. >>> I keep getting Error 60 with >>> Peer certificate cannot be authenticated with given CA certificates. >>> >>> Also, I had a look at Rob's blog. But I'm lost at what to do with >>> "... the fix was to reset the NSS trust flags in the Apache NSS database" >>> The curl command (with the date set back) seems to connect but it gives a >>> gnutls_handshake() failed: Illegal parameter >>> >>> And then I don't know what it looks like if >>> " You should get client certificate not found." >>> So I didn't try the certutil -M commands. >>> >>> BTW. While setting the date back I have bind (DNS) switched off because it >>> gave a lot of messages in my /var/log/syslog. >>> >> Right, this is for a previous version of IPA on RHEL/Fedora. The RA >> agent cert used to be stored in the Apache NSS database. > That's a problem in general with notes and blogs. When you write > things you don't know what to write about possible validness in > the future. > >> >> curl linked aganst gnu_tls? I have no idea, but it isn't this. >> >> This error message originates in curl. >> >> What is the validity of your CA certificate itself? Is it still valid >> back in time? > Yes it is. Again, this is Ubuntu 16.04, I'm looking at ... > > certutil -L -d /etc/pki/pki-tomcat/alias/ -n 'caSigningCert cert-pki-ca' > certutil -L -d /etc/apache2/nssdb/ -n 'GHS.NL IPA CA' > certutil -L -d /etc/dirsrv/slapd-GHS-NL -n 'GHS.NL IPA CA' > > Validity: > Not Before: Thu Nov 03 09:45:18 2016 > Not After : Mon Nov 03 09:45:18 2036 >
Ok I'm just flying blind right now, reading curl code, but in the gnutls code it throws this error if the server cert validation fails. I don't know where gnutls reads its CA trust from or whether it honors the system-wide trust, or even if your IPA CA is in the global trust properly. When back in time do IPA operations work? ipa user-show admin, ipa cert-show 1, etc? rob _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
