Thanks for the detailed answer. >From strace I can see that I'm using /lib/x86_64-linux-gnu/libssl.so.1.1
When I use the eventmachine lib that uses the wrong cert chain I can see with strace : openat(AT_FDCWD, "/usr/lib/ssl/cert.pem", O_RDONLY) = -1 ENOENT (No such file or directory) stat("/usr/lib/ssl/certs/2e5ac55d.0", {st_mode=S_IFREG|0644, st_size=1200, ...}) = 0 openat(AT_FDCWD, "/usr/lib/ssl/certs/2e5ac55d.0", O_RDONLY) = 8 stat("/usr/lib/ssl/certs/2e5ac55d.1", 0x7ffe1a8e5d80) = -1 ENOENT (No such file or directory) When I use another lib that uses the correct cert chain I can see with strace : openat(AT_FDCWD, "/usr/lib/ssl/cert.pem", O_RDONLY) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/lib/ssl/cert.pem", O_RDONLY) = -1 ENOENT (No such file or directory) stat("/usr/lib/ssl/certs/8d33f237.0", 0x7fff1b4b0f90) = -1 ENOENT (No such file or directory) stat("/usr/lib/ssl/certs/4042bcee.0", {st_mode=S_IFREG|0644, st_size=1939, ...}) = 0 openat(AT_FDCWD, "/usr/lib/ssl/certs/4042bcee.0", O_RDONLY) = 8 In the second case I can see it tries to load the R3 certificate ( 8d33f237.0 ). I wonder why in the first case it does not try to find each certificate in the chain from the trust store at all. I wonder if it can be related to the fact I do not see any call to SSL_CTX_set_cert_store in the lib. On the other hand I suppose if we do not call this it would pick the "default" trust store from the system which seems to be the case here because it can find /usr/lib/ssl/certs/2e5ac55d.0 . This looks like to be an issue in the eventmachine lib itself. I need to brush up my C skills and have a deeper look at this :). Thanks On Sat, Oct 2, 2021 at 8:11 PM Viktor Dukhovni <openssl-us...@dukhovni.org> wrote: > On Sat, Oct 02, 2021 at 06:21:00PM +0100, Angus Robertson - Magenta > Systems Ltd wrote: > > > > Yes. To make things even more complex, a few sites also have an > > > older version of R3 that is directly signed by the DST root: > > > > > > - leaf <- R3 <- DST Root CA X3 (self-signed) > > > > > > but that's far from common at this point. > > > > That old R3 [CA] was issued last winter and got installed in Windows > > Server 2018 intermediate stores then, and was still being sent out on > > 29th and 30th, despite expiring on 29th. > > Not just Windows, at least one Unix system running Postfix is still > vending a chain with the R3 signed by DST that expired on 2021-09-29: > > issuer=O = Digital Signature Trust Co., CN = DST Root CA X3 > subject=C = US, O = Let's Encrypt, CN = R3 > notBefore=Oct 7 19:21:40 2020 GMT > notAfter=Sep 29 19:21:40 2021 GMT > SHA256 > Fingerprint=73:0C:1B:DC:D8:5F:57:CE:5D:C0:BB:A7:33:E5:F1:BA:5A:92:5B:2A:77:1D:64:0A:26:F7:A4:54:22:4D:AD:3B > -----BEGIN CERTIFICATE----- > MIIEZTCCA02gAwIBAgIQQAF1BIMUpMghjISpDBbN3zANBgkqhkiG9w0BAQsFADA/ > MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT > DkRTVCBSb290IENBIFgzMB4XDTIwMTAwNzE5MjE0MFoXDTIxMDkyOTE5MjE0MFow > MjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxCzAJBgNVBAMT > AlIzMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuwIVKMz2oJTTDxLs > jVWSw/iC8ZmmekKIp10mqrUrucVMsa+Oa/l1yKPXD0eUFFU1V4yeqKI5GfWCPEKp > Tm71O8Mu243AsFzzWTjn7c9p8FoLG77AlCQlh/o3cbMT5xys4Zvv2+Q7RVJFlqnB > U840yFLuta7tj95gcOKlVKu2bQ6XpUA0ayvTvGbrZjR8+muLj1cpmfgwF126cm/7 > gcWt0oZYPRfH5wm78Sv3htzB2nFd1EbjzK0lwYi8YGd1ZrPxGPeiXOZT/zqItkel > /xMY6pgJdz+dU/nPAeX1pnAXFK9jpP+Zs5Od3FOnBv5IhR2haa4ldbsTzFID9e1R > oYvbFQIDAQABo4IBaDCCAWQwEgYDVR0TAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8E > BAMCAYYwSwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5p > ZGVudHJ1c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTE > p7Gkeyxx+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEE > AYLfEwEBATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2Vu > Y3J5cHQub3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0 > LmNvbS9EU1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYf > r52LFMLGMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjANBgkqhkiG9w0B > AQsFAAOCAQEA2UzgyfWEiDcx27sT4rP8i2tiEmxYt0l+PAK3qB8oYevO4C5z70kH > ejWEHx2taPDY/laBL21/WKZuNTYQHHPD5b1tXgHXbnL7KqC401dk5VvCadTQsvd8 > S8MXjohyc9z9/G2948kLjmE6Flh9dDYrVYA9x2O+hEPGOaEOa1eePynBgPayvUfL > qjBstzLhWVQLGAkXXmNs+5ZnPBxzDJOLxhF2JIbeQAcH5H0tZrUlo5ZYyOqA7s9p > O5b85o3AM/OJ+CktFBQtfvBhcJVd9wvlwPsk+uyOy2HI7mNxKKgsBTt375teA2Tw > UdHkhVNcsAKX1H7GNNLOEADksd86wuoXvg== > -----END CERTIFICATE----- > > The EE (depth 0 server) certificate is not expired, and yet somehow the > server is building a chain with a fresh leaf cert, and a rather stale > issuer CA. > > It verifies via the DANE implementation in OpenSSL, because its "2 1 1" > record with a fresh RRSIG specifies the R3 CA as trusted, and its > expiration date is not in scope since it was signed by an entity outside > the effective trust chain. > > Validation would fail for the same chain with WebPKI, unless there's a > new improved R3 in the trust store (not just the roots). > > My DANE survey scan engine checks trust-anchor cert expiration date, > even when an intermediate CA, mostly because the implementation is > done that way, but I can retroactively justify it because it makes > sense to be more strict in tools that look for potential issues. > > Implementations other than OpenSSL might similarly reject such a > suboptimal chain. > > -- > Viktor. >