Odd problem, I've been overhauling our x509 certificate handling and I've run
into an issue where the standard openssl x509 verification mechanism fails to
verify a certificate with a 1024-bit key when run with the FIPS version of the
library loaded but which verifies it with the non-FIPS version. All other key
lengths and certificates appear to behave the same way with both.
By the standard mechanism I mean:
m_x509_store = X509_STORE_new_fn();
X509_STORE_set_verify_cb_func(m_x509_store, verify_callback);
result = X509_STORE_add_cert_fn(m_x509_store, ca_cert);
verify_ctx = X509_STORE_CTX_new_fn();
result = X509_STORE_CTX_init_fn(verify_ctx, m_x509_store, cert_to_verify,
NULL);
result = X509_verify_cert_fn(verify_ctx);
(the _fn suffix is because we populate a fn table depending on whether we're
using fips versions or not.)
The verify_callback(int ok, X509_STORE_CTX *stor) appears to be called twice,
the first time with ok == 1, the second time with ok == 0 and the error log:
"certificate signature failure".
When observing a variation of this earlier, I'd thought that a unit test had
left something in a funky state in switching from fips to non-fips (and I'd
asked about the fact that there wasn't an api to completely remove and cleanup
openssl but this is a much more specific issue.)
Two questions:
why is the verify callback called twice, once with ok==1 and the second with
ok==0 (when called with the non-fips library both oks are ==1) - why two
calls?
Why is it failing with the fips library and passing with the non-fips library -
does it have anything to do with the 1024 bit key? (i.e. 2048 and 4096-key
certs both work, and the ca cert has a 2048-bit key)
Thanks ... N
---
Nou Dadoun
ndad...@teradici.com
604-628-1215
______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openssl-users@openssl.org
Automated List Manager majord...@openssl.org
