On 06-07-2010 01:03, Dave Thompson wrote:
From: owner-openssl-us...@openssl.org On Behalf Of Ryan Smith
Sent: Friday, 02 July, 2010 18:31
I have written a FIPS-1.1.2 compliant (OpenSSL 0.9.7m) application
that validates certificates that are read in from files. It also loads
the CA certificates and corresponding CRLs from files. I am trying to
determine any limitations with loading large CRLs (~200-250 MB) and to
Not your actual problem but IMNVHO any CA that needs to issue CRLs
anywhere near that large should not be trusted in the first place.
Depends on how many certificates it issues. For example a CA that
issues certificates to almost every US citizen (with proper verification
and all) would get a steady stream of reports of lost or compromised
keys, which could easily grow to a huge size over the lifetime of a root
certificate. I guess this is also the reason why Verisign started
culling its CRLs of expired certificates that were not really
compromised, just discontinued or renewed.
characterize the resulting performance. I did not have any problem
using CRLs that are ~~100MB in size or smaller. However with the ~200MB
CRL,
I get the following error [in PEM_read_bio_X509_CRL]:
1418976:error:0D078064:asn1 encoding routines:ASN1_ITEM_EX_D2I:aux
error:tasn_dec.c:407:Type=X509_CRL_INFO
1418976:error:0D08303A:asn1 encoding
routines:ANS1_TEMPLATE_D2I:nested asn1 error:tasn_dec.c:567:Field=crl,
Type=X509_CRL
1418976:error:0906700D:PEM routines:PEM_ASN1_read_bio:ASN1
lib:pem_oth.c:82:
The only likely cause I see is the call to asn1_enc_save (line 397 in
0.9.7m).
That needs to allocate enough memory to copy the entire DER, and would fail
if the heap aka arena does not have sufficient available (contiguous) space.
The certificates and ~200MB CRL in question are all validated
successfully
using the OpenSSL command line
openssl verify -CAfile<cafile> -crl_check<certificates>
1. I wouldn't trust that definitely means no error, without checking. I have
seen
some commandline logic suppress errors it thinks are 'irrelevant' or
'unimportant'.
2. That's definitely a different executable, and may have been linked
differently
or have a different memory usage pattern. Does your program
CRYPTO_set_mem_functions
or CRYPTO_malloc_debug_init ? Do you have lots of other dynamic space
allocated,
or had space previously allocated and free'd in a way that leaves
fragmentation?
Does your program run in an environment that has a (stricter) quota on
memory?
2a. It looks to me like PEM_read_bio uses BUF_MEM_grow in a way that may
fragment.
Can you try, just before PEM_read__CRL, malloc'ing and free'ing at least two
and
I think three blocks of the size of the DER that will be implictly read plus
some
slop, say (filesize-60)/4*3+100 assuming the file contains only the PEMmed
CRL?
Hmmm. These particular items (CRLs) will not be reencoded, so the enc_save
looks unnecessary. It might be nice if this were somehow optional; as above,
I don't think CRLs should be so big this is an issue, but some other things
using (I assume) the same ASN1 mechanism like PKCS7 and mime reasonably
might.
Maybe the code should be enhanced so it can handle large structures from
a BIO in a one-pass streaming way. This (amongst others) implies
hashing the data for signing or signature checking while simultaneously
processing it and generating tentative results that will not be used
until the signature check is complete.
However this is non-trivial in general and may need to wait for a future
version of openssl.
______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openssl-users@openssl.org
Automated List Manager majord...@openssl.org
