Having stolen a few minutes, a bit closer...Backtracking, I believe George's error must be coming from openssl/ssl/s3_clnt.c: ssl3_send_client_verify(), the block of code starting 36 lines in, shown below.
There is a call to EVP_SignFinal, that I believe will turn out to dispatch to RSA_sign (openssl/crypto/rsa/rsa_sign.c ). There is an intermediate maze of twisty passages that abstracts the signature mechanisms, but I'm pretty sure that's where we'll end up.
RSA_sign emits RSA_R_DIGEST_TOO_BIG_FOR_RSA_KEY when the size of the signature generated by i2d_X509_SIG is larger than the size allocated for it in the RSA structure.
Again, this is dependent (indirectly) on TLS version (the check is for md5_sha1 digest, which is used before TLS1.2). The TLS1.1 path is too well-worn to be likely. The code (abbreviated) goes like this:
if(type == NID_md5_sha1) {
...
i = SSL_SIG_LENGTH;
s = m;
} else {
...
i=i2d_X509_SIG(&sig,NULL);
}
j=RSA_size(rsa);
if (i > (j-RSA_PKCS1_PADDING_SIZE))
{
RSAerr(RSA_F_RSA_SIGN,RSA_R_DIGEST_TOO_BIG_FOR_RSA_KEY);
return(0);
}
This leaves the question of why the RSA structure doesn't have enough space.
Getting further back would be a lot easier with a reproducer and a
debugger...
The ssl3_send_client_verify +36:
if (TLS1_get_version(s) >= TLS1_2_VERSION)
{
long hdatalen = 0;
void *hdata;
const EVP_MD *md = s->cert->key->digest;
hdatalen = BIO_get_mem_data(s->s3->handshake_buffer,
&hdata);
if (hdatalen <= 0 || !tls12_get_sigandhash(p, pkey, md))
{
SSLerr(SSL_F_SSL3_SEND_CLIENT_VERIFY,
ERR_R_INTERNAL_ERROR);
goto err;
}
p += 2;
#ifdef SSL_DEBUG
fprintf(stderr, "Using TLS 1.2 with client alg %s\n",
EVP_MD_name(md));
#endif
if (!EVP_SignInit_ex(&mctx, md, NULL)
|| !EVP_SignUpdate(&mctx, hdata, hdatalen)
|| !EVP_SignFinal(&mctx, p + 2, &u,
pkey)) <===== breakpoint goes here
{
SSLerr(SSL_F_SSL3_SEND_CLIENT_VERIFY,
ERR_R_EVP_LIB);
goto err;
}
s2n(u,p);
n = u + 4;
if (!ssl3_digest_cached_records(s))
goto err;
}
.
Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.
This communication may not represent my employer's views,
if any, on the matters discussed.
smime.p7s
Description: S/MIME Cryptographic Signature
