Your right, there is another length there which was not shown in the SSL/TLS specs... how strange. I took these out and it worked fine now. Thanks for the clue... The problem then was that the data I was trying to decrypt was larger than the MOD... by 2 bytes.
Thanks again! Will -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Ken Ballou Sent: Friday, October 31, 2003 4:45 PM To: [EMAIL PROTECTED] Subject: Re: RSA Decryption with OpenSSL Crypto Library The length of 258 bytes is very curious. If the server's key length is 2048 bits, then this length should be 256 bytes. Is it possible your code left in a two byte length field before the encrypted pre-master secret? - Ken On Fri, Oct 31, 2003 at 03:59:01PM -0800, William Korbe III wrote: > Hello, > > I have been trying to decrypt the "client pre master secret" sent in a SSL > handshake using the server's private key with OpenSSL's Crypto library. But > RSA_private_decrypt() always returns -1! > > I have set up a test apache2 server with modssl and have generated the > server's private key and certificate. I sniffed the packets off the network > from a single SSL session using a single TCP connection between a client and > my test server, and now I am trying to decrypt the "client pre-master > secret". > > In the code below I have opened the server's private key file, and read it > into a RSA object using PEM_read_RSAPrivateKey(). The RSA_check_key() > function does not return any errors. I pass this key with my encrypted data, > its length, a buffer to hold the decrypted data, and the padding setting to > RSA_private_decrypt() but this always returns -1. > > I have checked to make sure my encrypted and decrypted buffers are correct, > and the length is correct. Apparently the RSA key struct is correct since no > errors came from RSA_check_key(), and I am using the only private key for > the server, so I can't be using the wrong key. I'm pretty sure the padding > is RSA_PKCS1_PADDING, is this correct for apache2 w/ modssl using rsa > handshakes for ssl? I have also tried the RSA_PKCS1_OAEP_PADDING and the > RSA_SSLv3_PADDING, and RSA_NO_PADDING without succes. > > The cipher suite used was TLS_RSA_WITH_RC4_128_MD5. > > The length of the encrypted-client-pre-master-secret is 258 bytes, when > decrypted it should be 48 bytes. > > Can anyone see a step I'm missing, anyone know what's going on here? > > Thanks! > > Will > > > . > . > . > > //Steps > //1. Using serverPrivateKey, decrypt the client_pre_master_secret > > //1.1 Open server's private key file > if((fp = fopen("server.key", "rb")) == NULL) > { > printf("Cannot open server key file.\n"); > return; > } > > //1.2 Generate RSA struct from private key file > PEM_read_RSAPrivateKey(fp, &pRsa, NULL, NULL); > > //1.3 Check for succesfull key generation > if(RSA_check_key(pRsa) != 1) > { > printf("RSA_check_key(): PrivateKey check failed\n"); > return; > } > > //1.4 Using Private RSA Key, decode the client_pre_master_secret > check = RSA_private_decrypt(clientPreMasterLength, toDecrypt, > decryptedPreMasterSecret, pRsa, RSA_PKCS1_PADDING); > if(check == -1) > { > printf("RSA_private_decrypt() failed"); > exit(1); > } > > . > . > . > > ______________________________________________________________________ > OpenSSL Project http://www.openssl.org > User Support Mailing List [EMAIL PROTECTED] > Automated List Manager [EMAIL PROTECTED] > ______________________________________________________________________ OpenSSL Project http://www.openssl.org User Support Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED] ______________________________________________________________________ OpenSSL Project http://www.openssl.org User Support Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
