I'm trying to develop a package which can establish an ssl connection between a windows server and a client using openssl and I'm running into some serious road blocks - I'm a relative novice at both the openssl and wincrypt apis, I've done extensive searching for any hints at how this problem might be solved but I haven't found anything similar which might be applicable.
Here's a high-level description: On the server side, I'm using both openssl and the windows crypto api, I use the windows crypto api to access the windows certificate store and search for and load the certificate (with its private key) that I'm interested in using. I spent some time trying to extract the private key to use in openssl directly but discovered that the windows crypto api refuses to let go of it in a form that openssl can use directly. So I'm using the openssl RSA_set_method facility to replace the private key decrypt method (rsa_priv_dec) to pass off the heavy lifting to the cryptoapi (this seems like a reasonable thing to do and the kind of thing that would be done with a smart card or TCM or something where there's no direct access to the private key). I tell openssl to use the x509 encoding of the loaded certificate for incoming connections and wait. When a client SSL HELLO comes in, the server presents the x509 certificate, the client uses the public key in the certificate to encrypt its "secret" and sends the response to the server to decrypt with the private key. The server receives the response, my replaced method kicks in and tries to decrypt the 128-byte response using CryptDecrypt with the private key and fails with NTE_BAD_DATA (pretty opaque and not very informative). So the connection fails. I've done some unit testing on the api's and I can encrypt/decrypt sample messages using CryptEncrypt and CryptDecrypt, but anything that I've encrypted with RSA_public_encrypt gives me the same NTE_BAD_DATA error when I hand it to (private key) CryptDecrypt. I've extracted the PUBLICKEYBLOB to get the windows representation of the public key and I can see that the key bytes are in reverse order of the public key portion of the x509 certificate. I've read that the crypto api uses little-endian as opposed to everybody else's big-endian but (perhaps foolishly) assumed that that difference was only an internal representation issue, that something encrypted by openssl should be decryptable by CryptoAPI (I tried reversing the bytes of the encrypted message too and that didn't work.) So I'm a little stymied, am I missing some secret sauce somewhere? If openssl had an api to load certificates from windows cert stores that would be an alternate route but it seems as though this mechanism of using CryptoApi to handle the certificate access/management shouldn't be as difficult as it seems. And surely the interop problem I'm experiencing can't be unique to my setup. Anyone have any suggestions? I can post code on request but thought I'd start with a high-level description of the problem to avoid clouding the issue too much. thanks in advance ... Nou --- Nou Dadoun ndad...@teradici.com 604-628-1215 ______________________________________________________________________ OpenSSL Project http://www.openssl.org User Support Mailing List openssl-users@openssl.org Automated List Manager majord...@openssl.org
