On Wed, Mar 07, 2012, Nou Dadoun wrote: > I'm trying to develop a package which can establish an ssl connection between > a windows server and a client using openssl and I'm running into some serious > road blocks - I'm a relative novice at both the openssl and wincrypt apis, > I've done extensive searching for any hints at how this problem might be > solved but I haven't found anything similar which might be applicable. > > Here's a high-level description: > > On the server side, I'm using both openssl and the windows crypto api, I use > the windows crypto api to access the windows certificate store and search for > and load the certificate (with its private key) that I'm interested in using. > I spent some time trying to extract the private key to use in openssl > directly but discovered that the windows crypto api refuses to let go of it > in a form that openssl can use directly. > > So I'm using the openssl RSA_set_method facility to replace the private key > decrypt method (rsa_priv_dec) to pass off the heavy lifting to the cryptoapi > (this seems like a reasonable thing to do and the kind of thing that would be > done with a smart card or TCM or something where there's no direct access to > the private key). I tell openssl to use the x509 encoding of the loaded > certificate for incoming connections and wait. > > When a client SSL HELLO comes in, the server presents the x509 certificate, > the client uses the public key in the certificate to encrypt its "secret" and > sends the response to the server to decrypt with the private key. > > The server receives the response, my replaced method kicks in and tries to > decrypt the 128-byte response using CryptDecrypt with the private key and > fails with NTE_BAD_DATA (pretty opaque and not very informative). So the > connection fails. > > I've done some unit testing on the api's and I can encrypt/decrypt sample > messages using CryptEncrypt and CryptDecrypt, but anything that I've > encrypted with RSA_public_encrypt gives me the same NTE_BAD_DATA error when I > hand it to (private key) CryptDecrypt. > > I've extracted the PUBLICKEYBLOB to get the windows representation of the > public key and I can see that the key bytes are in reverse order of the > public key portion of the x509 certificate. I've read that the crypto api > uses little-endian as opposed to everybody else's big-endian but (perhaps > foolishly) assumed that that difference was only an internal representation > issue, that something encrypted by openssl should be decryptable by CryptoAPI > (I tried reversing the bytes of the encrypted message too and that didn't > work.) > > So I'm a little stymied, am I missing some secret sauce somewhere? > > If openssl had an api to load certificates from windows cert stores that > would be an alternate route but it seems as though this mechanism of using > CryptoApi to handle the certificate access/management shouldn't be as > difficult as it seems. And surely the interop problem I'm experiencing can't > be unique to my setup. Anyone have any suggestions? > > I can post code on request but thought I'd start with a high-level > description of the problem to avoid clouding the issue too much. >
If windows can export the key and certificate as a PKCS#12 (PFX) file you can use that with OpenSSL. If the key is unexportable you can't. OpenSSL has a CryptoAPI ENGINE you can use and that will handle the conversion between formats for you. It is supported on the command line which you can use as a test. Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: http://www.openssl.org ______________________________________________________________________ OpenSSL Project http://www.openssl.org User Support Mailing List openssl-users@openssl.org Automated List Manager majord...@openssl.org
