Hi, As known on the openssl mailinglists there is a difference in byte order between openssl and MS cryptoAPI, when doing RSA encryption. Googling around I find many references to claims that "openssl is PKCS#1 compliant (big-endian)" "MS is wrong" "just reverse your bytes". I was about to settle with that: "OK, MS got it reversed compared to the rest of the world... what else is new"
Then I decided to actually read the rfc. [1] as these kind of issues typically are very clearly specified. When carefully reading chapter 7.2.1 encryption It describes how to generate the ciphertext representative c. (an integer) And then how to encode that integer using I2OSP which is described in 4.1. Reading chapter 4.1 I2OSP It states that the encrypted output is an octet string X = X_1 X_2 ... X_xLen. Where X_1 is the *least *significant octet of the ciphertext representative. This seens to indicate that openssls output (RSA_public_encrypt) is wrong in its byte order. (well wrong.. at least not conforming to the PKCS#1 (both v2.1 as v2.0) specification.) openssl has the *most *significant byte of the ciphertext representative as first element. Am I missing something ? Did I misunderstand hte PKCS specification ? Is openssl (and java, and ... , and ) known to be NOT compliant ? NB: I didn't believe it myself, untill I reread PKCS#1 rfc again and again. NB: As the ciphertext is a byte array, this has nothing to do with endianess of machines etc., its purely a specification issue... [1] PKCS v2.1 specification http://www.ietf.org/rfc/rfc3447.txt Regards, Harm
