Using javax.crypto.Cipher with RSA and an OpenSSL generated public key

[PMicke]

Hi, 

we have a server using OpenSLL crypto library functions around RSA.
I want to write a Java client that sends an encrypted message to the 
server, which can be decrypted there.

To test this I use the following Java code below:

The encoded public key is created with OpenSSL.
                RSA* rsa_private= RSA_generate_key (keylength, RSA_F4, 
NULL, NULL);
                len = i2d_RSA_PUBKEY(rsa_private, NULL);
                pBuffer=(char *)OPENSSL_malloc(len);
                upBuffer = (unsigned char*)pBuffer;
                len = i2d_RSA_PUBKEY(rsa_private, &upBuffer);
 
The server decrypts the password with the same private key used to 
generate the public key:
                retval = RSA_private_decrypt(RSA_size(rsa_private), 
encryptedPasswordCharArrayfromJavaClient, cleartext-output, rsa_private, 
RSA_NO_PADDING);

I tested my code so that encryption /  decryption works in the C code 
using OpenSSL.
I tested my Java code so that encryption /  decryption works in the Java 
code using standard Java packages with a Sun provider for RSA.
I have trouble to get the two together. If I use RSA_private_decrypt with 
a byte sequence created from the Java client the output is wrong - no 
error occurs.

One thing that might point to a reason for the incompatibility:
I observe that the Java client always creates the same encrypted byte 
sequence from a given input string and a public key.
The OpenSSL C client produces different encrypted byte sequences for each 
separate run with a given input string and a public key. Decryption works 
with all of them.
If I let the C client encrypt the same given input string twice in the 
same run, it creates the same encrypted sequence.

The java client code:

import java.security.KeyFactory;
import java.security.PublicKey;
import java.security.spec.X509EncodedKeySpec;
import java.util.Date;

import javax.crypto.Cipher;

public class ClientX509Encryption {

        static final byte[] encodedPublicKey = {
            48, -127, -97, 48, 13, 6, 9, 42, -122, 72, -122, -9, 13, 1, 1, 
1,
            5, 0, 3, -127, -115, 0, 48, -127, -119, 2, -127, -127, 0, -77, 
119, -61,
            76, -66, -58, -102, -98, 84, 20, 83, 117, -9, 69, 109, -105, 
51, -58, 12,
            31, -113, 110, -118, 20, 115, 107, -86, -97, 14, -39, 62, 8, 
50, -22, 65,
            121, -124, -52, 27, 17, -104, -123, -78, -56, 92, 18, -31, 
116, 69, 105, -96,
            -87, 44, 60, -115, -110, 8, -124, 59, -68, 54, 33, 63, 108, 
24, 114, -11,
            -103, -31, 48, -89, 10, 98, 106, -80, 123, -94, -100, -24, 
-59, 18, 112, 74,
            112, 58, -94, -23, 70, -41, 35, 92, 92, 26, -27, -51, 127, 
-125, -109, 15,
            -30, -18, -12, 27, 101, 92, 4, 77, 11, -22, 123, 47, -18, 88, 
-96, 95,
            32, -118, -128, 82, 22, -55, -39, 9, -34, 4, 90, 88, -17, 2, 
3, 1,
            0, 1,
        };

        public static void main(String[] args) {
 
                byte[] encryptedpassword = null;
 
                try {
                        Cipher c = Cipher.getInstance("RSA/ECB/NoPadding"
);

                        X509EncodedKeySpec publicKeySpec = new 
X509EncodedKeySpec(encodedPublicKey);

                        KeyFactory keyFactory = KeyFactory.getInstance(
"RSA");
                        PublicKey publicKey = 
keyFactory.generatePublic(publicKeySpec);

                        c.init(Cipher.ENCRYPT_MODE, publicKey);

                        byte[] arg0Bytes = args[0].getBytes();
 
                        // encrypt input parameter
                        encryptedpassword = c.doFinal(arg0Bytes);

                        System.out.print("static signed char 
encryptedPasswordStringfromJavaClient[] = {");
                        for (int i = 0; i < encryptedpassword.length; i++) 
{
                                if (i % 16 == 0) {
                                        System.out.println("");
                                        System.out.print("    ");
                                }
                                System.out.print(encryptedpassword[i] + ", 
");
                        }
                        System.out.println("\n};");
 
                } catch (Exception e) {
                        e.printStackTrace();
                }

        }
}