Using javax.crypto.Cipher with RSA and an OpenSSL generated public key

[PMicke]

Hi, 

I want to write a .NET client that sends an encrypted message to a server 
written in C, and decrypt the message there.
I use RSA with PKCS1 padding..
But the server cannot decrypt the sequence from the .NET client.
error:2006E079:BIO routines:BIO_puts:unsupported method
 
I see 3 possible sources of error in my code. 
1. How big is the chance that the public key encoded by the server for the 
.NET client is not correct even so it is accepted by the .NET client 
without error?
2. The way how I encode the encrypted sequence in the client is wrong. 
That would be an obvious reason for the error during decryption. 
3. There are 2 different versions of PKCS1 padding used.

To test the scenario  I use the J# code at the end of this mail. First I 
made sure that encryption / decryption works in J# alone.
To get the public key generated with OpenSSL into a format that can be 
used in .NET I use the following crypto library calls:

        BIO *b64, *out, *wbio;
        b64 = BIO_new(BIO_f_base64());
        out = BIO_new(BIO_s_file());
        BIO_set_fp(out,stdout,BIO_NOCLOSE);
        wbio=BIO_push(b64,out);

        int val[1] = {65537};
        char *buf = (char*) val;
        BIO_write(wbio, val, 3);  /* exponent, only 3 bytes written */
        (void)BIO_flush(wbio);
        BIO_write(wbio, rsa_public->n->d, 128); /* modulus */
        (void)BIO_flush(wbio);

The server decrypts the password with the same private key used to 
generate the public key:
                retval = RSA_private_decrypt(RSA_size(rsa_private), 
encryptedPasswordCharArrayfromNetClient, cleartext-output, rsa_private, 
RSA_PKCS1_PADDING);

I tested my code so that encryption /  decryption works in the C code 
using OpenSSL.
I tested my J# code so that encryption /  decryption works using standard 
.NET class  for RSA.
I have trouble to get the two together. If I use RSA_private_decrypt with 
a byte sequence created from the .NET client I get the error:
Error: 537321593
error:2006E079:BIO routines:BIO_puts:unsupported method

Debugging shows that the error is returned from 
        case RSA_PKCS1_PADDING:
                r=RSA_padding_check_PKCS1_type_2(to,num,buf,j,num);


The J# client code:

package RSAEncryption.java;

import System.*;
import System.Security.Cryptography.*;
import System.Text.*;

class RSASample
{
    public static void main(String[] args)
    {
        try {
            // Create a UnicodeEncoder to convert between byte array and 
string.
            UnicodeEncoding byteConverter =  new UnicodeEncoding();
            ubyte unicodeDataToEncrypt[] = byteConverter.GetBytes(
"myPassword!");

                        Encoding ascii = Encoding.get_ASCII();
                        Encoding unicode = Encoding.get_Unicode();
                        ubyte dataToEncrypt[] = Encoding.Convert(unicode, 
ascii, unicodeDataToEncrypt);
 
                        RSACryptoServiceProvider rsa = new 
RSACryptoServiceProvider();

                        String publicRSAKeyXML = 
"<RSAKeyValue><Modulus>h...deleted...0=</Modulus><Exponent>AQAB</Exponent></RSAKeyValue>"
;
                        rsa.FromXmlString(publicRSAKeyXML);

                        RSAParameters rsaParameter = rsa.ExportParameters(
false);

            // 3. parameter: a boolean flag specifying no OAEP padding.
            ubyte [] encryptedData = RSAEncrypt(dataToEncrypt, 
rsaParameter, false);

        // Display the encrypted sequence to the console. 
        System.out.print("static signed char 
encryptedPasswordStringfromNetClient[] = {");
        for (int i = 0; i < encryptedData.length; i++)
        {
                if (i % 16 == 0)
                {
                        System.out.println("");
                        System.out.print("    ");
                }
                System.out.print((byte)encryptedData[i] + ", ");
        }
        System.out.println("\n};");
 
        }
        catch (ArgumentNullException exp) {
            //Catch this exception in case the encryption did
            //not succeed.
            Console.WriteLine("Encryption failed.");
        }
    } //main 

    public static ubyte[] RSAEncrypt(ubyte dataToEncrypt[], 
        RSAParameters rsaKeyInfo, boolean doOaepPadding) 
    {
        try {
            // Create a new instance of RSACryptoServiceProvider.
            RSACryptoServiceProvider rsa =  new RSACryptoServiceProvider
();
 
            // Import the rsa Key information. This only needs
            // to include the public key information.
            rsa.ImportParameters(rsaKeyInfo);
 
            // Encrypt the passed byte array and specify OAEP padding. 
            // OAEP padding is only available on Microsoft Windows XP or
            // later. 
            return rsa.Encrypt(dataToEncrypt, doOaepPadding);
        }
        // Catch and display a CryptographicException 
        // to the console.
        catch (CryptographicException e) {
            Console.WriteLine(e.get_Message());
            return null ;
        }
    } //RSAEncrypt

} //RSASample

Regards,
Paul