RSA padding problem

[Matt Geimer]

Hi,

I am working on a program that does bulk encryption key exchange similar to SSH.  I am running into a problem with RSA_private_decrypt, it returns -1 and tells me padding error, no matter which padding scheme I use.   The main goal is to encrypt a Blowfish key to send over the wire and then decrypt on the other side.  Any help is appreciated.

Regards,
Matt Geimer
Computer Science and Engineering Student
Michigan State University

The code I am trying this with is just a simple sample for me to make sure it works, which it doesn't... :
------
#include <iostream>
#include <openssl/rsa.h>
#include <openssl/err.h>

using namespace std;

int main(){

int returnvalue;

RSA * mykey = RSA_new();
mykey=RSA_generate_key(2048,65537,NULL,NULL);
if(mykey==NULL)
{
  cout << "Error initiating key generation" << endl;
  exit(1);
}

unsigned char pltxt[32]="This is the plain text";
unsigned char * pubenc,*privenc,*recovered,*plain;
plain=pltxt;
unsigned char tobuf1[256];
pubenc=tobuf1;
cout << "plain text is \"" << plain << "\"" << endl;

returnvalue=RSA_public_encrypt(8,plain,pubenc,mykey,RSA_PKCS1_OAEP_PADDING);
cout << returnvalue << endl;
if( returnvalue == -1)
{
  ERR_load_crypto_strings();
  char errbuf[180];
  char *buf=errbuf;
  ERR_error_string(ERR_get_error(),buf);
  cout << errbuf;
  ERR_free_strings();
}
cout << pubenc << endl;

cout << "now try to decrypt it..." << endl;
unsigned char tobuf2[256];
recovered=tobuf2;
returnvalue=RSA_private_decrypt(8,pubenc,recovered,mykey,RSA_PKCS1_OAEP_PADDING);
cout << returnvalue << endl;
if( returnvalue == -1)
{
  ERR_load_crypto_strings();
  char errbuf[180];
  char *buf=errbuf;
  ERR_error_string(ERR_get_error(),buf);
  cout << errbuf;
  ERR_free_strings();
}
cout << recovered << endl;

return 1;
}
-----
The output I get (only the error part):

error:0407A079:rsa routines:RSA_padding_check_PKCS1_OAEP:oaep decoding error