Hannes Schuller wrote:
> hash = (unsigned char *)malloc(RSA_size(rsa) * sizeof(unsigned char));
> ciphertext = (char *)malloc(RSA_size(rsa) * sizeof(char));
> signature = (char *)malloc(RSA_size(rsa) * sizeof(char));
> if (ciphertext != NULL && signature != NULL && hash != NULL) {
> memset(ciphertext, 0, RSA_size(rsa));
> ok = RSA_private_encrypt(strlen(reply), (unsigned char *)reply,
> (unsigned char *)ciphertext, rsa, RSA_PKCS1_PADDING);
> if (ok < 0) {
> derror1("Message encryption error: %s",
> ERR_error_string(ERR_get_error(), (char
> *)NULL)); return (true);
> } else {
> dtrace1("Message encryption successful; return value:
> %d", ok); }
> len = base64Encode(ciphertext, ok);
> memset(hash, 0, RSA_size(rsa));
> RIPEMD160((unsigned char *)ciphertext, len, hash);
> memset(signature, 0, RSA_size(rsa));
> ok = RSA_private_encrypt(RIPEMD160_DIGEST_LENGTH, hash,
> (unsigned char *)signature, rsa, RSA_PKCS1_PADDING);
I'm very puzzled here. Why do you sign the reply and then sign a hash of the
signature? You say "Message encryption successful", but that's a signature
you're doing, not an encryption.
> That final line causes a segmentation fault. Here is a backtrace:
>
> -----
> #0 0x00007ffff74d0ba6 in ?? () from /lib/libc.so.6
> #1 0x00007ffff74d2aa0 in malloc () from /lib/libc.so.6
> #2 0x00007ffff7abc962 in CRYPTO_malloc (num=-142946720,
Since the fault is in 'malloc', that indicates something is trashing the
heap. Tools like 'valgrind' can find this for you. My guess would be that
the problem lies in 'base64Encode'. It doesn't seem to have any place to put
its output (and if it operates in place, it may overflow the memory
allocated for 'ciphertext'), but for all I know it calls 'realloc'
internally.
I have to also give you a generic warning -- there are some subtle clues in
your code that suggest that you do not know what you're doing. If you, or
anyone else, is going to rely on this code to meet any security
requirements, I *strongly* urge you to have the code evaluated by a security
expert sooner rather than later. It appears that you have designed this code
such that only the public key is needed to perform an operation that you
think of as decryption, and that's usually a sign of a serious design flaw.
Why are you don't things using these low-level functions anyway? OpenSSL
provides high-level functions with well-defined security properties.
DS
______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openssl-users@openssl.org
Automated List Manager majord...@openssl.org
