"David Schwartz" <dav...@webmaster.com> wrote:
>
> Hannes Schuller wrote:
>
> > hash = (unsigned char *)malloc(RSA_size(rsa) * sizeof(unsigned
> > char)); ciphertext = (char *)malloc(RSA_size(rsa) * sizeof(char));
> > signature = (char *)malloc(RSA_size(rsa) * sizeof(char));
> > if (ciphertext != NULL && signature != NULL && hash != NULL) {
> > memset(ciphertext, 0, RSA_size(rsa));
> > ok = RSA_private_encrypt(strlen(reply), (unsigned char
> > *)reply, (unsigned char *)ciphertext, rsa, RSA_PKCS1_PADDING);
> > if (ok < 0) {
> > derror1("Message encryption error: %s",
> > ERR_error_string(ERR_get_error(), (char
> > *)NULL)); return (true);
> > } else {
> > dtrace1("Message encryption successful; return
> > value: %d", ok); }
> > len = base64Encode(ciphertext, ok);
> > memset(hash, 0, RSA_size(rsa));
> > RIPEMD160((unsigned char *)ciphertext, len, hash);
> > memset(signature, 0, RSA_size(rsa));
> > ok = RSA_private_encrypt(RIPEMD160_DIGEST_LENGTH, hash,
> > (unsigned char *)signature, rsa, RSA_PKCS1_PADDING);
>
> I'm very puzzled here. Why do you sign the reply and then sign a hash
> of the signature? You say "Message encryption successful", but that's
> a signature you're doing, not an encryption.
I was under the impression that RSA_private_encrypt and
RSA_public_encrypt do nothing but encrypt the given payload. The
(non-quoted) code before this ensures the reply is shorter than the
regular message digest length.
What would be the prefered way for encryption?
> > That final line causes a segmentation fault. Here is a backtrace:
> >
> > -----
> > #0 0x00007ffff74d0ba6 in ?? () from /lib/libc.so.6
> > #1 0x00007ffff74d2aa0 in malloc () from /lib/libc.so.6
> > #2 0x00007ffff7abc962 in CRYPTO_malloc (num=-142946720,
>
> Since the fault is in 'malloc', that indicates something is trashing
> the heap. Tools like 'valgrind' can find this for you. My guess would
> be that the problem lies in 'base64Encode'. It doesn't seem to have
> any place to put its output (and if it operates in place, it may
> overflow the memory allocated for 'ciphertext'), but for all I know
> it calls 'realloc' internally.
Good point, I'll investigate.
> I have to also give you a generic warning -- there are some subtle
> clues in your code that suggest that you do not know what you're
> doing. If you, or anyone else, is going to rely on this code to meet
> any security requirements, I *strongly* urge you to have the code
> evaluated by a security expert sooner rather than later. It appears
> that you have designed this code such that only the public key is
> needed to perform an operation that you think of as decryption, and
> that's usually a sign of a serious design flaw.
I'm aware this is not the way it is intended and that it will only have
any security effect as far as signing the payload is concerned. Don't
worry. It's a long (boring) story.
Hannes
______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openssl-users@openssl.org
Automated List Manager majord...@openssl.org
