RE: Issue in DH Algorithm Keys Generation in OpenSSL 3.3.0

[Vishal Kevat via openssl-users]

Hi OpenSSL users,

I am using OpenSSL source version 3.3.0 and facing an issue in key generation 
part of Diffie Hellman (DH) Algorithm. Below are the APIs I am using for 
generating Public and Private Keys:

static unsigned char DH_PRIME_128[] = { /* 128 bit prime */
      0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
      0xc9, 0x0f, 0xda, 0xa2, 0x21, 0x68, 0xc2, 0x34
};

static unsigned char dh_g[] = {
      0x02,
};

dhContext->handle = DH_new(); // Allocate DH Context

BIGNUM *dhp_bn, *dhg_bn;

dhp_bn = BN_bin2bn(DH_PRIME_128, sizeof(DH_PRIME_128), NULL);
dhg_bn = BN_bin2bn(dh_g, sizeof(dh_g), NULL);
DH_set0_pqg(dhContext->handle, dhp_bn, NULL, dhg_bn)

int ret = DH_generate_key(dhContext->handle);  // Generate DH Pub and Priv keys

const BIGNUM* pub_key = DH_get0_pub_key(dhContext->handle);
const BIGNUM* pri_key = DH_get0_priv_key(dhContext->handle);


When I set the prime number DH_PRIME_128 as above, the API DH_generate_key() is 
failing to generate the Public and Private Keys and returning 0.
Using the above prime number Mocana Crypto Library is able to generate the DH 
Pub and Priv keys but OpenSSL is failing to do so.



When I use the below prime number instead:
static unsigned char DH_PRIME_128[] = { /* 128 bit prime */
      0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
      0x34, 0xC2, 0x68, 0x21, 0xA2, 0xDA, 0x0F, 0xC9
};

The DH_generate_key() API is successfully able to generate the keys.

Please let me know, what is the reason of this failing API when I use the first 
prime number for DH algorithm key generation or I am doing wrong anything here?

Regards,
Vishal Kevat




General