> Though the RSA_sign function signature doesn't seem to allow specifying
the padding.
Sorry I meant the hashing algorithm.
So I went back to this and I think the problem is we are forced to
create a RSA_private_encrypt function even if all we want to do is sign.
https://github.com/openssl/openssl/blob/master/crypto/rsa/rsa_pmeth.c#L184
That branch gets hit on doing a TLS connection but not on creating a
certificat
I implemented some custom engines and RSA_meth_set_priv_enc seems to map
to other libraries' RSA decrypt operation (NCryptDecrypt on Windows
CNG, Cipher class with Cipher.DECRYPT_MODE on Android). They can do a
TLS connection just fine with a self-signed cert.
I looked at another custom e
Thanks, I'll read up on your discussion there.
My original use for this is to share the same certificate and key on a
process that has both a TLS and DTLS connection. I went with just making
the DTLS derive a PSK from the keying material of the TLS.
It's DTLS-OK according to IANA.
https://www.iana.org/assignments/tls-parameters/tls-parameters.xhtml#tls-parameters-16
I tested ED25519 certificates on TLS 1.2 and it worked fine.
openssl s_server -port 4321 -cert server-cert.pem -key server-key.pem
-CAfile client-cert.pem -tls1_2 -sigalgs ed255
