The server error is correct - the signature_algorithms_cert extension does not offer rsa_pkcs1_sha256 (0x0401) which is the server certificate signing algorithm. If the client is written in Java, check java.security for "jdk.certpath.disabledAlgorithms" and check the constraints.
On Fri, 2021-12-31 at 15:05 +0000, Michael Wojcik wrote: > > From: openssl-users <openssl-users-boun...@openssl.org> On Behalf > > Of Ma Zhenhua > > Sent: Thursday, 30 December, 2021 23:59 > > On the SSL/TLS server, there's one error as follows. > > "SSL Error(118) - no suitable signature algorithm" > > Debugging handshake failures isn't my area of expertise, but I note > both ClientHellos include a signature_algorithms extension, and the > contents are quite different. In particular, the successful > ClientHello includes the Signature Hash Algorithm Hash and Signature > Hash Algorithm Signature parameters, while the failing one doesn't. > > The failing one also includes a signature_algorithms_cert extension, > while the successful one does not. I don't know offhand how the > algorithms specified in that extension correspond to the signature- > algorithm OIDs in signatures, but the server's certificate has > 1.2.840.113549.1.1.11 (sha256WithRSAEncryption) which seems like it > ought to correspond to either rsa_pss_rsae_sha256 or > rsa_pss_pss_sha256. (Apparently those are both RSA-PSS with SHA256, > as the name implies, and the difference between the two of them is > whether the public key is encoded using the rsaEncryption format in > the certificate, or the id-RSASSA-PSS format. The failing client is > saying it understands both, AIUI.) > > So my guess would be the server is unhappy that the failing client's > ClientHello doesn't include the parameters for the various supported > signature schemes in its signature_algorithms extension. But that's > just a guess, and I don't know how you'd fix it. >
