1. If you use certs, you are authenticating in SSL, perhaps unnecessarily if you doing some other auth and using that.
If you don't need SSL auth, why not just use anonymous-DH or anonymous-ECDH? That's exactly what they exist for. 2. To be clear, setting FAIL_IF_NO_PEER in server doesn't actually force the client to send a cert, but rejects handshake from a client that doesn't. It, and CLIENT_ONCE, are ignored on client side. OpenSSL client will always send cert (and use corresponding key) if allowed (server sends CertReq) and suitable for negotiated ciphersuite (and if you only offer ciphersuites that match your key&cert(s) that's guaranteed). 3. Using the same name for different selfsigned certs is just asking for confusion and mistakes. And if you need an OpenSSL server to accept connections from more than one selfsigned client, or one client to connect to more than one selfsigned server, it won't work at all. 4. The (local) error codes for (remote) fatal alerts are constructed by adding AD_REASON_OFFSET = 1000 to the alert code; see s3_pkt.c. Note that xx_srvr.c and xx_clnt.c generally contain only the logic that is specific to one side or the other; common and utility code is in xx_both xx_pkt xx_lib etc. 5. Server getting alert 48 unknown_ca from client means client doesn't have server's cert as trusted. You talked about creating selfsigned certs but you didn't say anything about securely transporting them to and installing them on peers; did you do that? Per above, for an OpenSSL client, if there is more than one server with a unique/different selfsigned cert but the same Subject (and Issuer) name, that won't work. From: owner-openssl-us...@openssl.org [mailto:owner-openssl-us...@openssl.org] On Behalf Of Troyanker, Vlad Sent: Tuesday, October 01, 2013 18:01 To: openssl-users@openssl.org Subject: tlsv1 alert unknown ca We are building a peer-to-peer system that uses SSL for connection privacy and performs authentication outside of SSL. The system creates self-signed certificates as needed on every node. Since we use those self-signed certficates in the authentication I have to force peer cert validation as the only means to forcing SSL to send client cert to the server. So I set the following flags on both ends of the connection: SSL_VERIFY_PEER | SSL_VERIFY_FAIL_IF_NO_PEER_CERT | SSL_VERIFY_CLIENT_ONCE Here is the code which creates the self-signed certificates: // Generates a self-signed x509 certificate static X509 * generate_x509(EVP_PKEY * pkey) { X509 * x509 = X509_new(); if(!x509) { throw CryptoException("Unable to create X509 structure"); } X509_set_version(x509, 2); ASN1_INTEGER_set(X509_get_serialNumber(x509), 1); X509_gmtime_adj(X509_get_notBefore(x509), 0); X509_gmtime_adj(X509_get_notAfter(x509), 31536000L); // one year X509_set_pubkey(x509, pkey); X509_NAME * name = X509_get_subject_name(x509); X509_NAME_add_entry_by_txt(name, "C", MBSTRING_ASC, (unsigned char *)"CA", -1, -1, 0); X509_NAME_add_entry_by_txt(name, "O", MBSTRING_ASC, (unsigned char *)"Amazon", -1, -1, 0); X509_NAME_add_entry_by_txt(name, "CN", MBSTRING_ASC, (unsigned char *)"localhost", -1, -1, 0); X509_set_issuer_name(x509, name); /* Inform openSSL this certificate can be used as CA */ add_ext(x509, NID_basic_constraints, "critical,CA:TRUE"); //add_ext(x509, NID_key_usage, "critical,keyCertSign,cRLSign"); if(!X509_sign(x509, pkey, EVP_sha1())) { X509_free(x509); throw CryptoException("Error signing certificate"); } return x509; } PROBLEM: the server fails SSL connection with SSL_accept: tlsv1 alert unknown ca The funny part I cannot even find where in source code that error (code SSL_R_TLSV1_ALERT_UNKNOWN_CA) gets thrown. I am looking through openssl-1.0.1e/ssl/s3_srvr.c What am I missing? Thank you for your time
