I did the following 3 basic tests, purpose of which was to determine whether browser client certificate validation is failing at browser or haproxy:
1. Test one is openssl s_client, result is successful connection 2. Test two is browser with haproxy ssl verify disabled, result is successful connection 3. Test three is browser with haproxy ssl verify enabled, result is failure to connect with error indicating peer did not send a certificate. Browser is Safari with CA root and client cert installed and valid, with preference configuration set to return client certificate for the website being fetched. Result is the same in Chrome. SSLDump at haproxy indicates negotiation of TLSv1.2 and also shows receipt of client certificate in case 1 and case 3. haproxy cli "@1 show errors" shows nothing in case 3. Server and Client certificates in Test 1, Test 2, and Test 3 are the same. Build of proxy 2.2 is on Centos-7 with gcc 8.x via devtoolset-8. Would be very useful to get this working. Any suggestions would be appreciated. This looks like a bug to me, but am not sure. I can provide access to the proxy if necessary. Thank you, details follow Dennis ---------------------------------- For all tests below request is: https://xx.xx.xx.82/stats ===TEST 1=========================================== SSLDump using opennssl s_client bind xx.xx.xx.82:443 alpn h2,http/1.1 ssl crt /etc/haproxy/ssl.d/ ca-file /etc/haproxy/ca_all.pem verify required ---------------------------------------------- New TCP connection #1: yy.yy.yy.167(47279) <-> xx.xx.xx.82(443) 1 1 0.0820 (0.0820) C>S Handshake ClientHello Version 3.3 cipher suites TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA TLS_DHE_DSS_WITH_AES_256_GCM_SHA384 TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 TLS_DHE_RSA_WITH_AES_256_CBC_SHA TLS_DHE_DSS_WITH_AES_256_CBC_SHA TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA TLS_DHE_DSS_WITH_CAMELLIA_256_CBC_SHA TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384 TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384 TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 TLS_ECDH_RSA_WITH_AES_256_CBC_SHA TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_AES_256_GCM_SHA384 TLS_RSA_WITH_AES_256_CBC_SHA256 TLS_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_CAMELLIA_256_CBC_SHA TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA TLS_DHE_DSS_WITH_AES_128_GCM_SHA256 TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 TLS_DHE_RSA_WITH_AES_128_CBC_SHA TLS_DHE_DSS_WITH_AES_128_CBC_SHA TLS_DHE_RSA_WITH_SEED_CBC_SHA TLS_DHE_DSS_WITH_SEED_CBC_SHA TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA TLS_DHE_DSS_WITH_CAMELLIA_128_CBC_SHA TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256 TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256 TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256 TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256 TLS_ECDH_RSA_WITH_AES_128_CBC_SHA TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_AES_128_GCM_SHA256 TLS_RSA_WITH_AES_128_CBC_SHA256 TLS_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_SEED_CBC_SHA TLS_RSA_WITH_CAMELLIA_128_CBC_SHA TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA TLS_RSA_WITH_3DES_EDE_CBC_SHA TLS_RSA_WITH_IDEA_CBC_SHA TLS_ECDHE_RSA_WITH_RC4_128_SHA TLS_ECDHE_ECDSA_WITH_RC4_128_SHA TLS_ECDH_RSA_WITH_RC4_128_SHA TLS_ECDH_ECDSA_WITH_RC4_128_SHA TLS_RSA_WITH_RC4_128_SHA TLS_RSA_WITH_RC4_128_MD5 TLS_EMPTY_RENEGOTIATION_INFO_SCSV compression methods NULL 1 2 0.0874 (0.0053) S>C Handshake ServerHello Version 3.3 session_id[0]= cipherSuite TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 compressionMethod NULL 1 3 0.0874 (0.0000) S>C Handshake Certificate 1 4 0.0874 (0.0000) S>C Handshake ServerKeyExchange Not enough data. Found 327 bytes (expecting 32767) 1 5 0.0874 (0.0000) S>C Handshake CertificateRequest certificate_types rsa_sign certificate_types dss_sign certificate_types ServerHelloDone 1 6 0.1230 (0.0355) C>S Handshake Certificate 1 7 0.1230 (0.0000) C>S Handshake ClientKeyExchange Not enough data. Found 64 bytes (expecting 16384) 1 8 0.1230 (0.0000) C>S Handshake CertificateVerify Not enough data. Found 258 bytes (expecting 16384) 1 9 0.1230 (0.0000) C>S ChangeCipherSpec 1 10 0.1230 (0.0000) C>S Handshake 1 11 0.1240 (0.0009) S>C Handshake 1 12 0.1240 (0.0000) S>C ChangeCipherSpec 1 13 0.1240 (0.0000) S>C Handshake 1 15.5871 (15.4631) S>C TCP FIN 1 15.6109 (0.0237) C>S TCP FIN ============================================ HAPROXY REPORTS 2.2.0, centos 7 ============================================ 00000000:https_front.accept(0009)=002d from [yy.yy.yy.167:47279] ALPN=<none> ===TEST 2=========================================== SSLDump using browser with verify disabled. bind xx.xx.xx.82:443 alpn h2,http/1.1 ssl crt /etc/haproxy/ssl.d/ ---------------------------------------------- [root@db0 haproxy]# ssldump -k ssl.d/admin.domain.net.pem -i enp1s0f1 -H -n -S crypto port 443 and host yy.yy.yy.167 New TCP connection #1: yy.yy.yy.167(39312) <-> xx.xx.xx.82(443) 1 1 0.0238 (0.0238) C>S Handshake ClientHello Version 3.3 resume [32]= 4b 93 70 2f fc d8 2c 69 d0 6b ab 20 b7 42 b8 19 73 7d 74 2e 4a bd 21 6a 94 79 07 d1 02 41 f8 08 cipher suites Unknown value 0x1301 Unknown value 0x1302 Unknown value 0x1303 TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA Unknown value 0xcca9 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA Unknown value 0xcca8 TLS_RSA_WITH_AES_256_GCM_SHA384 TLS_RSA_WITH_AES_128_GCM_SHA256 TLS_RSA_WITH_AES_256_CBC_SHA256 TLS_RSA_WITH_AES_128_CBC_SHA256 TLS_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_AES_128_CBC_SHA TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA TLS_RSA_WITH_3DES_EDE_CBC_SHA compression methods NULL 1 2 0.0242 (0.0004) S>C Handshake ServerHello Version 3.3 session_id[32]= 4b 93 70 2f fc d8 2c 69 d0 6b ab 20 b7 42 b8 19 73 7d 74 2e 4a bd 21 6a 94 79 07 d1 02 41 f8 08 cipherSuite TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 compressionMethod NULL 1 3 0.0242 (0.0000) S>C ChangeCipherSpec 1 4 0.0242 (0.0000) S>C Handshake 1 5 0.0485 (0.0242) C>S ChangeCipherSpec 1 6 0.0485 (0.0000) C>S Handshake 1 7 0.0485 (0.0000) C>S application_data 1 8 0.0485 (0.0000) C>S application_data 1 9 0.0485 (0.0000) C>S application_data 1 10 0.0485 (0.0000) C>S application_data 1 11 0.0490 (0.0004) S>C application_data 1 12 0.0505 (0.0015) S>C application_data 1 13 0.0755 (0.0250) C>S application_data 1 14 0.0834 (0.0078) C>S application_data 1 15 0.1079 (0.0244) S>C application_data 1 16 0.1080 (0.0001) S>C application_data 1 17 0.1080 (0.0000) S>C application_data 1 18 30.0851 (29.9771) S>C application_data 1 30.0852 (0.0000) S>C TCP FIN 1 19 30.1923 (0.1071) C>S application_data 1 30.2149 (0.0225) C>S TCP RST ===TEST 3=========================================== SSLDump using browser with verify enabled. bind xx.xx.xx.82:443 alpn h2,http/1.1 ssl crt /etc/haproxy/ssl.d/ ca-file /etc/haproxy/ca_all.pem verify required ---------------------------------------------- New TCP connection #2: yy.yy.yy.167(41842) <-> xx.xx.xx.82(443) 2 1 0.0237 (0.0237) C>S Handshake ClientHello Version 3.3 resume [32]= 14 b9 38 6d 1e ab 18 99 f7 d3 bb fe 7e 17 5a e1 e9 8f d8 c7 b8 ef c4 fd 25 59 27 fd 54 9c f6 55 cipher suites Unknown value 0x1301 Unknown value 0x1302 Unknown value 0x1303 TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA Unknown value 0xcca9 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA Unknown value 0xcca8 TLS_RSA_WITH_AES_256_GCM_SHA384 TLS_RSA_WITH_AES_128_GCM_SHA256 TLS_RSA_WITH_AES_256_CBC_SHA256 TLS_RSA_WITH_AES_128_CBC_SHA256 TLS_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_AES_128_CBC_SHA TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA TLS_RSA_WITH_3DES_EDE_CBC_SHA compression methods NULL 2 2 0.0288 (0.0050) S>C Handshake ServerHello Version 3.3 session_id[32]= e8 3d 9d 0c 9d ae be f2 4e aa 21 70 0a d0 ad 1c 85 73 6d 79 0d 3c cc 23 1c 5e d7 a9 c5 0e 8b 7b cipherSuite TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 compressionMethod NULL 2 3 0.0288 (0.0000) S>C Handshake Certificate 2 4 0.0288 (0.0000) S>C Handshake ServerKeyExchange Not enough data. Found 327 bytes (expecting 32767) 2 5 0.0288 (0.0000) S>C Handshake CertificateRequest certificate_types rsa_sign certificate_types dss_sign certificate_types ServerHelloDone 2 6 0.5969 (0.5681) C>S Handshake Certificate 2 7 0.5969 (0.0000) C>S Handshake ClientKeyExchange Not enough data. Found 64 bytes (expecting 16384) 2 8 0.5969 (0.0000) C>S ChangeCipherSpec 2 9 0.5969 (0.0000) C>S Handshake 2 10 0.5972 (0.0002) S>C Alert level fatal value handshake_failure 2 0.5977 (0.0005) S>C TCP FIN 2 0.6225 (0.0247) C>S TCP FIN New TCP connection #3: yy.yy.yy.167(40914) <-> xx.xx.xx.82(443) 3 1 0.0232 (0.0232) C>S Handshake ClientHello Version 3.3 resume [32]= d7 40 c0 b4 d4 05 ea f3 e5 ad 59 ae 32 88 0f 9e 4d cb 1f 24 46 e5 b8 4c d6 43 57 9a 6f da 92 7c cipher suites Unknown value 0x1301 Unknown value 0x1302 Unknown value 0x1303 TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA Unknown value 0xcca9 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA Unknown value 0xcca8 TLS_RSA_WITH_AES_256_GCM_SHA384 TLS_RSA_WITH_AES_128_GCM_SHA256 TLS_RSA_WITH_AES_256_CBC_SHA256 TLS_RSA_WITH_AES_128_CBC_SHA256 TLS_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_AES_128_CBC_SHA TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA TLS_RSA_WITH_3DES_EDE_CBC_SHA Unknown value 0x5600 compression methods NULL 3 2 0.0281 (0.0048) S>C Handshake ServerHello Version 3.3 session_id[32]= 85 78 e3 bf db da e3 9b 02 43 26 08 9a eb ab 46 15 c7 2f 01 3f 22 cf 1c f3 f8 e1 00 8c 01 26 f5 cipherSuite TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 compressionMethod NULL 3 3 0.0281 (0.0000) S>C Handshake Certificate 3 4 0.0281 (0.0000) S>C Handshake ServerKeyExchange Not enough data. Found 327 bytes (expecting 32767) 3 5 0.0281 (0.0000) S>C Handshake CertificateRequest certificate_types rsa_sign certificate_types dss_sign certificate_types ServerHelloDone ===================================================== > PEER XMITS CERTIFICATE ----------------------------------------------------- 3 6 0.5678 (0.5396) C>S Handshake Certificate 3 7 0.5678 (0.0000) C>S Handshake ClientKeyExchange Not enough data. Found 64 bytes (expecting 16384) 3 8 0.5678 (0.0000) C>S ChangeCipherSpec 3 9 0.5678 (0.0000) C>S Handshake ===================================================== < SERVER XMITS fatal error ----------------------------------------------------- 3 10 0.5679 (0.0001) S>C Alert level fatal value handshake_failure ----------------------------------------------------- 3 0.5683 (0.0003) S>C TCP FIN 3 0.5940 (0.0257) C>S TCP FIN ============================================ HAPROXY LOG SHOWS ============================================ fd[002d] OpenSSL error[0x140890c7] ssl3_get_client_certificate: peer did not return a certificate fd[002d] OpenSSL error[0x140890c7] ssl3_get_client_certificate: peer did not return a certificate fd[002d] OpenSSL error[0x1408a10b] ssl3_get_client_hello: wrong version number ============================================ HAPROXY 2.2.0 build config ============================================ [root@db0 haproxy]# /usr/local/sbin/haproxy -vv HA-Proxy version 2.2.0 2020/07/07 - https://haproxy.org/ Status: long-term supported branch - will stop receiving fixes around Q2 2025. Known bugs: http://www.haproxy.org/bugs/bugs-2.2.0.html Running on: Linux 3.10.0-1127.13.1.el7.x86_64 #1 SMP Tue Jun 23 15:46:38 UTC 2020 x86_64 Build options : TARGET = linux-glibc CPU = generic CC = gcc CFLAGS = -O2 -g -Wall -Wextra -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits OPTIONS = USE_OPENSSL=1 USE_SYSTEMD=1 Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H +GETADDRINFO +OPENSSL -LUA +FUTEX +ACCEPT4 -ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS Default settings : bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Built with multi-threading support (MAX_THREADS=64, default=12). Built with OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017 Running on OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2 Built with network namespace support. Built without compression support (neither USE_ZLIB nor USE_SLZ are set). Compression algorithms supported : identity("identity") Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Built without PCRE or PCRE2 support (using libc's regex instead) Encrypted password support via crypt(3): yes Built with gcc compiler version 4.8.5 20150623 (Red Hat 4.8.5-39) Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. Available multiplexer protocols : (protocols marked as <default> cannot be specified using 'proto' keyword) fcgi : mode=HTTP side=BE mux=FCGI <default> : mode=HTTP side=FE|BE mux=H1 h2 : mode=HTTP side=FE|BE mux=H2 <default> : mode=TCP side=FE|BE mux=PASS Available services : none Available filters : [SPOE] spoe [COMP] compression [TRACE] trace [CACHE] cache [FCGI] fcgi-app
