Hi,
we're having a weird error on Apache httpd server that I can't
understand how to troubleshoot it and not clear to me if it is an our problem
(apache http server) or a problem of the caller.
We have actually this configuration:
Server version: Apache/2.4.46 (Unix)
Server built: May 13 2021 05:46:31
Server's Module Magic Number: 20120211:93
Server loaded: APR 1.6.5, APR-UTIL 1.6.1
Compiled using: APR 1.6.5, APR-UTIL 1.6.1
Architecture: 64-bit
Server MPM: event
threaded: yes (fixed thread count)
forked: yes (variable process count)
Server compiled with....
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=256
-D HTTPD_ROOT="/data/apache2_frontend"
-D SUEXEC_BIN="/data/apache2_frontend/bin/suexec"
-D DEFAULT_PIDLOG="logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
The problem we have is that during ssl handshake we can see (only with debug or
tcpdump) an "SSL Library Error: error:1408F119:SSL
routines:SSL3_GET_RECORD:decryption failed or bad record mac" into apache httpd
error_log.
No other logs are written into access_log.
How is possible to troubleshoot it and understand where is the problem (caller?
network? receiver?)
Some logs from trace:
[Wed May 12 17:52:04.134409 2021] [ssl:debug] [pid 10532:tid 140112100849408]
ssl_engine_kernel.c(1741): [client ip:port] AH02275: Certificate Verification,
depth 2, CRL checking mode: none (0) [subject: CN=etc etc etc]
[Wed May 12 17:52:04.134553 2021] [ssl:debug] [pid 10532:tid 140112100849408]
ssl_engine_kernel.c(1741): [client ip:port] AH02275: Certificate Verification,
depth 1, CRL checking mode: none (0) [subject: CN=etc etc etc]
[Wed May 12 17:52:04.134681 2021] [ssl:debug] [pid 10532:tid 140112100849408]
ssl_engine_kernel.c(1741): [client ip:port] AH02275: Certificate Verification,
depth 0, CRL checking mode: none (0) [subject: CN=etc etc etc]
[Wed May 12 17:52:04.134705 2021] [ssl:trace3] [pid 10532:tid 140112100849408]
ssl_engine_kernel.c(2192): [client ip:port] OpenSSL: Loop: SSLv3 read client
certificate A
[Wed May 12 17:52:04.138368 2021] [ssl:trace3] [pid 10532:tid 140112100849408]
ssl_engine_kernel.c(2192): [client ip:port] OpenSSL: Loop: SSLv3 read client
key exchange A
[Wed May 12 17:52:04.138492 2021] [ssl:trace3] [pid 10532:tid 140112100849408]
ssl_engine_kernel.c(2192): [client ip:port] OpenSSL: Loop: SSLv3 read
certificate verify A
[Wed May 12 17:52:04.138513 2021] [ssl:trace4] [pid 10532:tid 140112100849408]
ssl_engine_io.c(2214): [client ip:port] OpenSSL: read 5/5 bytes from
BIO#7f6e2000ff60 [mem: 7f6e2c06f653]
[Wed May 12 17:52:04.138519 2021] [ssl:trace4] [pid 10532:tid 140112100849408]
ssl_engine_io.c(2214): [client ip:port] OpenSSL: read 1/1 bytes from
BIO#7f6e2000ff60 [mem: 7f6e2c06f658]
[Wed May 12 17:52:04.138568 2021] [ssl:trace4] [pid 10532:tid 140112100849408]
ssl_engine_io.c(2214): [client ip:port] OpenSSL: read 5/5 bytes from
BIO#7f6e2000ff60 [mem: 7f6e2c06f653]
[Wed May 12 17:52:04.138586 2021] [ssl:trace4] [pid 10532:tid 140112100849408]
ssl_engine_io.c(2214): [client ip:port] OpenSSL: read 40/40 bytes from
BIO#7f6e2000ff60 [mem: 7f6e2c06f658]
[Wed May 12 17:52:04.138600 2021] [ssl:trace4] [pid 10532:tid 140112100849408]
ssl_engine_io.c(2214): [client ip:port] OpenSSL: write 7/7 bytes to
BIO#7f6e20010f50 [mem: 7f6e3c03f763]
[Wed May 12 17:52:04.138607 2021] [core:trace6] [pid 10532:tid 140112100849408]
core_filters.c(525): [client ip:port] will flush because of FLUSH bucket
[Wed May 12 17:52:04.138639 2021] [ssl:trace4] [pid 10532:tid 140112100849408]
ssl_engine_io.c(2214): [client ip:port] OpenSSL: write 7/7 bytes to
BIO#7f6e20011d50 [mem: 7f6e20004950]
[Wed May 12 17:52:04.138669 2021] [core:trace6] [pid 10532:tid 140112100849408]
core_filters.c(525): [client ip:port] will flush because of FLUSH bucket
[Wed May 12 17:52:04.138676 2021] [ssl:trace3] [pid 10532:tid 140112100849408]
ssl_engine_kernel.c(2202): [client ip:port] OpenSSL: Write: error
[Wed May 12 17:52:04.138680 2021] [ssl:trace3] [pid 10532:tid 140112100849408]
ssl_engine_kernel.c(2221): [client ip:port] OpenSSL: Exit: error in error
[Wed May 12 17:52:04.138690 2021] [ssl:info] [pid 10532:tid 140112100849408]
[client ip:port] AH02008: SSL library error 1 in handshake (server server:port)
[Wed May 12 17:52:04.138711 2021] [ssl:info] [pid 10532:tid 140112100849408]
SSL Library Error: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption
failed or bad record mac
[Wed May 12 17:52:04.138720 2021] [ssl:info] [pid 10532:tid 140112100849408]
[client ip:port] AH01998: Connection closed to child 448 with abortive shutdown
(server server:port)
Thank you
Matteo Piemonti
________________________________
This message is for the designated recipient only and may contain privileged,
proprietary, or otherwise confidential information. If you have received it in
error, please notify the sender immediately and delete the original. Any other
use of the e-mail by you is prohibited. Where allowed by local law, electronic
communications with Accenture and its affiliates, including e-mail and instant
messaging (including content), may be scanned by our systems for the purposes
of information security and assessment of internal compliance with Accenture
policy. Your privacy is important to us. Accenture uses your personal data only
in compliance with data protection laws. For further information on how
Accenture processes your personal data, please see our privacy statement at
https://www.accenture.com/us-en/privacy-policy.
______________________________________________________________________________________
www.accenture.com
