I'm running to the exact same issue, and I've done exactly the same
troubleshoot, yet I don't have any more ideas of what to try ....
I'm with Ubuntu 22.04.1 LTS , Linux HOST 5.15.0-47-generic #51-Ubuntu SMP
Thu Aug 11 07:51:15 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Product Name: ConnectX-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, PCIe
4.0 x16, Crypto and Secure Boot
[PN] Part number: MCX623106AC-CDAT
Running the latest firmware and drivers
ethtool -i enp193s0f1np1
driver: mlx5_core
version: 5.7-1.0.2
firmware-version: 22.34.4000 (MT_0000000436)
expansion-rom-version:
bus-info: 0000:c1:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
ethtool -k enp193s0f1np1 | grep tls
tls-hw-tx-offload: on
tls-hw-rx-offload: on
tls-hw-record: off [fixed]
I have almost everything the same as @liwuliu, yet I'm unable to use NIC
kTLS.
I tried nginx 1.23.1, and 1.22.0
I tried openSSL 3.0.1 and 3.0.5
I tried static content only, I tried reverse proxy, yet unable to make HW
kTLS work (based on TLS stats, and based on Ethtool -S stats)
SW kTLS works:
cat /proc/net/tls_stat
TlsCurrTxSw 0
TlsCurrRxSw 0
TlsCurrTxDevice 0
TlsCurrRxDevice 0
TlsTxSw 11
TlsRxSw 0
TlsTxDevice 0
TlsRxDevice 0
TlsDecryptError 0
TlsRxDeviceResync 0
inline/nic kTLS doesn't seem to work
tx_tls_encrypted_packets: 0
tx_tls_encrypted_bytes: 0
tx_tls_ooo: 0
tx_tls_dump_packets: 0
tx_tls_dump_bytes: 0
tx_tls_resync_bytes: 0
tx_tls_skip_no_sync_data: 0
tx_tls_drop_no_sync_data: 0
tx_tls_drop_bypass_req: 0
rx_tls_decrypted_packets: 0
rx_tls_decrypted_bytes: 0
rx_tls_resync_req_pkt: 0
rx_tls_resync_req_start: 0
rx_tls_resync_req_end: 0
rx_tls_resync_req_skip: 0
rx_tls_resync_res_ok: 0
rx_tls_resync_res_retry: 0
rx_tls_resync_res_skip: 0
rx_tls_err: 0
tx_tls_ctx: 0
tx_tls_del: 0
rx_tls_ctx: 0
rx_tls_del: 0
rx0_tls_decrypted_packets: 0
rx0_tls_decrypted_bytes: 0
rx0_tls_resync_req_pkt: 0
rx0_tls_resync_req_start: 0
rx0_tls_resync_req_end: 0
rx0_tls_resync_req_skip: 0
rx0_tls_resync_res_ok: 0
rx0_tls_resync_res_retry: 0
rx0_tls_resync_res_skip: 0
rx0_tls_err: 0
All the settings @liwuliu wrote, I have the same.
Only thing I'm not sure when @liwuliu wrote he made it work, if it was typo
OpenSSL "3.1.0" that he said, cuz I can't find that version, so maybe he
tried 3.0.1? not quite sure, Latest I was able to find is 3.0.5.
I checked his Cipher list, and its 100% exactly the same as his
(/nginx/openssl-3.0.5/.openssl/bin] ./openssl ciphers)
my Nginx is built the same as well.
nginx version: nginx/1.22.0
built by gcc 11.2.0 (Ubuntu 11.2.0-19ubuntu1)
built with OpenSSL 3.0.5 5 Jul 2022
TLS SNI support enabled
configure arguments: --with-debug --with-http_slice_module
--with-http_ssl_module --with-http_realip_module --with-http_mp4_module
--with-http_flv_module --with-threads --with-http_stub_status_module
--with-http_secure_link_module --with-http_gzip_static_module
--with-http_v2_module --with-http_gunzip_module --with-http_geoip_module
--with-pcre-jit --with-compat --with-file-aio --with-http_addition_module
--with-http_auth_request_module --with-http_dav_module
--with-http_random_index_module --with-http_sub_module --with-stream
--with-stream_realip_module --with-stream_ssl_module
--with-stream_ssl_preread_module --with-openssl=../openssl-3.0.5
--with-openssl-opt=enable-ktls --with-cc-opt='-g -O2
-fstack-protector-strong -Wformat -Werror=format-security
-Wp,-D_FORTIFY_SOURCE=2 -fPIC'
This is the output of strace, downloading 400MB file via TLS
strace -e trace=network -p `pidof nginx | sed -e 's/ /,/g'` 2>&1 :
https://dpaste.com/HSU5QY2PY
This is "curl -v https://domain.com/data/1"; output:
https://dpaste.com/29DSYBQU2
my nginx config is the following:
pcre_jit on;
error_log /home/logs/error.log debug;
user www-data;
worker_processes auto;
worker_rlimit_nofile 50000;
worker_cpu_affinity auto;
events {
worker_connections 50000;
multi_accept on;
}
http {
include mime.types;
# tcp_nodelay on;
# tcp_nopush on;
sendfile on;
# sendfile_max_chunk 1m;
keepalive_timeout 60;
server {
listen 443 ssl reuseport;
server_name *.domain;
ssl_conf_command Options KTLS;
ssl_certificate /usr/local/nginx/cert/certificate.cer;
ssl_certificate_key /usr/local/nginx/cert/certificate.key;
ssl_protocols TLSv1.3;
#ssl_session_cache shared:SSL:10m;
#ssl_session_timeout 5m;
#ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
#ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
#ssl_prefer_server_ciphers on;
#access_log /home/logs/access.log;
#error_log /home/logs/error.log debug;
location / {
root html;
}
}
}
I would appreciate any help.
Posted at Nginx Forum:
https://forum.nginx.org/read.php?2,294477,295200#msg-295200
_______________________________________________
nginx mailing list -- nginx@nginx.org
To unsubscribe send an email to nginx-le...@nginx.org
