Hi, i try to enable RGW access log for calculation byte_send, byte_reseved, operation_count, audit_detail, request_tracing and ... with set these options
ceph config set client.rgw.vanak rgw_enable_usage_log true ceph config set client.rgw rgw_enable_usage_log true ceph config set client.rgw.vanak rgw_ops_log_rados true RGW Beast access logs show the HAProxy node IP instead of the real client IP, despite X-Forwarded-For being correctly set by the cephadm ingress service. Root Cause Analysis: 1. Logging Mechanism: By design, the Beast frontend logs the TCP source address (Layer 4) rather than parsing the X-Forwarded-For HTTP header (Layer 7). This ensures audit logs reflect the trusted transport peer. 2. Missing PROXY Protocol: To preserve the real client IP at the transport layer, PROXY protocol must be used (send-proxy-v2 in HAProxy and proxy_protocol=true in RGW). 3. Orchestration Blocker: The current version of cephadm ingress (e.g., Reef) does not support the enable_proxy_protocol parameter in its service specification. Consequently, it cannot generate the required send-proxy-v2 directive in the managed HAProxy configuration. Technical Impact: - Observability: RGW logs are insufficient for client-level auditing, forensics, or per-IP bandwidth accounting. - Workaround Needed: Client attribution must currently be performed at the HAProxy log layer rather than the RGW log layer. Proposed Resolution / Solution Paths: 1. Upgrade: Verify and move to a Ceph version where cephadm ingress supports enable_proxy_protocol. 2. Manual Ingress: Replace the cephadm-managed ingress with a manual HAProxy deployment configured with send-proxy-v2 to satisfy RGW’s proxy_protocol=true requirement. 3. Upstream Change: Propose a feature update for cephadm to include first-class PROXY protocol support in the ingress spec to ensure parity with Beast’s logging architecture. Captured evidence: - tcpdump confirms X-Forwarded-For is present but no PROXY preamble is sent. - RGW is configured for proxy_protocol=true but receives plain HTTP streams. 07:09:44.247960 ens160 In IP 172.29.244.7.44332 > ceph-csb-02.8000: Flags [.], ack 1, win 502, options [nop,nop,TS val 735749891 ecr 4199462555], length 0 E..4..@.@.;..........,[email protected]@f... ....#...... +....N.. 07:09:44.248068 ens160 In IP 172.29.244.7.44332 > ceph-csb-02.8000: Flags [P.], seq 1:902, ack 1, win 502, options [nop,nop,TS val 735749891 ecr 4199462555], length 901 E.....@[email protected].........,[email protected]@f... .....G..... +....N..PUT /test/test2 HTTP/1.1 host: 172.29.244.8 accept-encoding: identity user-agent: aws-cli/2.31.18 md/awscrt#0.27.6 ua/2.1 os/linux#6.8.0-86-generic md/arch#x86_64 lang/python#3.13.7 md/pyimpl#CPython m/Z,E,n,b,W,N cfg/retry-mode#standard md/installer#exe md/distr ib#ubuntu.24 md/prompt#off md/command#s3api.put-object expect: 100-continue x-amz-checksum-crc64nvme: DxVUd/EiBAY= x-amz-sdk-checksum-algorithm: CRC64NVME x-amz-date: 20260510T070943Z x-amz-content-sha256: e951419ac0d10cc76f3e5eb415e68f7d7d6dd99e016d1463f0cb1ce18680d8e1 authorization: AWS4-HMAC-SHA256 Credential=MPQOYU9DYBV012QXHQ0H/20260510/tehran/s3/aws4_request, SignedHeaders=host;x-amz-checksum-crc64nvme;x-amz-content-sha256;x-amz-date;x-amz-sdk-checksum-a lgorithm, Signature=59da15f49a75490adf3e90829616416ed0abac757272146692c3a6149849e1c3 content-length: 52428800 x-forwarded-for: 172.29.244.19 connection: close 07:09:44.248088 ens160 Out IP ceph-csb-02.8000 > 172.29.244.7.44332: Flags [.], ack 902, win 503, options [nop,nop,TS val 4199462556 ecr 735749891], length 0 E..4I.@.@............@.,... .CC.....@p..... .N..+... 07:09:44.253519 ens160 Out IP ceph-csb-02.8000 > 172.29.244.7.44332: Flags [P.], seq 1:26, ack 902, win 503, options [nop,nop,TS val 4199462561 ecr 735749891], length 25 E..MI.@.@............@.,... .CC.....@...... .N..+...HTTP/1.1 100 CONTINUE 07:09:44.253912 ens160 In IP 172.29.244.7.44332 > ceph-csb-02.8000: Flags [.], ack 26, win 502, options [nop,nop,TS val 735749897 ecr 4199462561], length 0 E..4..@.@.;..........,[email protected]........... +.. .N.. 07:09:44.255059 ens160 In IP 172.29.244.7.44332 > ceph-csb-02.8000: Flags [P.], seq 902:8142, ack 26, win 502, options [nop,nop,TS val 735749898 ecr 4199462561], length 7240 E..|>.@[email protected].........,[email protected]....\...... +.. .N.._>.z..?....b.T}g1Wa^yc..wWq$...S.....(..0...yf''...`..O....P.....Y.H.....?#.:f...z..v....{w_fjx..a....v...>+.............J;...M.......e...OE.....#.. .h...U.UW.`...d.,..6...2e .i(.;.... + my client_ip is '172.29.244.19' but show in log haproxy ip '192.29.244.7' :| {"log":"debug 2026-05-09T11:09:27.398+0000 7fc897ac4640 1 beast: 0x7fc7b18d6310: 172.29.244.7 - farhad$root [09/May/2026:11:09:27.394 +0000] \"GET / HTTP/1.1\" 200 510 - \"aws-cli/1.22.34 Python/3.10.12 Linux/5.15.0-164-generic botocore/1.23.34\" - latency=0.004000204s\n","stream":"stdout","time":"2026-05-09T11:09:27.405683471Z"} how can i fix this problem ? anyone have solution? _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
