Re: [ceph-users] Civetweb log format

2018-03-13 Thread Aaron Bassett
Well I have it mostly wrapped up and writing to graylog, however the ops log has a `remote_addr` field, but as far as I can tell it's always blank. I found this fix but it seems to only be in v13.0.1 https://github.com/ceph/ceph/pull/16860 Is there any chance we'd see backports of this to Jewel

Re: [ceph-users] Civetweb log format

2018-03-12 Thread Aaron Bassett
Quick update: adding the following to your config: rgw log http headers = "http_authorization" rgw ops log socket path = /tmp/rgw rgw enable ops log = true rgw enable usage log = true and you can now nc -U /tmp/rgw |./jq --stream 'fromstream(1|truncate_stream(inputs))' { "time": "2018-03-12

Re: [ceph-users] Civetweb log format

2018-03-09 Thread Konstantin Shalygin
Unfortunately I can't quite figure out how to use it. I've got "rgw log http headers = "authorization" in my ceph.conf but I'm getting no love in the rgw log. I think this shold have 'http_' prefix, like: rgw log http headers = "http_host,http_x_forwarded_for" k ___

Re: [ceph-users] Civetweb log format

2018-03-09 Thread Aaron Bassett
David, that's exactly my goal as well. On closer reading of the docs, I see that this setting is to be used for writing these headers to the ops log. I guess it's time for me to learn what that's about. I've never quite been able to figure out how to get my hands on it. I also see an option for

Re: [ceph-users] Civetweb log format

2018-03-09 Thread David Turner
Matt, my only goal is to be able to have something that can be checked to see which key was used to access which resource. The closest I was able to get in Jewel was rgw debug logging 10/10, but it generates 100+ lines of logs for every request and as Aaron points out takes some logic to combine th

Re: [ceph-users] Civetweb log format

2018-03-09 Thread Aaron Bassett
Ah yes, I found it: https://github.com/ceph/ceph/commit/3192ef6a034bf39becead5f87a0e48651fcab705 Unfortunately I can't quite figure out how to use it. I've got "rgw log http headers = "authorization" in my ceph.conf but I'm getting no love in the rgw log. Also, setting rgw debug level to 10 d

Re: [ceph-users] Civetweb log format

2018-03-08 Thread Matt Benjamin
Hi Yehuda, I did add support for logging arbitrary headers, but not a configurable log record a-la webservers. To level set, David, are you speaking about a file or pipe log sync on the RGW host? Matt On Thu, Mar 8, 2018 at 7:55 PM, Yehuda Sadeh-Weinraub wrote: > On Thu, Mar 8, 2018 at 2:22 PM

Re: [ceph-users] Civetweb log format

2018-03-08 Thread Yehuda Sadeh-Weinraub
On Thu, Mar 8, 2018 at 2:22 PM, David Turner wrote: > I remember some time ago Yehuda had commented on a thread like this saying > that it would make sense to add a logging/auditing feature like this to RGW. > I haven't heard much about it since then, though. Yehuda, do you remember > that and/or

Re: [ceph-users] Civetweb log format

2018-03-08 Thread David Turner
I remember some time ago Yehuda had commented on a thread like this saying that it would make sense to add a logging/auditing feature like this to RGW. I haven't heard much about it since then, though. Yehuda, do you remember that and/or think that logging like this might become viable. On Thu,

Re: [ceph-users] Civetweb log format

2018-03-08 Thread Aaron Bassett
Yea thats what I was afraid of. I'm looking at possibly patching to add it, but i really dont want to support my own builds. I suppose other alternatives are to use proxies to log stuff, but that makes me sad. Aaron On Mar 8, 2018, at 12:36 PM, David Turner mailto:drakonst...@gmail.com>> wrote

Re: [ceph-users] Civetweb log format

2018-03-08 Thread David Turner
Setting radosgw debug logging to 10/10 is the only way I've been able to get the access key in the logs for requests. It's very unfortunate as it DRASTICALLY increases the amount of log per request, but it's what we needed to do to be able to have the access key in the logs along with the request.

[ceph-users] Civetweb log format

2018-03-06 Thread Aaron Bassett
Hey all, I'm trying to get something of an audit log out of radosgw. To that end I was wondering if theres a mechanism to customize the log format of civetweb. It's already writing IP, HTTP Verb, path, response and time, but I'm hoping to get it to print the Authorization header of the request,