It should not be best effort. As written, exactly rgw_usage_log_flush_threshold outstanding log entries will be buffered. The default value for this parameter is 1024, which is probably not high for a sustained workload, but you could experiment with reducing it.
Matt On Fri, Apr 12, 2019 at 11:21 AM Aaron Bassett <aaron.bass...@nantomics.com> wrote: > > Ok thanks. Is the expectation that events will be available on that socket as > soon as the occur or is it more of a best effort situation? I'm just trying > to nail down which side of the socket might be lagging. It's pretty difficult > to recreate this as I have to hit the cluster very hard to get it to start > lagging. > > Thanks, Aaron > > > On Apr 12, 2019, at 11:16 AM, Matt Benjamin <mbenj...@redhat.com> wrote: > > > > Hi Aaron, > > > > I don't think that exists currently. > > > > Matt > > > > On Fri, Apr 12, 2019 at 11:12 AM Aaron Bassett > > <aaron.bass...@nantomics.com> wrote: > >> > >> I have an radogw log centralizer that we use to for an audit trail for > >> data access in our ceph clusters. We've enabled the ops log socket and > >> added logging of the http_authorization header to it: > >> > >> rgw log http headers = "http_authorization" > >> rgw ops log socket path = /var/run/ceph/rgw-ops.sock > >> rgw enable ops log = true > >> > >> We have a daemon that listens on the ops socket, extracts/manipulates some > >> information from the ops log, and sends it off to our log aggregator. > >> > >> This setup works pretty well for the most part, except when the cluster > >> comes under heavy load, it can get _very_ laggy - sometimes up to several > >> hours behind. I'm having a hard time nailing down whats causing this lag. > >> The daemon is rather naive, basically just some nc with jq in between, but > >> the log aggregator has plenty of spare capacity, so I don't think its > >> slowing down how fast the daemon is consuming from the socket. > >> > >> I was revisiting the documentation about this ops log and noticed the > >> following which I hadn't seen previously: > >> > >> When specifying a UNIX domain socket, it is also possible to specify the > >> maximum amount of memory that will be used to keep the data backlog: > >> rgw ops log data backlog = <size in bytes> > >> Any backlogged data in excess to the specified size will be lost, so the > >> socket needs to be read constantly. > >> > >> I'm wondering if theres a way I can query radosgw for the current size of > >> that backlog to help me narrow down where the bottleneck may be occuring. > >> > >> Thanks, > >> Aaron > >> > >> > >> > >> CONFIDENTIALITY NOTICE > >> This e-mail message and any attachments are only for the use of the > >> intended recipient and may contain information that is privileged, > >> confidential or exempt from disclosure under applicable law. If you are > >> not the intended recipient, any disclosure, distribution or other use of > >> this e-mail message or attachments is prohibited. If you have received > >> this e-mail message in error, please delete and notify the sender > >> immediately. Thank you. > >> > >> _______________________________________________ > >> ceph-users mailing list > >> ceph-users@lists.ceph.com > >> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.ceph.com_listinfo.cgi_ceph-2Dusers-2Dceph.com&d=DwIFaQ&c=Tpa2GKmmYSmpYS4baANxQwQYqA0vwGXwkJOPBegaiTs&r=5nKer5huNDFQXjYpOR4o_7t5CRI8wb5Vb_v1pBywbYw&m=sIK_aBR3PrR2olfXOZWgvPVm7jIoZtvEk2YHofl4TDU&s=FzFoCJ8qtZ66OKdL1Ph10qjZbCEjvMg9JyS_9LwEpSg&e= > >> > >> > > > > > > -- > > > > Matt Benjamin > > Red Hat, Inc. > > 315 West Huron Street, Suite 140A > > Ann Arbor, Michigan 48103 > > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.redhat.com_en_technologies_storage&d=DwIFaQ&c=Tpa2GKmmYSmpYS4baANxQwQYqA0vwGXwkJOPBegaiTs&r=5nKer5huNDFQXjYpOR4o_7t5CRI8wb5Vb_v1pBywbYw&m=sIK_aBR3PrR2olfXOZWgvPVm7jIoZtvEk2YHofl4TDU&s=hi6_HiZS0D_nzAqKsvJPPfmi8nZSv4lZCRFZ1ru9CxM&e= > > > > tel. 734-821-5101 > > fax. 734-769-8938 > > cel. 734-216-5309 > > -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
