[ceph-users] Federated gateways

Hello everyone, I am attempted to setup a two cluster situation for object storage disaster recovery. I have two physically separate sites so using 1 big cluster isn’t an option. I’m attempting to follow the guide at: http://ceph.com/docs/v0.80.5/radosgw/federated-config/

Re: [ceph-users] Federated gateways

fully those logs will tell > you what's wrong. > > > On Wed, Nov 5, 2014 at 11:39 AM, Aaron Bassett <mailto:aa...@five3genomics.com>> wrote: > Hello everyone, > I am attempted to setup a two cluster situation for object storage disaster > recovery. I ha

Re: [ceph-users] Federated gateways

16.10.103:0/1007381 done calling dispatch on 0x7f51b4001460 2014-11-11 14:37:06.701815 7f54447f0700 0 WARNING: set_req_state_err err_no=5 resorting to 500 2014-11-11 14:37:06.701894 7f54447f0700 1 == req done req=0x7f546800f3b0 http_status=500 == Any information you could give me

Re: [ceph-users] Federated gateways

earlier when I had mismatched keys. The .us-nh.rgw.buckets.index pool is syncing properly, as are the users. It seems like really the only thing that isn’t syncing is the .zone.rgw.buckets pool. Thanks, Aaron > > > > > On Tue, Nov 11, 2014 at 6:51 AM, Aaron Bassett <mai

Re: [ceph-users] Federated gateways

In playing around with this a bit more, I noticed that the two users on the secondary node cant see each others buckets. Is this a problem? > On Nov 11, 2014, at 6:56 PM, Craig Lewis wrote: > >> I see you're running 0.80.5. Are you using Apache 2.4? There is a known >> issue with Apache 2.4 o

Re: [ceph-users] Federated gateways

665.html> > > > IIRC, the system uses didn't see the other user's bucket in a bucket listing, > but they could read and write the objects fine. > > > > On Wed, Nov 12, 2014 at 11:16 AM, Aaron Bassett <mailto:aa...@five3genomics.com>> wrote: >

[ceph-users] Incomplete PGs

Hi all, I have a problem with some incomplete pgs. Here’s the backstory: I had a pool that I had accidently left with a size of 2. On one of the ods nodes, the system hdd started to fail and I attempted to rescue it by sacrificing one of my osd nodes. That went ok and I was able to bring the nod

Re: [ceph-users] Incomplete PGs

x27;48910307", "log_tail": "50495'48906592", The log tail seems to have lagged behind the last_update/last_complete. I suspect this is whats causing the cluster to reject these pgs. Anyone know how i can go about cleaning this up? Aaron > On Dec 1, 2014, at 8:12 PM, Aar

[ceph-users] Balancing erasure crush rule

I’m trying to set up an erasure coded pool with k=9 m=6 on 13 osd hosts. I’m trying to write a crush rule for this which will balance this between hosts as much as possible. I understand that having 9+6=15 > 13, I will need to parse the tree twice in order to find enough pgs. So what I’m trying

Re: [ceph-users] Balancing erasure crush rule

ing in multiple placements on 2 or 3 osds per pg. It turns out what I'm trying to do is described here: https://www.mail-archive.com/ceph-users%40lists.ceph.com/msg01076.html But I can't find any other references to anything like this. Thanks, Aaron > On Dec 23, 2014, at 9:23 AM, Aaron Bas

[ceph-users] RadosGW slow gc

I’m doing some load testing on radosgw to get ready for production and I had a problem with it stalling out. I had 100 cores from several nodes doing multipart uploads in parallel. This ran great for about two days, managing to upload about 2000 objects with an average size of 100GB. Then it sta

[ceph-users] RGW/Civet: Reads too much data when client doesn't close the connection

I have a situation where a client is GET'ing a large key (100GB) from RadosGW and just reading the first few bytes to determine if it's a gzip file or not, and then just moving on without closing the connection. I'm RadosGW then goes on to read the rest of the object out of the cluster, while se

Re: [ceph-users] RGW/Civet: Reads too much data when client doesn't close the connection

Yup already working on fixing the client, but it seems like a potentially nasty issue for RGW, as a malicious client could potentially DOS an endpoint pretty easily this way. Aaron > On Jul 12, 2017, at 11:48 AM, Jens Rosenboom wrote: > > 2017-07-12 15:23 GMT+00:00 Aaron Bassett : &g

[ceph-users] PG stuck inconsistent, but appears ok?

Good Morning, I have an odd situation where a pg is listed inconsistent, but rados is struggling to tell me about it: # ceph health detail HEALTH_ERR 1 pgs inconsistent; 1 requests are blocked > 32 sec; 1 osds have slow requests; 1 scrub errors pg 22.1611 is active+clean+inconsistent, acting [2

Re: [ceph-users] PG stuck inconsistent, but appears ok?

Ok good to hear, I just kicked one off on the acting primary so I guess I'll be patient now... Thanks, Aaron > On Jul 13, 2017, at 10:28 AM, Dan van der Ster wrote: > > On Thu, Jul 13, 2017 at 4:23 PM, Aaron Bassett > wrote: >> Because it was a read error I check SMA

Re: [ceph-users] PG stuck inconsistent, but appears ok?

) log [INF] : 21.1ae9 deep-scrub ok each time I run it, its the same pg. Is there some reason its not scrubbing all the pgs? Aaron > On Jul 13, 2017, at 10:29 AM, Aaron Bassett > wrote: > > Ok good to hear, I just kicked one off on the acting primary so I guess I'll >

Re: [ceph-users] PG stuck inconsistent, but appears ok?

2017-07-13 16:57:53.785261 7f40ce09a700 0 log_channel(cluster) log [INF] : 21.1ae9 deep-scrub ok each time I run it, its the same pg. Is there some reason its not scrubbing all the pgs? Aaron > On Jul 13, 2017, at 10:29 AM, Aaron Bassett > mailto:aaron.bass...@nantomics.com>> wr

[ceph-users] Stuck down+peering after host failure.

Morning All, I have a large-ish (16 node, 1100 osds) cluster I recent had to move from one DC to another. Before shutting everything down, I set noout, norecover, and nobackfill, thinking this would help everything stand back up again. Upon installation at the new DC, one of the nodes refused to

[ceph-users] Civetweb log format

Hey all, I'm trying to get something of an audit log out of radosgw. To that end I was wondering if theres a mechanism to customize the log format of civetweb. It's already writing IP, HTTP Verb, path, response and time, but I'm hoping to get it to print the Authorization header of the request,

Re: [ceph-users] Civetweb log format

in the logs along with the request. On Tue, Mar 6, 2018 at 3:09 PM Aaron Bassett mailto:aaron.bass...@nantomics.com>> wrote: Hey all, I'm trying to get something of an audit log out of radosgw. To that end I was wondering if theres a mechanism to customize the log format of civetweb. It&#

Re: [ceph-users] Civetweb log format

read like this saying that it would make sense to add a logging/auditing feature like this to RGW. I haven't heard much about it since then, though. Yehuda, do you remember that and/or think that logging like this might become viable. I vaguely remember Matt was working on this. Matt? Yehud

Re: [ceph-users] Civetweb log format

t and as Aaron points out takes some logic to combine the object, the key, and the action as well that it doesn't actually catch every type of request. It sounds like you've done some work with this. How can we utilize what you've done to be able to have audit logging on bucket

Re: [ceph-users] Civetweb log format

Quick update: adding the following to your config: rgw log http headers = "http_authorization" rgw ops log socket path = /tmp/rgw rgw enable ops log = true rgw enable usage log = true and you can now nc -U /tmp/rgw |./jq --stream 'fromstream(1|truncate_stream(inputs))' { "time": "2018-03-12

Re: [ceph-users] Civetweb log format

of this to Jewel and/or luminous? Aaron On Mar 12, 2018, at 5:50 PM, Aaron Bassett mailto:aaron.bass...@nantomics.com>> wrote: Quick update: adding the following to your config: rgw log http headers = "http_authorization" rgw ops log socket path = /tmp/rgw rgw enable ops log =

Re: [ceph-users] Ceph with Clos IP fabric

FWIW, I use a CLOS fabric with layer 3 right down to the hosts and multiple ToRs to enable HA/ECMP to each node. I'm using Cumulus Linux's "redistribute neighbor" feature, which advertises a /32 for any ARP'ed neighbor. I set up the hosts with an IP on each physical interface and on an aliased l

Re: [ceph-users] Ceph with Clos IP fabric

off is. Also, why are you putting your "cluster network" on the same physical interfaces but on separate VLANs? Traffic shaping/policing? What's your link speed there on the hosts? (25/40gbps?) On Sat, Apr 22, 2017 at 12:13 PM, Aaron Bassett mailto:aaron.bass...@nantomics.com>>

Re: [ceph-users] Ceph with Clos IP fabric

ical > interfaces but on separate VLANs? Traffic shaping/policing? What's your > link speed there on the hosts? (25/40gbps?) > > On Sat, Apr 22, 2017 at 12:13 PM, Aaron Bassett > mailto:aaron.bass...@nantomics.com> > > wrote: > > > FWIW, I use a CLOS fabric w

[ceph-users] RadosGW ops log lag?

I have an radogw log centralizer that we use to for an audit trail for data access in our ceph clusters. We've enabled the ops log socket and added logging of the http_authorization header to it: rgw log http headers = "http_authorization" rgw ops log socket path = /var/run/ceph/rgw-ops.sock rgw

Re: [ceph-users] RadosGW ops log lag?

ard to get it to start lagging. Thanks, Aaron > On Apr 12, 2019, at 11:16 AM, Matt Benjamin wrote: > > Hi Aaron, > > I don't think that exists currently. > > Matt > > On Fri, Apr 12, 2019 at 11:12 AM Aaron Bassett > wrote: >> >> I have an rad

[ceph-users] msgr2 and cephfs

Hi, I'm standing up a new cluster on nautilus to play with some of the new features, and I've somehow got my monitors only listening on msgrv2 port (3300) and not the legacy port (6789). I'm running kernel 4.15 on my clients. Can I mount cephfs via port 3300 or do I have to figure out how to get

Re: [ceph-users] msgr2 and cephfs

upport msgr2 yet. > > On Wed, Apr 24, 2019 at 4:19 PM Aaron Bassett > wrote: >> >> Hi, >> I'm standing up a new cluster on nautilus to play with some of the new >> features, and I've somehow got my monitors only listening on msgrv2 port >> (3300) an

Re: [ceph-users] msgr2 and cephfs

Ah nevermind, I found ceph mon set addrs and I'm good to go. Aaron > On Apr 24, 2019, at 4:36 PM, Aaron Bassett > wrote: > > Yea ok thats what I guessed. I'm struggling to get my mons to listen on both > ports. On startup they report: > > 2019-04-24 19:58

[ceph-users] radosgw index all keys in all buckets

Hello, I'm trying to write a tool to index all keys in all buckets stored in radosgw. I've created a user with the following caps: "caps": [ { "type": "buckets", "perm": "read" }, { "type": "metadata", "perm": "read"

Re: [ceph-users] [Ceph-large] Large Omap Warning on Log pool

as a bug prior to 12.2.12 that > prevented datalog trimming > (https://urldefense.proofpoint.com/v2/url?u=http-3A__tracker.ceph.com_issues_38412&d=DwICAg&c=Tpa2GKmmYSmpYS4baANxQwQYqA0vwGXwkJOPBegaiTs&r=5nKer5huNDFQXjYpOR4o_7t5CRI8wb5Vb_v1pBywbYw&m=v4DUT5hhECo7oEd5wRUGTpZor7RdHML6WBqg4ShUkD4&