Re: [ceph-users] RGW Replication

2014-03-18 Thread Craig Lewis
For the record, I have one bucket in my slave zone that caught up to the master zone. I stopped adding new data to my first bucket, and replication stopped. I started tickling the bucket by uploading and deleting a 0 byte file every 5 minutes. Now the slave has all of the files in that bucke

Re: [ceph-users] RGW Replication

2014-02-07 Thread Craig Lewis
I have confirmed this in production, with the default max-entries. I have a bucket that I'm no longer writing to. Radosgw-agent had stopped replicating this bucket. radosgw-admin bucket stats shows that the slave is missing ~600k objects. I uploaded a 1 byte file to the bucket. On the nex

Re: [ceph-users] RGW Replication

2014-02-06 Thread Craig Lewis
On 2/4/14 17:06 , Craig Lewis wrote: Now that I've started seeing missing objects, I'm not able to download objects that should be on the slave if replication is up to date. Either it's not up to date, or it's skipping objects every pass. Using my --max-entries fix (https://github.com/ce

Re: [ceph-users] RGW Replication

2014-02-05 Thread Craig Lewis
*Craig Lewis* Senior Systems Engineer Office +1.714.602.1309 Email cle...@centraldesktop.com *Central Desktop. Work together in ways you never thought possible.* Connect with us Website | Twitter

Re: [ceph-users] RGW Replication

2014-02-05 Thread Yehuda Sadeh
On Wed, Feb 5, 2014 at 2:21 PM, Josh Durgin wrote: > On 02/05/2014 01:23 PM, Craig Lewis wrote: >> >> >> On 2/4/14 20:02 , Josh Durgin wrote: >>> >>> >>> From the log it looks like you're hitting the default maximum number of >>> entries to be processed at once per shard. This was intended to prev

Re: [ceph-users] RGW Replication

2014-02-05 Thread Josh Durgin
On 02/05/2014 01:23 PM, Craig Lewis wrote: On 2/4/14 20:02 , Josh Durgin wrote: From the log it looks like you're hitting the default maximum number of entries to be processed at once per shard. This was intended to prevent one really busy shard from blocking progress on syncing other shards,

Re: [ceph-users] RGW Replication

2014-02-05 Thread Craig Lewis
On 2/4/14 20:02 , Josh Durgin wrote: From the log it looks like you're hitting the default maximum number of entries to be processed at once per shard. This was intended to prevent one really busy shard from blocking progress on syncing other shards, since the remainder will be synced the next

Re: [ceph-users] RGW Replication

2014-02-04 Thread Josh Durgin
On 02/04/2014 07:44 PM, Craig Lewis wrote: On 2/4/14 17:06 , Craig Lewis wrote: On 2/4/14 14:43 , Yehuda Sadeh wrote: Does it ever catching up? You mentioned before that most of the writes went to the same two buckets, so that's probably one of them. Note that writes to the same bucket are b

Re: [ceph-users] RGW Replication

2014-02-04 Thread Craig Lewis
On 2/4/14 17:06 , Craig Lewis wrote: On 2/4/14 14:43 , Yehuda Sadeh wrote: Does it ever catching up? You mentioned before that most of the writes went to the same two buckets, so that's probably one of them. Note that writes to the same bucket are being handled in-order by the agent. Yehuda

Re: [ceph-users] RGW Replication

2014-02-04 Thread Yehuda Sadeh
On Tue, Feb 4, 2014 at 5:06 PM, Craig Lewis wrote: > > > On 2/4/14 14:43 , Yehuda Sadeh wrote: > > Now that objects are missing in the slave, how do I fix it? radosgw-agent > --sync-scope=full ? > > That would do it, yes. > > > I'm hesitant to do this, at least until I understand what's going on

Re: [ceph-users] RGW Replication

2014-02-04 Thread Yehuda Sadeh
On Tue, Feb 4, 2014 at 2:21 PM, Craig Lewis wrote: > > Craig Lewis > Senior Systems Engineer > Office +1.714.602.1309 > Email cle...@centraldesktop.com > > Central Desktop. Work together in ways you never thought possible. > Connect with us Website | Twitter | Facebook | LinkedIn | Blog

Re: [ceph-users] RGW Replication

2014-02-04 Thread Craig Lewis
*Craig Lewis* Senior Systems Engineer Office +1.714.602.1309 Email cle...@centraldesktop.com *Central Desktop. Work together in ways you never thought possible.* Connect with us Website | Twitter

Re: [ceph-users] RGW Replication

2014-02-04 Thread Yehuda Sadeh
On Tue, Feb 4, 2014 at 10:07 AM, Craig Lewis wrote: > > > > On 2/3/14 14:34 , Craig Lewis wrote: > > > On 2/3/14 10:51 , Gregory Farnum wrote: > > On Mon, Feb 3, 2014 at 10:43 AM, Craig Lewis > wrote: > > I've been noticing somethings strange with my RGW federation. I added some > statistics to

Re: [ceph-users] RGW Replication

2014-02-03 Thread Craig Lewis
On 2/3/14 10:51 , Gregory Farnum wrote: On Mon, Feb 3, 2014 at 10:43 AM, Craig Lewis wrote: I've been noticing somethings strange with my RGW federation. I added some statistics to radosgw-agent to try and get some insight (https://github.com/ceph/radosgw-agent/pull/7), but that just showed

Re: [ceph-users] RGW Replication

2014-02-03 Thread Gregory Farnum
On Mon, Feb 3, 2014 at 10:43 AM, Craig Lewis wrote: > I've been noticing somethings strange with my RGW federation. I added some > statistics to radosgw-agent to try and get some insight > (https://github.com/ceph/radosgw-agent/pull/7), but that just showed me that > I don't understand how replic

[ceph-users] RGW Replication

2014-02-03 Thread Craig Lewis
I've been noticing somethings strange with my RGW federation. I added some statistics to radosgw-agent to try and get some insight (https://github.com/ceph/radosgw-agent/pull/7), but that just showed me that I don't understand how replication works. When PUT traffic was relatively slow to the