Re: [Openstack] [Swift] Unexplained 404s

I was able to reproduce the issue with some manual intervention on the same 1 node setup. 1. Using swift-get-nodes, I found the exact order of nodes in which Swift was going to attempt to write an object. 2. Then I manually unmounted the primary and first handoff disk. 3. Then I wrote the object u

Re: [Openstack] [Swift] Unexplained 404s

On 25/05/16 15:10, Clay Gerrard wrote: At the risk of repeating myself: On Tue, May 24, 2016 at 5:30 PM, Clay Gerrard > wrote: This inconsistency in search depth based on the per-worker error limiting may be something worth looking into generally - but i

Re: [Openstack] [Swift] Unexplained 404s

At the risk of repeating myself: On Tue, May 24, 2016 at 5:30 PM, Clay Gerrard wrote: > > This inconsistency in search depth based on the per-worker error limiting > may be something worth looking into generally - but it's probably mostly > hidden on clusters that are going 3 or more nodes deep

Re: [Openstack] [Swift] Unexplained 404s

> > I think in a four device single node single replica setup I'd probably just > run request_node_count = 4 and call it a day. I'll give this a shot right away. But there are two questions that remain unanswered. 1. Why is there a discrepancy in the way writes vs reads are handled? Isn't reques

Re: [Openstack] [Swift] Unexplained 404s

On Tue, May 24, 2016 at 5:38 PM, Shrinand Javadekar wrote: > Here's my test setup: > > - Single node > - Single replica > - 4 disks: /srv/node/r1, r2, r3 and r4. > - Backed by SSDs > > I think in a four device single node single replica setup I'd probably just run request_node_count = 4 and call

Re: [Openstack] [Swift] Unexplained 404s

So on review of a few code paths I see that the default search depth (request_node_count) is `2 x ` which is smallish on single replica systems. There are some error limiting situations that might have allowed the write to go deeper than the expected 2 nodes (primary, and one handoff) which may no

Re: [Openstack] [Swift] Unexplained 404s

Here's my test setup: - Single node - Single replica - 4 disks: /srv/node/r1, r2, r3 and r4. - Backed by SSDs Unfortunately, I don't have the logs when the object was first written. But can definitely say that it returned 201. This is done using an application (not manually). We also logged that

Re: [Openstack] [Swift] Unexplained 404s

On Tue, May 24, 2016 at 4:51 PM, Shrinand Javadekar wrote: > Thanks for the detailed explanation... > Well, you're welcome - my apologies if it was overly verbose. > This is unlike what I've seen in this setup. I have some code that > tried to read the object 5 times from Swift with exponentia

Re: [Openstack] [Swift] Unexplained 404s

On Tue, May 24, 2016 at 4:56 PM, Shrinand Javadekar wrote: > Sorry... I missed the first question > > no worries > > Yes, I was running on a single replica system. Ah! That's great information. I have *zero* experience with single replica systems. The logs should be even *more* interesting

Re: [Openstack] [Swift] Unexplained 404s

Sorry... I missed the first question >> Are you running only a single replica!? Was the object data *only* on the >> second handoff?! If the original PUT request did not return success it's >> much more likely that you would have an unspecified behavior on the read >> path. Yes, I was runnin

Re: [Openstack] [Swift] Unexplained 404s

Thanks for the detailed explanation... >> >> >> 1. So when the replicator catches up, it will move the object back to >> the correct location. Is that right? > > > The read path will find the object on any primary or any handoff location. > The replicator *will* copy the data files to the primary

Re: [Openstack] [Swift] Unexplained 404s

On Tue, May 24, 2016 at 11:59 AM, Shrinand Javadekar < shrin...@maginatics.com> wrote: > > I found the object written into the second handoff node. > Are you running only a single replica!? Was the object data *only* on the second handoff?! If the original PUT request did not return success it'

Re: [Openstack] [Swift] Unexplained 404s

Thanks for the explanations everyone! I found the object written into the second handoff node. 1. So when the replicator catches up, it will move the object back to the correct location. Is that right? 2. Is there a way to disable handoffs? -Shri On Tue, May 24, 2016 at 3:20 AM, Mark Kirkwood

Re: [Openstack] [Swift] Unexplained 404s

On 24/05/16 11:20, Clay Gerrard wrote: On Mon, May 23, 2016 at 1:49 PM, Shrinand Javadekar mailto:shrin...@maginatics.com>> wrote: If objects are placed on different devices than the computed ones, they will be unavailable until the replication places them at the correct location

Re: [Openstack] [Swift] Unexplained 404s

On Mon, May 23, 2016 at 1:49 PM, Shrinand Javadekar wrote: > > If objects are placed on different devices than the computed ones, > they will be unavailable until the replication places them at the > correct location. This part doesn't sound quite right to me, but the transaction logs will tell

Re: [Openstack] [Swift] Unexplained 404s

Thanks John, Mark, I'll try these steps. If objects are placed on different devices than the computed ones, they will be unavailable until the replication places them at the correct location. And this could take a really long time. Is that right? -Shri On Fri, May 20, 2016 at 4:53 PM, Mark Kirk

Re: [Openstack] [Swift] Unexplained 404s

On 21/05/16 05:27, Shrinand Javadekar wrote: Hi, I am troubleshooting a test setup where Swift returned a 201 for objects that were put in it but later when I tried to read it, I got back 404s. The system has been under load. I see lots of connection errors, lock-timeouts, etc. However, I am no

Re: [Openstack] [Swift] Unexplained 404s

On 20 May 2016, at 10:27, Shrinand Javadekar wrote: > Hi, > > I am troubleshooting a test setup where Swift returned a 201 for > objects that were put in it but later when I tried to read it, I got > back 404s. > > The system has been under load. I see lots of connection errors, > lock-timeouts,

[Openstack] [Swift] Unexplained 404s

Hi, I am troubleshooting a test setup where Swift returned a 201 for objects that were put in it but later when I tried to read it, I got back 404s. The system has been under load. I see lots of connection errors, lock-timeouts, etc. However, I am not sure if ever Swift should be returning a 404.