Re: [Openstack] Troubleshooting Swift 1.7.4 on mini servers

2012-10-30 Thread Nathan Trueblood
Ok, if I was giving out t-shirts for finding this issue then the prize would go to Pete. Thank you Disabling fallocate did the trick. I was slowly working my way through all the object-server config options and hadn't gotten to that one yet. Turning features on and off by brute force is ad

Re: [Openstack] Troubleshooting Swift 1.7.4 on mini servers

2012-10-30 Thread Pete Zaitcev
On Tue, 30 Oct 2012 11:07:55 -0700 Nathan Trueblood wrote: > The file size seems to have no bearing on the issue, although I haven't > tried really tiny files. Bigfile3 is only 200K. Okay. BTW, do not forget to use curl and issue the same PUT that proxy does, see if it throws 507 repeateably.

Re: [Openstack] Troubleshooting Swift 1.7.4 on mini servers

2012-10-30 Thread Nathan Trueblood
The filesystem is XFS, and I used the recommended mkfs and mount options for Swift. The file size seems to have no bearing on the issue, although I haven't tried really tiny files. Bigfile3 is only 200K. I'll try disabling fallocate... On Mon, Oct 29, 2012 at 7:37 PM, Pete Zaitcev wrote: > O

Re: [Openstack] Troubleshooting Swift 1.7.4 on mini servers

2012-10-30 Thread Nathan Trueblood
No disk errors in the kern.log. The filesystem is fine. I really think this will turn out to be a bug or a timing (slowness) issue. I will try some of the other recent suggestions, and failing those try to track this down with strace. Thx. On Mon, Oct 29, 2012 at 7:02 PM, Alex Yang wrote:

Re: [Openstack] Troubleshooting Swift 1.7.4 on mini servers

2012-10-30 Thread Rick Jones
On 10/29/2012 07:37 PM, Pete Zaitcev wrote: On Mon, 29 Oct 2012 18:16:52 -0700 Nathan Trueblood wrote: Definitely NOT a problem with the filesystem, but something is causing the object-server to think there is a problem with the filesystem. If you are willing to go all-out, you can probably

Re: [Openstack] Troubleshooting Swift 1.7.4 on mini servers

2012-10-29 Thread Pete Zaitcev
On Mon, 29 Oct 2012 18:16:52 -0700 Nathan Trueblood wrote: > Definitely NOT a problem with the filesystem, but something is causing the > object-server to think there is a problem with the filesystem. If you are willing to go all-out, you can probably catch the error with strace, if it works on

Re: [Openstack] Troubleshooting Swift 1.7.4 on mini servers

2012-10-29 Thread Alex Yang
There are any error about disk in the kern.log? 2012/10/30 Nathan Trueblood > Still no further clues. I re-created all the volumes I'm using for > Swift. Plenty of Inodes free: > > lab@data02:~$ df -i > FilesystemInodes IUsed IFree IUse% Mounted on > /dev/sda2 12214272 39290

Re: [Openstack] Troubleshooting Swift 1.7.4 on mini servers

2012-10-29 Thread Nathan Trueblood
Still no further clues. I re-created all the volumes I'm using for Swift. Plenty of Inodes free: lab@data02:~$ df -i FilesystemInodes IUsed IFree IUse% Mounted on /dev/sda2 12214272 39290 121749821% / none 107979 4821074971% /dev none 1

Re: [Openstack] Troubleshooting Swift 1.7.4 on mini servers

2012-10-29 Thread John Dickinson
Also check the number of inodes used: `df -i` --John On Oct 29, 2012, at 8:31 AM, Nathan Trueblood wrote: > Yeah, I read about the 507 error.However, when the error occurs on my I > can see with 'df' that the drive is only 1% full and is definitely not > unmounted. I can write files t

Re: [Openstack] Troubleshooting Swift 1.7.4 on mini servers

2012-10-29 Thread Nathan Trueblood
Yeah, I read about the 507 error.However, when the error occurs on my I can see with 'df' that the drive is only 1% full and is definitely not unmounted. I can write files to the mounted filesystem directly before, during, and after the Swift error occurs. So the problem must be some kind o

Re: [Openstack] Troubleshooting Swift 1.7.4 on mini servers

2012-10-27 Thread Diego Parrilla
Sorry for my off topic question, it's the first time I heard of swift storage servers running an ARM processor. I think ARM architectures can be an interesting alternative to replace classic computing servers, but swift storage nodes did not catch my attention as a valid alternative, until now

Re: [Openstack] Troubleshooting Swift 1.7.4 on mini servers

2012-10-26 Thread John Dickinson
A 507 is returned by the object servers in 2 situations: 1) the drives are full or 2) the drives have been unmounted because of disk error. It's highly likely that you simply have full drives. Remember that the usable space in your cluster is 1/N where N = replica count. As an example, with 3 r

Re: [Openstack] Troubleshooting Swift 1.7.4 on mini servers

2012-10-26 Thread Pete Zaitcev
On Fri, 26 Oct 2012 17:26:07 -0700 Nathan Trueblood wrote: > I'm trying to figure out what's going wrong with my Swift deployment on a > small cluster of "mini" servers. I have a small test cluster (5 storage > nodes, 1 proxy) of mini-servers that are ARM-based. The proxy is a > regular, Inte

[Openstack] Troubleshooting Swift 1.7.4 on mini servers

2012-10-26 Thread Nathan Trueblood
Hey folks- I'm trying to figure out what's going wrong with my Swift deployment on a small cluster of "mini" servers. I have a small test cluster (5 storage nodes, 1 proxy) of mini-servers that are ARM-based. The proxy is a regular, Intel-based server with plenty of RAM. The object/account/c