Re: [ceph-users] Infernalis upgrade breaks when journal on separate partition

2016-01-15 Thread Michał Chybowski
In my case one server was also non-GPT installed and in /usr/sbin/ceph-disk I've added line: os.chmod(os.path.join(path,'journal'), 0777) after line 1926. I know that it's very ugly and shouldn't be made on production, but I had no time to search for proper way to fix this. Regards Michał Chy

[ceph-users] cephfs triggers warnings "tar: file changed as we read it"

2016-01-15 Thread HMLTH
Hello, I'm evaluating cephfs on a virtual machines cluster. I'm using Infernalis (9.2.0) on debian Jessie as client and server. I'm trying to get some performance numbers on operations like tar/untar on things like the linux kernel. I have an issue where tar displays this warning : 'file

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

2016-01-15 Thread Yehuda Sadeh-Weinraub
On Thu, Jan 14, 2016 at 10:51 PM, seapasu...@uchicago.edu wrote: > It looks like the gateway is experiencing a similar race condition to what > we reported before. > > The rados object has a size of 0 bytes but the bucket index shows the object > listed and the object metadata shows a size of > 71

Re: [ceph-users] Odd single VM ceph error

2016-01-15 Thread Sage Weil
On Thu, 14 Jan 2016, Robert LeBlanc wrote: > We have a single VM that is acting odd. We had 7 SSD OSDs (out of 40) go > down over a period of about 12 hours. These are a cache tier and have size > 4, min_size 2. I'm not able to make heads or tails of the error and hoped > someone here could help. >

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

2016-01-15 Thread seapasu...@uchicago.edu
Hello Yehuda, Here it is:: radosgw-admin object stat --bucket="noaa-nexrad-l2" --object="2015/01/01/PAKC/NWS_NEXRAD_NXL2DP_PAKC_2015010111_20150101115959.tar" { "name": "2015\/01\/01\/PAKC\/NWS_NEXRAD_NXL2DP_PAKC_2015010111_20150101115959.tar", "size": 7147520, "policy":

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

2016-01-15 Thread Yehuda Sadeh-Weinraub
On Fri, Jan 15, 2016 at 9:36 AM, seapasu...@uchicago.edu wrote: > Hello Yehuda, > > Here it is:: > > radosgw-admin object stat --bucket="noaa-nexrad-l2" > --object="2015/01/01/PAKC/NWS_NEXRAD_NXL2DP_PAKC_2015010111_20150101115959.tar" > { > "name": > "2015\/01\/01\/PAKC\/NWS_NEXRAD_NXL2DP_

Re: [ceph-users] Odd single VM ceph error

2016-01-15 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Exactly the problem. Thanks for the point in the right direction. - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Fri, Jan 15, 2016 at 10:28 AM, Sage Weil wrote: > On Thu, 14 Jan 2016, Robert

Re: [ceph-users] Infernalis upgrade breaks when journal on separate partition

2016-01-15 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 If you are not booting from the GPT disk, you don't need the EFI partition (or any special boot partition). The required backup FAT is usually put at the end where there is usually some free space anyway. It has been a long time since I've converted

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

2016-01-15 Thread seapasu...@uchicago.edu
lacadmin@kh28-10:~$ rados -p .rgw.buckets ls | grep 'pcu5Hz6' lacadmin@kh28-10:~$ Nothing was found. That said when I run the command with another prefix snippet:: lacadmin@kh28-10:~$ rados -p .rgw.buckets ls | grep 'wksHvto' default.384153.1__shadow_2015/01/01/KABR/NWS_NEXRAD_NXL2DP_KABR_20150

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

2016-01-15 Thread Yehuda Sadeh-Weinraub
That's interesting, and might point at the underlying issue that caused it. Could be a racing upload that somehow ended up with the wrong object head. The 'multipart' object should be 4M in size, and the 'shadow' one should have the remainder of the data. You can run 'rados stat -p .rgw.buckets ' t

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

2016-01-15 Thread seapasu...@uchicago.edu
Sorry I am a bit confused. The successful list that I provided is from a different object of the same size to show that I could indeed get a list. Are you saying to copy the working object to the missing object? Sorry for the confusion. On 1/15/16 3:20 PM, Yehuda Sadeh-Weinraub wrote: That's

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

2016-01-15 Thread Yehuda Sadeh-Weinraub
Ah, I see. Misread that and the object names were very similar. No, don't copy it. You can try to grep for the specific object name and see if there are pieces of it lying around under a different upload id. Yehuda On Fri, Jan 15, 2016 at 1:44 PM, seapasu...@uchicago.edu wrote: > Sorry I am a bi

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

2016-01-15 Thread seapasu...@uchicago.edu
Sorry for the confusion:: When I grepped for the prefix of the missing object:: "2015\/01\/01\/PAKC\/NWS_NEXRAD_NXL2DP_PAKC_2015010111_20150101115959.tar.2~pcu5Hz6foFXjlSxBat22D8YMcHlQOBD" I am not able to find any chunks of the object:: lacadmin@kh28-10:~$ rados -p .rgw.buckets ls | grep '

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

2016-01-15 Thread Yehuda Sadeh-Weinraub
The head object of a multipart object has 0 size, so it's expected. What's missing is the tail of the object. I don't assume you have any logs from when the object was uploaded? Yehuda On Fri, Jan 15, 2016 at 2:12 PM, seapasu...@uchicago.edu wrote: > Sorry for the confusion:: > > When I grepped

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

2016-01-15 Thread seapasu...@uchicago.edu
I have looked all over and I do not see any explicit mention of "NWS_NEXRAD_NXL2DP_PAKC_2015010111_20150101115959" in the logs nor do I see a timestamp from November 4th although I do see log rotations dating back to october 15th. I don't think it's possible it wasn't logged so I am going t

[ceph-users] OSDs are down, don't know why

2016-01-15 Thread Jeff Epstein
Hello, I'm setting up a small test instance of ceph and I'm running into a situation where the OSDs are being shown as down, but I don't know why. Connectivity seems to be working. The OSD hosts are able to communicate with the MON hosts; running "ceph status" and "ceph osd in" from an OSD h

Re: [ceph-users] Inconsistent PG / Impossible deep-scrub

2016-01-15 Thread Jérôme Poulin
Finally, after having corruption with the MDS I had no choice but to try to manually repair the PG. Following the procedure on the blog post from Ceph at http://ceph.com/planet/ceph-manually-repair-object/ I was able to get the PG back to active+clean, ceph pg repair wasn't still not working and a