Re: [ceph-users] Corruption by missing blocks

2013-04-23 Thread Bryan Stillwell
On Tue, Apr 23, 2013 at 5:54 PM, Gregory Farnum wrote: > On Tue, Apr 23, 2013 at 4:45 PM, Sage Weil wrote: >> On Tue, 23 Apr 2013, Bryan Stillwell wrote: >>> On Tue, Apr 23, 2013 at 5:24 PM, Sage Weil wrote: >>> > >>> > On Tue, 23 Apr 2013, Bryan Stillwell wrote: >>> > > I'm testing this now, bu

Re: [ceph-users] Corruption by missing blocks

2013-04-23 Thread Bryan Stillwell
On Tue, Apr 23, 2013 at 5:45 PM, Sage Weil wrote: > On Tue, 23 Apr 2013, Bryan Stillwell wrote: >> On Tue, Apr 23, 2013 at 5:24 PM, Sage Weil wrote: >> > >> > On Tue, 23 Apr 2013, Bryan Stillwell wrote: >> > > I'm testing this now, but while going through the logs I saw something >> > > that migh

Re: [ceph-users] Corruption by missing blocks

2013-04-23 Thread Gregory Farnum
On Tue, Apr 23, 2013 at 4:45 PM, Sage Weil wrote: > On Tue, 23 Apr 2013, Bryan Stillwell wrote: >> On Tue, Apr 23, 2013 at 5:24 PM, Sage Weil wrote: >> > >> > On Tue, 23 Apr 2013, Bryan Stillwell wrote: >> > > I'm testing this now, but while going through the logs I saw something >> > > that migh

Re: [ceph-users] Corruption by missing blocks

2013-04-23 Thread Sage Weil
On Tue, 23 Apr 2013, Bryan Stillwell wrote: > On Tue, Apr 23, 2013 at 5:24 PM, Sage Weil wrote: > > > > On Tue, 23 Apr 2013, Bryan Stillwell wrote: > > > I'm testing this now, but while going through the logs I saw something > > > that might have something to do with this: > > > > > > Apr 23 16:35

Re: [ceph-users] Corruption by missing blocks

2013-04-23 Thread Bryan Stillwell
On Tue, Apr 23, 2013 at 5:24 PM, Sage Weil wrote: > > On Tue, 23 Apr 2013, Bryan Stillwell wrote: > > I'm testing this now, but while going through the logs I saw something > > that might have something to do with this: > > > > Apr 23 16:35:28 a1 kernel: [692455.496594] libceph: corrupt inc osdmap

Re: [ceph-users] Corruption by missing blocks

2013-04-23 Thread Sage Weil
On Tue, 23 Apr 2013, Bryan Stillwell wrote: > I'm testing this now, but while going through the logs I saw something > that might have something to do with this: > > Apr 23 16:35:28 a1 kernel: [692455.496594] libceph: corrupt inc osdmap > epoch 22146 off 102 (88021e0dc802 of > 88021e0dc79c

Re: [ceph-users] Corruption by missing blocks

2013-04-23 Thread Bryan Stillwell
I'm testing this now, but while going through the logs I saw something that might have something to do with this: Apr 23 16:35:28 a1 kernel: [692455.496594] libceph: corrupt inc osdmap epoch 22146 off 102 (88021e0dc802 of 88021e0dc79c-88021e0dc802) Apr 23 16:35:28 a1 kernel: [692455.50

Re: [ceph-users] Corruption by missing blocks

2013-04-23 Thread Gregory Farnum
On Tue, Apr 23, 2013 at 3:37 PM, Bryan Stillwell wrote: > I'm using the kernel client that's built into precise & quantal. > > I could give the ceph-fuse client a try and see if it has the same > issue. I haven't used it before, so I'll have to do some reading > first. If you've got the time tha

Re: [ceph-users] Corruption by missing blocks

2013-04-23 Thread Bryan Stillwell
I'm using the kernel client that's built into precise & quantal. I could give the ceph-fuse client a try and see if it has the same issue. I haven't used it before, so I'll have to do some reading first. Bryan On Tue, Apr 23, 2013 at 4:04 PM, Gregory Farnum wrote: > Sorry, I meant kernel clien

Re: [ceph-users] Corruption by missing blocks

2013-04-23 Thread Gregory Farnum
Sorry, I meant kernel client or ceph-fuse? Client logs would be enough to start with, I suppose — "debug client = 20" and "debug ms = 1" if using ceph-fuse; if using the kernel client things get tricker; I'd have to look at what logging is available without the debugfs stuff being enabled. :/ -Greg

Re: [ceph-users] Corruption by missing blocks

2013-04-23 Thread Bryan Stillwell
I've tried a few different ones: 1. cp to cephfs mounted filesystem on Ubuntu 12.10 (quantal) 2. rsync over ssh to cephfs mounted filesystem on Ubuntu 12.04.2 (precise) 3. scp to cephfs mounted filesystem on Ubuntu 12.04.2 (precise) It's fairly reproducible, so I can collect logs for you. Which

Re: [ceph-users] Corruption by missing blocks

2013-04-23 Thread Gregory Farnum
On Tue, Apr 23, 2013 at 11:38 AM, Bryan Stillwell wrote: > I've run into an issue where after copying a file to my cephfs cluster > the md5sums no longer match. I believe I've tracked it down to some > parts of the file which are missing: > > $ obj_name=$(cephfs "title1.mkv" show_location -l 0 |