Re: [ceph-users] Ceph and my use case - is it a fit?
Thanks John, I really mean my files are too small for HDFS, as the majority of them will be under 64M, which I think is (still?) the default HDFS block size, *and also,* they will be very numerous. As such, they would quickly consume a huge aggregate amount of RAM on the HDFS name node, which is designed to store a certain amount of bytes per file. The name node in that sense it may seem, had been initially designed to "manage" for a collection of huge files, not a huge collection of small files. Or at least it may seem from documentation it's not optimized for that. A constructive approach may suggest I'd just have to allocate a large server instance for the HDFS name node, which may a first step on a path towards learning the next bottleneck using HDFS for such files, the hard / long way. Yes, I am aware HDFS has some special dedicated API for handling small files, and some community wrappers for managing with small files, but they seem a bit hackish, or feel like "too many moving parts" for a simple scenario. What do you think, and what do you think about Ceph for this scenario? Thanks in advance! Matan On Thu, Jul 31, 2014 at 7:20 PM, John Spray wrote: > On Wed, Jul 30, 2014 at 5:08 PM, Matan Safriel > wrote: > > I'm looking for a distributed file system, for large JSON documents. My > file > > sizes are roughly between 20M and 100M, so they are too small for > couchbase, > > mongodb, even possibly Riak, but too small (by an order of magnitude) for > > HDFS. Would you recommend Ceph for this kind of scenario? > > When you say they're too small for HDFS, do you really mean they're > too numerous? How many are we talking about? > > If your use case calls for just puts and gets of named serialized > blobs, you may be best off with the RGW or librados object store > interfaces to Ceph, rather than the file system per se. > > > Additional question - will it also install and behave gracefully as a > > single-node cluster running on a single linux machine, in a dev scenario > > and/or a unit test machine scenario? > > Yes, that's how some of the ceph tests themselves operate. > > Cheers, > John > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] 0.80.5-1precise Not Able to Map RBD & CephFS
On Sat, Aug 2, 2014 at 1:41 AM, Christopher O'Connell wrote: > So I've been having a seemingly similar problem and while trying to follow > the steps in this thread, things have gone very south for me. Show me where in this thread have I said to set tunables to optimal ;) optimal (== firefly for firefly) is actually the opposite of what you are going to need. > > Kernal on OSDs and MONs: 2.6.32-431.20.3.0.1.el6.centos.plus.x86_64 #1 SMP > Wed Jul 16 21:27:52 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux > > Kernal on RBD host: 3.2.0-61-generic #93-Ubuntu SMP Fri May 2 21:31:50 UTC > 2014 x86_64 x86_64 x86_64 GNU/Linux > > All are running 0.80.5 Is this a new firefly cluster or was it created before firely (specifically before v0.78) and then upgraded? > > I updated the tunables as per this article > http://cephnotes.ksperis.com/blog/2014/01/16/set-tunables-optimal-on-ceph-crushmap > > Here's what's happening: > > 1) On the rbd client node, trying to map rbd produces > $ sudo rbd --conf /etc/ceph/mia1.conf --keyring > /etc/ceph/mia1.client.admin.keyring map poolname > > rbd: add failed: (5) Input/output error > > Dmesg: > > [331172.147289] libceph: mon0 10.103.11.132:6789 feature set mismatch, my 2 > < server's 20042040002, missing 2004204 > [331172.154059] libceph: mon0 10.103.11.132:6789 missing required protocol > features > [331182.176604] libceph: mon1 10.103.11.141:6789 feature set mismatch, my 2 > < server's 20042040002, missing 2004204 > [331182.183535] libceph: mon1 10.103.11.141:6789 missing required protocol > features > [331192.192630] libceph: mon2 10.103.11.152:6789 feature set mismatch, my 2 > < server's 20042040002, missing 2004204 > [331192.199810] libceph: mon2 10.103.11.152:6789 missing required protocol > features > [331202.209324] libceph: mon0 10.103.11.132:6789 feature set mismatch, my 2 > < server's 20042040002, missing 2004204 > [331202.216957] libceph: mon0 10.103.11.132:6789 missing required protocol > features > [331212.224540] libceph: mon0 10.103.11.132:6789 feature set mismatch, my 2 > < server's 20042040002, missing 2004204 > [331212.232276] libceph: mon0 10.103.11.132:6789 missing required protocol > features > [331222.240605] libceph: mon2 10.103.11.152:6789 feature set mismatch, my 2 > < server's 20042040002, missing 2004204 > [331222.248660] libceph: mon2 10.103.11.152:6789 missing required protocol > features > > However, running > $ sudo rbd --conf /etc/ceph/mia1.conf --keyring > /etc/ceph/mia1.client.admin.keyring ls > poolname > > works fine and shows the expected pool name. > > 2) On the monitor where I ran the command to update the tunables, I can no > longer run the ceph console: > $ ceph -c /etc/ceph/mia1.conf --keyring /etc/ceph/mia1.client.admin.keyring > 2014-08-01 17:32:05.026960 7f21943d2700 0 -- 10.103.11.132:0/1030058 >> > 10.103.11.141:6789/0 pipe(0x7f2190028440 sd=3 :42360 s=1 pgs=0 cs=0 l=1 > c=0x7f21900286a0).connect protocol feature mismatch, my f < peer > 20f missing 200 > 2014-08-01 17:32:05.027024 7f21943d2700 0 -- 10.103.11.132:0/1030058 >> > 10.103.11.141:6789/0 pipe(0x7f2190028440 sd=3 :42360 s=1 pgs=0 cs=0 l=1 > c=0x7f21900286a0).fault > 2014-08-01 17:32:05.027544 7f21943d2700 0 -- 10.103.11.132:0/1030058 >> > 10.103.11.141:6789/0 pipe(0x7f2190028440 sd=3 :42361 s=1 pgs=0 cs=0 l=1 > c=0x7f21900286a0).connect protocol feature mismatch, my f < peer > 20f missing 200 > > and it just keeps spitting out a similar message. However I *can* run the > ceph console and execute basic commands (status, at the very least) from > other nodes. What does ceph -s from those other nodes say? Check versions of all monitors with ceph daemon mon. version > > At this point, I'm reluctant to continue without some advice from someone > else. I can certainly try upgrading the kernal on the rbd client, but I'm > worried I may just make things worse. Upgrading the kernel won't make things worse, it's just a client. I'm pretty sure we can make this work with 3.2, but if you actually plan on using krbd for anything serious, I'd recommend an upgrade to 3.14. 3.13 will do too, if you don't plan on having any erasure pools in your cluster. Thanks, Ilya ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] 0.80.5-1precise Not Able to Map RBD & CephFS
On Sat, Aug 2, 2014 at 10:03 PM, Christopher O'Connell wrote: > On Sat, Aug 2, 2014 at 6:27 AM, Ilya Dryomov > wrote: >> >> On Sat, Aug 2, 2014 at 1:41 AM, Christopher O'Connell >> wrote: >> > So I've been having a seemingly similar problem and while trying to >> > follow >> > the steps in this thread, things have gone very south for me. >> >> Show me where in this thread have I said to set tunables to optimal ;) >> optimal (== firefly for firefly) is actually the opposite of what you >> are going to need. > > > So what should tunables be set to? Optimal? Ordinarily yes, but not if you are going to use older kernels. In that case you'd want "default" or "legacy". > >> >> >> > >> > Kernal on OSDs and MONs: 2.6.32-431.20.3.0.1.el6.centos.plus.x86_64 #1 >> > SMP >> > Wed Jul 16 21:27:52 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux >> > >> > Kernal on RBD host: 3.2.0-61-generic #93-Ubuntu SMP Fri May 2 21:31:50 >> > UTC >> > 2014 x86_64 x86_64 x86_64 GNU/Linux >> > >> > All are running 0.80.5 >> >> Is this a new firefly cluster or was it created before firely >> (specifically before v0.78) and then upgraded? > > > It was created before 0.78 and upgraded. It has also been expanded several > times. > >> >> >> > >> > I updated the tunables as per this article >> > >> > http://cephnotes.ksperis.com/blog/2014/01/16/set-tunables-optimal-on-ceph-crushmap >> > >> > Here's what's happening: >> > >> > 1) On the rbd client node, trying to map rbd produces >> > $ sudo rbd --conf /etc/ceph/mia1.conf --keyring >> > /etc/ceph/mia1.client.admin.keyring map poolname >> > >> > rbd: add failed: (5) Input/output error >> > >> > Dmesg: >> > >> > [331172.147289] libceph: mon0 10.103.11.132:6789 feature set mismatch, >> > my 2 >> > < server's 20042040002, missing 2004204 >> > [331172.154059] libceph: mon0 10.103.11.132:6789 missing required >> > protocol >> > features >> > [331182.176604] libceph: mon1 10.103.11.141:6789 feature set mismatch, >> > my 2 >> > < server's 20042040002, missing 2004204 >> > [331182.183535] libceph: mon1 10.103.11.141:6789 missing required >> > protocol >> > features >> > [331192.192630] libceph: mon2 10.103.11.152:6789 feature set mismatch, >> > my 2 >> > < server's 20042040002, missing 2004204 >> > [331192.199810] libceph: mon2 10.103.11.152:6789 missing required >> > protocol >> > features >> > [331202.209324] libceph: mon0 10.103.11.132:6789 feature set mismatch, >> > my 2 >> > < server's 20042040002, missing 2004204 >> > [331202.216957] libceph: mon0 10.103.11.132:6789 missing required >> > protocol >> > features >> > [331212.224540] libceph: mon0 10.103.11.132:6789 feature set mismatch, >> > my 2 >> > < server's 20042040002, missing 2004204 >> > [331212.232276] libceph: mon0 10.103.11.132:6789 missing required >> > protocol >> > features >> > [331222.240605] libceph: mon2 10.103.11.152:6789 feature set mismatch, >> > my 2 >> > < server's 20042040002, missing 2004204 >> > [331222.248660] libceph: mon2 10.103.11.152:6789 missing required >> > protocol >> > features >> > >> > However, running >> > $ sudo rbd --conf /etc/ceph/mia1.conf --keyring >> > /etc/ceph/mia1.client.admin.keyring ls >> > poolname >> > >> > works fine and shows the expected pool name. >> > >> > 2) On the monitor where I ran the command to update the tunables, I can >> > no >> > longer run the ceph console: >> > $ ceph -c /etc/ceph/mia1.conf --keyring >> > /etc/ceph/mia1.client.admin.keyring >> > 2014-08-01 17:32:05.026960 7f21943d2700 0 -- 10.103.11.132:0/1030058 >> >> > 10.103.11.141:6789/0 pipe(0x7f2190028440 sd=3 :42360 s=1 pgs=0 cs=0 l=1 >> > c=0x7f21900286a0).connect protocol feature mismatch, my f < peer >> > 20f missing 200 >> > 2014-08-01 17:32:05.027024 7f21943d2700 0 -- 10.103.11.132:0/1030058 >> >> > 10.103.11.141:6789/0 pipe(0x7f2190028440 sd=3 :42360 s=1 pgs=0 cs=0 l=1 >> > c=0x7f21900286a0).fault >> > 2014-08-01 17:32:05.027544 7f21943d2700 0 -- 10.103.11.132:0/1030058 >> >> > 10.103.11.141:6789/0 pipe(0x7f2190028440 sd=3 :42361 s=1 pgs=0 cs=0 l=1 >> > c=0x7f21900286a0).connect protocol feature mismatch, my f < peer >> > 20f missing 200 >> > >> > and it just keeps spitting out a similar message. However I *can* run >> > the >> > ceph console and execute basic commands (status, at the very least) from >> > other nodes. >> >> What does ceph -s from those other nodes say? Check versions of all >> monitors with >> >> ceph daemon mon. version > > > So with some suggestions from people on IRC last night, it seems that > several nodes didn't get librados upgraded, but still had 0.72. I'm not > entirely sure how this happened, but I had to use yum-transaction to sort > out the fact that python-librados went away for 0.80, and it's quite > possible that I made a mistake and didn't upgrade these libraries. After > manually getting all of the libraries up to date the problems went away. > >> >> >> > >> > At this point, I'm reluctant to continue without s
Re: [ceph-users] 0.80.5-1precise Not Able to Map RBD & CephFS
Hi Ilya, Short of building a 3.14 kernel from scratch, are there any centos/EL kernels of 3.14 but less than 3.15? Is the fix in 3.15 yet? I just installed 3.15.8. All the best, ~ Christopher Al On Sat, Aug 2, 2014 at 11:20 AM, Ilya Dryomov wrote: > On Sat, Aug 2, 2014 at 10:03 PM, Christopher O'Connell > wrote: > > On Sat, Aug 2, 2014 at 6:27 AM, Ilya Dryomov > > wrote: > >> > >> On Sat, Aug 2, 2014 at 1:41 AM, Christopher O'Connell > >> wrote: > >> > So I've been having a seemingly similar problem and while trying to > >> > follow > >> > the steps in this thread, things have gone very south for me. > >> > >> Show me where in this thread have I said to set tunables to optimal ;) > >> optimal (== firefly for firefly) is actually the opposite of what you > >> are going to need. > > > > > > So what should tunables be set to? Optimal? > > Ordinarily yes, but not if you are going to use older kernels. In that > case you'd want "default" or "legacy". > > > > >> > >> > >> > > >> > Kernal on OSDs and MONs: 2.6.32-431.20.3.0.1.el6.centos.plus.x86_64 #1 > >> > SMP > >> > Wed Jul 16 21:27:52 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux > >> > > >> > Kernal on RBD host: 3.2.0-61-generic #93-Ubuntu SMP Fri May 2 21:31:50 > >> > UTC > >> > 2014 x86_64 x86_64 x86_64 GNU/Linux > >> > > >> > All are running 0.80.5 > >> > >> Is this a new firefly cluster or was it created before firely > >> (specifically before v0.78) and then upgraded? > > > > > > It was created before 0.78 and upgraded. It has also been expanded > several > > times. > > > >> > >> > >> > > >> > I updated the tunables as per this article > >> > > >> > > http://cephnotes.ksperis.com/blog/2014/01/16/set-tunables-optimal-on-ceph-crushmap > >> > > >> > Here's what's happening: > >> > > >> > 1) On the rbd client node, trying to map rbd produces > >> > $ sudo rbd --conf /etc/ceph/mia1.conf --keyring > >> > /etc/ceph/mia1.client.admin.keyring map poolname > >> > > >> > rbd: add failed: (5) Input/output error > >> > > >> > Dmesg: > >> > > >> > [331172.147289] libceph: mon0 10.103.11.132:6789 feature set > mismatch, > >> > my 2 > >> > < server's 20042040002, missing 2004204 > >> > [331172.154059] libceph: mon0 10.103.11.132:6789 missing required > >> > protocol > >> > features > >> > [331182.176604] libceph: mon1 10.103.11.141:6789 feature set > mismatch, > >> > my 2 > >> > < server's 20042040002, missing 2004204 > >> > [331182.183535] libceph: mon1 10.103.11.141:6789 missing required > >> > protocol > >> > features > >> > [331192.192630] libceph: mon2 10.103.11.152:6789 feature set > mismatch, > >> > my 2 > >> > < server's 20042040002, missing 2004204 > >> > [331192.199810] libceph: mon2 10.103.11.152:6789 missing required > >> > protocol > >> > features > >> > [331202.209324] libceph: mon0 10.103.11.132:6789 feature set > mismatch, > >> > my 2 > >> > < server's 20042040002, missing 2004204 > >> > [331202.216957] libceph: mon0 10.103.11.132:6789 missing required > >> > protocol > >> > features > >> > [331212.224540] libceph: mon0 10.103.11.132:6789 feature set > mismatch, > >> > my 2 > >> > < server's 20042040002, missing 2004204 > >> > [331212.232276] libceph: mon0 10.103.11.132:6789 missing required > >> > protocol > >> > features > >> > [331222.240605] libceph: mon2 10.103.11.152:6789 feature set > mismatch, > >> > my 2 > >> > < server's 20042040002, missing 2004204 > >> > [331222.248660] libceph: mon2 10.103.11.152:6789 missing required > >> > protocol > >> > features > >> > > >> > However, running > >> > $ sudo rbd --conf /etc/ceph/mia1.conf --keyring > >> > /etc/ceph/mia1.client.admin.keyring ls > >> > poolname > >> > > >> > works fine and shows the expected pool name. > >> > > >> > 2) On the monitor where I ran the command to update the tunables, I > can > >> > no > >> > longer run the ceph console: > >> > $ ceph -c /etc/ceph/mia1.conf --keyring > >> > /etc/ceph/mia1.client.admin.keyring > >> > 2014-08-01 17:32:05.026960 7f21943d2700 0 -- 10.103.11.132:0/1030058 > >> > >> > 10.103.11.141:6789/0 pipe(0x7f2190028440 sd=3 :42360 s=1 pgs=0 cs=0 > l=1 > >> > c=0x7f21900286a0).connect protocol feature mismatch, my f < > peer > >> > 20f missing 200 > >> > 2014-08-01 17:32:05.027024 7f21943d2700 0 -- 10.103.11.132:0/1030058 > >> > >> > 10.103.11.141:6789/0 pipe(0x7f2190028440 sd=3 :42360 s=1 pgs=0 cs=0 > l=1 > >> > c=0x7f21900286a0).fault > >> > 2014-08-01 17:32:05.027544 7f21943d2700 0 -- 10.103.11.132:0/1030058 > >> > >> > 10.103.11.141:6789/0 pipe(0x7f2190028440 sd=3 :42361 s=1 pgs=0 cs=0 > l=1 > >> > c=0x7f21900286a0).connect protocol feature mismatch, my f < > peer > >> > 20f missing 200 > >> > > >> > and it just keeps spitting out a similar message. However I *can* run > >> > the > >> > ceph console and execute basic commands (status, at the very least) > from > >> > other nodes. > >> > >> What does ceph -s from those other nodes say? Check versions of all > >> monitors with >
Re: [ceph-users] 0.80.5-1precise Not Able to Map RBD & CephFS
To be more clear on my question, we currently use ELRepo for those rare occasions when we need a 3.x kernel on centos. Are you aware of anyone maintaining a 3.14 kernel. On Sat, Aug 2, 2014 at 3:01 PM, Christopher O'Connell wrote: > Hi Ilya, > > Short of building a 3.14 kernel from scratch, are there any centos/EL > kernels of 3.14 but less than 3.15? > > Is the fix in 3.15 yet? I just installed 3.15.8. > > All the best, > > ~ Christopher > > Al > > > On Sat, Aug 2, 2014 at 11:20 AM, Ilya Dryomov > wrote: > >> On Sat, Aug 2, 2014 at 10:03 PM, Christopher O'Connell >> wrote: >> > On Sat, Aug 2, 2014 at 6:27 AM, Ilya Dryomov >> > wrote: >> >> >> >> On Sat, Aug 2, 2014 at 1:41 AM, Christopher O'Connell >> >> wrote: >> >> > So I've been having a seemingly similar problem and while trying to >> >> > follow >> >> > the steps in this thread, things have gone very south for me. >> >> >> >> Show me where in this thread have I said to set tunables to optimal ;) >> >> optimal (== firefly for firefly) is actually the opposite of what you >> >> are going to need. >> > >> > >> > So what should tunables be set to? Optimal? >> >> Ordinarily yes, but not if you are going to use older kernels. In that >> case you'd want "default" or "legacy". >> >> > >> >> >> >> >> >> > >> >> > Kernal on OSDs and MONs: 2.6.32-431.20.3.0.1.el6.centos.plus.x86_64 >> #1 >> >> > SMP >> >> > Wed Jul 16 21:27:52 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux >> >> > >> >> > Kernal on RBD host: 3.2.0-61-generic #93-Ubuntu SMP Fri May 2 >> 21:31:50 >> >> > UTC >> >> > 2014 x86_64 x86_64 x86_64 GNU/Linux >> >> > >> >> > All are running 0.80.5 >> >> >> >> Is this a new firefly cluster or was it created before firely >> >> (specifically before v0.78) and then upgraded? >> > >> > >> > It was created before 0.78 and upgraded. It has also been expanded >> several >> > times. >> > >> >> >> >> >> >> > >> >> > I updated the tunables as per this article >> >> > >> >> > >> http://cephnotes.ksperis.com/blog/2014/01/16/set-tunables-optimal-on-ceph-crushmap >> >> > >> >> > Here's what's happening: >> >> > >> >> > 1) On the rbd client node, trying to map rbd produces >> >> > $ sudo rbd --conf /etc/ceph/mia1.conf --keyring >> >> > /etc/ceph/mia1.client.admin.keyring map poolname >> >> > >> >> > rbd: add failed: (5) Input/output error >> >> > >> >> > Dmesg: >> >> > >> >> > [331172.147289] libceph: mon0 10.103.11.132:6789 feature set >> mismatch, >> >> > my 2 >> >> > < server's 20042040002, missing 2004204 >> >> > [331172.154059] libceph: mon0 10.103.11.132:6789 missing required >> >> > protocol >> >> > features >> >> > [331182.176604] libceph: mon1 10.103.11.141:6789 feature set >> mismatch, >> >> > my 2 >> >> > < server's 20042040002, missing 2004204 >> >> > [331182.183535] libceph: mon1 10.103.11.141:6789 missing required >> >> > protocol >> >> > features >> >> > [331192.192630] libceph: mon2 10.103.11.152:6789 feature set >> mismatch, >> >> > my 2 >> >> > < server's 20042040002, missing 2004204 >> >> > [331192.199810] libceph: mon2 10.103.11.152:6789 missing required >> >> > protocol >> >> > features >> >> > [331202.209324] libceph: mon0 10.103.11.132:6789 feature set >> mismatch, >> >> > my 2 >> >> > < server's 20042040002, missing 2004204 >> >> > [331202.216957] libceph: mon0 10.103.11.132:6789 missing required >> >> > protocol >> >> > features >> >> > [331212.224540] libceph: mon0 10.103.11.132:6789 feature set >> mismatch, >> >> > my 2 >> >> > < server's 20042040002, missing 2004204 >> >> > [331212.232276] libceph: mon0 10.103.11.132:6789 missing required >> >> > protocol >> >> > features >> >> > [331222.240605] libceph: mon2 10.103.11.152:6789 feature set >> mismatch, >> >> > my 2 >> >> > < server's 20042040002, missing 2004204 >> >> > [331222.248660] libceph: mon2 10.103.11.152:6789 missing required >> >> > protocol >> >> > features >> >> > >> >> > However, running >> >> > $ sudo rbd --conf /etc/ceph/mia1.conf --keyring >> >> > /etc/ceph/mia1.client.admin.keyring ls >> >> > poolname >> >> > >> >> > works fine and shows the expected pool name. >> >> > >> >> > 2) On the monitor where I ran the command to update the tunables, I >> can >> >> > no >> >> > longer run the ceph console: >> >> > $ ceph -c /etc/ceph/mia1.conf --keyring >> >> > /etc/ceph/mia1.client.admin.keyring >> >> > 2014-08-01 17:32:05.026960 7f21943d2700 0 -- >> 10.103.11.132:0/1030058 >> >> >> > 10.103.11.141:6789/0 pipe(0x7f2190028440 sd=3 :42360 s=1 pgs=0 cs=0 >> l=1 >> >> > c=0x7f21900286a0).connect protocol feature mismatch, my f < >> peer >> >> > 20f missing 200 >> >> > 2014-08-01 17:32:05.027024 7f21943d2700 0 -- >> 10.103.11.132:0/1030058 >> >> >> > 10.103.11.141:6789/0 pipe(0x7f2190028440 sd=3 :42360 s=1 pgs=0 cs=0 >> l=1 >> >> > c=0x7f21900286a0).fault >> >> > 2014-08-01 17:32:05.027544 7f21943d2700 0 -- >> 10.103.11.132:0/1030058 >> >> >> > 10.103.11.141:6789/0 pipe(0x7f2190028440 sd=3 :42361 s=1 pgs=0 cs=0 >> l=1 >> >> > c=0x7f219
Re: [ceph-users] Firefly OSDs stuck in creating state forever
Yes I looked at tcpdump on each of the OSDs and saw communications between all
3 OSDs before I sent my first question to this list. When I disabled selinux on
the one offending server based on your feedback (typically we have this
disabled on lab systems that are only on the lab net) the 10 pages in my test
pool all went to 'active+clean' almost immediately. Unfortunately the 3 default
pools still remain in the creating states and are not health_ok. The OSDs all
stayed UP/IN after the selinux change for the rest of the day until I made the
mistake of creating a RBD image on demo-pool and it's 10 'active+clean' pages.
I created the rbd, but when I attempted to look at it with 'rbd info' the
cluster went into an endless loop trying to read a placement group and loop
that I left running overnight. This morning ceph-mon was crashed again. I'll
probably start all over from scratch once again on Monday.
I deleted ceph-mds and got rid of the 'laggy' comments from 'ceph health'. The
"official" online Ceph docs on that "coming soon" and most references I could
find were pre firefly so it was a little trail and error to figure out to use
the pool number and not it's name to get the removal to work. Same with 'ceph
mds newfs' to get rid of 'laggy-ness' in the 'ceph health' output.
[root@essperf3 Ceph]# ceph mds rm 0 mds.essperf3
mds gid 0 dne
[root@essperf3 Ceph]# ceph health
HEALTH_WARN 96 pgs incomplete; 96 pgs peering; 192 pgs stuck inactive; 192 pgs
stuck unclean mds essperf3 is laggy
[root@essperf3 Ceph]# ceph mds newfs 1 0 --yes-i-really-mean-it
new fs with metadata pool 1 and data pool 0
[root@essperf3 Ceph]# ceph health
HEALTH_WARN 96 pgs incomplete; 96 pgs peering; 192 pgs stuck inactive; 192 pgs
stuck unclean
[root@essperf3 Ceph]#
From: Brian Rak [mailto:b...@gameservers.com]
Sent: Friday, August 01, 2014 6:14 PM
To: Bruce McFarland; ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Firefly OSDs stuck in creating state forever
What happens if you remove nodown? I'd be interested to see what OSDs it
thinks are down. My next thought would be tcpdump on the private interface.
See if the OSDs are actually managing to connect to each other.
For comparison, when I bring up a cluster of 3 OSDs it goes to HEALTH_OK nearly
instantly (definitely under a minute!), so it's probably not just taking awhile.
Does 'ceph osd dump' show the proper public and private IPs?
On 8/1/2014 6:13 PM, Bruce McFarland wrote:
MDS: I assumed that I'd need to bring up a ceph-mds for my cluster at initial
bringup. We also intended to modify the CRUSH map such that it's pool is
resident to SSD(s). It is one of the areas of the online docs there doesn't
seem to be a lot of info on and I haven't spent a lot of time researching. I'll
stop it.
OSD connectivity: The connectivity is good for both 1GE and 10GE. I thought
moving to 10GE with nothing else on that net might help with group placement
etc and bring up the pages quicker. I've checked 'tcpdump' output on all boxes.
Firewall: Thanks for that one - it's the "basic" I over looked in my ceph
learning curve. One of the OSDs had selinux=enforcing - all others were
disabled. Changing that box and the 10 pages in my demo-pool (kept page count
very small for sanity) are now 'active+clean'. The pages for the default pools
- data, metadata, rbd - are still stuck in creating+peering or
creating+incomplete. I did have to use manually set 'osd pool default min size
= 1' from it's default of 2 for these 3 pools to eliminate a bunch of warnings
in the 'ceph health detail' output.
I'm adding the [mon] setting you suggested below and stopping ceph-mds and
bringing everything up now.
[root@essperf3 Ceph]# ceph -s
cluster 4b3ffe60-73f4-4512-b7da-b04e4775dd73
health HEALTH_WARN 96 pgs incomplete; 96 pgs peering; 192 pgs stuck
inactive; 192 pgs stuck unclean; 28 requests are blocked > 32 sec;
nodown,noscrub flag(s) set
monmap e1: 1 mons at {essperf3=209.243.160.35:6789/0}, election epoch 1,
quorum 0 essperf3
mdsmap e43: 1/1/1 up {0=essperf3=up:creating}
osdmap e752: 3 osds: 3 up, 3 in
flags nodown,noscrub
pgmap v1483: 202 pgs, 4 pools, 0 bytes data, 0 objects
134 MB used, 1158 GB / 1158 GB avail
96 creating+peering
10 active+clean
Subject: Re: [ceph-users] Firefly OSDs stuck in creating state forever
Why do you have a MDS active? I'd suggest getting rid of that at least until
you have everything else working.
I see you've set nodown on the OSDs, did you have problems with the OSDs
flapping? Do the OSDs have broken connectivity between themselves? Do you
have some kind of firewall interfering here?
I've seen odd issues when the OSDs have broken private networking, you'll get
