[ceph-users] How to start/stop ceph daemons separately?

[Da Chun]

On Ubuntu, we can start/stop ceph daemons separately as below:
start ceph-mon id=ceph0
stop ceph-mon id=ceph0


How to do this on Centos or rhel? Thanks!___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph pgs stuck unclean

[Howarth, Chris]

Hi,
One of our OSD disks failed on a cluster and I replaced it, but when it 
failed it did not completely recover and I have a number of pgs which are stuck 
unclean:

# ceph health detail
HEALTH_WARN 7 pgs stuck unclean
pg 3.5a is stuck unclean for 335339.172516, current state active, last acting 
[5,4]
pg 3.54 is stuck unclean for 335339.157608, current state active, last acting 
[15,7]
pg 3.55 is stuck unclean for 335339.167154, current state active, last acting 
[16,9]
pg 3.1c is stuck unclean for 335339.174150, current state active, last acting 
[8,16]
pg 3.a is stuck unclean for 335339.177001, current state active, last acting 
[0,8]
pg 3.4 is stuck unclean for 335339.165377, current state active, last acting 
[17,4]
pg 3.5 is stuck unclean for 335339.149507, current state active, last acting 
[2,6]

Does anyone know how to fix these ? I tried the following, but this does not 
seem to work:


# ceph pg 3.5 mark_unfound_lost revert
pg has no unfound objects

thanks

Chris
__
Chris Howarth
OS Platforms Engineering
Citi Architecture & Technology Engineering
(e) chris.howa...@citi.com
(t) +44 (0) 20 7508 3848
(f) +44 (0) 20 7508 0964
(mail-drop) CGC-06-3A

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How to start/stop ceph daemons separately?

[Wido den Hollander]

Op 7 aug. 2013 om 10:20 heeft "Da Chun"  het volgende geschreven:

> On Ubuntu, we can start/stop ceph daemons separately as below:
> start ceph-mon id=ceph0
> stop ceph-mon id=ceph0
> 
> How to do this on Centos or rhel? Thanks!

I think this should work:

$ service ceph stop mon.ceph0
$ service ceph stop osd.0

And replace stop by start if you want to start ;)

Wido

> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph-deploy with partition, lvm or dm-crypt

[Pierre BLONDEAU]

Hello,

I read in the documentation that it is more recommended to use 
ceph-deploy the configuration files.


But I can not:
 Use a partition as OSD (and not a full hard drive)
 Give a logical volume (LVM) as log (SSD hardware raid 1)
 Using dm-crypt

My version of Ceph-depoly is 1.0-1 on http://ceph.com/debian-cuttlefish/

Thank you in advance for your help.
Regards

Pierre

--
--
Pierre BLONDEAU
Administrateur Systèmes & réseaux
Université de Caen
Laboratoire GREYC, Département d'informatique

tel : 02 31 56 75 42
bureau  : Campus 2, Science 3, 406
--
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] fuse or kernel fs?

[Sage Weil]

On Wed, 7 Aug 2013, James Harper wrote:
> Are the fuse and kernel filesystem drivers about the same or is one 
> definitely better than the other?

Both are actively maintained.  I would say the kernel one is faster and a 
bit more robust, but it is also necessary to run a recent kernel to get 
all the latest bug fixes.

sage
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] journal on ssd

[Tren Blackburn]

On Tue, Aug 6, 2013 at 11:14 AM, Joao Pedras  wrote:

> Greetings all.
>
> I am installing a test cluster using one ssd (/dev/sdg) to hold the
> journals. Ceph's version is 0.61.7 and I am using ceph-deploy obtained from
> ceph's git yesterday. This is on RHEL6.4, fresh install.
>
> When preparing the first 2 drives, sda and sdb, all goes well and the
> journals get created in sdg1 and sdg2:
>
> $> ceph-deploy osd prepare ceph00:sda:sdg ceph00:sdb:sdg
> [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks
> ceph00:/dev/sda:/dev/sdg ceph00:/dev/sdb:/dev/sdg
> [ceph_deploy.osd][DEBUG ] Deploying osd to ceph00
> [ceph_deploy.osd][DEBUG ] Host ceph00 is now ready for osd use.
> [ceph_deploy.osd][DEBUG ] Preparing host ceph00 disk /dev/sda journal
> /dev/sdg activate False
> [ceph_deploy.osd][DEBUG ] Preparing host ceph00 disk /dev/sdb journal
> /dev/sdg activate False
>
> When preparing sdc or any disk after the first 2 I get the following in
> that osd's log but no errors on ceph-deploy:
>
> # tail -f /var/log/ceph/ceph-osd.2.log
> 2013-08-06 10:51:36.655053 7f5ba701a780  0 ceph version 0.61.7
> (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 11596
> 2013-08-06 10:51:36.658671 7f5ba701a780  1
> filestore(/var/lib/ceph/tmp/mnt.i2NK47) mkfs in /var/lib/ceph/tmp/mnt.i2NK47
> 2013-08-06 10:51:36.658697 7f5ba701a780  1
> filestore(/var/lib/ceph/tmp/mnt.i2NK47) mkfs fsid is already set to
> 5d1beb09-1f80-421d-a88c-57789e2fc33e
> 2013-08-06 10:51:36.813783 7f5ba701a780  1
> filestore(/var/lib/ceph/tmp/mnt.i2NK47) leveldb db exists/created
> 2013-08-06 10:51:36.813964 7f5ba701a780 -1 journal FileJournal::_open:
> disabling aio for non-block journal.  Use journal_force_aio to force use of
> aio anyway
> 2013-08-06 10:51:36.813999 7f5ba701a780  1 journal _open
> /var/lib/ceph/tmp/mnt.i2NK47/journal fd 10: 0 bytes, block size 4096 bytes,
> directio = 1, aio = 0
> 2013-08-06 10:51:36.814035 7f5ba701a780 -1 journal check: ondisk fsid
> ---- doesn't match expected
> 5d1beb09-1f80-421d-a88c-57789e2fc33e, invalid (someone else's?) journal
> 2013-08-06 10:51:36.814093 7f5ba701a780 -1
> filestore(/var/lib/ceph/tmp/mnt.i2NK47) mkjournal error creating journal on
> /var/lib/ceph/tmp/mnt.i2NK47/journal: (22) Invalid argument
> 2013-08-06 10:51:36.814125 7f5ba701a780 -1 OSD::mkfs: FileStore::mkfs
> failed with error -22
> 2013-08-06 10:51:36.814185 7f5ba701a780 -1  ** ERROR: error creating empty
> object store in /var/lib/ceph/tmp/mnt.i2NK47: (22) Invalid argument
>
> I have cleaned the disks with dd, zapped them and so forth but this always
> occurs. If doing sdc/sdd first, for example, then sda or whatever follows
> fails with similar errors.
>
> Does anyone have any insight on this issue?
>

Is SELinux disabled?

t.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] minimum object size in ceph

[Nulik Nol]

thanks Dan,
i meant like PRIMARY KEY in a RDBMS, or Key for NoSQL (key-value pair)
database to perform put() get() operations. Well, if it is string then
it's ok, I can print binary keys in HEX or uuencode or something like
that.
Is there a limit on maximum string length for object name?

Regards
Nulik

On Tue, Aug 6, 2013 at 4:08 PM, Dan Mick  wrote:
> No minumum object size.  As for key, not sure what you mean; the closest
> thing to an object 'key' is its name, but it's obvious from routines like
> rados_read() and rados_write() that that's a const char *.  Did you mean
> some other key?
>
>
> On 08/06/2013 12:13 PM, Nulik Nol wrote:
>>
>> Hi,
>>
>> when using the C api (RADOS) what is the minimum object size ? And
>> what is the key type ? (uint64_t, char[], or something like that ?)
>>
>> TIA
>> Nulik
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
> --
> Dan Mick, Filesystem Engineering
> Inktank Storage, Inc.   http://inktank.com
> Ceph docs: http://ceph.com/docs
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] journal on ssd

[Joao Pedras]

Hello Tren,

It is indeed:

$> sestatus
SELinux status: disabled

Thanks,


On Wed, Aug 7, 2013 at 9:33 AM, Tren Blackburn  wrote:

> On Tue, Aug 6, 2013 at 11:14 AM, Joao Pedras  wrote:
>
>> Greetings all.
>>
>> I am installing a test cluster using one ssd (/dev/sdg) to hold the
>> journals. Ceph's version is 0.61.7 and I am using ceph-deploy obtained from
>> ceph's git yesterday. This is on RHEL6.4, fresh install.
>>
>> When preparing the first 2 drives, sda and sdb, all goes well and the
>> journals get created in sdg1 and sdg2:
>>
>> $> ceph-deploy osd prepare ceph00:sda:sdg ceph00:sdb:sdg
>> [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks
>> ceph00:/dev/sda:/dev/sdg ceph00:/dev/sdb:/dev/sdg
>> [ceph_deploy.osd][DEBUG ] Deploying osd to ceph00
>> [ceph_deploy.osd][DEBUG ] Host ceph00 is now ready for osd use.
>> [ceph_deploy.osd][DEBUG ] Preparing host ceph00 disk /dev/sda journal
>> /dev/sdg activate False
>> [ceph_deploy.osd][DEBUG ] Preparing host ceph00 disk /dev/sdb journal
>> /dev/sdg activate False
>>
>> When preparing sdc or any disk after the first 2 I get the following in
>> that osd's log but no errors on ceph-deploy:
>>
>> # tail -f /var/log/ceph/ceph-osd.2.log
>> 2013-08-06 10:51:36.655053 7f5ba701a780  0 ceph version 0.61.7
>> (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 11596
>> 2013-08-06 10:51:36.658671 7f5ba701a780  1
>> filestore(/var/lib/ceph/tmp/mnt.i2NK47) mkfs in /var/lib/ceph/tmp/mnt.i2NK47
>> 2013-08-06 10:51:36.658697 7f5ba701a780  1
>> filestore(/var/lib/ceph/tmp/mnt.i2NK47) mkfs fsid is already set to
>> 5d1beb09-1f80-421d-a88c-57789e2fc33e
>> 2013-08-06 10:51:36.813783 7f5ba701a780  1
>> filestore(/var/lib/ceph/tmp/mnt.i2NK47) leveldb db exists/created
>> 2013-08-06 10:51:36.813964 7f5ba701a780 -1 journal FileJournal::_open:
>> disabling aio for non-block journal.  Use journal_force_aio to force use of
>> aio anyway
>> 2013-08-06 10:51:36.813999 7f5ba701a780  1 journal _open
>> /var/lib/ceph/tmp/mnt.i2NK47/journal fd 10: 0 bytes, block size 4096 bytes,
>> directio = 1, aio = 0
>> 2013-08-06 10:51:36.814035 7f5ba701a780 -1 journal check: ondisk fsid
>> ---- doesn't match expected
>> 5d1beb09-1f80-421d-a88c-57789e2fc33e, invalid (someone else's?) journal
>> 2013-08-06 10:51:36.814093 7f5ba701a780 -1
>> filestore(/var/lib/ceph/tmp/mnt.i2NK47) mkjournal error creating journal on
>> /var/lib/ceph/tmp/mnt.i2NK47/journal: (22) Invalid argument
>> 2013-08-06 10:51:36.814125 7f5ba701a780 -1 OSD::mkfs: FileStore::mkfs
>> failed with error -22
>> 2013-08-06 10:51:36.814185 7f5ba701a780 -1  ** ERROR: error creating
>> empty object store in /var/lib/ceph/tmp/mnt.i2NK47: (22) Invalid argument
>>
>> I have cleaned the disks with dd, zapped them and so forth but this
>> always occurs. If doing sdc/sdd first, for example, then sda or whatever
>> follows fails with similar errors.
>>
>> Does anyone have any insight on this issue?
>>
>
> Is SELinux disabled?
>
> t.
>
>


-- 
Joao Pedras
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] journal on ssd

[Joao Pedras]

Some more info about this...

The subject should have been journal on another device. The issue also
occurs if using another disk to hold the journal.
If doing something like 'ceph-deploy node:sda:sdk' a subsequent run like
'ceph-deploy:sdb:sdk' will show the error regarding sdb's osd. If doing
'ceph-deply node:sda:sdk node:sdb:sdk node:sdc:sdk [...]' the first 2 osds
will be created and launched fine, sdc's and any others won't.

Thanks.


On Wed, Aug 7, 2013 at 10:55 AM, Joao Pedras  wrote:

> Hello Tren,
>
> It is indeed:
>
> $> sestatus
> SELinux status: disabled
>
> Thanks,
>
>
> On Wed, Aug 7, 2013 at 9:33 AM, Tren Blackburn wrote:
>
>> On Tue, Aug 6, 2013 at 11:14 AM, Joao Pedras  wrote:
>>
>>> Greetings all.
>>>
>>> I am installing a test cluster using one ssd (/dev/sdg) to hold the
>>> journals. Ceph's version is 0.61.7 and I am using ceph-deploy obtained from
>>> ceph's git yesterday. This is on RHEL6.4, fresh install.
>>>
>>> When preparing the first 2 drives, sda and sdb, all goes well and the
>>> journals get created in sdg1 and sdg2:
>>>
>>> $> ceph-deploy osd prepare ceph00:sda:sdg ceph00:sdb:sdg
>>> [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks
>>> ceph00:/dev/sda:/dev/sdg ceph00:/dev/sdb:/dev/sdg
>>> [ceph_deploy.osd][DEBUG ] Deploying osd to ceph00
>>> [ceph_deploy.osd][DEBUG ] Host ceph00 is now ready for osd use.
>>> [ceph_deploy.osd][DEBUG ] Preparing host ceph00 disk /dev/sda journal
>>> /dev/sdg activate False
>>> [ceph_deploy.osd][DEBUG ] Preparing host ceph00 disk /dev/sdb journal
>>> /dev/sdg activate False
>>>
>>> When preparing sdc or any disk after the first 2 I get the following in
>>> that osd's log but no errors on ceph-deploy:
>>>
>>> # tail -f /var/log/ceph/ceph-osd.2.log
>>> 2013-08-06 10:51:36.655053 7f5ba701a780  0 ceph version 0.61.7
>>> (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 11596
>>> 2013-08-06 10:51:36.658671 7f5ba701a780  1
>>> filestore(/var/lib/ceph/tmp/mnt.i2NK47) mkfs in /var/lib/ceph/tmp/mnt.i2NK47
>>> 2013-08-06 10:51:36.658697 7f5ba701a780  1
>>> filestore(/var/lib/ceph/tmp/mnt.i2NK47) mkfs fsid is already set to
>>> 5d1beb09-1f80-421d-a88c-57789e2fc33e
>>> 2013-08-06 10:51:36.813783 7f5ba701a780  1
>>> filestore(/var/lib/ceph/tmp/mnt.i2NK47) leveldb db exists/created
>>> 2013-08-06 10:51:36.813964 7f5ba701a780 -1 journal FileJournal::_open:
>>> disabling aio for non-block journal.  Use journal_force_aio to force use of
>>> aio anyway
>>> 2013-08-06 10:51:36.813999 7f5ba701a780  1 journal _open
>>> /var/lib/ceph/tmp/mnt.i2NK47/journal fd 10: 0 bytes, block size 4096 bytes,
>>> directio = 1, aio = 0
>>> 2013-08-06 10:51:36.814035 7f5ba701a780 -1 journal check: ondisk fsid
>>> ---- doesn't match expected
>>> 5d1beb09-1f80-421d-a88c-57789e2fc33e, invalid (someone else's?) journal
>>> 2013-08-06 10:51:36.814093 7f5ba701a780 -1
>>> filestore(/var/lib/ceph/tmp/mnt.i2NK47) mkjournal error creating journal on
>>> /var/lib/ceph/tmp/mnt.i2NK47/journal: (22) Invalid argument
>>> 2013-08-06 10:51:36.814125 7f5ba701a780 -1 OSD::mkfs: FileStore::mkfs
>>> failed with error -22
>>> 2013-08-06 10:51:36.814185 7f5ba701a780 -1  ** ERROR: error creating
>>> empty object store in /var/lib/ceph/tmp/mnt.i2NK47: (22) Invalid argument
>>>
>>> I have cleaned the disks with dd, zapped them and so forth but this
>>> always occurs. If doing sdc/sdd first, for example, then sda or whatever
>>> follows fails with similar errors.
>>>
>>> Does anyone have any insight on this issue?
>>>
>>
>> Is SELinux disabled?
>>
>> t.
>>
>>
>
>
> --
> Joao Pedras
>



-- 
Joao Pedras
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Why is my mon store.db is 220GB?

[Jeppesen, Nelson]

Joao,

Have you had a chance to look at my monitor issues? I Ran ''ceph-mon -i FOO 
-compact'  last week but it did not improve disk usage.

Let me know if there's anything else I dig up. The monitor still at 0.67-rc2 
with the OSDs at .0.61.7.


On 08/02/2013 12:15 AM, Jeppesen, Nelson wrote:
> Thanks for the reply, but how can I fix this without an outage?
>
> I tired adding 'mon compact on start = true' but the monitor just hung. 
> Unfortunately this is a production cluster and can't take the outages (I'm 
> assuming the cluster will fail without a monitor). I had three monitors I was 
> hit with the store.db bug and lost two of the three.
>
> I have tried running with 0.61.5, .0.61.7 and 0.67-rc2. None of them seem to 
> shrink the DB.

My guess is that the compaction policies we are enforcing won't cover 
the portions of the store that haven't been compacted *prior* to the 
upgrade.

Even today we still know of users with stores growing over dozens of 
GBs, requiring occasional restarts to compact (which is far from an 
acceptable fix).  Some of these stores can take several minutes to 
compact when the monitors are restarted, although these guys can often 
mitigate any down time by restarting monitors one at a time while 
maintaining quorum.  Unfortunately you don't have that luxury. :-\

If however you are willing to manually force a compaction, you should be 
able to do so with 'ceph-mon -i FOO --compact'.

Now, there is a possibility this is why you've been unable to add other 
monitors to the cluster.  Chances are that the iterators used to 
synchronize the store get stuck, or move slowly enough to make all sorts 
of funny timeouts to be triggered.

I intend to look into your issue (especially the problems with adding 
new monitors) in the morning to better assess what's happening.

   -Joao


> -Original Message-
> From: Mike Dawson [mailto:mike.dawson at cloudapt.com]
> Sent: Thursday, August 01, 2013 4:10 PM
> To: Jeppesen, Nelson
> Cc: ceph-users at lists.ceph.com
> Subject: Re: [ceph-users] Why is my mon store.db is 220GB?
>
> 220GB is way, way too big. I suspect your monitors need to go through a 
> successful leveldb compaction. The early releases of Cuttlefish suffered 
> several issues with store.db growing unbounded. Most were fixed by 0.61.5, I 
> believe.
>
> You may have luck stoping all Ceph daemons, then starting the monitor by 
> itself. When there were bugs, leveldb compaction tended work better without 
> OSD traffic hitting the monitors. Also, there are some settings to force a 
> compact on startup like 'mon compact on start = true' and mon compact on trim 
> = true". I don't think either are required anymore though. See some history 
> here:
>
> http://tracker.ceph.com/issues/4895
>
>
> Thanks,
>
> Mike Dawson
> Co-Founder & Director of Cloud Architecture Cloudapt LLC
> 6330 East 75th Street, Suite 170
> Indianapolis, IN 46250
>
> On 8/1/2013 6:52 PM, Jeppesen, Nelson wrote:
>> My Mon store.db has been at 220GB for a few months now. Why is this
>> and how can I fix it? I have one monitor in this cluster and I suspect
>> that I can't  add monitors to the cluster because it is too big. Thank you.
>>
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users at lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> ___
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>


-- 
Joao Eduardo Luis
Software Engineer | http://inktank.com | http://ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] journal on ssd

[Joao Pedras]

The journal devices entries beyond the 2nd (ie. /dev/sdg2) are not created
under /dev. Basically doing the following addresses the issue:

--- /usr/sbin/ceph-disk 2013-07-25 00:55:41.0 -0700
+++ /root/ceph-disk 2013-08-07 15:54:17.538542684 -0700
@@ -857,6 +857,14 @@
 'settle',
 ],
 )
+subprocess.call(
+args=[
+# wait for udev event queue to clear
+'partx',
+'-a',
+   '{journal}'.format(journal=journal)
+],
+)

 journal_symlink = '/dev/disk/by-partuuid/{journal_uuid}'.format(
 journal_uuid=journal_uuid,

This is RHEL 6.4.

Thanks for the help,


On Wed, Aug 7, 2013 at 12:48 PM, Joao Pedras  wrote:

> Some more info about this...
>
> The subject should have been journal on another device. The issue also
> occurs if using another disk to hold the journal.
> If doing something like 'ceph-deploy node:sda:sdk' a subsequent run like
> 'ceph-deploy:sdb:sdk' will show the error regarding sdb's osd. If doing
> 'ceph-deply node:sda:sdk node:sdb:sdk node:sdc:sdk [...]' the first 2 osds
> will be created and launched fine, sdc's and any others won't.
>
> Thanks.
>
>
> On Wed, Aug 7, 2013 at 10:55 AM, Joao Pedras  wrote:
>
>> Hello Tren,
>>
>> It is indeed:
>>
>> $> sestatus
>> SELinux status: disabled
>>
>> Thanks,
>>
>>
>> On Wed, Aug 7, 2013 at 9:33 AM, Tren Blackburn wrote:
>>
>>> On Tue, Aug 6, 2013 at 11:14 AM, Joao Pedras  wrote:
>>>
 Greetings all.

 I am installing a test cluster using one ssd (/dev/sdg) to hold the
 journals. Ceph's version is 0.61.7 and I am using ceph-deploy obtained from
 ceph's git yesterday. This is on RHEL6.4, fresh install.

 When preparing the first 2 drives, sda and sdb, all goes well and the
 journals get created in sdg1 and sdg2:

 $> ceph-deploy osd prepare ceph00:sda:sdg ceph00:sdb:sdg
 [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks
 ceph00:/dev/sda:/dev/sdg ceph00:/dev/sdb:/dev/sdg
 [ceph_deploy.osd][DEBUG ] Deploying osd to ceph00
 [ceph_deploy.osd][DEBUG ] Host ceph00 is now ready for osd use.
 [ceph_deploy.osd][DEBUG ] Preparing host ceph00 disk /dev/sda journal
 /dev/sdg activate False
 [ceph_deploy.osd][DEBUG ] Preparing host ceph00 disk /dev/sdb journal
 /dev/sdg activate False

 When preparing sdc or any disk after the first 2 I get the following in
 that osd's log but no errors on ceph-deploy:

 # tail -f /var/log/ceph/ceph-osd.2.log
 2013-08-06 10:51:36.655053 7f5ba701a780  0 ceph version 0.61.7
 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 11596
 2013-08-06 10:51:36.658671 7f5ba701a780  1
 filestore(/var/lib/ceph/tmp/mnt.i2NK47) mkfs in 
 /var/lib/ceph/tmp/mnt.i2NK47
 2013-08-06 10:51:36.658697 7f5ba701a780  1
 filestore(/var/lib/ceph/tmp/mnt.i2NK47) mkfs fsid is already set to
 5d1beb09-1f80-421d-a88c-57789e2fc33e
 2013-08-06 10:51:36.813783 7f5ba701a780  1
 filestore(/var/lib/ceph/tmp/mnt.i2NK47) leveldb db exists/created
 2013-08-06 10:51:36.813964 7f5ba701a780 -1 journal FileJournal::_open:
 disabling aio for non-block journal.  Use journal_force_aio to force use of
 aio anyway
 2013-08-06 10:51:36.813999 7f5ba701a780  1 journal _open
 /var/lib/ceph/tmp/mnt.i2NK47/journal fd 10: 0 bytes, block size 4096 bytes,
 directio = 1, aio = 0
 2013-08-06 10:51:36.814035 7f5ba701a780 -1 journal check: ondisk fsid
 ---- doesn't match expected
 5d1beb09-1f80-421d-a88c-57789e2fc33e, invalid (someone else's?) journal
 2013-08-06 10:51:36.814093 7f5ba701a780 -1
 filestore(/var/lib/ceph/tmp/mnt.i2NK47) mkjournal error creating journal on
 /var/lib/ceph/tmp/mnt.i2NK47/journal: (22) Invalid argument
 2013-08-06 10:51:36.814125 7f5ba701a780 -1 OSD::mkfs: FileStore::mkfs
 failed with error -22
 2013-08-06 10:51:36.814185 7f5ba701a780 -1  ** ERROR: error creating
 empty object store in /var/lib/ceph/tmp/mnt.i2NK47: (22) Invalid argument

 I have cleaned the disks with dd, zapped them and so forth but this
 always occurs. If doing sdc/sdd first, for example, then sda or whatever
 follows fails with similar errors.

 Does anyone have any insight on this issue?

>>>
>>> Is SELinux disabled?
>>>
>>> t.
>>>
>>>
>>
>>
>> --
>> Joao Pedras
>>
>
>
>
> --
> Joao Pedras
>



-- 
Joao Pedras
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] journal on ssd

[Joao Pedras]

Disregard the udev comment above. Copy/paste mistake. :)


On Wed, Aug 7, 2013 at 4:43 PM, Joao Pedras  wrote:

> The journal devices entries beyond the 2nd (ie. /dev/sdg2) are not created
> under /dev. Basically doing the following addresses the issue:
>
> --- /usr/sbin/ceph-disk 2013-07-25 00:55:41.0 -0700
> +++ /root/ceph-disk 2013-08-07 15:54:17.538542684 -0700
> @@ -857,6 +857,14 @@
>  'settle',
>  ],
>  )
> +subprocess.call(
> +args=[
> +# wait for udev event queue to clear
> +'partx',
> +'-a',
> +   '{journal}'.format(journal=journal)
> +],
> +)
>
>  journal_symlink = '/dev/disk/by-partuuid/{journal_uuid}'.format(
>  journal_uuid=journal_uuid,
>
> This is RHEL 6.4.
>
> Thanks for the help,
>
>
> On Wed, Aug 7, 2013 at 12:48 PM, Joao Pedras  wrote:
>
>> Some more info about this...
>>
>> The subject should have been journal on another device. The issue also
>> occurs if using another disk to hold the journal.
>> If doing something like 'ceph-deploy node:sda:sdk' a subsequent run like
>> 'ceph-deploy:sdb:sdk' will show the error regarding sdb's osd. If doing
>> 'ceph-deply node:sda:sdk node:sdb:sdk node:sdc:sdk [...]' the first 2 osds
>> will be created and launched fine, sdc's and any others won't.
>>
>> Thanks.
>>
>>
>> On Wed, Aug 7, 2013 at 10:55 AM, Joao Pedras  wrote:
>>
>>> Hello Tren,
>>>
>>> It is indeed:
>>>
>>> $> sestatus
>>> SELinux status: disabled
>>>
>>> Thanks,
>>>
>>>
>>> On Wed, Aug 7, 2013 at 9:33 AM, Tren Blackburn wrote:
>>>
 On Tue, Aug 6, 2013 at 11:14 AM, Joao Pedras wrote:

> Greetings all.
>
> I am installing a test cluster using one ssd (/dev/sdg) to hold the
> journals. Ceph's version is 0.61.7 and I am using ceph-deploy obtained 
> from
> ceph's git yesterday. This is on RHEL6.4, fresh install.
>
> When preparing the first 2 drives, sda and sdb, all goes well and the
> journals get created in sdg1 and sdg2:
>
> $> ceph-deploy osd prepare ceph00:sda:sdg ceph00:sdb:sdg
> [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks
> ceph00:/dev/sda:/dev/sdg ceph00:/dev/sdb:/dev/sdg
> [ceph_deploy.osd][DEBUG ] Deploying osd to ceph00
> [ceph_deploy.osd][DEBUG ] Host ceph00 is now ready for osd use.
> [ceph_deploy.osd][DEBUG ] Preparing host ceph00 disk /dev/sda journal
> /dev/sdg activate False
> [ceph_deploy.osd][DEBUG ] Preparing host ceph00 disk /dev/sdb journal
> /dev/sdg activate False
>
> When preparing sdc or any disk after the first 2 I get the following
> in that osd's log but no errors on ceph-deploy:
>
> # tail -f /var/log/ceph/ceph-osd.2.log
> 2013-08-06 10:51:36.655053 7f5ba701a780  0 ceph version 0.61.7
> (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 11596
> 2013-08-06 10:51:36.658671 7f5ba701a780  1
> filestore(/var/lib/ceph/tmp/mnt.i2NK47) mkfs in 
> /var/lib/ceph/tmp/mnt.i2NK47
> 2013-08-06 10:51:36.658697 7f5ba701a780  1
> filestore(/var/lib/ceph/tmp/mnt.i2NK47) mkfs fsid is already set to
> 5d1beb09-1f80-421d-a88c-57789e2fc33e
> 2013-08-06 10:51:36.813783 7f5ba701a780  1
> filestore(/var/lib/ceph/tmp/mnt.i2NK47) leveldb db exists/created
> 2013-08-06 10:51:36.813964 7f5ba701a780 -1 journal FileJournal::_open:
> disabling aio for non-block journal.  Use journal_force_aio to force use 
> of
> aio anyway
> 2013-08-06 10:51:36.813999 7f5ba701a780  1 journal _open
> /var/lib/ceph/tmp/mnt.i2NK47/journal fd 10: 0 bytes, block size 4096 
> bytes,
> directio = 1, aio = 0
> 2013-08-06 10:51:36.814035 7f5ba701a780 -1 journal check: ondisk fsid
> ---- doesn't match expected
> 5d1beb09-1f80-421d-a88c-57789e2fc33e, invalid (someone else's?) journal
> 2013-08-06 10:51:36.814093 7f5ba701a780 -1
> filestore(/var/lib/ceph/tmp/mnt.i2NK47) mkjournal error creating journal 
> on
> /var/lib/ceph/tmp/mnt.i2NK47/journal: (22) Invalid argument
> 2013-08-06 10:51:36.814125 7f5ba701a780 -1 OSD::mkfs: FileStore::mkfs
> failed with error -22
> 2013-08-06 10:51:36.814185 7f5ba701a780 -1  ** ERROR: error creating
> empty object store in /var/lib/ceph/tmp/mnt.i2NK47: (22) Invalid argument
>
> I have cleaned the disks with dd, zapped them and so forth but this
> always occurs. If doing sdc/sdd first, for example, then sda or whatever
> follows fails with similar errors.
>
> Does anyone have any insight on this issue?
>

 Is SELinux disabled?

 t.


>>>
>>>
>>> --
>>> Joao Pedras
>>>
>>
>>
>>
>> --
>> Joao Pedras
>>
>
>
>
> --
> Joao Pedras
>



-- 
Joao Pedras
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/li