[ceph-users] Replace OSD drive without remove/re-add OSD

[Indra Pramana]

Hi,

May I know if it's possible to replace an OSD drive without removing /
re-adding back the OSD? I want to avoid the time and the excessive I/O load
which will happen during the recovery process at the time when:

- the OSD is removed; and
- the OSD is being put back into the cluster.

I read David Zafman's comment on this thread, that we can set "noout", take
OSD "down", replace the drive, and then bring the OSD back "up" and unset
"noout".

http://www.spinics.net/lists/ceph-users/msg05959.html

May I know if it's possible to do this?

- ceph osd set noout
- sudo stop ceph-osd id=12
- Replace the drive, and once done:
- sudo start ceph-osd id=12
- ceph osd unset noout

The cluster was built using ceph-deploy, can we just replace a drive like
that without zapping and preparing the disk using ceph-deploy?

Looking forward to your reply, thank you.

Cheers.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph editable failure domains

[Fabrizio G. Ventola]

Hello everybody,
I'm making some tests with ceph and its editable cluster map and I'm
trying to define a "rack" layer for its hierarchy in this way:

ceph osd tree:

# id weight type name up/down reweight
-1 0.84 root default
-7 0.28 rack rack1
-2 0.14 host cephosd1-dev
0 0.14 osd.0 up 1
-3 0.14 host cephosd2-dev
1 0.14 osd.1 up 1
-8 0.28 rack rack2
-4 0.14 host cephosd3-dev
2 0.14 osd.2 up 1
-5 0.14 host cephosd4-dev
3 0.14 osd.3 up 1
-9 0.28 rack rack3
-6 0.28 host cephosd5-dev
4 0.28 osd.4 up 1

Those are my pools:
pool 0 'data' rep size 3 min_size 2 crush_ruleset 0 object_hash
rjenkins pg_num 333 pgp_num 333 last_change 2545 owner 0
crash_replay_interval 45
pool 1 'metadata' rep size 3 min_size 2 crush_ruleset 1 object_hash
rjenkins pg_num 333 pgp_num 333 last_change 2548 owner 0
pool 2 'rbd' rep size 3 min_size 2 crush_ruleset 2 object_hash
rjenkins pg_num 333 pgp_num 333 last_change 2529 owner 0
pool 4 'pool_01' rep size 3 min_size 2 crush_ruleset 0 object_hash
rjenkins pg_num 333 pgp_num 333 last_change 2542 owner 0

I configured replica 3 for all pools and min_size 2, thus I'm
expecting when I write new data on ceph-fs (through FUSE) or when I
make a new RBD to see the same amount of data on every rack (3 racks,
3 replicas -> 1 replica per rack). But as you can see the third rack
has just one OSD (the first two have two by the way) and should have
the rack1+rack2 amount of data. Instead it has less data than the
other racks (but more than one single OSD of the first two racks).
Where am I wrong?

Thank you in advance,
Fabrizio
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Replace OSD drive without remove/re-add OSD

[Andrey Korolyov]

On 05/02/2014 03:27 PM, Indra Pramana wrote:
> Hi,
> 
> May I know if it's possible to replace an OSD drive without removing /
> re-adding back the OSD? I want to avoid the time and the excessive I/O
> load which will happen during the recovery process at the time when:
> 
> - the OSD is removed; and
> - the OSD is being put back into the cluster.
> 
> I read David Zafman's comment on this thread, that we can set "noout",
> take OSD "down", replace the drive, and then bring the OSD back "up" and
> unset "noout".
> 
> http://www.spinics.net/lists/ceph-users/msg05959.html
> 
> May I know if it's possible to do this?
> 
> - ceph osd set noout
> - sudo stop ceph-osd id=12
> - Replace the drive, and once done:
> - sudo start ceph-osd id=12
> - ceph osd unset noout
> 
> The cluster was built using ceph-deploy, can we just replace a drive
> like that without zapping and preparing the disk using ceph-deploy?
> 

There will be absolutely no quirks except continuous remapping with
peering along entire recovery process. If your cluster may meet this
well, there is absolutely no problem to go through this flow. Otherwise,
in longer out+in flow, there are only two short intensive recalculations
which can be done at the scheduled time, comparing with peering during
remap, which can introduce unnecessary I/O spikes.

> Looking forward to your reply, thank you.
> 
> Cheers.
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Replace OSD drive without remove/re-add OSD

[Henrik Korkuc]

On 2014.05.02 16:01, Andrey Korolyov wrote:
> On 05/02/2014 03:27 PM, Indra Pramana wrote:
>> Hi,
>>
>> May I know if it's possible to replace an OSD drive without removing /
>> re-adding back the OSD? I want to avoid the time and the excessive I/O
>> load which will happen during the recovery process at the time when:
>>
>> - the OSD is removed; and
>> - the OSD is being put back into the cluster.
>>
>> I read David Zafman's comment on this thread, that we can set "noout",
>> take OSD "down", replace the drive, and then bring the OSD back "up" and
>> unset "noout".
>>
>> http://www.spinics.net/lists/ceph-users/msg05959.html
>>
>> May I know if it's possible to do this?
>>
>> - ceph osd set noout
>> - sudo stop ceph-osd id=12
>> - Replace the drive, and once done:
>> - sudo start ceph-osd id=12
>> - ceph osd unset noout
>>
>> The cluster was built using ceph-deploy, can we just replace a drive
>> like that without zapping and preparing the disk using ceph-deploy?
>>
> There will be absolutely no quirks except continuous remapping with
> peering along entire recovery process. If your cluster may meet this
> well, there is absolutely no problem to go through this flow. Otherwise,
> in longer out+in flow, there are only two short intensive recalculations
> which can be done at the scheduled time, comparing with peering during
> remap, which can introduce unnecessary I/O spikes.

I am interested in such replacements too. What about preparing new disk?
I think that ceph-deploy osd prepare does not suit in this case, because
only disk needs to be prepared, not OSD (or is there an option to
prepare disk for specific OSD?)
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Manual emperor monitor installation hangs at ceph-mon --mkfs

[Stefan U. Walter]

Hello,

I have been trying the manual installation method of a ceph monitor (emperor release) with the goal of integrating ceph 
installation with a configuration management utility.

I pretty much followed the docs at

  http://ceph.com/docs/master/install/manual-deployment/

but for me the 'ceph-mon --mkfs' command always hangs indefinitely.

Here is what I do as root on an up-to-date RHEL6.5 system, ready to copy-paste 
on a vanilla installation:

=== SNIP ===
yum clean all

cat >/etc/yum.repos.d/ceph.repo <<__END__
[ceph]
name=Ceph packages for $basearch
baseurl=http://ceph.com/rpm-emperor/rhel6/\$basearch
enabled=1
gpgcheck=1
priority=1
type=rpm-md
gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc

[ceph-noarch]
name=Ceph noarch packages
baseurl=http://ceph.com/rpm-emperor/rhel6/noarch
enabled=1
gpgcheck=1
priority=1
type=rpm-md
gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
__END__

yum install -y snappy leveldb python-argparse gperftools-libs ceph

uuidgen >/etc/ceph/fsid

cat >/etc/ceph/ceph.conf <<__END__
[global]
fsid = `cat /etc/ceph/fsid`
mon initial members = `hostname -s`
mon host = `ip -f inet -o addr show eth0 2>/dev/null | cut -d" " -f 7 | cut -d/ 
-f 1`
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
osd journal size = 1024
filestore xattr use omap = true
osd pool default size = 2
osd pool default min size = 1
osd pool default pg num = 333
osd pool default pgp num = 333
osd crush chooseleaf type = 1
__END__

ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap 
mon 'allow *'
ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --set-uid=0 --cap mon 
'allow *' --cap osd 'allow *' --cap mds 'allow'

ceph-authtool /tmp/ceph.mon.keyring --import-keyring 
/etc/ceph/ceph.client.admin.keyring

monmaptool --create --add `hostname -s` `ip -f inet -o addr show eth0 2>/dev/null | cut -d" " -f 7 | cut -d/ -f 1` 
--fsid `cat /etc/ceph/fsid` /tmp/monmap


mkdir /var/lib/ceph/mon/ceph-`hostname -s`

ceph-mon --mkfs -i `hostname -s` --monmap /tmp/monmap --keyring 
/tmp/ceph.mon.keyring
=== SNIP ===

Any idea how to get past this step would be greatly appreciated. :)


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Migrate system VMs from local storage to CEPH

[Andrija Panic]

Hi.

I was wondering what would be correct way to migrate system VMs
(storage,console,VR) from local storage to CEPH.

I'm on CS 4.2.1 and will be soon updating to 4.3...

Is it enough to just change global setting system.vm.use.local.storage =
true, to FALSE, and then destroy system VMs (cloudstack will recreate them
in 1-2 minutes)

Also how to make sure that system VMs will NOT end up on NFS storage ?

Thanks for any input...

-- 

Andrija Panić
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] help to tune ceph

[Matteo Favaro]

Hi to all,

I need your help to understand how to tune my ceph configuration to achieve 
some performance result.

my installation is built as follow:

5 server with 16gb ram and 8 core
5 client (same machine)

Each computer is connected with a 1gb/s ethernet to the same switch.
The device storage export 20 luns with 7.2TB of capacity.

Every server is connected to a device storace with a FC connection at 4Gb/s  
and export 2 osds.

I always has a healthy ceph cluster up and running before start any test.

I have tried 3 different configuration of storage to achieve the best result.

The test for the performance is always the same: 5 dd from /dev/zero to a file 
the dimension of the dd is 25GB, each dd is run from a different client 
concurrently and the result file is stored intoceph posix mounted folder.

1) first configuration 
every luns is been partitioned with data and journal, every 
/var/lib/caph/osd/[osd folder]/journal  was linked to the partition journal (on 
the same lun).

In this configuration I saw that the san storage was stormed by a lot of iops 
and the performance was very slow, I investigated and I discovered that a 
possible reason could be the journal of every osd,
http://pastebin.com/GdkngHTd

so i decide to change the configuration as follow.

2) second 

Every server use 4 luns one for data and one for journal, changing the link of 
journal dir into the var/lib/caph/osd/[osd fold]/journal. Obviously I have 
flushed and recreated journal before start the cluster again.

again this configuration was not optimal and the performance was a little 
better from the previous experiment but again very far from the nominal 
possibility of the storage... i haven't ssds where to put the journal... so I 
decided to change the configuration again.
http://pastebin.com/VhmAt7Sf

3) third
Every server use a ram disk for journal.

I thought that since it is a test configuration I could use a ram disk that is 
the fastest device that I have and put there the journal... so i did. I launch 
again the same test... but the performance is still the same...

http://pastebin.com/tdAdGGnM


What am I doing wrong?

Could anyone help me to understand what I can do to achieve the best 
performance from this device?

ps. this device with the gpfs filesystem reached 500/600 MB/s in write and 
600/700MB/s in read with ceph only 250/300 in almost all configurations...

pps. i know that with 5 servers and 5 client I will never reach the same 
performance but I will expect to saturate the ethernet bandwidth.

thanks a lot 
Matteo


ceph.conf: http://pastebin.com/ZQsA1i7A
ceph -w during last configuration test: http://pastebin.com/zzJ6atj1
ceph-osd tree: http://pastebin.com/UxzqmSN8


speed during test inboud from server: http://pastebin.com/d8FT8UZE
speed during test outbound from client: http://pastebin.com/2W1DWaPP







---
Matteo Favaro
mail: matteo.favaro (at) cnaf.infn.it
alias: mfavaro (at) cnaf.infn.it



smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph mom help

[Jimmy Lu]

Hello All,

I am using “ceph-deploy” to setup my ceph cluster with 3 nodes. I am getting an 
error when running “sudo ceph-deploy mon create gfsnode5”. Would you someone 
please give me a pointer what the problem is?

Thanks in advance!

-Jimmy


[cuser@gfsnode5 my-cluster]$ sudo ceph-deploy mon create gfsnode5

[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf

[ceph_deploy.cli][INFO  ] Invoked (1.5.1): /usr/bin/ceph-deploy mon create 
gfsnode5

[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts gfsnode5

[ceph_deploy.mon][DEBUG ] detecting platform for host gfsnode5 ...

[gfsnode5][DEBUG ] connected to host: gfsnode5

[gfsnode5][DEBUG ] detect platform information from remote host

[gfsnode5][DEBUG ] detect machine type

[ceph_deploy.mon][INFO  ] distro info: Red Hat Enterprise Linux Server 6.5 
Santiago

[gfsnode5][DEBUG ] determining if provided host has same hostname in remote

[gfsnode5][DEBUG ] get remote short hostname

[gfsnode5][DEBUG ] deploying mon to gfsnode5

[gfsnode5][DEBUG ] get remote short hostname

[gfsnode5][DEBUG ] remote hostname: gfsnode5

[gfsnode5][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf

[gfsnode5][DEBUG ] create the mon path if it does not exist

[gfsnode5][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-gfsnode5/done

[gfsnode5][DEBUG ] done path does not exist: 
/var/lib/ceph/mon/ceph-gfsnode5/done

[gfsnode5][INFO  ] creating keyring file: 
/var/lib/ceph/tmp/ceph-gfsnode5.mon.keyring

[gfsnode5][DEBUG ] create the monitor keyring file

[gfsnode5][INFO  ] Running command: sudo ceph-mon --cluster ceph --mkfs -i 
gfsnode5 --keyring /var/lib/ceph/tmp/ceph-gfsnode5.mon.keyring

[gfsnode5][WARNIN] pthread lock: Invalid argument

[gfsnode5][WARNIN] *** Caught signal (Aborted) **

[gfsnode5][WARNIN]  in thread 7f22e8d737a0

[gfsnode5][WARNIN]  ceph version 0.72.2 
(a913ded2ff138aefb8cb84d347d72164099cfd60)

[gfsnode5][WARNIN]  1: ceph-mon() [0x808571]

[gfsnode5][WARNIN]  2: /lib64/libpthread.so.0() [0x34aec0f710]

[gfsnode5][WARNIN]  3: (gsignal()+0x35) [0x34ae832925]

[gfsnode5][WARNIN]  4: (abort()+0x175) [0x34ae834105]

[gfsnode5][WARNIN]  5: /usr/lib64/libleveldb.so.1() [0x33bf034d71]

[gfsnode5][WARNIN]  6: (leveldb::DBImpl::Get(leveldb::ReadOptions const&, 
leveldb::Slice const&, leveldb::Value*)+0x50) [0x33bf01a120]

[gfsnode5][WARNIN]  7: (LevelDBStore::_get_iterator()+0x41) [0x7773b1]

[gfsnode5][WARNIN]  8: (MonitorDBStore::exists(std::string const&, std::string 
const&)+0x28) [0x5276f8]

[gfsnode5][WARNIN]  9: (Monitor::check_fsid()+0x73) [0x550bb3]

[gfsnode5][WARNIN]  10: (Monitor::mkfs(ceph::buffer::list&)+0x47) [0x55f027]

[gfsnode5][WARNIN]  11: (main()+0x11d7) [0x521f97]

[gfsnode5][WARNIN]  12: (__libc_start_main()+0xfd) [0x34ae81ed1d]

[gfsnode5][WARNIN]  13: ceph-mon() [0x51fa79]

[gfsnode5][WARNIN] 2014-05-02 11:19:46.335854 7f22e8d737a0 -1 *** Caught signal 
(Aborted) **

[gfsnode5][WARNIN]  in thread 7f22e8d737a0

[gfsnode5][WARNIN]

[gfsnode5][WARNIN]  ceph version 0.72.2 
(a913ded2ff138aefb8cb84d347d72164099cfd60)

[gfsnode5][WARNIN]  1: ceph-mon() [0x808571]

[gfsnode5][WARNIN]  2: /lib64/libpthread.so.0() [0x34aec0f710]

[gfsnode5][WARNIN]  3: (gsignal()+0x35) [0x34ae832925]

[gfsnode5][WARNIN]  4: (abort()+0x175) [0x34ae834105]

[gfsnode5][WARNIN]  5: /usr/lib64/libleveldb.so.1() [0x33bf034d71]

[gfsnode5][WARNIN]  6: (leveldb::DBImpl::Get(leveldb::ReadOptions const&, 
leveldb::Slice const&, leveldb::Value*)+0x50) [0x33bf01a120]

[gfsnode5][WARNIN]  7: (LevelDBStore::_get_iterator()+0x41) [0x7773b1]

[gfsnode5][WARNIN]  8: (MonitorDBStore::exists(std::string const&, std::string 
const&)+0x28) [0x5276f8]

[gfsnode5][WARNIN]  9: (Monitor::check_fsid()+0x73) [0x550bb3]

[gfsnode5][WARNIN]  10: (Monitor::mkfs(ceph::buffer::list&)+0x47) [0x55f027]

[gfsnode5][WARNIN]  11: (main()+0x11d7) [0x521f97]

[gfsnode5][WARNIN]  12: (__libc_start_main()+0xfd) [0x34ae81ed1d]

[gfsnode5][WARNIN]  13: ceph-mon() [0x51fa79]

[gfsnode5][WARNIN]  NOTE: a copy of the executable, or `objdump -rdS 
` is needed to interpret this.

[gfsnode5][WARNIN]

[gfsnode5][WARNIN]  0> 2014-05-02 11:19:46.335854 7f22e8d737a0 -1 *** 
Caught signal (Aborted) **

[gfsnode5][WARNIN]  in thread 7f22e8d737a0

[gfsnode5][WARNIN]

[gfsnode5][WARNIN]  ceph version 0.72.2 
(a913ded2ff138aefb8cb84d347d72164099cfd60)

[gfsnode5][WARNIN]  1: ceph-mon() [0x808571]

[gfsnode5][WARNIN]  2: /lib64/libpthread.so.0() [0x34aec0f710]

[gfsnode5][WARNIN]  3: (gsignal()+0x35) [0x34ae832925]

[gfsnode5][WARNIN]  4: (abort()+0x175) [0x34ae834105]

[gfsnode5][WARNIN]  5: /usr/lib64/libleveldb.so.1() [0x33bf034d71]

[gfsnode5][WARNIN]  6: (leveldb::DBImpl::Get(leveldb::ReadOptions const&, 
leveldb::Slice const&, leveldb::Value*)+0x50) [0x33bf01a120]

[gfsnode5][WARNIN]  7: (LevelDBStore::_get_iterator()+0x41) [0x7773b1]

[gfsnode5][WARNIN]  8: (MonitorDBStore::exists(std::string const&, std::string 
const&)+0x28) [0x5276f8]

[gfsnode5][WARNIN]  9:

Re: [ceph-users] ceph mom help

[Jimmy Lu]

This is a possible bug which was resolved. It was due to leveldb version. My 
node is already running on version 1.12.


[root@gfsnode5 my-cluster]# rpm -qa | grep -i leveldb

leveldb-1.12.0-3.el6.x86_64

[root@gfsnode5 my-cluster]#

Thanks,
Jimmy

From: J L mailto:j...@yahoo-inc.com>>
Date: Friday, May 2, 2014 at 11:21 AM
To: "ceph-us...@ceph.com" 
mailto:ceph-us...@ceph.com>>
Subject: ceph mom help

Hello All,

I am using “ceph-deploy” to setup my ceph cluster with 3 nodes. I am getting an 
error when running “sudo ceph-deploy mon create gfsnode5”. Would you someone 
please give me a pointer what the problem is?

Thanks in advance!

-Jimmy


[cuser@gfsnode5 my-cluster]$ sudo ceph-deploy mon create gfsnode5

[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf

[ceph_deploy.cli][INFO  ] Invoked (1.5.1): /usr/bin/ceph-deploy mon create 
gfsnode5

[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts gfsnode5

[ceph_deploy.mon][DEBUG ] detecting platform for host gfsnode5 ...

[gfsnode5][DEBUG ] connected to host: gfsnode5

[gfsnode5][DEBUG ] detect platform information from remote host

[gfsnode5][DEBUG ] detect machine type

[ceph_deploy.mon][INFO  ] distro info: Red Hat Enterprise Linux Server 6.5 
Santiago

[gfsnode5][DEBUG ] determining if provided host has same hostname in remote

[gfsnode5][DEBUG ] get remote short hostname

[gfsnode5][DEBUG ] deploying mon to gfsnode5

[gfsnode5][DEBUG ] get remote short hostname

[gfsnode5][DEBUG ] remote hostname: gfsnode5

[gfsnode5][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf

[gfsnode5][DEBUG ] create the mon path if it does not exist

[gfsnode5][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-gfsnode5/done

[gfsnode5][DEBUG ] done path does not exist: 
/var/lib/ceph/mon/ceph-gfsnode5/done

[gfsnode5][INFO  ] creating keyring file: 
/var/lib/ceph/tmp/ceph-gfsnode5.mon.keyring

[gfsnode5][DEBUG ] create the monitor keyring file

[gfsnode5][INFO  ] Running command: sudo ceph-mon --cluster ceph --mkfs -i 
gfsnode5 --keyring /var/lib/ceph/tmp/ceph-gfsnode5.mon.keyring

[gfsnode5][WARNIN] pthread lock: Invalid argument

[gfsnode5][WARNIN] *** Caught signal (Aborted) **

[gfsnode5][WARNIN]  in thread 7f22e8d737a0

[gfsnode5][WARNIN]  ceph version 0.72.2 
(a913ded2ff138aefb8cb84d347d72164099cfd60)

[gfsnode5][WARNIN]  1: ceph-mon() [0x808571]

[gfsnode5][WARNIN]  2: /lib64/libpthread.so.0() [0x34aec0f710]

[gfsnode5][WARNIN]  3: (gsignal()+0x35) [0x34ae832925]

[gfsnode5][WARNIN]  4: (abort()+0x175) [0x34ae834105]

[gfsnode5][WARNIN]  5: /usr/lib64/libleveldb.so.1() [0x33bf034d71]

[gfsnode5][WARNIN]  6: (leveldb::DBImpl::Get(leveldb::ReadOptions const&, 
leveldb::Slice const&, leveldb::Value*)+0x50) [0x33bf01a120]

[gfsnode5][WARNIN]  7: (LevelDBStore::_get_iterator()+0x41) [0x7773b1]

[gfsnode5][WARNIN]  8: (MonitorDBStore::exists(std::string const&, std::string 
const&)+0x28) [0x5276f8]

[gfsnode5][WARNIN]  9: (Monitor::check_fsid()+0x73) [0x550bb3]

[gfsnode5][WARNIN]  10: (Monitor::mkfs(ceph::buffer::list&)+0x47) [0x55f027]

[gfsnode5][WARNIN]  11: (main()+0x11d7) [0x521f97]

[gfsnode5][WARNIN]  12: (__libc_start_main()+0xfd) [0x34ae81ed1d]

[gfsnode5][WARNIN]  13: ceph-mon() [0x51fa79]

[gfsnode5][WARNIN] 2014-05-02 11:19:46.335854 7f22e8d737a0 -1 *** Caught signal 
(Aborted) **

[gfsnode5][WARNIN]  in thread 7f22e8d737a0

[gfsnode5][WARNIN]

[gfsnode5][WARNIN]  ceph version 0.72.2 
(a913ded2ff138aefb8cb84d347d72164099cfd60)

[gfsnode5][WARNIN]  1: ceph-mon() [0x808571]

[gfsnode5][WARNIN]  2: /lib64/libpthread.so.0() [0x34aec0f710]

[gfsnode5][WARNIN]  3: (gsignal()+0x35) [0x34ae832925]

[gfsnode5][WARNIN]  4: (abort()+0x175) [0x34ae834105]

[gfsnode5][WARNIN]  5: /usr/lib64/libleveldb.so.1() [0x33bf034d71]

[gfsnode5][WARNIN]  6: (leveldb::DBImpl::Get(leveldb::ReadOptions const&, 
leveldb::Slice const&, leveldb::Value*)+0x50) [0x33bf01a120]

[gfsnode5][WARNIN]  7: (LevelDBStore::_get_iterator()+0x41) [0x7773b1]

[gfsnode5][WARNIN]  8: (MonitorDBStore::exists(std::string const&, std::string 
const&)+0x28) [0x5276f8]

[gfsnode5][WARNIN]  9: (Monitor::check_fsid()+0x73) [0x550bb3]

[gfsnode5][WARNIN]  10: (Monitor::mkfs(ceph::buffer::list&)+0x47) [0x55f027]

[gfsnode5][WARNIN]  11: (main()+0x11d7) [0x521f97]

[gfsnode5][WARNIN]  12: (__libc_start_main()+0xfd) [0x34ae81ed1d]

[gfsnode5][WARNIN]  13: ceph-mon() [0x51fa79]

[gfsnode5][WARNIN]  NOTE: a copy of the executable, or `objdump -rdS 
` is needed to interpret this.

[gfsnode5][WARNIN]

[gfsnode5][WARNIN]  0> 2014-05-02 11:19:46.335854 7f22e8d737a0 -1 *** 
Caught signal (Aborted) **

[gfsnode5][WARNIN]  in thread 7f22e8d737a0

[gfsnode5][WARNIN]

[gfsnode5][WARNIN]  ceph version 0.72.2 
(a913ded2ff138aefb8cb84d347d72164099cfd60)

[gfsnode5][WARNIN]  1: ceph-mon() [0x808571]

[gfsnode5][WARNIN]  2: /lib64/libpthread.so.0() [0x34aec0f710]

[gfsnode5][WARNIN]  3: (gsignal()+0x35) [0x34ae832925]

[gfsnode5][WARNIN]  4: (abo

Re: [ceph-users] ceph mom help

[Ian Colle]

That¹s what it looks like ­ I¹d go back to 1.7, sorry about that.

Ian R. Colle
Director of Engineering
Inktank
Delivering the Future of Storage
http://www.linkedin.com/in/ircolle
http://www.twitter.com/ircolle
Cell: +1.303.601.7713
Email: i...@inktank.com

On 5/2/14, 12:30 PM, "Jimmy Lu"  wrote:

> This is a possible bug which was resolved. It was due to leveldb version. My
> node is already running on version 1.12.
> 
> [root@gfsnode5 my-cluster]# rpm -qa | grep -i leveldb
> 
> leveldb-1.12.0-3.el6.x86_64
> 
> [root@gfsnode5 my-cluster]#
> 
> 
> Thanks,
> Jimmy
> 
> From: J L 
> Date: Friday, May 2, 2014 at 11:21 AM
> To: "ceph-us...@ceph.com" 
> Subject: ceph mom help
> 
> Hello All,
> 
> I am using ³ceph-deploy² to setup my ceph cluster with 3 nodes. I am getting
> an error when running ³sudo ceph-deploy mon create gfsnode5². Would you
> someone please give me a pointer what the problem is?
> 
> Thanks in advance!
> 
> -Jimmy
> 
> [cuser@gfsnode5 my-cluster]$ sudo ceph-deploy mon create gfsnode5
> 
> [ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
> 
> [ceph_deploy.cli][INFO  ] Invoked (1.5.1): /usr/bin/ceph-deploy mon create
> gfsnode5
> 
> [ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts gfsnode5
> 
> [ceph_deploy.mon][DEBUG ] detecting platform for host gfsnode5 ...
> 
> [gfsnode5][DEBUG ] connected to host: gfsnode5
> 
> [gfsnode5][DEBUG ] detect platform information from remote host
> 
> [gfsnode5][DEBUG ] detect machine type
> 
> [ceph_deploy.mon][INFO  ] distro info: Red Hat Enterprise Linux Server 6.5
> Santiago
> 
> [gfsnode5][DEBUG ] determining if provided host has same hostname in remote
> 
> [gfsnode5][DEBUG ] get remote short hostname
> 
> [gfsnode5][DEBUG ] deploying mon to gfsnode5
> 
> [gfsnode5][DEBUG ] get remote short hostname
> 
> [gfsnode5][DEBUG ] remote hostname: gfsnode5
> 
> [gfsnode5][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
> 
> [gfsnode5][DEBUG ] create the mon path if it does not exist
> 
> [gfsnode5][DEBUG ] checking for done path:
> /var/lib/ceph/mon/ceph-gfsnode5/done
> 
> [gfsnode5][DEBUG ] done path does not exist:
> /var/lib/ceph/mon/ceph-gfsnode5/done
> 
> [gfsnode5][INFO  ] creating keyring file:
> /var/lib/ceph/tmp/ceph-gfsnode5.mon.keyring
> 
> [gfsnode5][DEBUG ] create the monitor keyring file
> 
> [gfsnode5][INFO  ] Running command: sudo ceph-mon --cluster ceph --mkfs -i
> gfsnode5 --keyring /var/lib/ceph/tmp/ceph-gfsnode5.mon.keyring
> 
> [gfsnode5][WARNIN] pthread lock: Invalid argument
> 
> [gfsnode5][WARNIN] *** Caught signal (Aborted) **
> 
> [gfsnode5][WARNIN]  in thread 7f22e8d737a0
> 
> [gfsnode5][WARNIN]  ceph version 0.72.2
> (a913ded2ff138aefb8cb84d347d72164099cfd60)
> 
> [gfsnode5][WARNIN]  1: ceph-mon() [0x808571]
> 
> [gfsnode5][WARNIN]  2: /lib64/libpthread.so.0() [0x34aec0f710]
> 
> [gfsnode5][WARNIN]  3: (gsignal()+0x35) [0x34ae832925]
> 
> [gfsnode5][WARNIN]  4: (abort()+0x175) [0x34ae834105]
> 
> [gfsnode5][WARNIN]  5: /usr/lib64/libleveldb.so.1() [0x33bf034d71]
> 
> [gfsnode5][WARNIN]  6: (leveldb::DBImpl::Get(leveldb::ReadOptions const&,
> leveldb::Slice const&, leveldb::Value*)+0x50) [0x33bf01a120]
> 
> [gfsnode5][WARNIN]  7: (LevelDBStore::_get_iterator()+0x41) [0x7773b1]
> 
> [gfsnode5][WARNIN]  8: (MonitorDBStore::exists(std::string const&, std::string
> const&)+0x28) [0x5276f8]
> 
> [gfsnode5][WARNIN]  9: (Monitor::check_fsid()+0x73) [0x550bb3]
> 
> [gfsnode5][WARNIN]  10: (Monitor::mkfs(ceph::buffer::list&)+0x47) [0x55f027]
> 
> [gfsnode5][WARNIN]  11: (main()+0x11d7) [0x521f97]
> 
> [gfsnode5][WARNIN]  12: (__libc_start_main()+0xfd) [0x34ae81ed1d]
> 
> [gfsnode5][WARNIN]  13: ceph-mon() [0x51fa79]
> 
> [gfsnode5][WARNIN] 2014-05-02 11:19:46.335854 7f22e8d737a0 -1 *** Caught
> signal (Aborted) **
> 
> [gfsnode5][WARNIN]  in thread 7f22e8d737a0
> 
> [gfsnode5][WARNIN]
> 
> [gfsnode5][WARNIN]  ceph version 0.72.2
> (a913ded2ff138aefb8cb84d347d72164099cfd60)
> 
> [gfsnode5][WARNIN]  1: ceph-mon() [0x808571]
> 
> [gfsnode5][WARNIN]  2: /lib64/libpthread.so.0() [0x34aec0f710]
> 
> [gfsnode5][WARNIN]  3: (gsignal()+0x35) [0x34ae832925]
> 
> [gfsnode5][WARNIN]  4: (abort()+0x175) [0x34ae834105]
> 
> [gfsnode5][WARNIN]  5: /usr/lib64/libleveldb.so.1() [0x33bf034d71]
> 
> [gfsnode5][WARNIN]  6: (leveldb::DBImpl::Get(leveldb::ReadOptions const&,
> leveldb::Slice const&, leveldb::Value*)+0x50) [0x33bf01a120]
> 
> [gfsnode5][WARNIN]  7: (LevelDBStore::_get_iterator()+0x41) [0x7773b1]
> 
> [gfsnode5][WARNIN]  8: (MonitorDBStore::exists(std::string const&, std::string
> const&)+0x28) [0x5276f8]
> 
> [gfsnode5][WARNIN]  9: (Monitor::check_fsid()+0x73) [0x550bb3]
> 
> [gfsnode5][WARNIN]  10: (Monitor::mkfs(ceph::buffer::list&)+0x47) [0x55f027]
> 
> [gfsnode5][WARNIN]  11: (main()+0x11d7) [0x521f97]
> 
> [gfsnode5][WARNIN]  12: (__libc_start_main()+0xfd) [0x34ae81ed1d]
> 
> [gfsnode5][WARNIN]  13: ceph-mon() [0x51fa79]
> 
> [gfsnode5][WARNIN]  NOTE: a copy of the executable, or `objdump 

[ceph-users] Rados Gateway pagination

[Fabricio Archanjo]

Hi All,

Someone knows if the rados-gw does have pagination support on S3 API?

I'm using S3 Object to browse my structure, then it's too slow to list one
bucket that have too much objects.


Thanks,

Fabricio
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph mom help

[Jimmy Lu]

I was able to dig up an archive of an IRC chat from Sage. The suggestion from 
the chat was to downgrade leveldb from 1.12 to 1.7.0. After the downgrade, I 
was able to run .

-Jimmy

From: J L mailto:j...@yahoo-inc.com>>
Date: Friday, May 2, 2014 at 11:30 AM
To: "ceph-us...@ceph.com" 
mailto:ceph-us...@ceph.com>>
Subject: Re: ceph mom help

This is a possible bug which was resolved. It was due to leveldb version. My 
node is already running on version 1.12.


[root@gfsnode5 my-cluster]# rpm -qa | grep -i leveldb

leveldb-1.12.0-3.el6.x86_64

[root@gfsnode5 my-cluster]#

Thanks,
Jimmy

From: J L mailto:j...@yahoo-inc.com>>
Date: Friday, May 2, 2014 at 11:21 AM
To: "ceph-us...@ceph.com" 
mailto:ceph-us...@ceph.com>>
Subject: ceph mom help

Hello All,

I am using “ceph-deploy” to setup my ceph cluster with 3 nodes. I am getting an 
error when running “sudo ceph-deploy mon create gfsnode5”. Would you someone 
please give me a pointer what the problem is?

Thanks in advance!

-Jimmy


[cuser@gfsnode5 my-cluster]$ sudo ceph-deploy mon create gfsnode5

[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf

[ceph_deploy.cli][INFO  ] Invoked (1.5.1): /usr/bin/ceph-deploy mon create 
gfsnode5

[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts gfsnode5

[ceph_deploy.mon][DEBUG ] detecting platform for host gfsnode5 ...

[gfsnode5][DEBUG ] connected to host: gfsnode5

[gfsnode5][DEBUG ] detect platform information from remote host

[gfsnode5][DEBUG ] detect machine type

[ceph_deploy.mon][INFO  ] distro info: Red Hat Enterprise Linux Server 6.5 
Santiago

[gfsnode5][DEBUG ] determining if provided host has same hostname in remote

[gfsnode5][DEBUG ] get remote short hostname

[gfsnode5][DEBUG ] deploying mon to gfsnode5

[gfsnode5][DEBUG ] get remote short hostname

[gfsnode5][DEBUG ] remote hostname: gfsnode5

[gfsnode5][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf

[gfsnode5][DEBUG ] create the mon path if it does not exist

[gfsnode5][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-gfsnode5/done

[gfsnode5][DEBUG ] done path does not exist: 
/var/lib/ceph/mon/ceph-gfsnode5/done

[gfsnode5][INFO  ] creating keyring file: 
/var/lib/ceph/tmp/ceph-gfsnode5.mon.keyring

[gfsnode5][DEBUG ] create the monitor keyring file

[gfsnode5][INFO  ] Running command: sudo ceph-mon --cluster ceph --mkfs -i 
gfsnode5 --keyring /var/lib/ceph/tmp/ceph-gfsnode5.mon.keyring

[gfsnode5][WARNIN] pthread lock: Invalid argument

[gfsnode5][WARNIN] *** Caught signal (Aborted) **

[gfsnode5][WARNIN]  in thread 7f22e8d737a0

[gfsnode5][WARNIN]  ceph version 0.72.2 
(a913ded2ff138aefb8cb84d347d72164099cfd60)

[gfsnode5][WARNIN]  1: ceph-mon() [0x808571]

[gfsnode5][WARNIN]  2: /lib64/libpthread.so.0() [0x34aec0f710]

[gfsnode5][WARNIN]  3: (gsignal()+0x35) [0x34ae832925]

[gfsnode5][WARNIN]  4: (abort()+0x175) [0x34ae834105]

[gfsnode5][WARNIN]  5: /usr/lib64/libleveldb.so.1() [0x33bf034d71]

[gfsnode5][WARNIN]  6: (leveldb::DBImpl::Get(leveldb::ReadOptions const&, 
leveldb::Slice const&, leveldb::Value*)+0x50) [0x33bf01a120]

[gfsnode5][WARNIN]  7: (LevelDBStore::_get_iterator()+0x41) [0x7773b1]

[gfsnode5][WARNIN]  8: (MonitorDBStore::exists(std::string const&, std::string 
const&)+0x28) [0x5276f8]

[gfsnode5][WARNIN]  9: (Monitor::check_fsid()+0x73) [0x550bb3]

[gfsnode5][WARNIN]  10: (Monitor::mkfs(ceph::buffer::list&)+0x47) [0x55f027]

[gfsnode5][WARNIN]  11: (main()+0x11d7) [0x521f97]

[gfsnode5][WARNIN]  12: (__libc_start_main()+0xfd) [0x34ae81ed1d]

[gfsnode5][WARNIN]  13: ceph-mon() [0x51fa79]

[gfsnode5][WARNIN] 2014-05-02 11:19:46.335854 7f22e8d737a0 -1 *** Caught signal 
(Aborted) **

[gfsnode5][WARNIN]  in thread 7f22e8d737a0

[gfsnode5][WARNIN]

[gfsnode5][WARNIN]  ceph version 0.72.2 
(a913ded2ff138aefb8cb84d347d72164099cfd60)

[gfsnode5][WARNIN]  1: ceph-mon() [0x808571]

[gfsnode5][WARNIN]  2: /lib64/libpthread.so.0() [0x34aec0f710]

[gfsnode5][WARNIN]  3: (gsignal()+0x35) [0x34ae832925]

[gfsnode5][WARNIN]  4: (abort()+0x175) [0x34ae834105]

[gfsnode5][WARNIN]  5: /usr/lib64/libleveldb.so.1() [0x33bf034d71]

[gfsnode5][WARNIN]  6: (leveldb::DBImpl::Get(leveldb::ReadOptions const&, 
leveldb::Slice const&, leveldb::Value*)+0x50) [0x33bf01a120]

[gfsnode5][WARNIN]  7: (LevelDBStore::_get_iterator()+0x41) [0x7773b1]

[gfsnode5][WARNIN]  8: (MonitorDBStore::exists(std::string const&, std::string 
const&)+0x28) [0x5276f8]

[gfsnode5][WARNIN]  9: (Monitor::check_fsid()+0x73) [0x550bb3]

[gfsnode5][WARNIN]  10: (Monitor::mkfs(ceph::buffer::list&)+0x47) [0x55f027]

[gfsnode5][WARNIN]  11: (main()+0x11d7) [0x521f97]

[gfsnode5][WARNIN]  12: (__libc_start_main()+0xfd) [0x34ae81ed1d]

[gfsnode5][WARNIN]  13: ceph-mon() [0x51fa79]

[gfsnode5][WARNIN]  NOTE: a copy of the executable, or `objdump -rdS 
` is needed to interpret this.

[gfsnode5][WARNIN]

[gfsnode5][WARNIN]  0> 2014-05-02 11:19:46.335854 7f22e8d737a0 -1 ***

[ceph-users] Ceph User Committee elections

[Loic Dachary]

Hi,

It is time for the first elections of the Ceph User Committee ! I've enjoyed 
the position for the past six months. It is a little time consuming (about 
eight hours a week) but it's also a great opportunity to be in the center of 
the storage (r)evolution. If you're tempted, feel free to add your platform at:

  https://wiki.ceph.com/Community/2014-04_Ceph_User_Committee_Elections

Feedback about how to improve the representation of the Ceph users so our voice 
can be heard is welcome. To address the elephant in the room : yes, RedHat 
acquisition of Inktank makes the Ceph User Committee more important than ever. 

Cheers

-- 
Loïc Dachary, Artisan Logiciel Libre



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph editable failure domains

[Craig Lewis]

On 5/2/14 05:15 , Fabrizio G. Ventola wrote:

Hello everybody,
I'm making some tests with ceph and its editable cluster map and I'm
trying to define a "rack" layer for its hierarchy in this way:

ceph osd tree:

# id weight type name up/down reweight
-1 0.84 root default
-7 0.28 rack rack1
-2 0.14 host cephosd1-dev
0 0.14 osd.0 up 1
-3 0.14 host cephosd2-dev
1 0.14 osd.1 up 1
-8 0.28 rack rack2
-4 0.14 host cephosd3-dev
2 0.14 osd.2 up 1
-5 0.14 host cephosd4-dev
3 0.14 osd.3 up 1
-9 0.28 rack rack3
-6 0.28 host cephosd5-dev
4 0.28 osd.4 up 1

Those are my pools:
pool 0 'data' rep size 3 min_size 2 crush_ruleset 0 object_hash
rjenkins pg_num 333 pgp_num 333 last_change 2545 owner 0
crash_replay_interval 45
pool 1 'metadata' rep size 3 min_size 2 crush_ruleset 1 object_hash
rjenkins pg_num 333 pgp_num 333 last_change 2548 owner 0
pool 2 'rbd' rep size 3 min_size 2 crush_ruleset 2 object_hash
rjenkins pg_num 333 pgp_num 333 last_change 2529 owner 0
pool 4 'pool_01' rep size 3 min_size 2 crush_ruleset 0 object_hash
rjenkins pg_num 333 pgp_num 333 last_change 2542 owner 0

I configured replica 3 for all pools and min_size 2, thus I'm
expecting when I write new data on ceph-fs (through FUSE) or when I
make a new RBD to see the same amount of data on every rack (3 racks,
3 replicas -> 1 replica per rack). But as you can see the third rack
has just one OSD (the first two have two by the way) and should have
the rack1+rack2 amount of data. Instead it has less data than the
other racks (but more than one single OSD of the first two racks).
Where am I wrong?

Thank you in advance,
Fabrizio
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



You also need to edit the crush rules to tell it to choose a leaf from 
each rack, instead of the default host.  If you run

ceph osd crush dump

You'll see that the rules 0, 1, and 2 are operation chooseleaf_firstn, 
type host.  Those rule numbers are referenced in the pool data's 
crush_ruleset above.



This should get you started on editing the crush map:
https://ceph.com/docs/master/rados/operations/crush-map/#editing-a-crush-map

In the rules section of the decompiled map, change your
step chooseleaf firstn 0 type host
to
step chooseleaf firstn 0 type rack


Then compile and set the new crushmap.

A lot of data is going to start moving.  This will give you a chance to 
use your cluster during a heavy recovery operation.



--

*Craig Lewis*
Senior Systems Engineer
Office +1.714.602.1309
Email cle...@centraldesktop.com 

*Central Desktop. Work together in ways you never thought possible.*
Connect with us Website   | Twitter 
  | Facebook 
  | LinkedIn 
  | Blog 



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Replace OSD drive without remove/re-add OSD

[Indra Pramana]

Sorry forgot to cc the list.
 On 3 May 2014 08:00, "Indra Pramana"  wrote:

> Hi Andrey,
>
> I actually wanted to try this (instead of remove and readd OSD) to avoid
> remapping of PGs to other OSDs and the unnecessary I/O load.
>
> Are you saying that doing this will also trigger remapping? I thought it
> will just do recovery to replace missing PGs as a result of the drive
> replacement?
>
> Thank you.
>  On 2 May 2014 21:02, "Andrey Korolyov"  wrote:
>
>> On 05/02/2014 03:27 PM, Indra Pramana wrote:
>> > Hi,
>> >
>> > May I know if it's possible to replace an OSD drive without removing /
>> > re-adding back the OSD? I want to avoid the time and the excessive I/O
>> > load which will happen during the recovery process at the time when:
>> >
>> > - the OSD is removed; and
>> > - the OSD is being put back into the cluster.
>> >
>> > I read David Zafman's comment on this thread, that we can set "noout",
>> > take OSD "down", replace the drive, and then bring the OSD back "up" and
>> > unset "noout".
>> >
>> > http://www.spinics.net/lists/ceph-users/msg05959.html
>> >
>> > May I know if it's possible to do this?
>> >
>> > - ceph osd set noout
>> > - sudo stop ceph-osd id=12
>> > - Replace the drive, and once done:
>> > - sudo start ceph-osd id=12
>> > - ceph osd unset noout
>> >
>> > The cluster was built using ceph-deploy, can we just replace a drive
>> > like that without zapping and preparing the disk using ceph-deploy?
>> >
>>
>> There will be absolutely no quirks except continuous remapping with
>> peering along entire recovery process. If your cluster may meet this
>> well, there is absolutely no problem to go through this flow. Otherwise,
>> in longer out+in flow, there are only two short intensive recalculations
>> which can be done at the scheduled time, comparing with peering during
>> remap, which can introduce unnecessary I/O spikes.
>>
>> > Looking forward to your reply, thank you.
>> >
>> > Cheers.
>> >
>> >
>> > ___
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >
>>
>>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com