[ceph-users] rados gateway pools for users

[Shashank Puntamkar]

I  am planning to use ceph object gateway to store data in ceph
cluster.I need two different users of Rados gateway to store data in
different pools.How can I create and assign different pools to
different ceph object gateway users?
Thanks in advance
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Use case: one-way RADOS "replication" between two clusters by time period

[Anthony Alba]

Hi list,

Can RADOS fulfil the following use case:

I wish to have a radosgw-S3 object store that is "LIVE",
this represents "current" objects of users.

Separated by an air-gap is another radosgw-S3 object store that is "ARCHIVE".

The objects will only be created and manipulated by radosgw.

Periodically, (on the order of 3-6 months), I want to connect the two
clusters and replicate all objects from LIVE to ARCHIVE created from
"time period DDMM1 - DDMM2" or better yet "from
the last timestamp" . This is a "one" way replication and the objects
are transferred only in the LIVE ==> ARCHIVE direction.

Can this be done easily?

Thanks
Anthony
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Replacing a disk: Best practices?

[Udo Lembke]

Am 15.10.2014 22:08, schrieb Iban Cabrillo:
> HI Cephers,
> 
>  I have an other question related to this issue, What would be the
> procedure to restore a server fail (a whole server for example due to a
> mother board trouble with no damage on disk).
> 
> Regards, I 
> 
Hi,
- change serverboard.
- perhaps adapt /etc/udev/rules.d/70-persistent-net.rules (to get the
same devices (eth0/1...) for your network.
boot and wait for resync.

To avoid to much traffic I set noout if a whole server is lost.


Udo
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] mds isn't working anymore after osd's running full

[Jasper Siero]

Hi John,

Thanks I will look into it. Is there already a new Giant release date?

Jasper

Van: john.sp...@inktank.com [john.sp...@inktank.com] namens John Spray 
[john.sp...@redhat.com]
Verzonden: donderdag 16 oktober 2014 12:23
Aan: Jasper Siero
CC: Gregory Farnum; ceph-users
Onderwerp: Re: [ceph-users] mds isn't working anymore after osd's running full

Following up: firefly fix for undump is: https://github.com/ceph/ceph/pull/2734

Jasper: if you still need to try undumping on this existing firefly
cluster, then you can download ceph-mds packages from this
wip-firefly-undump branch from
http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-basic/ref/

Cheers,
John

On Wed, Oct 15, 2014 at 8:15 PM, John Spray  wrote:
> Sadly undump has been broken for quite some time (it was fixed in
> giant as part of creating cephfs-journal-tool).  If there's a one line
> fix for this then it's probably worth putting in firefly since it's a
> long term supported branch -- I'll do that now.
>
> John
>
> On Wed, Oct 15, 2014 at 8:23 AM, Jasper Siero
>  wrote:
>> Hello Greg,
>>
>> The dump and reset of the journal was succesful:
>>
>> [root@th1-mon001 ~]# /usr/bin/ceph-mds -i th1-mon001 --pid-file 
>> /var/run/ceph/mds.th1-mon001.pid -c /etc/ceph/ceph.conf --cluster ceph 
>> --dump-journal 0 journaldumptgho-mon001
>> journal is 9483323613~134215459
>> read 134213311 bytes at offset 9483323613
>> wrote 134213311 bytes at offset 9483323613 to journaldumptgho-mon001
>> NOTE: this is a _sparse_ file; you can
>> $ tar cSzf journaldumptgho-mon001.tgz journaldumptgho-mon001
>>   to efficiently compress it while preserving sparseness.
>>
>> [root@th1-mon001 ~]# /usr/bin/ceph-mds -i th1-mon001 --pid-file 
>> /var/run/ceph/mds.th1-mon001.pid -c /etc/ceph/ceph.conf --cluster ceph 
>> --reset-journal 0
>> old journal was 9483323613~134215459
>> new journal start will be 9621733376 (4194304 bytes past old end)
>> writing journal head
>> writing EResetJournal entry
>> done
>>
>>
>> Undumping the journal was not successful and looking into the error 
>> "client_lock.is_locked()" is showed several times. The mds is not running 
>> when I start the undumping so maybe have forgot something?
>>
>> [root@th1-mon001 ~]# /usr/bin/ceph-mds -i th1-mon001 --pid-file 
>> /var/run/ceph/mds.th1-mon001.pid -c /etc/ceph/ceph.conf --cluster ceph 
>> --undump-journal 0 journaldumptgho-mon001
>> undump journaldumptgho-mon001
>> start 9483323613 len 134213311
>> writing header 200.
>> osdc/Objecter.cc: In function 'ceph_tid_t 
>> Objecter::op_submit(Objecter::Op*)' thread 7fec3e5ad7a0 time 2014-10-15 
>> 09:09:32.020287
>> osdc/Objecter.cc: 1225: FAILED assert(client_lock.is_locked())
>>  ceph version 0.80.5 (38b73c67d375a2552d8ed67843c8a65c2c0feba6)
>>  1: /usr/bin/ceph-mds() [0x80f15e]
>>  2: (Dumper::undump(char const*)+0x65d) [0x56c7ad]
>>  3: (main()+0x1632) [0x569c62]
>>  4: (__libc_start_main()+0xfd) [0x7fec3ca68d5d]
>>  5: /usr/bin/ceph-mds() [0x567d99]
>>  NOTE: a copy of the executable, or `objdump -rdS ` is needed to 
>> interpret this.
>> 2014-10-15 09:09:32.021313 7fec3e5ad7a0 -1 osdc/Objecter.cc: In function 
>> 'ceph_tid_t Objecter::op_submit(Objecter::Op*)' thread 7fec3e5ad7a0 time 
>> 2014-10-15 09:09:32.020287
>> osdc/Objecter.cc: 1225: FAILED assert(client_lock.is_locked())
>>
>>  ceph version 0.80.5 (38b73c67d375a2552d8ed67843c8a65c2c0feba6)
>>  1: /usr/bin/ceph-mds() [0x80f15e]
>>  2: (Dumper::undump(char const*)+0x65d) [0x56c7ad]
>>  3: (main()+0x1632) [0x569c62]
>>  4: (__libc_start_main()+0xfd) [0x7fec3ca68d5d]
>>  5: /usr/bin/ceph-mds() [0x567d99]
>>  NOTE: a copy of the executable, or `objdump -rdS ` is needed to 
>> interpret this.
>>
>>  0> 2014-10-15 09:09:32.021313 7fec3e5ad7a0 -1 osdc/Objecter.cc: In 
>> function 'ceph_tid_t Objecter::op_submit(Objecter::Op*)' thread 7fec3e5ad7a0 
>> time 2014-10-15 09:09:32.020287
>> osdc/Objecter.cc: 1225: FAILED assert(client_lock.is_locked())
>>
>>  ceph version 0.80.5 (38b73c67d375a2552d8ed67843c
>> [root@th1-mon001 ~]# /usr/bin/ceph-mds -i th1-mon001 --p8a65c2c0feba6)
>>  1: /usr/bin/ceph-mds() [0x80f15e]
>>  2: (Dumper::undump(char const*)+0x65d) [0x56c7ad]
>>  3: (main()+0x1632) [0x569c62]
>>  4: (__libc_start_main()+0xfd) [0x7fec3ca68d5d]
>>  5: /usr/bin/ceph-mds() [0x567d99]
>>  NOTE: a copy of the executable, or `objdump -rdS ` is needed to 
>> interpret this.
>>
>> terminate called after throwing an instance of 'ceph::FailedAssertion'
>> *** Caught signal (Aborted) **
>>  in thread 7fec3e5ad7a0
>>  ceph version 0.80.5 (38b73c67d375a2552d8ed67843c8a65c2c0feba6)
>>  1: /usr/bin/ceph-mds() [0x82ef61]
>>  2: (()+0xf710) [0x7fec3d9a6710]
>>  3: (gsignal()+0x35) [0x7fec3ca7c635]
>>  4: (abort()+0x175) [0x7fec3ca7de15]
>>  5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x7fec3d336a5d]
>>  6: (()+0xbcbe6) [0x7fec3d334be6]
>>  7: (()+0xbcc13) [0x7fec3d334c13]
>>  8: (()+0xbcd0e) [0x7fec3d334d0e]
>>  9: (ceph::__ceph_assert

[ceph-users] pool size/min_size does not make any effect on erasure-coded pool, right?

[yuelongguang]

hi,all
 
pool size/min_size does not make any effect on erasure-coded pool,right?
 
thanks___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Replacing a disk: Best practices?

[Iban Cabrillo]

HI Udo,
  Thanks a lot! The resync flag have solved my doubts.

Regards, I

2014-10-16 12:21 GMT+02:00 Udo Lembke :

> Am 15.10.2014 22:08, schrieb Iban Cabrillo:
> > HI Cephers,
> >
> >  I have an other question related to this issue, What would be the
> > procedure to restore a server fail (a whole server for example due to a
> > mother board trouble with no damage on disk).
> >
> > Regards, I
> >
> Hi,
> - change serverboard.
> - perhaps adapt /etc/udev/rules.d/70-persistent-net.rules (to get the
> same devices (eth0/1...) for your network.
> boot and wait for resync.
>
> To avoid to much traffic I set noout if a whole server is lost.
>
>
> Udo
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 

Iban Cabrillo Bartolome
Instituto de Fisica de Cantabria (IFCA)
Santander, Spain
Tel: +34942200969
PGP PUBLIC KEY:
http://pgp.mit.edu/pks/lookup?op=get&search=0xD9DF0B3D6C8C08AC

Bertrand Russell:
*"El problema con el mundo es que los estúpidos están seguros de todo y los
inteligentes están llenos de dudas*"
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] mds isn't working anymore after osd's running full

[John Spray]

Following up: firefly fix for undump is: https://github.com/ceph/ceph/pull/2734

Jasper: if you still need to try undumping on this existing firefly
cluster, then you can download ceph-mds packages from this
wip-firefly-undump branch from
http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-basic/ref/

Cheers,
John

On Wed, Oct 15, 2014 at 8:15 PM, John Spray  wrote:
> Sadly undump has been broken for quite some time (it was fixed in
> giant as part of creating cephfs-journal-tool).  If there's a one line
> fix for this then it's probably worth putting in firefly since it's a
> long term supported branch -- I'll do that now.
>
> John
>
> On Wed, Oct 15, 2014 at 8:23 AM, Jasper Siero
>  wrote:
>> Hello Greg,
>>
>> The dump and reset of the journal was succesful:
>>
>> [root@th1-mon001 ~]# /usr/bin/ceph-mds -i th1-mon001 --pid-file 
>> /var/run/ceph/mds.th1-mon001.pid -c /etc/ceph/ceph.conf --cluster ceph 
>> --dump-journal 0 journaldumptgho-mon001
>> journal is 9483323613~134215459
>> read 134213311 bytes at offset 9483323613
>> wrote 134213311 bytes at offset 9483323613 to journaldumptgho-mon001
>> NOTE: this is a _sparse_ file; you can
>> $ tar cSzf journaldumptgho-mon001.tgz journaldumptgho-mon001
>>   to efficiently compress it while preserving sparseness.
>>
>> [root@th1-mon001 ~]# /usr/bin/ceph-mds -i th1-mon001 --pid-file 
>> /var/run/ceph/mds.th1-mon001.pid -c /etc/ceph/ceph.conf --cluster ceph 
>> --reset-journal 0
>> old journal was 9483323613~134215459
>> new journal start will be 9621733376 (4194304 bytes past old end)
>> writing journal head
>> writing EResetJournal entry
>> done
>>
>>
>> Undumping the journal was not successful and looking into the error 
>> "client_lock.is_locked()" is showed several times. The mds is not running 
>> when I start the undumping so maybe have forgot something?
>>
>> [root@th1-mon001 ~]# /usr/bin/ceph-mds -i th1-mon001 --pid-file 
>> /var/run/ceph/mds.th1-mon001.pid -c /etc/ceph/ceph.conf --cluster ceph 
>> --undump-journal 0 journaldumptgho-mon001
>> undump journaldumptgho-mon001
>> start 9483323613 len 134213311
>> writing header 200.
>> osdc/Objecter.cc: In function 'ceph_tid_t 
>> Objecter::op_submit(Objecter::Op*)' thread 7fec3e5ad7a0 time 2014-10-15 
>> 09:09:32.020287
>> osdc/Objecter.cc: 1225: FAILED assert(client_lock.is_locked())
>>  ceph version 0.80.5 (38b73c67d375a2552d8ed67843c8a65c2c0feba6)
>>  1: /usr/bin/ceph-mds() [0x80f15e]
>>  2: (Dumper::undump(char const*)+0x65d) [0x56c7ad]
>>  3: (main()+0x1632) [0x569c62]
>>  4: (__libc_start_main()+0xfd) [0x7fec3ca68d5d]
>>  5: /usr/bin/ceph-mds() [0x567d99]
>>  NOTE: a copy of the executable, or `objdump -rdS ` is needed to 
>> interpret this.
>> 2014-10-15 09:09:32.021313 7fec3e5ad7a0 -1 osdc/Objecter.cc: In function 
>> 'ceph_tid_t Objecter::op_submit(Objecter::Op*)' thread 7fec3e5ad7a0 time 
>> 2014-10-15 09:09:32.020287
>> osdc/Objecter.cc: 1225: FAILED assert(client_lock.is_locked())
>>
>>  ceph version 0.80.5 (38b73c67d375a2552d8ed67843c8a65c2c0feba6)
>>  1: /usr/bin/ceph-mds() [0x80f15e]
>>  2: (Dumper::undump(char const*)+0x65d) [0x56c7ad]
>>  3: (main()+0x1632) [0x569c62]
>>  4: (__libc_start_main()+0xfd) [0x7fec3ca68d5d]
>>  5: /usr/bin/ceph-mds() [0x567d99]
>>  NOTE: a copy of the executable, or `objdump -rdS ` is needed to 
>> interpret this.
>>
>>  0> 2014-10-15 09:09:32.021313 7fec3e5ad7a0 -1 osdc/Objecter.cc: In 
>> function 'ceph_tid_t Objecter::op_submit(Objecter::Op*)' thread 7fec3e5ad7a0 
>> time 2014-10-15 09:09:32.020287
>> osdc/Objecter.cc: 1225: FAILED assert(client_lock.is_locked())
>>
>>  ceph version 0.80.5 (38b73c67d375a2552d8ed67843c
>> [root@th1-mon001 ~]# /usr/bin/ceph-mds -i th1-mon001 --p8a65c2c0feba6)
>>  1: /usr/bin/ceph-mds() [0x80f15e]
>>  2: (Dumper::undump(char const*)+0x65d) [0x56c7ad]
>>  3: (main()+0x1632) [0x569c62]
>>  4: (__libc_start_main()+0xfd) [0x7fec3ca68d5d]
>>  5: /usr/bin/ceph-mds() [0x567d99]
>>  NOTE: a copy of the executable, or `objdump -rdS ` is needed to 
>> interpret this.
>>
>> terminate called after throwing an instance of 'ceph::FailedAssertion'
>> *** Caught signal (Aborted) **
>>  in thread 7fec3e5ad7a0
>>  ceph version 0.80.5 (38b73c67d375a2552d8ed67843c8a65c2c0feba6)
>>  1: /usr/bin/ceph-mds() [0x82ef61]
>>  2: (()+0xf710) [0x7fec3d9a6710]
>>  3: (gsignal()+0x35) [0x7fec3ca7c635]
>>  4: (abort()+0x175) [0x7fec3ca7de15]
>>  5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x7fec3d336a5d]
>>  6: (()+0xbcbe6) [0x7fec3d334be6]
>>  7: (()+0xbcc13) [0x7fec3d334c13]
>>  8: (()+0xbcd0e) [0x7fec3d334d0e]
>>  9: (ceph::__ceph_assert_fail(char const*, char const*, int, char 
>> const*)+0x7f2) [0x94b812]
>>  10: /usr/bin/ceph-mds() [0x80f15e]
>>  11: (Dumper::undump(char const*)+0x65d) [0x56c7ad]
>>  12: (main()+0x1632) [0x569c62]
>>  13: (__libc_start_main()+0xfd) [0x7fec3ca68d5d]
>>  14: /usr/bin/ceph-mds() [0x567d99]
>> 2014-10-15 09:09:32.024248 7fec3e5ad7a0 -1 *** Caught signal (Aborted) **
>>  in thread 7fec3e5ad7a0
>>

[ceph-users] urgent- object unfound

[Ta Ba Tuan]

Hi eveyone,  I use replicate 3, many unfound object and Ceph very slow.

pg 6.9d8 is active+recovery_wait+degraded+remapped, acting [22,93], 4 
unfound
pg 6.766 is active+recovery_wait+degraded+remapped, acting [21,36], 1 
unfound
pg 6.73f is active+recovery_wait+degraded+remapped, acting [19,84], 2 
unfound
pg 6.63c is active+recovery_wait+degraded+remapped, acting [10,37], 2 
unfound
pg 6.56c is active+recovery_wait+degraded+remapped, acting [124,93], 2 
unfound

pg 6.4d3 is active+recovering+degraded+remapped, acting [33,94], 2 unfound
pg 6.4a5 is active+recovery_wait+degraded+remapped, acting [11,94], 2 
unfound
pg 6.2f9 is active+recovery_wait+degraded+remapped, acting [22,34], 2 
unfound
recovery 535673/52672768 objects degraded (1.017%); 17/17470639 unfound 
(0.000%)


ceph pg map  6.766
osdmap e94990 pg 6.766 (6.766) -> up [49,36,21] acting [21,36]


I can't resolve it. I need data on those objects. Guide me, please!

Thank you!

--
Tuan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph storage pool definition with KVM/libvirt

[Dan Ryder (daryder)]

Hi Dan,



Maybe I misunderstand what you are trying to do, but I think you are trying to 
add your Ceph RBD pool into libvirt as a storage pool?



If so, it's relatively straightforward - here's an example from my setup:





  

  







  



Related libvirt storage pool definition is:





LibvirtStoragePoolName



  volumes















Hope this helps,



Dan Ryder



-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Dan 
Geist
Sent: Wednesday, October 15, 2014 4:37 PM
To: ceph-users@lists.ceph.com
Subject: [ceph-users] Ceph storage pool definition with KVM/libvirt



I'm leveraging Ceph in a vm prototyping environment currently and am having 
issues abstracting my VM definitions from the storage pool (to use a libvirt 
convention).



I'm able to use the rbd support within the disk configuration of individual VMs 
but am struggling to find a good reference for abstracting it to a storage 
pool. How do I pull the source definition from below to the pool definition?







  

  



  

  







  

  

   






Thanks.

Dan



--

Dan Geist dan(@)polter.net

___

ceph-users mailing list

ceph-users@lists.ceph.com

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] urgent- object unfound

[Andrei Mikhailovsky]

Tuan, 

I had a similar behaviour when I've connected the cache pool tier. I resolved 
the issues by restarting all my osds. If your case is the same, try it and see 
if it works. If not, I guess the guys here and on the ceph irc might be able to 
help you. 

Cheers 

Andrei 
- Original Message -

> From: "Ta Ba Tuan" 
> To: ceph-users@lists.ceph.com
> Sent: Thursday, 16 October, 2014 1:36:01 PM
> Subject: [ceph-users] urgent- object unfound

> Hi eveyone, I use replicate 3, many unfound object and Ceph very
> slow.

> pg 6.9d8 is active+recovery_wait+degraded+remapped, acting [22,93], 4
> unfound
> pg 6.766 is active+recovery_wait+degraded+remapped, acting [21,36], 1
> unfound
> pg 6.73f is active+recovery_wait+degraded+remapped, acting [19,84], 2
> unfound
> pg 6.63c is active+recovery_wait+degraded+remapped, acting [10,37], 2
> unfound
> pg 6.56c is active+recovery_wait+degraded+remapped, acting [124,93],
> 2
> unfound
> pg 6.4d3 is active+recovering+degraded+remapped, acting [33,94], 2
> unfound
> pg 6.4a5 is active+recovery_wait+degraded+remapped, acting [11,94], 2
> unfound
> pg 6.2f9 is active+recovery_wait+degraded+remapped, acting [22,34], 2
> unfound
> recovery 535673/52672768 objects degraded (1.017%); 17/17470639
> unfound
> (0.000%)

> ceph pg map 6.766
> osdmap e94990 pg 6.766 (6.766) -> up [49,36,21] acting [21,36]

> I can't resolve it. I need data on those objects. Guide me, please!

> Thank you!

> --
> Tuan
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph storage pool definition with KVM/libvirt

[Dan Geist]

Thanks Dan (Doctor, doctor...)

Correct. I'd like to abstract the details of the rbd storage from the VM 
definitions as much as possible (like not having the monitor IPs/ports 
defined). I plan on experimenting with monitors and so forth on ceph and would 
like to not have to touch every single VM when changes are made. Small mods to 
the storage pool on each hypervisor are not so bad...

In your example, do you still need the "host" definitions in both disk (per VM) 
and source (per pool) stanzas? Also, do you not use cephx for authentication? 
I'd love to have that defined in the pool as well if possible, allowing 
per-hypervisor authentication instead of per-host (not necessarily for 
security, but for less complex managability).

Dan


- Original Message -
From: "Dan Ryder (daryder)" 
To: "Dan Geist" 
Cc: ceph-users@lists.ceph.com
Sent: Thursday, October 16, 2014 8:41:50 AM
Subject: RE: Ceph storage pool definition with KVM/libvirt

Hi Dan,



Maybe I misunderstand what you are trying to do, but I think you are trying to 
add your Ceph RBD pool into libvirt as a storage pool?



If so, it's relatively straightforward - here's an example from my setup:





  

  







  



Related libvirt storage pool definition is:





LibvirtStoragePoolName



  volumes















Hope this helps,



Dan Ryder



-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Dan 
Geist
Sent: Wednesday, October 15, 2014 4:37 PM
To: ceph-users@lists.ceph.com
Subject: [ceph-users] Ceph storage pool definition with KVM/libvirt



I'm leveraging Ceph in a vm prototyping environment currently and am having 
issues abstracting my VM definitions from the storage pool (to use a libvirt 
convention).



I'm able to use the rbd support within the disk configuration of individual VMs 
but am struggling to find a good reference for abstracting it to a storage 
pool. How do I pull the source definition from below to the pool definition?







  

  



  

  







  

  

   






Thanks.

Dan



--

Dan Geist dan(@)polter.net

___

ceph-users mailing list

ceph-users@lists.ceph.com

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
-- 
Dan Geist dan(@)polter.net

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Usage of journal on balance operations

[Emmanuel Lacour]

Hi,


does anyone know if journals are involved when ceph rebalance data (on
osd crash for example) or when replication size is changed?


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph storage pool definition with KVM/libvirt

[Brian Rak]

What I've found is the nicest way of handling this is to add all the 
mons to your ceph.conf file.  The QEMU client will use these if you 
don't define any in the libvirt config.


Similarly, define a libvirt 'secret' and you can use that for auth, so 
you only have one place to change it.  My entire libvirt config (for 
attaching an iso) looks like:



  
  

  
  
  
  
  



On 10/16/2014 9:21 AM, Dan Geist wrote:

Thanks Dan (Doctor, doctor...)

Correct. I'd like to abstract the details of the rbd storage from the VM 
definitions as much as possible (like not having the monitor IPs/ports 
defined). I plan on experimenting with monitors and so forth on ceph and would 
like to not have to touch every single VM when changes are made. Small mods to 
the storage pool on each hypervisor are not so bad...

In your example, do you still need the "host" definitions in both disk (per VM) 
and source (per pool) stanzas? Also, do you not use cephx for authentication? I'd love to 
have that defined in the pool as well if possible, allowing per-hypervisor authentication 
instead of per-host (not necessarily for security, but for less complex managability).

Dan


- Original Message -
From: "Dan Ryder (daryder)" 
To: "Dan Geist" 
Cc: ceph-users@lists.ceph.com
Sent: Thursday, October 16, 2014 8:41:50 AM
Subject: RE: Ceph storage pool definition with KVM/libvirt

Hi Dan,



Maybe I misunderstand what you are trying to do, but I think you are trying to 
add your Ceph RBD pool into libvirt as a storage pool?



If so, it's relatively straightforward - here's an example from my setup:





   

   

 

 

 

   



Related libvirt storage pool definition is:





 LibvirtStoragePoolName

 

   volumes

 

 

 

 







Hope this helps,



Dan Ryder



-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Dan 
Geist
Sent: Wednesday, October 15, 2014 4:37 PM
To: ceph-users@lists.ceph.com
Subject: [ceph-users] Ceph storage pool definition with KVM/libvirt



I'm leveraging Ceph in a vm prototyping environment currently and am having 
issues abstracting my VM definitions from the storage pool (to use a libvirt 
convention).



I'm able to use the rbd support within the disk configuration of individual VMs 
but am struggling to find a good reference for abstracting it to a storage 
pool. How do I pull the source definition from below to the pool definition?







   

   

 

   

   

 

 

 

   

   








Thanks.

Dan



--

Dan Geist dan(@)polter.net

___

ceph-users mailing list

ceph-users@lists.ceph.com

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Usage of journal on balance operations

[Loic Dachary]

Bonjour Emmanuel,

The journals are used for all write operations on an OSD. They will be used if 
an OSD crashes and others need to create new copies of objects.

Cheers

On 16/10/2014 06:51, Emmanuel Lacour wrote:
> 
> Hi,
> 
> 
> does anyone know if journals are involved when ceph rebalance data (on
> osd crash for example) or when replication size is changed?
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

-- 
Loïc Dachary, Artisan Logiciel Libre



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ssh; cannot resolve hostname errors

[Marco Garcês]

The best way to setup SSH is to use a ~/.ssh/config file It solves
a lot of issues!
Example:

~/.ssh/config


Host ceph1 cephosd1
   HostName 192.168.1.10
   User ceph

Host ceph2 cephosd2
   HostName 192.168.1.11
   User ceph


With that you can just do a "ssh ceph1" for example... All other SSH
options are available in the config file.
You can also spread this file with automation tools like ansible.

Marco Garcês
#sysadmin
Maputo - Mozambique

On Thu, Oct 16, 2014 at 1:15 AM, JIten Shah  wrote:
> Please send your /etc/hosts contents here.
>
> --Jiten
>
> On Oct 15, 2014, at 7:27 AM, Support - Avantek 
> wrote:
>
> I may be completely overlooking something here but I keep getting “ssh;
> cannot resolve hostname” when I try to contact my OSD node’s from my monitor
> node. I have set the ipaddress’s of the 3 nodes in /etc/hosts as suggested
> on the website.
>
> Thanks in advance
>
> James
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] the state of cephfs in giant

[Ric Wheeler]

On 10/15/2014 08:43 AM, Amon Ott wrote:

Am 14.10.2014 16:23, schrieb Sage Weil:

On Tue, 14 Oct 2014, Amon Ott wrote:

Am 13.10.2014 20:16, schrieb Sage Weil:

We've been doing a lot of work on CephFS over the past few months. This
is an update on the current state of things as of Giant.

...

* Either the kernel client (kernel 3.17 or later) or userspace (ceph-fuse
   or libcephfs) clients are in good working order.

Thanks for all the work and specially for concentrating on CephFS! We
have been watching and testing for years by now and really hope to
change our Clusters to CephFS soon.

For kernel maintenance reasons, we only want to run longterm stable
kernels. And for performance reasons and because of severe known
problems we want to avoid Fuse. How good are our chances of a stable
system with the kernel client in the latest longterm kernel 3.14? Will
there be further bugfixes or feature backports?

There are important bug fixes missing from 3.14.  IIRC, the EC, cache
tiering, and firefly CRUSH changes aren't there yet either (they landed in
3.15), and that is not appropriate for a stable series.

They can be backported, but no commitment yet on that :)

If the bugfixes are easily identified in one of your Ceph git branches,
I would even try to backport them myself. Still, I would rather see
someone from the Ceph team with deeper knowledge of the code port them.

IMHO, it would be good for Ceph to have stable support in at least the
latest longterm kernel. No need for new features, but bugfixes should be
there.

Amon Ott


Long term support and aggressive, tedious backports are what you go to distro 
vendors for normally - I don't think that it is generally a good practice to 
continually backport anything to stable series kernels that is not a 
bugfix/security issue (or else, the stable branches rapidly just a stale version 
of the upstream tip :)).


Not meant as a commercial for RH, other vendors also do this kind of thing of 
course...


Regards,

Ric

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ssh; cannot resolve hostname errors

[Support - Avantek]

It's ok, I resolved the problem. I made a typo in the hosts file! But thanks 
for the response

James

From: JIten Shah [mailto:jshah2...@me.com]
Sent: 16 October 2014 00:16
To: Support - Avantek
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] ssh; cannot resolve hostname errors

Please send your /etc/hosts contents here.

--Jiten

On Oct 15, 2014, at 7:27 AM, Support - Avantek 
mailto:supp...@avantek.co.uk>> wrote:


I may be completely overlooking something here but I keep getting "ssh; cannot 
resolve hostname" when I try to contact my OSD node's from my monitor node. I 
have set the ipaddress's of the 3 nodes in /etc/hosts as suggested on the 
website.

Thanks in advance

James
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] pool size/min_size does not make any effect on erasure-coded pool, right?

[yuelongguang]

 
hi,all
 
pool size/min_size does not make any effect on erasure-coded pool,right?
and  erasure-coded pool does support rbd?
 
thanks


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] mkcephfs error

[Sakhi Hadebe]

Hi, 

I am deploying a 3 node ceph storagollowing thee cluster for my company, 
following the webinar: http://www.youtube.com/watch?v=R3gnLrsZSno 

I am stuck at formating the osd's and making them ready to mount the 
directories. Below is the error thrown back: 
root@ceph-node1:~# mkcephfs -a -c /etc/ceph/ceph.conf -k ceph.keyring --mkfs 
unrecognized option '--mkfs' 
usage: /sbin/mkcephfs -a -c ceph.conf [-k adminkeyring] [--mkbtrfs] 
   to generate a new ceph cluster on all nodes; for advanced usage see man page 
   ** be careful, this WILL clobber old data; check your ceph.conf carefully ** 

Changing the --mkfs flag to --mkbtrfs results in the error below: 
root@ceph-node1:~# mkcephfs -a -c /etc/ceph/ceph.conf -k ceph.keyring --mkbtrfs 
temp dir is /tmp/mkcephfs.By7pV0aY0W 
preparing monmap in /tmp/mkcephfs.By7pV0aY0W/monmap 
/usr/bin/monmaptool --create --clobber --add a 192.168.56.21:6789 --add b 
192.168.56.22:6789 --add c 192.168.56.23:6789 --print 
/tmp/mkcephfs.By7pV0aY0W/monmap 
/usr/bin/monmaptool: monmap file /tmp/mkcephfs.By7pV0aY0W/monmap 
/usr/bin/monmaptool: generated fsid de3258d8-1a1f-427d-91ca-cbc679f75305 
epoch 0 
fsid de3258d8-1a1f-427d-91ca-cbc679f75305 
last_changed 2014-10-15 09:35:41.950988 
created 2014-10-15 09:35:41.950988 
0: 192.168.56.21:6789/0 mon.a 
1: 192.168.56.22:6789/0 mon.b 
2: 192.168.56.23:6789/0 mon.c 
/usr/bin/monmaptool: writing epoch 0 to /tmp/mkcephfs.By7pV0aY0W/monmap (3 
monitors) 
=== osd.0 ===  
no btrfs devs defined for osd.0 
2014-10-15 09:35:42.183383 7f9a06bca780 must specify '--osd-data=foo' data path 
2014-10-15 09:35:42.192312 7f9a06bca780 usage: ceph-osd -i osdid 
[--osd-data=path] [--osd-journal=path] [--mkfs] [--mkjournal] 
[--convert-filestore] 
2014-10-15 09:35:42.192602 7f9a06bca780--debug_osd N   set debug level 
(e.g. 10) 
--conf/-cRead configuration from the given configuration file 
-d   Run in foreground, log to stderr. 
-f   Run in foreground, log to usual location. 
--id/-i  set ID portion of my name 
--name/-nset name (TYPE.ID) 
--versionshow version and quit 

   --debug_ms N 
set message debug level (e.g. 1) 
failed: '/sbin/mkcephfs -d /tmp/mkcephfs.By7pV0aY0W --init-daemon osd.0' 


Please help 
Regards,
Sakhi Hadebe
Engineer: South African National Research Network (SANReN)Competency Area, 
Meraka, CSIR

Tel:   +27 12 841 2308 
Fax:   +27 12 841 4223 
Cell:  +27 71 331 9622 
Email: shad...@csir.co.za



-- 
This message is subject to the CSIR's copyright terms and conditions, e-mail 
legal notice, and implemented Open Document Format (ODF) standard. 
The full disclaimer details can be found at 
http://www.csir.co.za/disclaimer.html.

This message has been scanned for viruses and dangerous content by MailScanner, 
and is believed to be clean.

Please consider the environment before printing this email.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Error deploying Ceph

[Support - Avantek]

Hi,

I am trying to deploy ceph and I am just getting an runtime error (please see 
attached). Any ideas?

James Rothero


Ceph Deploy error
Description: Ceph Deploy error
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] mkcephfs error

[Wido den Hollander]

On 10/15/2014 09:37 AM, Sakhi Hadebe wrote:
> 
> Hi, 
> 
> I am deploying a 3 node ceph storagollowing thee cluster for my company, 
> following the webinar: http://www.youtube.com/watch?v=R3gnLrsZSno 
> 

I recommend using ceph-deploy. mkcephfs is deprecated and should not be
used anymore.

Wido

> I am stuck at formating the osd's and making them ready to mount the 
> directories. Below is the error thrown back: 
> root@ceph-node1:~# mkcephfs -a -c /etc/ceph/ceph.conf -k ceph.keyring --mkfs 
> unrecognized option '--mkfs' 
> usage: /sbin/mkcephfs -a -c ceph.conf [-k adminkeyring] [--mkbtrfs] 
>to generate a new ceph cluster on all nodes; for advanced usage see man 
> page 
>** be careful, this WILL clobber old data; check your ceph.conf carefully 
> ** 
> 
> Changing the --mkfs flag to --mkbtrfs results in the error below: 
> root@ceph-node1:~# mkcephfs -a -c /etc/ceph/ceph.conf -k ceph.keyring 
> --mkbtrfs 
> temp dir is /tmp/mkcephfs.By7pV0aY0W 
> preparing monmap in /tmp/mkcephfs.By7pV0aY0W/monmap 
> /usr/bin/monmaptool --create --clobber --add a 192.168.56.21:6789 --add b 
> 192.168.56.22:6789 --add c 192.168.56.23:6789 --print 
> /tmp/mkcephfs.By7pV0aY0W/monmap 
> /usr/bin/monmaptool: monmap file /tmp/mkcephfs.By7pV0aY0W/monmap 
> /usr/bin/monmaptool: generated fsid de3258d8-1a1f-427d-91ca-cbc679f75305 
> epoch 0 
> fsid de3258d8-1a1f-427d-91ca-cbc679f75305 
> last_changed 2014-10-15 09:35:41.950988 
> created 2014-10-15 09:35:41.950988 
> 0: 192.168.56.21:6789/0 mon.a 
> 1: 192.168.56.22:6789/0 mon.b 
> 2: 192.168.56.23:6789/0 mon.c 
> /usr/bin/monmaptool: writing epoch 0 to /tmp/mkcephfs.By7pV0aY0W/monmap (3 
> monitors) 
> === osd.0 ===  
> no btrfs devs defined for osd.0 
> 2014-10-15 09:35:42.183383 7f9a06bca780 must specify '--osd-data=foo' data 
> path 
> 2014-10-15 09:35:42.192312 7f9a06bca780 usage: ceph-osd -i osdid 
> [--osd-data=path] [--osd-journal=path] [--mkfs] [--mkjournal] 
> [--convert-filestore] 
> 2014-10-15 09:35:42.192602 7f9a06bca780--debug_osd N   set debug level 
> (e.g. 10) 
> --conf/-cRead configuration from the given configuration file 
> -d   Run in foreground, log to stderr. 
> -f   Run in foreground, log to usual location. 
> --id/-i  set ID portion of my name 
> --name/-nset name (TYPE.ID) 
> --versionshow version and quit 
> 
>--debug_ms N 
> set message debug level (e.g. 1) 
> failed: '/sbin/mkcephfs -d /tmp/mkcephfs.By7pV0aY0W --init-daemon osd.0' 
> 
> 
> Please help 
> Regards,
> Sakhi Hadebe
> Engineer: South African National Research Network (SANReN)Competency Area, 
> Meraka, CSIR
> 
> Tel:   +27 12 841 2308 
> Fax:   +27 12 841 4223 
> Cell:  +27 71 331 9622 
> Email: shad...@csir.co.za
> 
> 
> 
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 


-- 
Wido den Hollander
Ceph consultant and trainer
42on B.V.

Phone: +31 (0)20 700 9902
Skype: contact42on
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Error deploying Ceph

[Loic Dachary]

Hi,

Did you get this error on a freshly installed ubuntu-14.04 ?

[node1monitor][DEBUG ]
[node1monitor][WARNIN] E: Package 'ceph-mds' has no installation candidate
[node1monitor][WARNIN] E: Unable to locate package ceph-fs-common
[node1monitor][ERROR ] RuntimeError: command returned non-zero exit status: 100
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: env 
DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get -q -o 
Dpkg::Options::=--force-confnew --no-install-recommends --assume-yes install -- 
ceph ceph-mds ceph-common ceph-fs-common gdisk

Cheers

On 16/10/2014 05:21, Support - Avantek wrote:
> Hi,
> 
>  
> 
> I am trying to deploy ceph and I am just getting an runtime error (please see 
> attached). Any ideas?
> 
>  
> 
> James Rothero
> 
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

-- 
Loïc Dachary, Artisan Logiciel Libre



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Usage of journal on balance operations

[Emmanuel Lacour]

Le 16/10/2014 16:16, Loic Dachary a écrit :
> Bonjour Emmanuel,
> 
> The journals are used for all write operations on an OSD. They will be used 
> if an OSD crashes and others need to create new copies of objects.
> 
>

thanks Loic, I had doubt on this ;)


-- 
Easter-eggs  Spécialiste GNU/Linux
44-46 rue de l'Ouest  -  75014 Paris  -  France -  Métro Gaité
Phone: +33 (0) 1 43 35 00 37-   Fax: +33 (0) 1 43 35 00 76
mailto:elac...@easter-eggs.com  -   http://www.easter-eggs.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rados gateway pools for users

[Yehuda Sadeh]

See this discussion:

http://comments.gmane.org/gmane.comp.file-systems.ceph.user/4992

Yehuda

On Thu, Oct 16, 2014 at 12:11 AM, Shashank Puntamkar
 wrote:
> I  am planning to use ceph object gateway to store data in ceph
> cluster.I need two different users of Rados gateway to store data in
> different pools.How can I create and assign different pools to
> different ceph object gateway users?
> Thanks in advance
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Radosgw refusing to even attempt to use keystone auth

[lakshmi k s]

Hello Mark - 

Can you please paste your keystone.conf? Also It seems that Icehouse install 
that I have does not have keystone.conf. Do we need to create one? Like I said, 
adding WSGIChunkedRequest On in Keystone.conf did not solve my issue.

Thanks,
Lakshmi.
On Wednesday, October 15, 2014 10:17 PM, lakshmi k s  wrote:
 


Has anyone seen this issue? Appreciate your time.


On Wednesday, October 15, 2014 4:50 PM, lakshmi k s  wrote:
 


I still think that there is problem with the way radosgw is setup. Two things I 
want to point out - 

1. rgw keystone url - If this flag is under radosgw section of ceph.conf file, 
I do not see the packets being exchanged between keystone and gateway node when 
radosgw is restarted. I tried to run tcpdump on both the nodes. 

2. rgw.keystone url - If this is in global section (which is wrong), I do see 
the packets being exchanged between the nodes when radosgw is restarted. 

I have tried my best to follow the instructions as per 
http://ceph.com/docs/master/radosgw/config/ to setup radosgw. Also with this 
setup, I can still create users using radosgw-admin and make swift v1.0 calls 
from swift-client.

How should I go about resolving this issue? Please help.
Thanks,
Lakshmi.
 




On Wednesday, October 15, 2014 2:58 PM, Mark Kirkwood 
 wrote:
 


On 16/10/14 10:37, Mark Kirkwood wrote:
> On 16/10/14 09:08, lakshmi k s wrote:
>> I am trying to integrate Openstack keystone with
 radosgw. I have
>> followed the instructions as per the link -
>> http://ceph.com/docs/master/radosgw/keystone/. But for some reason,
>> keystone flags under [client.radosgw.gateway] section are not being
>> honored. That means, presence of these flags never attempt
 to use
>> keystone. Hence, any swift v2.0 calls results in 401-Authorization
>> problem. But If I move the keystone url outside under global section, I
>> see that there is initial keystone handshake between keystone and
>> gateway nodes.
>>
>> Please note that swift v1 calls (without using keystone) work great.
>> Any thoughts on how to resolve this problem?
>>
>> ceph.conf
>>
>>
 [global]
>> fsid = f216cbe1-fa49-42ed-b28a-322aa3d48fff
>>
 mon_initial_members = node1
>> mon_host = 192.168.122.182
>> auth_cluster_required = cephx
>> auth_service_required = cephx
>> auth_client_required = cephx
>> filestore_xattr_use_omap = true
>>
>> [client.admin]
>> keyring = /etc/ceph/ceph.client.admin.keyring
>>
>> [client.radosgw.gateway]
>> host = radosgw
>> keyring =
 /etc/ceph/ceph.client.radosgw.keyring
>> rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
>> log file = /var/log/ceph/client.radosgw.gateway.log
>> rgw dns name = radosgw
>>
>> rgw keystone url = http://192.168.122.165:5000
>> rgw keystone admin token = faedf7bc53e3371924e7b3ddb9d13ddd
>> rgw keystone accepted roles = admin Member _member_
>> rgw keystone token cache size = 500
>> rgw keystone revocation interval = 500
>> rgw s3 auth use keystone = true
>> nss db path = /var/ceph/nss
>>
>>
>
> I have managed to to reproduce this:
>
> If I copy your [client.radosgw.gateway] section and amend the obvious
> differences (hostnames and ips, and socket paths), then I too see auth
> failed and no sign of any attempt to use keystone auth logged. Making
> the following change:
>
> - rgw keystone url = http://192.168.122.165:5000
> + rgw keystone url = http://192.168.122.165:35357
>
> makes it work again. I'm guessing it is tied up with with the fact we
> needed to add WSGI Chunked encoding... and we did that only for the
> 35357 keystone virtualhost (I guess I can add it to 5000 too and see if
> that fixes it). I does seem odd that there is no log entry on the rgw...
> but it may be failing before the call gets logged (will look).
>
>


So amending the keystone site config:


 ...
 
WSGIChunkedRequest On
 ...


makes the original keystone url with port 5000 work too.

The logging business is a bit more tricky - I'd copied your 
[client.radosgw.gateway] section which lacks

debug rgw = 20

line, which explains *my* lack of seeing the keystone auth log lines. 
When I add that line I'm seeing the debug auth info (even if I remove 
the WSGI chunking for 5000 and make it fail again).

So
 Lakshmi, can you add the 'WSGIChunkedRequest On' as inidicated, and 
make sure you have the debug line in there and retest?


Regards

Mark___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Error deploying Ceph

[Loic Dachary]

Hi Support,

On 16/10/2014 08:11, Support - Avantek wrote:
> Yes that's right. I've set up a small 32bit arm processor cluster and just 
> experimenting with Ceph running Ubuntu 14.04

I don't think there are arm repositories on ceph.com. Maybe there is a way to 
instruct ceph-deploy to use the native ubuntu repositories ?

Cheers

> 
> Thanks,
> James
> 
> -Original Message-
> From: Loic Dachary [mailto:l...@dachary.org] 
> Sent: 16 October 2014 16:09
> To: Support - Avantek; ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Error deploying Ceph
> 
> Hi,
> 
> Did you get this error on a freshly installed ubuntu-14.04 ?
> 
> [node1monitor][DEBUG ]
> [node1monitor][WARNIN] E: Package 'ceph-mds' has no installation candidate 
> [node1monitor][WARNIN] E: Unable to locate package ceph-fs-common 
> [node1monitor][ERROR ] RuntimeError: command returned non-zero exit status: 
> 100 [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: env 
> DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get -q -o 
> Dpkg::Options::=--force-confnew --no-install-recommends --assume-yes install 
> -- ceph ceph-mds ceph-common ceph-fs-common gdisk
> 
> Cheers
> 
> On 16/10/2014 05:21, Support - Avantek wrote:
>> Hi,
>>
>>  
>>
>> I am trying to deploy ceph and I am just getting an runtime error (please 
>> see attached). Any ideas?
>>
>>  
>>
>> James Rothero
>>
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> 
> --
> Loïc Dachary, Artisan Logiciel Libre
> 

-- 
Loïc Dachary, Artisan Logiciel Libre



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Performance doesn't scale well on a full ssd cluster.

[Mark Wu]

Hi list,

During my test, I found ceph doesn't scale as I expected on a 30 osds
cluster.
The following is the information of my setup:
HW configuration:
   15 Dell R720 servers, and each server has:
  Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, 20 cores and hyper-thread
enabled.
  128GB memory
  two Intel 3500 SSD disks, connected with MegaRAID SAS 2208
controller, each disk is configured as raid0 separately.
  bonding with two 10GbE nics, used for both the public network and
cluster network.

SW configuration:
   OS CentOS 6.5, Kernel 3.17,  Ceph 0.86
   XFS as file system for data.
   each SSD disk has two partitions, one is osd data and the other is osd
journal.
   the pool has 2048 pgs. 2 replicas.
   5 monitors running on 5 of the 15 servers.
   Ceph configuration (in memory debugging options are disabled)

[osd]
osd data = /var/lib/ceph/osd/$cluster-$id
osd journal = /var/lib/ceph/osd/$cluster-$id/journal
osd mkfs type = xfs
osd mkfs options xfs = -f -i size=2048
osd mount options xfs = rw,noatime,logbsize=256k,delaylog
osd journal size = 20480
osd mon heartbeat interval = 30 # Performance tuning filestore
osd_max_backfills = 10
osd_recovery_max_active = 15
merge threshold = 40
filestore split multiple = 8
filestore fd cache size = 1024
osd op threads = 64 # Recovery tuning osd recovery max active = 1 osd max
backfills = 1
osd recovery op priority = 1
throttler perf counter = false
osd enable op tracker = false
filestore_queue_max_ops = 5000
filestore_queue_committing_max_ops = 5000
journal_max_write_entries = 1000
journal_queue_max_ops = 5000
objecter_inflight_ops = 8192


  When I test with 7 servers (14 osds),  the maximum iops of 4k random
write I saw is 17k on single volume and 44k on the whole cluster.
I expected the number of 30 osds cluster could approximate 90k. But
unfornately,  I found that with 30 osds, it almost provides the performce
as 14 osds, even worse sometime. I checked the iostat output on all the
nodes, which have similar numbers. It's well distributed but disk
utilization is low.
In the test with 14 osds, I can see higher utilization of disk (80%~90%).
So do you have any tunning suggestion to improve the performace with 30
osds?
Any feedback is appreciated.


iostat output:
Device: rrqm/s   wrqm/s r/s w/s   rsec/s   wsec/s avgrq-sz
avgqu-sz   await  svctm  %util
sda   0.00 0.000.000.00 0.00 0.00 0.00
0.000.00   0.00   0.00
sdb   0.0088.500.00 5188.00 0.00 93397.0018.00
0.900.17   0.09  47.85
sdc   0.00   443.500.00 5561.50 0.00 97324.0017.50
4.060.73   0.09  47.90
dm-0  0.00 0.000.000.00 0.00 0.00 0.00
0.000.00   0.00   0.00
dm-1  0.00 0.000.000.00 0.00 0.00 0.00
0.000.00   0.00   0.00

Device: rrqm/s   wrqm/s r/s w/s   rsec/s   wsec/s avgrq-sz
avgqu-sz   await  svctm  %util
sda   0.0017.500.00   28.00 0.00  3948.00   141.00
0.010.29   0.05   0.15
sdb   0.0069.500.00 4932.00 0.00 87067.5017.65
2.270.46   0.09  43.45
sdc   0.0069.000.00 4855.50 0.00 105771.5021.78
0.950.20   0.10  46.40
dm-0  0.00 0.000.000.00 0.00 0.00 0.00
0.000.00   0.00   0.00
dm-1  0.00 0.000.00   42.50 0.00  3948.0092.89
0.010.19   0.04   0.15

Device: rrqm/s   wrqm/s r/s w/s   rsec/s   wsec/s avgrq-sz
avgqu-sz   await  svctm  %util
sda   0.0012.000.008.00 0.00   568.0071.00
0.000.12   0.12   0.10
sdb   0.0072.500.00 5046.50 0.00 113198.5022.43
1.090.22   0.10  51.40
sdc   0.0072.500.00 4912.00 0.00 91204.5018.57
2.250.46   0.09  43.60
dm-0  0.00 0.000.000.00 0.00 0.00 0.00
0.000.00   0.00   0.00
dm-1  0.00 0.000.00   18.00 0.00   568.0031.56
0.000.17   0.06   0.10



Regards,
Mark Wu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.

[Gregory Farnum]

If you're running a single client to drive these tests, that's your
bottleneck. Try running multiple clients and aggregating their numbers.
-Greg

On Thursday, October 16, 2014, Mark Wu  wrote:

> Hi list,
>
> During my test, I found ceph doesn't scale as I expected on a 30 osds
> cluster.
> The following is the information of my setup:
> HW configuration:
>15 Dell R720 servers, and each server has:
>   Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, 20 cores and hyper-thread
> enabled.
>   128GB memory
>   two Intel 3500 SSD disks, connected with MegaRAID SAS 2208
> controller, each disk is configured as raid0 separately.
>   bonding with two 10GbE nics, used for both the public network and
> cluster network.
>
> SW configuration:
>OS CentOS 6.5, Kernel 3.17,  Ceph 0.86
>XFS as file system for data.
>each SSD disk has two partitions, one is osd data and the other is osd
> journal.
>the pool has 2048 pgs. 2 replicas.
>5 monitors running on 5 of the 15 servers.
>Ceph configuration (in memory debugging options are disabled)
>
> [osd]
> osd data = /var/lib/ceph/osd/$cluster-$id
> osd journal = /var/lib/ceph/osd/$cluster-$id/journal
> osd mkfs type = xfs
> osd mkfs options xfs = -f -i size=2048
> osd mount options xfs = rw,noatime,logbsize=256k,delaylog
> osd journal size = 20480
> osd mon heartbeat interval = 30 # Performance tuning filestore
> osd_max_backfills = 10
> osd_recovery_max_active = 15
> merge threshold = 40
> filestore split multiple = 8
> filestore fd cache size = 1024
> osd op threads = 64 # Recovery tuning osd recovery max active = 1 osd max
> backfills = 1
> osd recovery op priority = 1
> throttler perf counter = false
> osd enable op tracker = false
> filestore_queue_max_ops = 5000
> filestore_queue_committing_max_ops = 5000
> journal_max_write_entries = 1000
> journal_queue_max_ops = 5000
> objecter_inflight_ops = 8192
>
>
>   When I test with 7 servers (14 osds),  the maximum iops of 4k random
> write I saw is 17k on single volume and 44k on the whole cluster.
> I expected the number of 30 osds cluster could approximate 90k. But
> unfornately,  I found that with 30 osds, it almost provides the performce
> as 14 osds, even worse sometime. I checked the iostat output on all the
> nodes, which have similar numbers. It's well distributed but disk
> utilization is low.
> In the test with 14 osds, I can see higher utilization of disk (80%~90%).
> So do you have any tunning suggestion to improve the performace with 30
> osds?
> Any feedback is appreciated.
>
>
> iostat output:
> Device: rrqm/s   wrqm/s r/s w/s   rsec/s   wsec/s avgrq-sz
> avgqu-sz   await  svctm  %util
> sda   0.00 0.000.000.00 0.00 0.00 0.00
> 0.000.00   0.00   0.00
> sdb   0.0088.500.00 5188.00 0.00 93397.0018.00
> 0.900.17   0.09  47.85
> sdc   0.00   443.500.00 5561.50 0.00 97324.0017.50
> 4.060.73   0.09  47.90
> dm-0  0.00 0.000.000.00 0.00 0.00 0.00
> 0.000.00   0.00   0.00
> dm-1  0.00 0.000.000.00 0.00 0.00 0.00
> 0.000.00   0.00   0.00
>
> Device: rrqm/s   wrqm/s r/s w/s   rsec/s   wsec/s avgrq-sz
> avgqu-sz   await  svctm  %util
> sda   0.0017.500.00   28.00 0.00  3948.00   141.00
> 0.010.29   0.05   0.15
> sdb   0.0069.500.00 4932.00 0.00 87067.5017.65
> 2.270.46   0.09  43.45
> sdc   0.0069.000.00 4855.50 0.00 105771.50
>  21.78 0.950.20   0.10  46.40
> dm-0  0.00 0.000.000.00 0.00 0.00 0.00
> 0.000.00   0.00   0.00
> dm-1  0.00 0.000.00   42.50 0.00  3948.0092.89
> 0.010.19   0.04   0.15
>
> Device: rrqm/s   wrqm/s r/s w/s   rsec/s   wsec/s avgrq-sz
> avgqu-sz   await  svctm  %util
> sda   0.0012.000.008.00 0.00   568.0071.00
> 0.000.12   0.12   0.10
> sdb   0.0072.500.00 5046.50 0.00 113198.50
>  22.43 1.090.22   0.10  51.40
> sdc   0.0072.500.00 4912.00 0.00 91204.5018.57
> 2.250.46   0.09  43.60
> dm-0  0.00 0.000.000.00 0.00 0.00 0.00
> 0.000.00   0.00   0.00
> dm-1  0.00 0.000.00   18.00 0.00   568.0031.56
> 0.000.17   0.06   0.10
>
>
>
> Regards,
> Mark Wu
>
>

-- 
Software Engineer #42 @ http://inktank.com | http://ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] slow requests - what is causing them?

[Andrei Mikhailovsky]

Hello cephers, 

I've been testing flashcache and enhanceio block device caching for the osds 
and i've noticed i have started getting the slow requests. The caching type 
that I use is ready only, so all writes bypass the caching ssds and go directly 
to osds, just like what it used to be before introducing the caching layer. 
Prior to introducing caching, i rarely had the slow requests. Judging by the 
logs, all slow requests are looking like these: 


2014-10-16 01:09:15.600807 osd.7 192.168.168.200:6836/32031 100 : [WRN] slow 
request 30.999641 seconds old, received at 2014-10-16 01:08:44.601040: 
osd_op(client.36035566.0:16626375 rbd_data.51da686763845e 
.5a15 [set-alloc-hint object_size 4194304 write_size 4194304,write 
2007040~16384] 5.7b16421b snapc c4=[c4] ack+ondisk+write e61892) v4 currently 
waiting for subops from 9 
2014-10-16 01:09:15.600811 osd.7 192.168.168.200:6836/32031 101 : [WRN] slow 
request 30.999581 seconds old, received at 2014-10-16 01:08:44.601100: 
osd_op(client.36035566.0:16626376 rbd_data.51da686763845e 
.5a15 [set-alloc-hint object_size 4194304 write_size 4194304,write 
2039808~16384] 5.7b16421b snapc c4=[c4] ack+ondisk+write e61892) v4 currently 
waiting for subops from 9 
2014-10-16 01:09:16.185530 osd.2 192.168.168.200:6811/31891 76 : [WRN] 20 slow 
requests, 1 included below; oldest blocked for > 57.003961 secs 
2014-10-16 01:09:16.185564 osd.2 192.168.168.200:6811/31891 77 : [WRN] slow 
request 30.098574 seconds old, received at 2014-10-16 01:08:46.086854: 
osd_op(client.38917806.0:3481697 rbd_data.251d05e3db45a54. 
0304 [stat,set-alloc-hint object_size 4194304 write_size 
4194304,write 2732032~8192] 5.e4683bbb ack+ondisk+write e61892) v4 currently 
waiting for subops from 11 
2014-10-16 01:09:16.601020 osd.7 192.168.168.200:6836/32031 102 : [WRN] 16 slow 
requests, 2 included below; oldest blocked for > 43.531516 secs 


In general, I see between 0 and about 2,000 slow request log entries per day. 
On one day I saw over 100k entries, but it only happened once. 

I am struggling to understand what is casing the slow requests? If all the 
writes go the same path as before caching was introduced, how come I am getting 
them? How can I investigate this further? 

Thanks 

Andrei 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Error deploying Ceph

[Ian Colle]

Loic,

You mean these? http://ceph.com/debian/dists/trusty/main/binary-armhf/

Ian 

- Original Message -
From: "Loic Dachary" 
To: "Support - Avantek" 
Cc: "ceph-users" 
Sent: Thursday, October 16, 2014 10:17:29 AM
Subject: Re: [ceph-users] Error deploying Ceph

Hi Support,

On 16/10/2014 08:11, Support - Avantek wrote:
> Yes that's right. I've set up a small 32bit arm processor cluster and just 
> experimenting with Ceph running Ubuntu 14.04

I don't think there are arm repositories on ceph.com. Maybe there is a way to 
instruct ceph-deploy to use the native ubuntu repositories ?

Cheers

> 
> Thanks,
> James
> 
> -Original Message-
> From: Loic Dachary [mailto:l...@dachary.org] 
> Sent: 16 October 2014 16:09
> To: Support - Avantek; ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Error deploying Ceph
> 
> Hi,
> 
> Did you get this error on a freshly installed ubuntu-14.04 ?
> 
> [node1monitor][DEBUG ]
> [node1monitor][WARNIN] E: Package 'ceph-mds' has no installation candidate 
> [node1monitor][WARNIN] E: Unable to locate package ceph-fs-common 
> [node1monitor][ERROR ] RuntimeError: command returned non-zero exit status: 
> 100 [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: env 
> DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get -q -o 
> Dpkg::Options::=--force-confnew --no-install-recommends --assume-yes install 
> -- ceph ceph-mds ceph-common ceph-fs-common gdisk
> 
> Cheers
> 
> On 16/10/2014 05:21, Support - Avantek wrote:
>> Hi,
>>
>>  
>>
>> I am trying to deploy ceph and I am just getting an runtime error (please 
>> see attached). Any ideas?
>>
>>  
>>
>> James Rothero
>>
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> 
> --
> Loïc Dachary, Artisan Logiciel Libre
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.

[Mark Wu]

Thanks for the detailed information. but I am already using fio with rbd
engine. Almost 4 volumes can reach the peak.
2014 年 10 月 17 日 上午 1:03于 wud...@gmail.com写道：

Thanks for the detailed information. but I am already using fio with rbd
engine. Almost 4 volumes can reach the peak.
2014 年 10 月 17 日 上午 12:55于 "Daniel Schwager" 写道：

 Hi Mark,



maybe you will check rbd-enabled fio


http://telekomcloud.github.io/ceph/2014/02/26/ceph-performance-analysis_fio_rbd.html



yum install ceph-devel

git clone git://git.kernel.dk/fio.git

cd fio ; ./configure ; make -j5 ; make install



Setup the number of jobs (==clients) inside fio config to

numjobs=8

for simulating multiple clients.





regards

Danny





my test.fio:



[global]

#logging

#write_iops_log=write_iops_log

#write_bw_log=write_bw_log

#write_lat_log=write_lat_log

ioengine=rbd

clientname=admin

pool=rbd

rbdname=myimage

invalidate=0# mandatory

rw=randwrite

bs=1m

runtime=120

iodepth=8

numjobs=8



time_based

#direct=0





[seq-write]

stonewall

rw=write



#[seq-read]

#stonewall

#rw=read
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.

[Mark Wu]

forgot to cc the list
-- 转发的邮件 --
发件人："Mark Wu" 
日期：2014 年 10 月 17 日 上午 12:51
主题：Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.
收件人："Gregory Farnum" 
抄送：

Thanks for the reply. I am not using single client. Writing 5 rbd volumes
on 3 host can reach the peak. The client is fio and also running on osd
nodes. But there're no bottlenecks on cpu or network. I also tried running
client on two non osd servers, but the same result.
2014 年 10 月 17 日 上午 12:29于 "Gregory Farnum" 写道：

> If you're running a single client to drive these tests, that's your
> bottleneck. Try running multiple clients and aggregating their numbers.
> -Greg
>
> On Thursday, October 16, 2014, Mark Wu  wrote:
>
>> Hi list,
>>
>> During my test, I found ceph doesn't scale as I expected on a 30 osds
>> cluster.
>> The following is the information of my setup:
>> HW configuration:
>>15 Dell R720 servers, and each server has:
>>   Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, 20 cores and
>> hyper-thread enabled.
>>   128GB memory
>>   two Intel 3500 SSD disks, connected with MegaRAID SAS 2208
>> controller, each disk is configured as raid0 separately.
>>   bonding with two 10GbE nics, used for both the public network and
>> cluster network.
>>
>> SW configuration:
>>OS CentOS 6.5, Kernel 3.17,  Ceph 0.86
>>XFS as file system for data.
>>each SSD disk has two partitions, one is osd data and the other is osd
>> journal.
>>the pool has 2048 pgs. 2 replicas.
>>5 monitors running on 5 of the 15 servers.
>>Ceph configuration (in memory debugging options are disabled)
>>
>> [osd]
>> osd data = /var/lib/ceph/osd/$cluster-$id
>> osd journal = /var/lib/ceph/osd/$cluster-$id/journal
>> osd mkfs type = xfs
>> osd mkfs options xfs = -f -i size=2048
>> osd mount options xfs = rw,noatime,logbsize=256k,delaylog
>> osd journal size = 20480
>> osd mon heartbeat interval = 30 # Performance tuning filestore
>> osd_max_backfills = 10
>> osd_recovery_max_active = 15
>> merge threshold = 40
>> filestore split multiple = 8
>> filestore fd cache size = 1024
>> osd op threads = 64 # Recovery tuning osd recovery max active = 1 osd max
>> backfills = 1
>> osd recovery op priority = 1
>> throttler perf counter = false
>> osd enable op tracker = false
>> filestore_queue_max_ops = 5000
>> filestore_queue_committing_max_ops = 5000
>> journal_max_write_entries = 1000
>> journal_queue_max_ops = 5000
>> objecter_inflight_ops = 8192
>>
>>
>>   When I test with 7 servers (14 osds),  the maximum iops of 4k random
>> write I saw is 17k on single volume and 44k on the whole cluster.
>> I expected the number of 30 osds cluster could approximate 90k. But
>> unfornately,  I found that with 30 osds, it almost provides the performce
>> as 14 osds, even worse sometime. I checked the iostat output on all the
>> nodes, which have similar numbers. It's well distributed but disk
>> utilization is low.
>> In the test with 14 osds, I can see higher utilization of disk
>> (80%~90%).  So do you have any tunning suggestion to improve the performace
>> with 30 osds?
>> Any feedback is appreciated.
>>
>>
>> iostat output:
>> Device: rrqm/s   wrqm/s r/s w/s   rsec/s   wsec/s
>> avgrq-sz avgqu-sz   await  svctm  %util
>> sda   0.00 0.000.000.00 0.00 0.00
>> 0.00 0.000.00   0.00   0.00
>> sdb   0.0088.500.00 5188.00 0.00 93397.00
>>  18.00 0.900.17   0.09  47.85
>> sdc   0.00   443.500.00 5561.50 0.00 97324.00
>>  17.50 4.060.73   0.09  47.90
>> dm-0  0.00 0.000.000.00 0.00 0.00
>> 0.00 0.000.00   0.00   0.00
>> dm-1  0.00 0.000.000.00 0.00 0.00
>> 0.00 0.000.00   0.00   0.00
>>
>> Device: rrqm/s   wrqm/s r/s w/s   rsec/s   wsec/s
>> avgrq-sz avgqu-sz   await  svctm  %util
>> sda   0.0017.500.00   28.00 0.00  3948.00
>> 141.00 0.010.29   0.05   0.15
>> sdb   0.0069.500.00 4932.00 0.00 87067.50
>>  17.65 2.270.46   0.09  43.45
>> sdc   0.0069.000.00 4855.50 0.00 105771.50
>>  21.78 0.950.20   0.10  46.40
>> dm-0  0.00 0.000.000.00 0.00 0.00
>> 0.00 0.000.00   0.00   0.00
>> dm-1  0.00 0.000.00   42.50 0.00  3948.00
>>  92.89 0.010.19   0.04   0.15
>>
>> Device: rrqm/s   wrqm/s r/s w/s   rsec/s   wsec/s
>> avgrq-sz avgqu-sz   await  svctm  %util
>> sda   0.0012.000.008.00 0.00   568.00
>>  71.00 0.000.12   0.12   0.10
>> sdb   0.0072.500.00 5046.50 0.00 113198.50
>>  22.43 1.090.22   0.10  51.40
>> sdc   0.0072.500.00 4912.00 0.00 91204.50
>>  18.57 2.250.46   0.09  43.60
>> dm-0  0.00 0.000.000.00 0.00 0.00
>>

Re: [ceph-users] CRUSH depends on host + OSD?

[Chad Seys]

Hi Dan,
  I'd like to decommission a node to reproduce the problem and post enough 
information for you (at least) to understand what is going on.
  Unfortunately I'm a ceph newbie, so I'm not sure what info would be of 
interest before/during the drain.
  Probably the crushmap would be of interest.  Pre-decommision (the 
interesting parts?):

root default {
  id -1   # do not change unnecessarily
  # weight 21.890
  alg straw
  hash 0  # rjenkins1
  item osd01 weight 2.700
  item osd03 weight 3.620
  item osd05 weight 1.350
  item osd06 weight 2.260
  item osd07 weight 2.710
  item osd08 weight 2.030
  item osd09 weight 1.800
  item osd02 weight 1.350
  item osd10 weight 4.070
}

# rules
rule data {
  ruleset 0
  type replicated
  min_size 1
  max_size 10
  step take default
  step chooseleaf firstn 0 type host
  step emit
}

Should I gather anything else?
Chad.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.

[Gregory Farnum]

[Re-added the list.]

I assume you added more clients and checked that it didn't scale past
that? You might look through the list archives; there are a number of
discussions about how and how far you can scale SSD-backed cluster
performance.
Just scanning through the config options you set, you might want to
bump up all the filestore and journal queue values a lot farther.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com


On Thu, Oct 16, 2014 at 9:51 AM, Mark Wu  wrote:
> Thanks for the reply. I am not using single client. Writing 5 rbd volumes on
> 3 host can reach the peak. The client is fio and also running on osd nodes.
> But there're no bottlenecks on cpu or network. I also tried running client
> on two non osd servers, but the same result.
>
> 2014 年 10 月 17 日 上午 12:29于 "Gregory Farnum" 写道：
>
>> If you're running a single client to drive these tests, that's your
>> bottleneck. Try running multiple clients and aggregating their numbers.
>> -Greg
>>
>> On Thursday, October 16, 2014, Mark Wu  wrote:
>>>
>>> Hi list,
>>>
>>> During my test, I found ceph doesn't scale as I expected on a 30 osds
>>> cluster.
>>> The following is the information of my setup:
>>> HW configuration:
>>>15 Dell R720 servers, and each server has:
>>>   Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, 20 cores and
>>> hyper-thread enabled.
>>>   128GB memory
>>>   two Intel 3500 SSD disks, connected with MegaRAID SAS 2208
>>> controller, each disk is configured as raid0 separately.
>>>   bonding with two 10GbE nics, used for both the public network and
>>> cluster network.
>>>
>>> SW configuration:
>>>OS CentOS 6.5, Kernel 3.17,  Ceph 0.86
>>>XFS as file system for data.
>>>each SSD disk has two partitions, one is osd data and the other is osd
>>> journal.
>>>the pool has 2048 pgs. 2 replicas.
>>>5 monitors running on 5 of the 15 servers.
>>>Ceph configuration (in memory debugging options are disabled)
>>>
>>> [osd]
>>> osd data = /var/lib/ceph/osd/$cluster-$id
>>> osd journal = /var/lib/ceph/osd/$cluster-$id/journal
>>> osd mkfs type = xfs
>>> osd mkfs options xfs = -f -i size=2048
>>> osd mount options xfs = rw,noatime,logbsize=256k,delaylog
>>> osd journal size = 20480
>>> osd mon heartbeat interval = 30 # Performance tuning filestore
>>> osd_max_backfills = 10
>>> osd_recovery_max_active = 15
>>> merge threshold = 40
>>> filestore split multiple = 8
>>> filestore fd cache size = 1024
>>> osd op threads = 64 # Recovery tuning osd recovery max active = 1 osd max
>>> backfills = 1
>>> osd recovery op priority = 1
>>> throttler perf counter = false
>>> osd enable op tracker = false
>>> filestore_queue_max_ops = 5000
>>> filestore_queue_committing_max_ops = 5000
>>> journal_max_write_entries = 1000
>>> journal_queue_max_ops = 5000
>>> objecter_inflight_ops = 8192
>>>
>>>
>>>   When I test with 7 servers (14 osds),  the maximum iops of 4k random
>>> write I saw is 17k on single volume and 44k on the whole cluster.
>>> I expected the number of 30 osds cluster could approximate 90k. But
>>> unfornately,  I found that with 30 osds, it almost provides the performce
>>> as 14 osds, even worse sometime. I checked the iostat output on all the
>>> nodes, which have similar numbers. It's well distributed but disk
>>> utilization is low.
>>> In the test with 14 osds, I can see higher utilization of disk (80%~90%).
>>> So do you have any tunning suggestion to improve the performace with 30
>>> osds?
>>> Any feedback is appreciated.
>>>
>>>
>>> iostat output:
>>> Device: rrqm/s   wrqm/s r/s w/s   rsec/s   wsec/s
>>> avgrq-sz avgqu-sz   await  svctm  %util
>>> sda   0.00 0.000.000.00 0.00 0.00
>>> 0.00 0.000.00   0.00   0.00
>>> sdb   0.0088.500.00 5188.00 0.00 93397.00
>>> 18.00 0.900.17   0.09  47.85
>>> sdc   0.00   443.500.00 5561.50 0.00 97324.00
>>> 17.50 4.060.73   0.09  47.90
>>> dm-0  0.00 0.000.000.00 0.00 0.00
>>> 0.00 0.000.00   0.00   0.00
>>> dm-1  0.00 0.000.000.00 0.00 0.00
>>> 0.00 0.000.00   0.00   0.00
>>>
>>> Device: rrqm/s   wrqm/s r/s w/s   rsec/s   wsec/s
>>> avgrq-sz avgqu-sz   await  svctm  %util
>>> sda   0.0017.500.00   28.00 0.00  3948.00
>>> 141.00 0.010.29   0.05   0.15
>>> sdb   0.0069.500.00 4932.00 0.00 87067.50
>>> 17.65 2.270.46   0.09  43.45
>>> sdc   0.0069.000.00 4855.50 0.00 105771.50
>>> 21.78 0.950.20   0.10  46.40
>>> dm-0  0.00 0.000.000.00 0.00 0.00
>>> 0.00 0.000.00   0.00   0.00
>>> dm-1  0.00 0.000.00   42.50 0.00  3948.00
>>> 92.89 0.010.19   0.04   0.15
>>>
>>> Device: rrqm/s   wrqm/s r/s w/s   rsec/s   wsec/s
>>> avgrq-sz avgqu-sz   await  svctm  %uti

Re: [ceph-users] Ceph storage pool definition with KVM/libvirt

[Dan Geist]

Thanks, Brian. That helps a lot. I suspect that wasn't needed if the MON hosts 
were defined within ceph.conf, but hadn't tried it previously. 

To go with the pools quesiton, I'm able to define a pool for my RBD cluster 
(and it obtains storage info, images present, etc). Is there a way to refer to 
the pool definition in the host vm's xml?

For example, is it possible to do something like this:

  
  

  
  


Paraphrasing or course, but can we leverage the contents of the pool 
definitions to abstract them at runtime on VMs or are they purely there for 
generation of the vm settings when a vm is instantiated?

Thanks
Dan

- Original Message -
From: "Brian Rak" 
To: "Dan Geist" , "Dan Ryder (daryder)" 
Cc: ceph-users@lists.ceph.com
Sent: Thursday, October 16, 2014 9:55:40 AM
Subject: Re: [ceph-users] Ceph storage pool definition with KVM/libvirt

What I've found is the nicest way of handling this is to add all the 
mons to your ceph.conf file.  The QEMU client will use these if you 
don't define any in the libvirt config.

Similarly, define a libvirt 'secret' and you can use that for auth, so 
you only have one place to change it.  My entire libvirt config (for 
attaching an iso) looks like:


   
   
 
   
   
   
   
   



On 10/16/2014 9:21 AM, Dan Geist wrote:
> Thanks Dan (Doctor, doctor...)
>
> Correct. I'd like to abstract the details of the rbd storage from the VM 
> definitions as much as possible (like not having the monitor IPs/ports 
> defined). I plan on experimenting with monitors and so forth on ceph and 
> would like to not have to touch every single VM when changes are made. Small 
> mods to the storage pool on each hypervisor are not so bad...
>
> In your example, do you still need the "host" definitions in both disk (per 
> VM) and source (per pool) stanzas? Also, do you not use cephx for 
> authentication? I'd love to have that defined in the pool as well if 
> possible, allowing per-hypervisor authentication instead of per-host (not 
> necessarily for security, but for less complex managability).
>
> Dan
>
>
> - Original Message -
> From: "Dan Ryder (daryder)" 
> To: "Dan Geist" 
> Cc: ceph-users@lists.ceph.com
> Sent: Thursday, October 16, 2014 8:41:50 AM
> Subject: RE: Ceph storage pool definition with KVM/libvirt
>
> Hi Dan,
>
>
>
> Maybe I misunderstand what you are trying to do, but I think you are trying 
> to add your Ceph RBD pool into libvirt as a storage pool?
>
>
>
> If so, it's relatively straightforward - here's an example from my setup:
>
>
>
> 
>
>
>
> name='volumes/volume-f3bcec3d-7daf-4eff-818e-0d8848c120d5'>
>
>  
>
>  
>
>  
>
>
>
>
>
> Related libvirt storage pool definition is:
>
>
>
> 
>
>  LibvirtStoragePoolName
>
>  
>
>volumes
>
>  
>
>  
>
>  
>
>  
>
> 
>
>
>
>
>
> Hope this helps,
>
>
>
> Dan Ryder
>
>
>
> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Dan 
> Geist
> Sent: Wednesday, October 15, 2014 4:37 PM
> To: ceph-users@lists.ceph.com
> Subject: [ceph-users] Ceph storage pool definition with KVM/libvirt
>
>
>
> I'm leveraging Ceph in a vm prototyping environment currently and am having 
> issues abstracting my VM definitions from the storage pool (to use a libvirt 
> convention).
>
>
>
> I'm able to use the rbd support within the disk configuration of individual 
> VMs but am struggling to find a good reference for abstracting it to a 
> storage pool. How do I pull the source definition from below to the pool 
> definition?
>
>
>
>
>
> 
>
>
>
>
>
>  
>
>
>
>
>
>  
>
>  
>
>  
>
>
>
>
>
> function='0x0'/> 
>
>
>
>
>
> Thanks.
>
> Dan
>
>
>
> --
>
> Dan Geist dan(@)polter.net
>
> ___
>
> ceph-users mailing list
>
> ceph-users@lists.ceph.com
>
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
-- 
Dan Geist dan(@)polter.net

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.

[Sebastien Han]

Mark, please read this: 
https://www.mail-archive.com/ceph-users@lists.ceph.com/msg12486.html

On 16 Oct 2014, at 19:19, Mark Wu  wrote:

> 
> Thanks for the detailed information. but I am already using fio with rbd 
> engine. Almost 4 volumes can reach the peak.
> 
> 2014 年 10 月 17 日 上午 1:03于 wud...@gmail.com写道：
> Thanks for the detailed information. but I am already using fio with rbd 
> engine. Almost 4 volumes can reach the peak.
> 
> 2014 年 10 月 17 日 上午 12:55于 "Daniel Schwager" 写道：
> Hi Mark,
> 
>  
> 
> maybe you will check rbd-enabled fio
> 
> 
> http://telekomcloud.github.io/ceph/2014/02/26/ceph-performance-analysis_fio_rbd.html
> 
>  
> 
> yum install ceph-devel
> 
> git clone git://git.kernel.dk/fio.git
> 
> cd fio ; ./configure ; make -j5 ; make install
> 
>  
> 
> Setup the number of jobs (==clients) inside fio config to
> 
> numjobs=8
> 
> for simulating multiple clients.
> 
>  
> 
>  
> 
> regards
> 
> Danny
> 
>  
> 
>  
> 
> my test.fio:
> 
>  
> 
> [global]
> 
> #logging
> 
> #write_iops_log=write_iops_log
> 
> #write_bw_log=write_bw_log
> 
> #write_lat_log=write_lat_log
> 
> ioengine=rbd
> 
> clientname=admin
> 
> pool=rbd
> 
> rbdname=myimage
> 
> invalidate=0# mandatory
> 
> rw=randwrite
> 
> bs=1m
> 
> runtime=120
> 
> iodepth=8
> 
> numjobs=8
> 
>  
> 
> time_based
> 
> #direct=0
> 
>  
> 
>  
> 
> [seq-write]
> 
> stonewall
> 
> rw=write
> 
>  
> 
> #[seq-read]
> 
> #stonewall
> 
> #rw=read
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


Cheers.
 
Sébastien Han 
Cloud Architect 

"Always give 100%. Unless you're giving blood."

Phone: +33 (0)1 49 70 99 72 
Mail: sebastien@enovance.com 
Address : 11 bis, rue Roquépine - 75008 Paris
Web : www.enovance.com - Twitter : @enovance 



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Radosgw refusing to even attempt to use keystone auth

[Mark Kirkwood]

Hi,

While I certainly can (attached) - if your install has keystone running 
it *must* have one. It will be hiding somewhere!


Cheers

Mark

On 17/10/14 05:12, lakshmi k s wrote:

Hello Mark -

Can you please paste your keystone.conf? Also It seems that Icehouse install 
that I have does not have keystone.conf. Do we need to create one? Like I said, 
adding WSGIChunkedRequest On in Keystone.conf did not solve my issue.



Listen 5000
Listen 35357


WSGIDaemonProcess keystone-public processes=5 threads=1 user=stack 
display-name=%{GROUP}
WSGIProcessGroup keystone-public
WSGIScriptAlias / /var/www/keystone/main
WSGIApplicationGroup %{GLOBAL}
WSGIChunkedRequest On
ErrorLogFormat "%{cu}t %M"
ErrorLog /var/log/apache2/keystone.log
CustomLog /var/log/apache2/keystone_access.log combined



WSGIDaemonProcess keystone-admin processes=5 threads=1 user=stack 
display-name=%{GROUP}
WSGIProcessGroup keystone-admin
WSGIScriptAlias / /var/www/keystone/admin
WSGIApplicationGroup %{GLOBAL}
WSGIChunkedRequest On
ErrorLogFormat "%{cu}t %M"
ErrorLog /var/log/apache2/keystone.log
CustomLog /var/log/apache2/keystone_access.log combined


# Workaround for missing path on RHEL6, see
#  https://bugzilla.redhat.com/show_bug.cgi?id=1121019
WSGISocketPrefix /var/run/apache2
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Radosgw refusing to even attempt to use keystone auth

[lakshmi k s]

Thank you Mark. Strangely, Icehouse install that I have didn't seem to have 
one. At least not in /etc/apache2/ sub-directories. Like I said earlier, I can 
make the keystone openstack integration work seamlessly if I move all the 
keystone related flags under global section. Not otherwise. I am still looking 
into this. Appreciate all your help.

Thanks,
Lakshmi.




On Thursday, October 16, 2014 3:17 PM, Mark Kirkwood 
 wrote:
 


Hi,

While I certainly can (attached) - if your install has keystone running 
it *must* have one. It will be hiding somewhere!

Cheers

Mark


On 17/10/14 05:12, lakshmi k s wrote:
> Hello Mark -
>
> Can you please paste your keystone.conf? Also It seems that Icehouse install 
> that I have does not have keystone.conf. Do we need to create one? Like I 
> said, adding WSGIChunkedRequest On in Keystone.conf did not solve my issue.
>___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.

[Shu, Xinxin]

We do observe the same issue on our 12 SSD setup, disable the all log maybe 
helpful.

Cheers,
xinxin

From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Mark Wu
Sent: Friday, October 17, 2014 12:18 AM
To: ceph-users@lists.ceph.com
Subject: [ceph-users] Performance doesn't scale well on a full ssd cluster.

Hi list,

During my test, I found ceph doesn't scale as I expected on a 30 osds cluster.
The following is the information of my setup:
HW configuration:
   15 Dell R720 servers, and each server has:
  Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, 20 cores and hyper-thread 
enabled.
  128GB memory
  two Intel 3500 SSD disks, connected with MegaRAID SAS 2208 controller, 
each disk is configured as raid0 separately.
  bonding with two 10GbE nics, used for both the public network and cluster 
network.

SW configuration:
   OS CentOS 6.5, Kernel 3.17,  Ceph 0.86
   XFS as file system for data.
   each SSD disk has two partitions, one is osd data and the other is osd 
journal.
   the pool has 2048 pgs. 2 replicas.
   5 monitors running on 5 of the 15 servers.
   Ceph configuration (in memory debugging options are disabled)

[osd]
osd data = /var/lib/ceph/osd/$cluster-$id
osd journal = /var/lib/ceph/osd/$cluster-$id/journal
osd mkfs type = xfs
osd mkfs options xfs = -f -i size=2048
osd mount options xfs = rw,noatime,logbsize=256k,delaylog
osd journal size = 20480
osd mon heartbeat interval = 30 # Performance tuning filestore
osd_max_backfills = 10
osd_recovery_max_active = 15
merge threshold = 40
filestore split multiple = 8
filestore fd cache size = 1024
osd op threads = 64 # Recovery tuning osd recovery max active = 1 osd max
backfills = 1
osd recovery op priority = 1
throttler perf counter = false
osd enable op tracker = false
filestore_queue_max_ops = 5000
filestore_queue_committing_max_ops = 5000
journal_max_write_entries = 1000
journal_queue_max_ops = 5000
objecter_inflight_ops = 8192


  When I test with 7 servers (14 osds),  the maximum iops of 4k random write I 
saw is 17k on single volume and 44k on the whole cluster.
I expected the number of 30 osds cluster could approximate 90k. But 
unfornately,  I found that with 30 osds, it almost provides the performce
as 14 osds, even worse sometime. I checked the iostat output on all the nodes, 
which have similar numbers. It's well distributed but disk utilization is low.
In the test with 14 osds, I can see higher utilization of disk (80%~90%).  So 
do you have any tunning suggestion to improve the performace with 30 osds?
Any feedback is appreciated.


iostat output:
Device: rrqm/s   wrqm/s r/s w/s   rsec/s   wsec/s avgrq-sz 
avgqu-sz   await  svctm  %util
sda   0.00 0.000.000.00 0.00 0.00 0.00 
0.000.00   0.00   0.00
sdb   0.0088.500.00 5188.00 0.00 93397.0018.00 
0.900.17   0.09  47.85
sdc   0.00   443.500.00 5561.50 0.00 97324.0017.50 
4.060.73   0.09  47.90
dm-0  0.00 0.000.000.00 0.00 0.00 0.00 
0.000.00   0.00   0.00
dm-1  0.00 0.000.000.00 0.00 0.00 0.00 
0.000.00   0.00   0.00

Device: rrqm/s   wrqm/s r/s w/s   rsec/s   wsec/s avgrq-sz 
avgqu-sz   await  svctm  %util
sda   0.0017.500.00   28.00 0.00  3948.00   141.00 
0.010.29   0.05   0.15
sdb   0.0069.500.00 4932.00 0.00 87067.5017.65 
2.270.46   0.09  43.45
sdc   0.0069.000.00 4855.50 0.00 105771.5021.78 
0.950.20   0.10  46.40
dm-0  0.00 0.000.000.00 0.00 0.00 0.00 
0.000.00   0.00   0.00
dm-1  0.00 0.000.00   42.50 0.00  3948.0092.89 
0.010.19   0.04   0.15

Device: rrqm/s   wrqm/s r/s w/s   rsec/s   wsec/s avgrq-sz 
avgqu-sz   await  svctm  %util
sda   0.0012.000.008.00 0.00   568.0071.00 
0.000.12   0.12   0.10
sdb   0.0072.500.00 5046.50 0.00 113198.5022.43 
1.090.22   0.10  51.40
sdc   0.0072.500.00 4912.00 0.00 91204.5018.57 
2.250.46   0.09  43.60
dm-0  0.00 0.000.000.00 0.00 0.00 0.00 
0.000.00   0.00   0.00
dm-1  0.00 0.000.00   18.00 0.00   568.0031.56 
0.000.17   0.06   0.10



Regards,
Mark Wu

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.

[Christian Balzer]

Hello (Greg in particular),

On Thu, 16 Oct 2014 10:06:58 -0700 Gregory Farnum wrote:

> [Re-added the list.]
> 
> I assume you added more clients and checked that it didn't scale past
> that? You might look through the list archives; there are a number of
> discussions about how and how far you can scale SSD-backed cluster
> performance.

Indeed there are and the first one (while not SSD backed, but close
enough) I remember is by yours truly:
https://www.mail-archive.com/ceph-users@lists.ceph.com/msg09537.html

In which you participated as well.

> Just scanning through the config options you set, you might want to
> bump up all the filestore and journal queue values a lot farther.
>
I did that back then, with little to no effect.

Which brings me to another point.
Only a fraction of these parameters (visible when doing a live config dump)
are documented and while one can guess what they probably do/mean and what
their values denote this is not how it should be.
Especially when you expect people to tune these parameters.

Christian

> -Greg
> Software Engineer #42 @ http://inktank.com | http://ceph.com
> 
> 
> On Thu, Oct 16, 2014 at 9:51 AM, Mark Wu  wrote:
> > Thanks for the reply. I am not using single client. Writing 5 rbd
> > volumes on 3 host can reach the peak. The client is fio and also
> > running on osd nodes. But there're no bottlenecks on cpu or network. I
> > also tried running client on two non osd servers, but the same result.
> >
> > 2014 年 10 月 17 日 上午 12:29于 "Gregory Farnum"
> > 写道：
> >
> >> If you're running a single client to drive these tests, that's your
> >> bottleneck. Try running multiple clients and aggregating their
> >> numbers. -Greg
> >>
> >> On Thursday, October 16, 2014, Mark Wu  wrote:
> >>>
> >>> Hi list,
> >>>
> >>> During my test, I found ceph doesn't scale as I expected on a 30 osds
> >>> cluster.
> >>> The following is the information of my setup:
> >>> HW configuration:
> >>>15 Dell R720 servers, and each server has:
> >>>   Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, 20 cores and
> >>> hyper-thread enabled.
> >>>   128GB memory
> >>>   two Intel 3500 SSD disks, connected with MegaRAID SAS 2208
> >>> controller, each disk is configured as raid0 separately.
> >>>   bonding with two 10GbE nics, used for both the public network
> >>> and cluster network.
> >>>
> >>> SW configuration:
> >>>OS CentOS 6.5, Kernel 3.17,  Ceph 0.86
> >>>XFS as file system for data.
> >>>each SSD disk has two partitions, one is osd data and the other
> >>> is osd journal.
> >>>the pool has 2048 pgs. 2 replicas.
> >>>5 monitors running on 5 of the 15 servers.
> >>>Ceph configuration (in memory debugging options are disabled)
> >>>
> >>> [osd]
> >>> osd data = /var/lib/ceph/osd/$cluster-$id
> >>> osd journal = /var/lib/ceph/osd/$cluster-$id/journal
> >>> osd mkfs type = xfs
> >>> osd mkfs options xfs = -f -i size=2048
> >>> osd mount options xfs = rw,noatime,logbsize=256k,delaylog
> >>> osd journal size = 20480
> >>> osd mon heartbeat interval = 30 # Performance tuning filestore
> >>> osd_max_backfills = 10
> >>> osd_recovery_max_active = 15
> >>> merge threshold = 40
> >>> filestore split multiple = 8
> >>> filestore fd cache size = 1024
> >>> osd op threads = 64 # Recovery tuning osd recovery max active = 1
> >>> osd max backfills = 1
> >>> osd recovery op priority = 1
> >>> throttler perf counter = false
> >>> osd enable op tracker = false
> >>> filestore_queue_max_ops = 5000
> >>> filestore_queue_committing_max_ops = 5000
> >>> journal_max_write_entries = 1000
> >>> journal_queue_max_ops = 5000
> >>> objecter_inflight_ops = 8192
> >>>
> >>>
> >>>   When I test with 7 servers (14 osds),  the maximum iops of 4k
> >>> random write I saw is 17k on single volume and 44k on the whole
> >>> cluster. I expected the number of 30 osds cluster could approximate
> >>> 90k. But unfornately,  I found that with 30 osds, it almost provides
> >>> the performce as 14 osds, even worse sometime. I checked the iostat
> >>> output on all the nodes, which have similar numbers. It's well
> >>> distributed but disk utilization is low.
> >>> In the test with 14 osds, I can see higher utilization of disk
> >>> (80%~90%). So do you have any tunning suggestion to improve the
> >>> performace with 30 osds?
> >>> Any feedback is appreciated.
> >>>
> >>>
> >>> iostat output:
> >>> Device: rrqm/s   wrqm/s r/s w/s   rsec/s   wsec/s
> >>> avgrq-sz avgqu-sz   await  svctm  %util
> >>> sda   0.00 0.000.000.00 0.00 0.00
> >>> 0.00 0.000.00   0.00   0.00
> >>> sdb   0.0088.500.00 5188.00 0.00 93397.00
> >>> 18.00 0.900.17   0.09  47.85
> >>> sdc   0.00   443.500.00 5561.50 0.00 97324.00
> >>> 17.50 4.060.73   0.09  47.90
> >>> dm-0  0.00 0.000.000.00 0.00 0.00
> >>> 0.00 0.000.00   0.00   0.00
> >>> dm-1  0.00 0.00  

[ceph-users] OSD (and probably other settings) not being picked up outside of the [global] section

[Christian Balzer]

Hello,

Consider this rather basic configuration file:
---
[global]
fsid = e6687ef7-54e1-44bd-8072-f9ecab00815
mon_initial_members = ceph-01, comp-01, comp-02
mon_host = 10.0.0.21,10.0.0.5,10.0.0.6
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
mon_osd_downout_subtree_limit = host
public_network = 10.0.0.0/8
osd_pool_default_pg_num = 2048
osd_pool_default_pgp_num = 2048
osd_crush_chooseleaf_type = 1

[osd]
osd_mkfs_type = ext4
osd_mkfs_options_ext4 = -J size=1024 -E lazy_itable_init=0,lazy_journal_init=0
osd_op_threads = 10
osd_scrub_load_threshold = 2.5
filestore_max_sync_interval = 10
---

Let us slide the annoying fact that ceph ignores the pg and pgp settings
when creating the initial pools. 
And that monitors are preferred based on IP address instead of the
sequence they're listed in the config file.

Interestingly ceph-deploy correctly picks up the mkfs_options but why it
fails to choose the mkfs_type as default is beyond me.

The real issue is that the other three OSD setting are NOT picked up by
ceph on startup.
But they sure are when moved to the global section.

Anybody else seeing this (both with 0.80.1 and 0.80.6)?

Regards,

Christian
-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Global OnLine Japan/Fusion Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] [radosgw] object copy implementation

[Blair Bethwaite]

Hi there rgw folks,

Just wondering if the server-side copy operation ties up the radosgw
host to actually proxy the data or if the copy is handled
transparently by rados and the backend OSDs?

-- 
Cheers,
~Blairo
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] monitoring tool for monitoring end-user

[pragya jain]

thanks zhu for your reply.

Regards
Pragya Jain



On Thursday, 16 October 2014 7:08 AM, zhu qiang  
wrote:
 

>
>
>Maybe you can try “ceph df detail”
>and sum the pool’s usage for your end-user
> 
>GLOBAL:
>SIZE  AVAIL RAW USED %RAW USED OBJECTS 
>3726G 3570G 3799M0.10  419 
>POOLS:
>NAME   ID CATEGORY USED  %USED OBJECTS 
> DIRTY READ  WRITE 
>data   0  -0 0 0   
> 0 0 0 
>metadata   1  -0 0 0   
> 0 0 0 
>rbd2  -0 0 0   
> 0 0 0 
>.rgw.root  3  -822   0 3   
> 3 180   3 
>.rgw.control   4  -0 0 8   
> 8 0 0 
>.rgw   5  -9834  0 56  
> 56509   208   
>.rgw.gc6  -0 0 32  
> 3230387 20692 
>.users.uid 7  -5288  0 20  
> 204187  337   
>.users 8  -392   0 13  
> 135419
>.rgw.buckets.index 9  -0 0 23  
> 2316390 1300  
>.log   10 -4458k 0 99  
> 990 18024 
>.rgw.buckets   11 -319M  0 157 
> 157   16074 5836  
>.usage 12 -0 0 8   
> 8 3091  2834  
>.users.email   13 -0 0 0   
> 0 0 0 
>   14 -0 0 0   
> 0 3 27
>and .rgw.* is rgw client used
> 
>From:ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of pragya 
>jain
>Sent: Tuesday, September 09, 2014 3:34 PM
>To: ceph-users@lists.ceph.com
>Subject: Re: [ceph-users] monitoring tool for monitoring end-user
> 
>please somebody reply to clarify it to me.
> 
>Regards
>Pragya Jain
> 
>On Wednesday, 3 September 2014 12:14 PM, pragya jain  
>wrote:
> 
>>hi all!
>> 
>>Is there any monitoring tool for ceph which monitor end-user level usage and 
>>data transfer for ceph object storage service?
>> 
>>Please help me  to know any type of information related to it. 
>> 
>>Regards
>>Pragya Jain
>> 
>
>___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.

[Alexandre DERUMIER]

Hi,
>>Thanks for the detailed information. but I am already using fio with rbd 
>>engine. Almost 4 volumes can reach the peak.

What is your cpu usage of fio-rbd ?
Myself I'm cpu bound on 8cores with around 4iops read 4K.



- Mail original -

De: "Mark Wu" 
À: "Daniel Schwager" 
Cc: ceph-users@lists.ceph.com
Envoyé: Jeudi 16 Octobre 2014 19:19:17
Objet: Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.



Thanks for the detailed information. but I am already using fio with rbd 
engine. Almost 4 volumes can reach the peak.
2014 年 10 月 17 日 上午 1:03于 wud...@gmail.com 写道：



Thanks for the detailed information. but I am already using fio with rbd 
engine. Almost 4 volumes can reach the peak.
2014 年 10 月 17 日 上午 12:55于 "Daniel Schwager" < daniel.schwa...@dtnet.de >写道：





Hi Mark,

maybe you will check rbd-enabled fio
http://telekomcloud.github.io/ceph/2014/02/26/ceph-performance-analysis_fio_rbd.html

yum install ceph-devel
git clone git:// git.kernel.dk/fio.git
cd fio ; ./configure ; make -j5 ; make install

Setup the number of jobs (==clients) inside fio config to
numjobs=8
for simulating multiple clients.


regards
Danny


my test.fio:

[global]
#logging
#write_iops_log=write_iops_log
#write_bw_log=write_bw_log
#write_lat_log=write_lat_log
ioengine=rbd
clientname=admin
pool=rbd
rbdname=myimage
invalidate=0 # mandatory
rw=randwrite
bs=1m
runtime=120
iodepth=8
numjobs=8

time_based
#direct=0


[seq-write]
stonewall
rw=write

#[seq-read]
#stonewall
#rw=read




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com