Re: [ceph-users] How to Configure Cinder to access multiple pools

[Vikrant Verma]

Hi Sébastien,

Thanks for reply!

As per the configuration in the link , now I am able to access multiple
ceph pools through Cinder.

But i have one question - As per the link
http://ceph.com/docs/master/rbd/rbd-openstack/#configuring-nova  we need to
provide the pool name for the parameter "*libvirt_images_rbd_pool*" in
nova.conf of compute node.
Do we need to provide multiple back-end (pools) details here also,
currently it is configured to only one pool which i believe is not the
correct setting.

If we need to have multi storage config in nova as well then please let me
know how to configure it.

Regards,
Vikrant



On Tue, Feb 25, 2014 at 8:21 PM, Sebastien Han
wrote:

> Hi,
>
> Please have a look at the cinder multi-backend functionality: examples
> here:
> http://www.sebastien-han.fr/blog/2013/04/25/ceph-and-cinder-multi-backend/
>
> Cheers.
> 
> Sébastien Han
> Cloud Engineer
>
> "Always give 100%. Unless you're giving blood."
>
> Phone: +33 (0)1 49 70 99 72
> Mail: sebastien@enovance.com
> Address : 10, rue de la Victoire - 75009 Paris
> Web : www.enovance.com - Twitter : @enovance
>
> On 25 Feb 2014, at 14:42, Vikrant Verma  wrote:
>
> > Hi All,
> >
> > I am using cinder as a front end for volume storage in Openstack
> configuration.
> > Ceph is used as storage back-end.
> >
> > Currently cinder uses only one pool (in my case pool name is "volumes" )
> for its volume storage.
> > I want cinder to use multiple ceph pools for volume storage
> >
> >
> > --following is the cinder.conf---
> > volume_driver=cinder.volume.drivers.rbd.RBDDriver
> > rbd_pool=volumes
> > rbd_ceph_conf=/etc/ceph/ceph.conf
> > rbd_flatten_volume_from_snapshot=false
> >
> >
> > Please let me know if it is possible to have multiple pools associated
> to cinder, let me know how to configure it.
> >
> > Regards,
> > Vikrant
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How to Configure Cinder to access multiple pools

[James Page]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 27/02/14 09:29, Vikrant Verma wrote:
> But i have one question - As per the link 
> http://ceph.com/docs/master/rbd/rbd-openstack/#configuring-nova
> we need to provide the pool name for the parameter 
> "*libvirt_images_rbd_pool*" in nova.conf of compute node. Do we
> need to provide multiple back-end (pools) details here also, 
> currently it is configured to only one pool which i believe is not
> the correct setting.

That's only used if you are using ceph as the default backing storage
for all instance root volumes.  The path to cinder volumes is passed
in full when attaching volumes to instances.


- -- 
James Page
Ubuntu and Debian Developer
james.p...@ubuntu.com
jamesp...@debian.org
-BEGIN PGP SIGNATURE-
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJTDzc9AAoJEL/srsug59jDaX4QAIAut1pvGFJnnDMS7+x2RBuf
cIDeB+AdyWzzTDFIxihu4DaMUrJPgbAKSgjTneT6jFfT0D40zX8FbfAV+PM/PxGz
D8j0HUOkzFmpiDWbDM4HtLmGCX6fcddJUQWCS0QBWIZe3GizJvCWGqX8pB4JuNEu
AdrxtaXoOfZyWAcVwXkU65RFQwJLAUi8wxxparlDxg0nu6izdaBgkKXA3d6T1qL3
R/9HreQ4HvoC2mtPcvSbmjlmaoBtHaIf+pxGfZjwq+ur+5NZojewJrqpahE76ov8
zStGEhSkJhzJr026OVxv7pK7+XuXRXZk3hrr2WYcaRho2ZGY6PNVrjEOaNmbST0J
Cr3D/WaLUmaRa/9nEhB+Dz7aj+XrNlHRBOUl2K8WLa7X/TBNJk/AoIXgxUyq1NeL
+vTFkLJ6vKFPfguVdt2zn528YN5CP1ztkDnTq7YV2khpNi2N1ZvTIimQQZw5TENr
Z2yAxjJ7nHuiTutkY8qfdYXgF3YOEI/7jdp7wAdCaWLDGugr04eDlf+NXGAUvy+s
NYHbauhjg2XBiWQKzeMz4NDjtf3uvH7NmeFjAc34soZ26zzOMI0ODFvptX7GLzxU
Iqbn2VsbB0ZR84auJrR42NMLUi/TSnigC5l9IdesbwByCyZuBquxJQiVVjpdYADk
UsC6qZGvhdZmUjWO8qzM
=GXST
-END PGP SIGNATURE-
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ubuntu 13.10 packages

[Tim Bishop]

Hi Michael,

On Tue, Feb 25, 2014 at 10:01:31PM +, Michael wrote:
> Just wondering if there was a reason for no packages for Ubuntu Saucy in 
> http://ceph.com/packages/ceph-extras/debian/dists/. Could do with 
> upgrading to fix a few bugs but would hate to have to drop Ceph from 
> being handled through the package manager!

The raring packages work fine with saucy. Not ideal, but it's the
easiest solution at the moment.

I'd like to see native saucy packages too, and actually it'd be good to
get trusty sorted soon too so it's all ready for the release.

Tim.

-- 
Tim Bishop
http://www.bishnet.net/tim/
PGP Key: 0x6C226B37FDF38D55

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ubuntu 13.10 packages

[Michael]

Thanks Tim, I'll give the raring packages a try.
Found a tracker for Saucy packages, looks like the person they were 
assigned to hasn't checking in for a fair while so they might have just 
been overlooked http://tracker.ceph.com/issues/6726.


-Michael

On 27/02/2014 13:33, Tim Bishop wrote:

Hi Michael,

On Tue, Feb 25, 2014 at 10:01:31PM +, Michael wrote:

Just wondering if there was a reason for no packages for Ubuntu Saucy in
http://ceph.com/packages/ceph-extras/debian/dists/. Could do with
upgrading to fix a few bugs but would hate to have to drop Ceph from
being handled through the package manager!

The raring packages work fine with saucy. Not ideal, but it's the
easiest solution at the moment.

I'd like to see native saucy packages too, and actually it'd be good to
get trusty sorted soon too so it's all ready for the release.

Tim.



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Swift APIs not authenticating Rados gateway !!!

[Srinivasa Rao Ragolu]

Hi Larry and Yehuda,

Feeling happy that my Rados gateway working fine.
I have done 3 mistakes

1) hostname is used instead of {fqdn}
2) radosgw keyring is generated only "+r" instead of "rw"
3) same keyring should be copied to all cluster nodes
4) run radosgw manually

thanks for support

Srinivas
On Feb 26, 2014 7:18 PM, "Liu, Larry"  wrote:

> All my ceph servers(3 nodes) are running ubuntu 13.04(Raring) which was
> recommended by ceph & inktank. I didn't do anything special except
> following ceph.com's documentation carefully.
>
> Swift client pkg, yes, apt-get install.
>
> From: Srinivasa Rao Ragolu 
> Date: Tuesday, February 25, 2014 10:17 PM
> To: Microsoft Office User 
> Cc: Yehuda Sadeh , "ceph-users@lists.ceph.com" <
> ceph-users@lists.ceph.com>
> Subject: Re: [ceph-users] Swift APIs not authenticating Rados gateway !!!
>
> Hi Larry,
>
> As you suggested I have changed to Ubuntu 10.04. Still I could not able to
> figure out what is this problem. I skipped only two sections in ceph
> documentation is 1) SSL 2) DNS , as I thought not needed any security to my
> gateway.
>
> 1) I strongly have a doubt in specifying hostname in ceph.conf, rgw.conf.
> Kindly share those files with me of your working setup.
> 2) How did you configure https link for rados gateway?
> 3) I have installed swift with sudo apt-get install swift. Is it ok?
>
> Will be very thankful to you
> Srinivas.
>
>
>
> On Sat, Feb 22, 2014 at 3:03 AM, Liu, Larry  wrote:
>
>> Srinivasa, I pretty much think your problem is your fedora systems are
>> missing some right lib files. I just got s3 working on my ubuntu raring
>> setup. Just follow exactly what is written on
>> http://ceph.com/docs/master/install/install-ceph-gateway/ .   Still a
>> question to everyone else:  for swift API, what is the auth url?  The
>> command swift -A  already works fine for me.  Can't fine the swift
>> auth url on the doc site.
>>
>> From: Srinivasa Rao Ragolu 
>> Date: Thursday, February 20, 2014 10:05 PM
>> To: Microsoft Office User 
>> Cc: Yehuda Sadeh , "ceph-users@lists.ceph.com" <
>> ceph-users@lists.ceph.com>
>> Subject: Re: [ceph-users] Swift APIs not authenticating Rados gateway !!!
>>
>> Please help in making it easy Rados gateway configurable with Swift. It
>> would be great support from you.
>>
>> I have skipped only two sections in
>> http://ceph.com/docs/master/install/install-ceph-gateway/
>>
>> a) Enable SSL and b) Add wildcard to DNS
>>
>> Apart from these steps I have followed all other instruction on fedora
>> 19..Please go through the attached configuration files.
>>
>> Still getting Authorisation failed : Http error 404
>>
>> Please help me.
>>
>> Srinivas.
>>
>>
>>
>> On Fri, Feb 21, 2014 at 1:06 AM, Liu, Larry  wrote:
>>
>>> Hi Yehuda,
>>>
>>> Is there any doc on how to set the swift url (rgw swift url)
>>> configurable?
>>>
>>> On 2/19/14 7:42 AM, "Yehuda Sadeh"  wrote:
>>>
>>> >On Wed, Feb 19, 2014 at 2:37 AM, Srinivasa Rao Ragolu
>>> > wrote:
>>> >> Hi all,
>>> >>
>>> >> I have setup cluster successfully and one node using to setup rados
>>> >>gateway.
>>> >> Machine is Fedora 19(all nodes)
>>> >>
>>> >> Steps I followed
>>> >>
>>> >> 1) Installed httpd, mod_fastcgi, ceph and ceph-radosgw using link
>>> >> http://ceph.com/docs/master/install/install-ceph-gateway/
>>> >>
>>> >> Note : Did not follow "Enable SSL" and "Add wild card DNS" sections
>>> >> 2) Made modifications in /etc/httpd/conf/httpd.conf,
>>> >> /etc/httpd/conf.d/fastcgi.conf
>>> >>
>>> >> 3) Created rgw.conf in /etc/httpd/conf.d/
>>> >>
>>> >> 4) Followed the link
>>> >> http://linuxmanpages.net/manpages/fedora19/man8/radosgw.8.html to
>>> create
>>> >> rgw.conf.
>>> >>
>>> >> 5) Added radosgw section in /etc/ceph/ceph.conf
>>> >>
>>> >> 6) Please see httpd.conf, fastcgi.conf, rgw.conf and ceph.conf as
>>> >> attachments.
>>> >>
>>> >> 7) Now followed below steps
>>> >>
>>> >> a)
>>> >>
>>> >>corresponding radosgw script (/var/www/s3gw.fcgi):
>>> >>
>>> >>#!/bin/sh
>>> >>exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n
>>> >> client.radosgw.gateway
>>> >>
>>> >>
>>> >> Gave execute permissions to s3gw.fcgi
>>> >>
>>> >>
>>> >>
>>> >>b)
>>> >>   ceph-authtool -C -n client.radosgw.gateway --gen-key
>>> >> /etc/ceph/keyring.radosgw.gateway
>>> >>ceph-authtool -n client.radosgw.gateway --cap mon 'allow r'
>>> >>--cap osd
>>> >> 'allow rwx' /etc/ceph/keyring.radosgw.gateway
>>> >> ceph auth add client.radosgw.gateway --in-file=keyring.radosgw.gateway
>>> >>
>>> >>
>>> >>
>>> >> 8) sudo service ceph restart
>>> >>
>>> >>sudo service httpd restart
>>> >>
>>> >>sudo /usr/bin/radosgw -c /etc/ceph/ceph.conf -n
>>> client.rados.gateway
>>> >>
>>> >> 9)  From the link http://ceph.com/docs/next/radosgw/config/, I
>>> executed
>>> >> "Create a gateway user", "Enabling swift access" sections
>>> >>
>>> >> 10) After above all steps if I run swift commands, I got following
>>> error
>>> >>
>>> >> [gateway@gateway ceph]$ s

Re: [ceph-users] Admin credentials

[Yehuda Sadeh]

On Thu, Feb 27, 2014 at 7:53 AM, Erik Tank  wrote:
> On http://ceph.com/docs/master/radosgw/adminops/ a "configurable 'admin' 
> resource entry point" is described.  I have the URI, however, I can not find 
> information on how to get/set the 'admin' credentials.
>
> Any help is greatly appreciated,
>

You need to specify the appropriate capabilities for the specific user. E.g.,

$ ./radosgw-admin caps add --uid=yehsad --caps="users=read, write; buckets=read"

Current options for caps are: users, buckets, usage, metadata, bilog,
mdlog, datalog, opstate.


Yehuda
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Admin credentials

[Erik Tank]

On http://ceph.com/docs/master/radosgw/adminops/ a "configurable 'admin' 
resource entry point" is described.  I have the URI, however, I can not find 
information on how to get/set the 'admin' credentials.

Any help is greatly appreciated,

Erik Tank
et...@liquidweb.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] recovery backfill still too aggressive

[Craig Lewis]

I recently added a 3rd node to my cluster, and increased the pool size to 3.

Latency was initially so bad that OSDs were being kicked out for being 
unresponsive.  I checked the list, and changed

osd max backfills = 1
osd recovery op priority = 1

That's helped.  OSDs aren't so slow that they get kicked out of the cluster.

I'm not super sensitive to latency, but I'm still seeing > 2s for RGW 
GET and PUT operations.  Does anybody have some suggestions for other 
ways I can give more priority to clients and less to backfill?


I know the ultimate answer is "add more nodes", and I'm planning on it.  
For the next expansion (in a few months), I'm thinking about only adding 
a single OSD at a time.  Since my cluster is so small, most objects are 
stripped across all of the disks.  One slow OSD vs. all slow OSDs will 
go a long way to reducing total latency.  It'll take two months to 
finish, but that's not a big deal.



I have 3 nodes, with 8x 4TB HDDs each.  Journal is on disk.  The Cluster 
network is gigabit, but it's only pushing ~300 Mbps per node.


In general, I'm more concerned with read performance than write. Using 
all the drive bays for spindles gave better read throughput and latency, 
at the expense of write latency.  I didn't think about how concerned 
with write latency I would be during a recovery.


Thanks for any suggestions.

--

*Craig Lewis*
Senior Systems Engineer
Office +1.714.602.1309
Email cle...@centraldesktop.com 

*Central Desktop. Work together in ways you never thought possible.*
Connect with us Website   | Twitter 
  | Facebook 
  | LinkedIn 
  | Blog 



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Trying to rescue a lost quorum

[Marc]

Hi,

I was handed a Ceph cluster that had just lost quorum due to 2/3 mons
(b,c) running out of disk space (using up 15GB each). We were trying to
rescue this cluster without service downtime. As such we freed up some
space to keep mon b running a while longer, which succeeded, quorum
restored (a,b), mon c remained offline. Even though we have freed up
some space on mon c's disk also, that mon just won't start. It's log
file does say

ceph version 0.61.2 (fea782543a844bb277ae94d3391788b76c5bee60), process
ceph-mon, pid 27846

and thats all she wrote. Even when starting ceph-mon with -d mind you.

So we had a cluster with 2/3 mons up and wanted to add another mon since
it was only a matter of time til mon b failed again due to disk space.

As such I added mon.g to the cluster, which took a long while to sync,
but now reports running.

Then mon.h got added for the same reason. mon.h fails to start much the
same as mon.c does.

Still that should leave us with 3/5 mons up. However running "ceph
daemon mon.{g,h} mon_status" on the respective node also blocks. The
only output we get from those are fault messages.

Ok so now mon.g apparantly crashed:

2014-02-28 00:11:48.861263 7f4728042700 -1 mon/Monitor.cc: In function
'void Monitor::sync_timeout(entity_inst_t&)' thread 7f4728042700 time
2014-02-28 00:11:48.782305 mon/Monitor.cc: 1099: FAILED
assert(sync_state == SYNC_STATE_CHUNKS)

... and now blocks trying to start much like c and h.

Long story short: is it possible to add .61.9 mons to a cluster running
.61.2 on the 2 alive mons and all the osds? I'm guessing this is the
last shot at trying to rescue the cluster without downtime.

KR,
Marc
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Trying to rescue a lost quorum

[Gregory Farnum]

On Thu, Feb 27, 2014 at 4:25 PM, Marc  wrote:
> Hi,
>
> I was handed a Ceph cluster that had just lost quorum due to 2/3 mons
> (b,c) running out of disk space (using up 15GB each). We were trying to
> rescue this cluster without service downtime. As such we freed up some
> space to keep mon b running a while longer, which succeeded, quorum
> restored (a,b), mon c remained offline. Even though we have freed up
> some space on mon c's disk also, that mon just won't start. It's log
> file does say
>
> ceph version 0.61.2 (fea782543a844bb277ae94d3391788b76c5bee60), process
> ceph-mon, pid 27846
>
> and thats all she wrote. Even when starting ceph-mon with -d mind you.
>
> So we had a cluster with 2/3 mons up and wanted to add another mon since
> it was only a matter of time til mon b failed again due to disk space.
>
> As such I added mon.g to the cluster, which took a long while to sync,
> but now reports running.
>
> Then mon.h got added for the same reason. mon.h fails to start much the
> same as mon.c does.
>
> Still that should leave us with 3/5 mons up. However running "ceph
> daemon mon.{g,h} mon_status" on the respective node also blocks. The
> only output we get from those are fault messages.
>
> Ok so now mon.g apparantly crashed:
>
> 2014-02-28 00:11:48.861263 7f4728042700 -1 mon/Monitor.cc: In function
> 'void Monitor::sync_timeout(entity_inst_t&)' thread 7f4728042700 time
> 2014-02-28 00:11:48.782305 mon/Monitor.cc: 1099: FAILED
> assert(sync_state == SYNC_STATE_CHUNKS)
>
> ... and now blocks trying to start much like c and h.
>
> Long story short: is it possible to add .61.9 mons to a cluster running
> .61.2 on the 2 alive mons and all the osds? I'm guessing this is the
> last shot at trying to rescue the cluster without downtime.

That should be fine, and is likely (though not guaranteed) to resolve
your sync issues -- although it's pretty unfortunate that you're that
far behind on the point releases; they fixed a whole lot of sync
issues and related things and you might need to upgrade the existing
monitors too in order to get the fixes you need... :/
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Trying to rescue a lost quorum

[Marc]

Hi,

thanks for the reply. I updated one of the new mons. And after a
resonably long init phase (inconsistent state), I am now seeing these:

2014-02-28 01:05:12.344648 7fe9d05cb700  0 cephx: verify_reply coudln't
decrypt with error: error decoding block for decryption
2014-02-28 01:05:12.345599 7fe9d05cb700  0 -- X.Y.Z.207:6789/0 >>
X.Y.Z.201:6789/0 pipe(0x14e1400 sd=21 :49082 s=1 pgs=5421935 cs=12
l=0).failed verifying authorize reply

with .207 being the updated mon and .201 being one of the "old" alive
mons. I guess they don't understand each other? I would rather not try
to update the mons running on servers that also host OSDs, especially
since there seem to be communication issues between those versions... or
am I reading this wrong?

KR,
Marc

On 28/02/2014 01:32, Gregory Farnum wrote:
> On Thu, Feb 27, 2014 at 4:25 PM, Marc  wrote:
>> Hi,
>>
>> I was handed a Ceph cluster that had just lost quorum due to 2/3 mons
>> (b,c) running out of disk space (using up 15GB each). We were trying to
>> rescue this cluster without service downtime. As such we freed up some
>> space to keep mon b running a while longer, which succeeded, quorum
>> restored (a,b), mon c remained offline. Even though we have freed up
>> some space on mon c's disk also, that mon just won't start. It's log
>> file does say
>>
>> ceph version 0.61.2 (fea782543a844bb277ae94d3391788b76c5bee60), process
>> ceph-mon, pid 27846
>>
>> and thats all she wrote. Even when starting ceph-mon with -d mind you.
>>
>> So we had a cluster with 2/3 mons up and wanted to add another mon since
>> it was only a matter of time til mon b failed again due to disk space.
>>
>> As such I added mon.g to the cluster, which took a long while to sync,
>> but now reports running.
>>
>> Then mon.h got added for the same reason. mon.h fails to start much the
>> same as mon.c does.
>>
>> Still that should leave us with 3/5 mons up. However running "ceph
>> daemon mon.{g,h} mon_status" on the respective node also blocks. The
>> only output we get from those are fault messages.
>>
>> Ok so now mon.g apparantly crashed:
>>
>> 2014-02-28 00:11:48.861263 7f4728042700 -1 mon/Monitor.cc: In function
>> 'void Monitor::sync_timeout(entity_inst_t&)' thread 7f4728042700 time
>> 2014-02-28 00:11:48.782305 mon/Monitor.cc: 1099: FAILED
>> assert(sync_state == SYNC_STATE_CHUNKS)
>>
>> ... and now blocks trying to start much like c and h.
>>
>> Long story short: is it possible to add .61.9 mons to a cluster running
>> .61.2 on the 2 alive mons and all the osds? I'm guessing this is the
>> last shot at trying to rescue the cluster without downtime.
> That should be fine, and is likely (though not guaranteed) to resolve
> your sync issues -- although it's pretty unfortunate that you're that
> far behind on the point releases; they fixed a whole lot of sync
> issues and related things and you might need to upgrade the existing
> monitors too in order to get the fixes you need... :/
> -Greg
> Software Engineer #42 @ http://inktank.com | http://ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] [Annonce]The progress of KeyValueStore in Firely

[Haomai Wang]

Hi all,

last release I propose a KeyValueStore prototype(get info from
http://sebastien-han.fr/blog/2013/12/02/ceph-performance-interesting-things-going-on).
It contains some performance results and problems. Now I'd like to
refresh our thoughts on KeyValueStore.

KeyValueStore is pursuing FileStore's performance during this release.
Now things go farther, KeyValueStore did better in rbd
situation(partial write) .

I test KeyValueStore compared to FileStore in a single OSD on Samsung
SSD 840. The config can be viewed
here(http://pad.ceph.com/p/KeyValueStore.conf). The same config file
is applied to both FileStore and KeyValueStore except "osd
objectstore" option.

I use fio which rbd supported from
TelekomCloud(https://github.com/TelekomCloud/fio/commits/rbd-engine)
to test rbd.

The fio command: fio -direct=1 -iodepth=64 -thread -rw=randwrite
-ioengine=rbd -bs=4k -size=19G -numjobs=1 -runtime=100
-group_reporting -name=ebs_test -pool=openstack -rbdname=image
-clientname=fio -invalidate=0



FileStore result:
ebs_test: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=rbd, iodepth=64
fio-2.1.4
Starting 1 thread
rbd engine: RBD version: 0.1.8

ebs_test: (groupid=0, jobs=1): err= 0: pid=30886: Thu Feb 27 08:09:18 2014
  write: io=283040KB, bw=6403.4KB/s, iops=1600, runt= 44202msec
slat (usec): min=116, max=2817, avg=195.78, stdev=56.45
clat (msec): min=8, max=661, avg=39.57, stdev=29.26
 lat (msec): min=9, max=661, avg=39.77, stdev=29.25
clat percentiles (msec):
 |  1.00th=[   15],  5.00th=[   20], 10.00th=[   23], 20.00th=[   28],
 | 30.00th=[   31], 40.00th=[   35], 50.00th=[   37], 60.00th=[   40],
 | 70.00th=[   43], 80.00th=[   46], 90.00th=[   51], 95.00th=[   58],
 | 99.00th=[  128], 99.50th=[  210], 99.90th=[  457], 99.95th=[  494],
 | 99.99th=[  545]
bw (KB  /s): min= 2120, max=12656, per=100.00%, avg=6464.27, stdev=1726.55
lat (msec) : 10=0.01%, 20=5.91%, 50=83.35%, 100=8.88%, 250=1.47%
lat (msec) : 500=0.34%, 750=0.05%
  cpu  : usr=29.83%, sys=1.36%, ctx=84002, majf=0, minf=216
  IO depths: 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=17.4%, >=64=82.6%
 submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete  : 0=0.0%, 4=99.1%, 8=0.5%, 16=0.3%, 32=0.1%, 64=0.1%, >=64=0.0%
 issued: total=r=0/w=70760/d=0, short=r=0/w=0/d=0
 latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: io=283040KB, aggrb=6403KB/s, minb=6403KB/s, maxb=6403KB/s,
mint=44202msec, maxt=44202msec

Disk stats (read/write):
  sdb: ios=5/9512, merge=0/69, ticks=4/10649, in_queue=10645, util=0.92%

===

KeyValueStore:
ebs_test: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=rbd, iodepth=64
fio-2.1.4
Starting 1 thread
rbd engine: RBD version: 0.1.8

ebs_test: (groupid=0, jobs=1): err= 0: pid=29137: Thu Feb 27 08:06:30 2014
  write: io=444376KB, bw=6280.2KB/s, iops=1570, runt= 70759msec
slat (usec): min=122, max=3237, avg=184.51, stdev=37.76
clat (msec): min=10, max=168, avg=40.57, stdev= 5.70
 lat (msec): min=11, max=168, avg=40.75, stdev= 5.71
clat percentiles (msec):
 |  1.00th=[   34],  5.00th=[   37], 10.00th=[   39], 20.00th=[   39],
 | 30.00th=[   40], 40.00th=[   40], 50.00th=[   41], 60.00th=[   41],
 | 70.00th=[   42], 80.00th=[   42], 90.00th=[   44], 95.00th=[   45],
 | 99.00th=[   48], 99.50th=[   50], 99.90th=[  163], 99.95th=[  167],
 | 99.99th=[  167]
bw (KB  /s): min= 4590, max= 7480, per=100.00%, avg=6285.69, stdev=374.22
lat (msec) : 20=0.02%, 50=99.58%, 100=0.23%, 250=0.17%
  cpu  : usr=29.11%, sys=1.10%, ctx=118564, majf=0, minf=194
  IO depths: 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.7%, >=64=99.3%
 submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.1%, 32=0.0%, 64=0.1%, >=64=0.0%
 issued: total=r=0/w=111094/d=0, short=r=0/w=0/d=0
 latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: io=444376KB, aggrb=6280KB/s, minb=6280KB/s, maxb=6280KB/s,
mint=70759msec, maxt=70759msec

Disk stats (read/write):
  sdb: ios=0/15936, merge=0/272, ticks=0/17157, in_queue=17146, util=0.94%


It's just a simple test, maybe exist some misleadings on the config or
results. But
we can obviously see the conspicuous improvement for KeyValueStore.

In the near future, performance still will be the first thing to
improve especially at write operation(The goal of KeyValueStore is
provided with powerful write performance compared to FileStore), such
as
1. Fine-grained lock in object-level to improve the degree of
parallelism, because KeyValueStore doesn't have Journal to quick the
latency of write transaction, we need to avoid block as far as
possible.
2. Header cache(like inode in filesystem) to quic

[ceph-users] Viewing CephFS Client Debugging

[Michael Sevilla]

I'm looking for the debug messages in Client.cc, which uses ldout
(library debugging). I increased the client debug level for all
daemons (i.e. under [global] in ceph.conf) and verified that it got
set:

$ ceph --admin-daemon /var/run/ceph/ceph-mon.issdm-3.asok config show
| grep client
"client": "20\/20",

After I mount and make a couple files, I expect to see some debugging
activity, but the log is empty. Logging into the client and checking
results in:

$ ls -l /var/log/ceph/ceph-client.admin.log
-rw-r--r-- 1 root root 0 Feb 27 10:19 /var/log/ceph/ceph-client.admin.log

Is there a flag I am missing? I'm a little confused since the client
isn't really managed by a daemon like the MDSs, OSDs, and MONs are.

Things I tried: changing the permissions of /var/log/ceph, using both
the kernel and FUSE client, redirecting the logs to syslog, setting
the name of the log (i.e. log file = x in ceph.conf), checking
dmesg/kern.log.


Thanks!

Michael
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [Annonce]The progress of KeyValueStore in Firely

[Alexandre DERUMIER]

Thanks for the report !

Results seem to be encouraging . (Is is leveldb keystore ?)

Thanks to fio-rbd, it'll be easier to do random io benchmark now !

(I'm waiting to see if rocksdb will improve things in the future)

Regards,

Alexandre
- Mail original - 

De: "Haomai Wang"  
À: ceph-users@lists.ceph.com, ceph-de...@vger.kernel.org 
Envoyé: Vendredi 28 Février 2014 03:45:01 
Objet: [ceph-users] [Annonce]The progress of KeyValueStore in Firely 

Hi all, 

last release I propose a KeyValueStore prototype(get info from 
http://sebastien-han.fr/blog/2013/12/02/ceph-performance-interesting-things-going-on).
 
It contains some performance results and problems. Now I'd like to 
refresh our thoughts on KeyValueStore. 

KeyValueStore is pursuing FileStore's performance during this release. 
Now things go farther, KeyValueStore did better in rbd 
situation(partial write) . 

I test KeyValueStore compared to FileStore in a single OSD on Samsung 
SSD 840. The config can be viewed 
here(http://pad.ceph.com/p/KeyValueStore.conf). The same config file 
is applied to both FileStore and KeyValueStore except "osd 
objectstore" option. 

I use fio which rbd supported from 
TelekomCloud(https://github.com/TelekomCloud/fio/commits/rbd-engine) 
to test rbd. 

The fio command: fio -direct=1 -iodepth=64 -thread -rw=randwrite 
-ioengine=rbd -bs=4k -size=19G -numjobs=1 -runtime=100 
-group_reporting -name=ebs_test -pool=openstack -rbdname=image 
-clientname=fio -invalidate=0 

 

FileStore result: 
ebs_test: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=rbd, iodepth=64 
fio-2.1.4 
Starting 1 thread 
rbd engine: RBD version: 0.1.8 

ebs_test: (groupid=0, jobs=1): err= 0: pid=30886: Thu Feb 27 08:09:18 2014 
write: io=283040KB, bw=6403.4KB/s, iops=1600, runt= 44202msec 
slat (usec): min=116, max=2817, avg=195.78, stdev=56.45 
clat (msec): min=8, max=661, avg=39.57, stdev=29.26 
lat (msec): min=9, max=661, avg=39.77, stdev=29.25 
clat percentiles (msec): 
| 1.00th=[ 15], 5.00th=[ 20], 10.00th=[ 23], 20.00th=[ 28], 
| 30.00th=[ 31], 40.00th=[ 35], 50.00th=[ 37], 60.00th=[ 40], 
| 70.00th=[ 43], 80.00th=[ 46], 90.00th=[ 51], 95.00th=[ 58], 
| 99.00th=[ 128], 99.50th=[ 210], 99.90th=[ 457], 99.95th=[ 494], 
| 99.99th=[ 545] 
bw (KB /s): min= 2120, max=12656, per=100.00%, avg=6464.27, stdev=1726.55 
lat (msec) : 10=0.01%, 20=5.91%, 50=83.35%, 100=8.88%, 250=1.47% 
lat (msec) : 500=0.34%, 750=0.05% 
cpu : usr=29.83%, sys=1.36%, ctx=84002, majf=0, minf=216 
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=17.4%, >=64=82.6% 
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% 
complete : 0=0.0%, 4=99.1%, 8=0.5%, 16=0.3%, 32=0.1%, 64=0.1%, >=64=0.0% 
issued : total=r=0/w=70760/d=0, short=r=0/w=0/d=0 
latency : target=0, window=0, percentile=100.00%, depth=64 

Run status group 0 (all jobs): 
WRITE: io=283040KB, aggrb=6403KB/s, minb=6403KB/s, maxb=6403KB/s, 
mint=44202msec, maxt=44202msec 

Disk stats (read/write): 
sdb: ios=5/9512, merge=0/69, ticks=4/10649, in_queue=10645, util=0.92% 

=== 

KeyValueStore: 
ebs_test: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=rbd, iodepth=64 
fio-2.1.4 
Starting 1 thread 
rbd engine: RBD version: 0.1.8 

ebs_test: (groupid=0, jobs=1): err= 0: pid=29137: Thu Feb 27 08:06:30 2014 
write: io=444376KB, bw=6280.2KB/s, iops=1570, runt= 70759msec 
slat (usec): min=122, max=3237, avg=184.51, stdev=37.76 
clat (msec): min=10, max=168, avg=40.57, stdev= 5.70 
lat (msec): min=11, max=168, avg=40.75, stdev= 5.71 
clat percentiles (msec): 
| 1.00th=[ 34], 5.00th=[ 37], 10.00th=[ 39], 20.00th=[ 39], 
| 30.00th=[ 40], 40.00th=[ 40], 50.00th=[ 41], 60.00th=[ 41], 
| 70.00th=[ 42], 80.00th=[ 42], 90.00th=[ 44], 95.00th=[ 45], 
| 99.00th=[ 48], 99.50th=[ 50], 99.90th=[ 163], 99.95th=[ 167], 
| 99.99th=[ 167] 
bw (KB /s): min= 4590, max= 7480, per=100.00%, avg=6285.69, stdev=374.22 
lat (msec) : 20=0.02%, 50=99.58%, 100=0.23%, 250=0.17% 
cpu : usr=29.11%, sys=1.10%, ctx=118564, majf=0, minf=194 
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.7%, >=64=99.3% 
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% 
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.1%, 32=0.0%, 64=0.1%, >=64=0.0% 
issued : total=r=0/w=111094/d=0, short=r=0/w=0/d=0 
latency : target=0, window=0, percentile=100.00%, depth=64 

Run status group 0 (all jobs): 
WRITE: io=444376KB, aggrb=6280KB/s, minb=6280KB/s, maxb=6280KB/s, 
mint=70759msec, maxt=70759msec 

Disk stats (read/write): 
sdb: ios=0/15936, merge=0/272, ticks=0/17157, in_queue=17146, util=0.94% 


It's just a simple test, maybe exist some misleadings on the config or 
results. But 
we can obviously see the conspicuous improvement for KeyValueStore. 

In the near future, performance still will be the first thing to 
improve especially at write operation(The goal of KeyValueStore is 
provided with powerful write performance compared