[ceph-users] status of glance/cinder/nova integration in openstack grizzly

[Darren Birkett]

Hi All,

tl;dr - does glance/rbd and cinder/rbd play together nicely in grizzly?

I'm currently testing a ceph/rados back end with an openstack installation.
 I have the following things working OK:

1. cinder configured to create volumes in RBD
2. nova configured to boot from RBD backed cinder volumes (libvirt UUID
secret set etc)
3. glance configured to use RBD as a back end store for images

With this setup, when I create a bootable volume in cinder, passing an id
of an image in glance, the image gets downloaded, converted to raw, and
then created as an RBD object and made available to cinder.  The correct
metadata field for the cinder volume is populated (volume_image_metadata)
and so the cinder client marks the volume as bootable.  This is all fine.

If I want to take advantage of the fact that both glance images and cinder
volumes are stored in RBD, I can add the following flag to the
glance-api.conf:

show_image_direct_url = True

This enables cinder to see that the glance image is stored in RBD, and the
cinder rbd driver then, instead of downloading the image and creating an
RBD image from it, just issues an 'rbd clone' command (seen in the
cinder-volume.log):

rbd clone --pool images --image dcb2f16d-a09d-4064-9198-1965274e214d --snap
snap --dest-pool volumes --dest volume-20987f9d-b4fb-463d-8b8f-fa667bd47c6d

This is all very nice, and the cinder volume is available immediately as
you'd expect.  The problem is that the metadata field is not populated so
it's not seen as bootable.  Even manually populating this field leaves the
volume unbootable.  The volume can not even be attached to another instance
for inspection.

libvirt doesn't seem to be able to access the rbd device. From
nova-compute.log:

qemu-system-x86_64: -drive
file=rbd:volumes/volume-20987f9d-b4fb-463d-8b8f-fa667bd47c6d:id=volumes:key=AQAnAy9ScPB4IRAAtxD/V1rDciqFiT9AMPPr+A==:auth_supported=cephx\;none,if=none,id=drive-virtio-disk0,format=raw,serial=20987f9d-b4fb-463d-8b8f-fa667bd47c6d,cache=none:
error reading header from volume-20987f9d-b4fb-463d-8b8f-fa667bd47c6d

qemu-system-x86_64: -drive
file=rbd:volumes/volume-20987f9d-b4fb-463d-8b8f-fa667bd47c6d:id=volumes:key=AQAnAy9ScPB4IRAAtxD/V1rDciqFiT9AMPPr+A==:auth_supported=cephx\;none,if=none,id=drive-virtio-disk0,format=raw,serial=20987f9d-b4fb-463d-8b8f-fa667bd47c6d,cache=none:
could not open disk image
rbd:volumes/volume-20987f9d-b4fb-463d-8b8f-fa667bd47c6d:id=volumes:key=AQAnAy9ScPB4IRAAtxD/V1rDciqFiT9AMPPr+A==:auth_supported=cephx\;none:
Operation not permitted

It's almost like a permission issue, but my ceph/rbd knowledge is still
fledgeling.

I know that the cinder rbd driver has been rewritten to use librbd in
havana, and I'm wondering if this will change any of this behaviour?  I'm
also wondering if anyone has actually got this working with grizzly, and
how?

Many thanks
Darren
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] status of glance/cinder/nova integration in openstack grizzly

[Darren Birkett]

Hi Mike,

Thanks - glad to hear it definitely works as expected!  Here's my
client.glance and client.volumes from 'ceph auth list':

client.glance
key: AQAWFi9SOKzAABAAPV1ZrpWkx72tmJ5E7nOi3A==
caps: [mon] allow r
caps: [osd] allow rwx pool=images, allow class-read object_prefix
rbd_children
client.volumes
key: AQAnAy9ScPB4IRAAtxD/V1rDciqFiT9AMPPr+A==
caps: [mon] allow r
caps: [osd] allow class-read object_prefix rbd_children, allow rwx
pool=volumes

Thanks
Darren


On 10 September 2013 20:08, Mike Dawson  wrote:

> Darren,
>
> I can confirm Copy on Write (show_image_direct_url = True) does work in
> Grizzly.
>
> It sounds like you are close. To check permissions, run 'ceph auth list',
> and reply with "client.images" and "client.volumes" (or whatever keys you
> use in Glance and Cinder).
>
> Cheers,
> Mike Dawson
>
>
>
> On 9/10/2013 10:12 AM, Darren Birkett wrote:
>
>> Hi All,
>>
>> tl;dr - does glance/rbd and cinder/rbd play together nicely in grizzly?
>>
>> I'm currently testing a ceph/rados back end with an openstack
>> installation.  I have the following things working OK:
>>
>> 1. cinder configured to create volumes in RBD
>> 2. nova configured to boot from RBD backed cinder volumes (libvirt UUID
>> secret set etc)
>> 3. glance configured to use RBD as a back end store for images
>>
>> With this setup, when I create a bootable volume in cinder, passing an
>> id of an image in glance, the image gets downloaded, converted to raw,
>> and then created as an RBD object and made available to cinder.  The
>> correct metadata field for the cinder volume is populated
>> (volume_image_metadata) and so the cinder client marks the volume as
>> bootable.  This is all fine.
>>
>> If I want to take advantage of the fact that both glance images and
>> cinder volumes are stored in RBD, I can add the following flag to the
>> glance-api.conf:
>>
>> show_image_direct_url = True
>>
>> This enables cinder to see that the glance image is stored in RBD, and
>> the cinder rbd driver then, instead of downloading the image and
>> creating an RBD image from it, just issues an 'rbd clone' command (seen
>> in the cinder-volume.log):
>>
>> rbd clone --pool images --image dcb2f16d-a09d-4064-9198-**1965274e214d
>> --snap snap --dest-pool volumes --dest
>> volume-20987f9d-b4fb-463d-**8b8f-fa667bd47c6d
>>
>> This is all very nice, and the cinder volume is available immediately as
>> you'd expect.  The problem is that the metadata field is not populated
>> so it's not seen as bootable.  Even manually populating this field
>> leaves the volume unbootable.  The volume can not even be attached to
>> another instance for inspection.
>>
>> libvirt doesn't seem to be able to access the rbd device. From
>> nova-compute.log:
>>
>> qemu-system-x86_64: -drive
>> file=rbd:volumes/volume-**20987f9d-b4fb-463d-8b8f-**
>> fa667bd47c6d:id=volumes:key=**AQAnAy9ScPB4IRAAtxD/**
>> V1rDciqFiT9AMPPr+A==:auth_**supported=cephx\;none,if=none,**
>> id=drive-virtio-disk0,format=**raw,serial=20987f9d-b4fb-463d-**
>> 8b8f-fa667bd47c6d,cache=none:
>> error reading header from volume-20987f9d-b4fb-463d-**8b8f-fa667bd47c6d
>>
>> qemu-system-x86_64: -drive
>> file=rbd:volumes/volume-**20987f9d-b4fb-463d-8b8f-**
>> fa667bd47c6d:id=volumes:key=**AQAnAy9ScPB4IRAAtxD/**
>> V1rDciqFiT9AMPPr+A==:auth_**supported=cephx\;none,if=none,**
>> id=drive-virtio-disk0,format=**raw,serial=20987f9d-b4fb-463d-**
>> 8b8f-fa667bd47c6d,cache=none:
>> could not open disk image
>> rbd:volumes/volume-20987f9d-**b4fb-463d-8b8f-fa667bd47c6d:**
>> id=volumes:key=**AQAnAy9ScPB4IRAAtxD/**V1rDciqFiT9AMPPr+A==:auth_**
>> supported=cephx\;none:
>> Operation not permitted
>>
>> It's almost like a permission issue, but my ceph/rbd knowledge is still
>> fledgeling.
>>
>> I know that the cinder rbd driver has been rewritten to use librbd in
>> havana, and I'm wondering if this will change any of this behaviour?
>>   I'm also wondering if anyone has actually got this working with
>> grizzly, and how?
>>
>> Many thanks
>> Darren
>>
>>
>>
>> __**_
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/**listinfo.cgi/ceph-users-ceph.**com<http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>
>>
>>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] status of glance/cinder/nova integration in openstack grizzly

[Darren Birkett]

Hi Mike,

That led me to realise what the issue was.  My cinder (volumes) client did
not have the correct perms on the images pool.  I ran the following to
update the perms for that client:

ceph auth caps client.volumes mon 'allow r' osd 'allow class-read
object_prefix rbd_children, allow rwx pool=volumes, allow rx pool=images'

...and was then able to successfully boot an instance from a cinder volume
that was created by cloning a glance image from the images pool!

One last question: I presume the fact that the 'volume_image_metadata'
field is not populated when cloning a glance image into a cinder volume is
a bug?  It means that the cinder client doesn't show the volume as
bootable, though I'm not sure what other detrimental effect it actually has
(clearly the volume can be booted from).

Thanks
Darren


On 10 September 2013 21:04, Darren Birkett  wrote:

> Hi Mike,
>
> Thanks - glad to hear it definitely works as expected!  Here's my
> client.glance and client.volumes from 'ceph auth list':
>
> client.glance
>  key: AQAWFi9SOKzAABAAPV1ZrpWkx72tmJ5E7nOi3A==
> caps: [mon] allow r
> caps: [osd] allow rwx pool=images, allow class-read object_prefix
> rbd_children
> client.volumes
> key: AQAnAy9ScPB4IRAAtxD/V1rDciqFiT9AMPPr+A==
> caps: [mon] allow r
> caps: [osd] allow class-read object_prefix rbd_children, allow rwx
> pool=volumes
>
> Thanks
> Darren
>
>
> On 10 September 2013 20:08, Mike Dawson  wrote:
>
>> Darren,
>>
>> I can confirm Copy on Write (show_image_direct_url = True) does work in
>> Grizzly.
>>
>> It sounds like you are close. To check permissions, run 'ceph auth list',
>> and reply with "client.images" and "client.volumes" (or whatever keys you
>> use in Glance and Cinder).
>>
>> Cheers,
>> Mike Dawson
>>
>>
>>
>> On 9/10/2013 10:12 AM, Darren Birkett wrote:
>>
>>> Hi All,
>>>
>>> tl;dr - does glance/rbd and cinder/rbd play together nicely in grizzly?
>>>
>>> I'm currently testing a ceph/rados back end with an openstack
>>> installation.  I have the following things working OK:
>>>
>>> 1. cinder configured to create volumes in RBD
>>> 2. nova configured to boot from RBD backed cinder volumes (libvirt UUID
>>> secret set etc)
>>> 3. glance configured to use RBD as a back end store for images
>>>
>>> With this setup, when I create a bootable volume in cinder, passing an
>>> id of an image in glance, the image gets downloaded, converted to raw,
>>> and then created as an RBD object and made available to cinder.  The
>>> correct metadata field for the cinder volume is populated
>>> (volume_image_metadata) and so the cinder client marks the volume as
>>> bootable.  This is all fine.
>>>
>>> If I want to take advantage of the fact that both glance images and
>>> cinder volumes are stored in RBD, I can add the following flag to the
>>> glance-api.conf:
>>>
>>> show_image_direct_url = True
>>>
>>> This enables cinder to see that the glance image is stored in RBD, and
>>> the cinder rbd driver then, instead of downloading the image and
>>> creating an RBD image from it, just issues an 'rbd clone' command (seen
>>> in the cinder-volume.log):
>>>
>>> rbd clone --pool images --image dcb2f16d-a09d-4064-9198-**1965274e214d
>>> --snap snap --dest-pool volumes --dest
>>> volume-20987f9d-b4fb-463d-**8b8f-fa667bd47c6d
>>>
>>> This is all very nice, and the cinder volume is available immediately as
>>> you'd expect.  The problem is that the metadata field is not populated
>>> so it's not seen as bootable.  Even manually populating this field
>>> leaves the volume unbootable.  The volume can not even be attached to
>>> another instance for inspection.
>>>
>>> libvirt doesn't seem to be able to access the rbd device. From
>>> nova-compute.log:
>>>
>>> qemu-system-x86_64: -drive
>>> file=rbd:volumes/volume-**20987f9d-b4fb-463d-8b8f-**
>>> fa667bd47c6d:id=volumes:key=**AQAnAy9ScPB4IRAAtxD/**
>>> V1rDciqFiT9AMPPr+A==:auth_**supported=cephx\;none,if=none,**
>>> id=drive-virtio-disk0,format=**raw,serial=20987f9d-b4fb-463d-**
>>> 8b8f-fa667bd47c6d,cache=none:
>>> error reading header from volume-20987f9d-b4fb-463d-**8b8f-fa667bd47c6d
>>>
>>> qemu-system-x86_64: -drive
>>> file=rbd:volumes/volume-**20987f9d-b4fb-463d-8b8f-**
>>> fa667bd47c6d:id=volumes:key=**AQAnAy9

[ceph-users] live migration with rbd/cinder/nova - not supported?

[Darren Birkett]

Hi,

It seems that the combination of libvirt and ceph will happily do live
migrations.

However, when using openstack, and a nova instance is booted from a cinder
volume that itself exists in rbd, it appears from the nova code that nova
itself does not have support for instance migration due to the fact it
expects shared storage mounted on /var/lib/nova/instances.

nova does not seem to be aware of the fact that an rbd type back end is
being used for an instance root (ephemeral) disk, and will fail the shared
storage check, and so fail the migration.

Block migrations will also not work as there is no disk file to actually
copy.

I know that there has been support added to nova's libvirt_image_type in
havana to allow nova to directly boot into rbd, without having to boot from
a cinder volume, but it does not appear that any of the live migration code
has been touched.

So is it fair to say that if you are using cinder/rbd or (in havana)
nova/rbd for your instance root disk, you lose the ability to perform live
migrations (as performed by nova)?


Thanks
Darren
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] live migration with rbd/cinder/nova - not supported?

[Darren Birkett]

Hi Maciej,

I'm using Grizzly, but the live migration doesn't appear to be changed even
in trunk.  It seems to check if you are using shared storage by writing a
test file on the destination host (in /var/lib/nova/instances) and then
trying to read it on the source host, and will fail if this test does not
succeed.  Given that we do not use a shared filesystem as such when using
RBD backed instances, I don't understand how this can succeed:

https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L3726-L3761

Seems like I might be missing something and am probably reading the code
wrong, as it sounds like you have it working.  Are there any particular
settings you had to add to nova to make it work?

Thanks
Darren


On 12 September 2013 13:48, Maciej Gałkiewicz wrote:

> Hi
>
> I dont know which version of openstack you are using but I ensure you
> that everything works fine with Grizzly. Let me know if you need some
> help.
>
>
> regards
> --
> Maciej Gałkiewicz
> Shelly Cloud Sp. z o. o., Sysadmin
> http://shellycloud.com/, mac...@shellycloud.com
> KRS: 440358 REGON: 101504426
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] live migration with rbd/cinder/nova - not supported?

[Darren Birkett]

Hi Maciej,

That's interesting.  The following also seems to suggest that nova has
those shared storage dependencies for live migration that I spoke about:

http://tracker.ceph.com/issues/5938

Thanks
Darren


On 12 September 2013 17:01, Maciej Gałkiewicz wrote:

> On 12 September 2013 15:15, Darren Birkett 
> wrote:
> > Hi Maciej,
> >
> > I'm using Grizzly, but the live migration doesn't appear to be changed
> even
> > in trunk.  It seems to check if you are using shared storage by writing a
> > test file on the destination host (in /var/lib/nova/instances) and then
> > trying to read it on the source host, and will fail if this test does not
> > succeed.  Given that we do not use a shared filesystem as such when using
> > RBD backed instances, I don't understand how this can succeed:
> >
> >
> https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L3726-L3761
>
> Apparently the code is not used in case of rbd. I didnt notice any
> issues caused by this method.
>
> > Seems like I might be missing something and am probably reading the code
> > wrong, as it sounds like you have it working.  Are there any particular
> > settings you had to add to nova to make it work?
>
> Nova does not require any changes. Only cinder.
>
> regards
> --
> Maciej Gałkiewicz
> Shelly Cloud Sp. z o. o., Sysadmin
> http://shellycloud.com/, mac...@shellycloud.com
> KRS: 440358 REGON: 101504426
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Help with radosGW

[Darren Birkett]

Hi Alexis,

Great to hear you fixed your problem! Would you care to describe in more
detail what the fix was, in case other people experience the same issues as
you did.

Thanks
Darren


On 18 September 2013 10:12, Alexis GÜNST HORN  wrote:

> Hello to all,
> Thanks for your answers.
>
> Well... after an awful night, I found the problem...
> It was a MTU mistake !
>
> No relation with Ceph !
> So sorry for the noise, and thanks again.
>
>
> Best Regards - Cordialement
>
> Alexis
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [SOLVED] Re: Ceph bock storage and Openstack Cinder Scheduler issue

[Darren Birkett]

On 19 September 2013 11:51, Gavin  wrote:

>
> Hi,
>
> Please excuse/disregard my previous email, I just needed a
> clarification on my understanding of how this all fits together.
>
> I was kindly pointed in the right direction by a friendly gentleman
> from Rackspace. Thanks Darren. :)
>
> The reason for my confusion was due to the way that the volumes are
> displayed in the Horizon dashboard.
>
> The dashboard shows that all volumes are attached to one Compute node,
> which obviously led to my initial concerns.
>
> Now that I know that the connections come from libvirt on the compute
> node where the instances live, I have one less thing to worry about.
>
> Thanks,
> Gavin
> ___
>

yes AFAIK, cinder-volume is largely only involved in brokering the initial
volume creation in RBD.  libvirt on the compute host where the instance
lives then connects to RBD.

Whilst this would suggest that the inital cinder-volume host that brokered
the creation is therefore no longer needed after creation, I do vaguely
remember there still being some sort of thin requirement on that host
remaining there, in grizzly at least.  That may be fixed now, but I'd be
interested to see your experiences with that.

Darren
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OpenStack Grizzly Authentication (Keystone PKI) with RADOS Gateway

[Darren Birkett]

Hi Amit,

It can, but at the moment there is some issue with keystone token caching
(in Dumpling), so every auth call hits keystone and does not cache the
token.

See here:

http://www.spinics.net/lists/ceph-users/msg04531.html

and here:

http://tracker.ceph.com/issues/6360


Thanks

Darren



On 28 September 2013 03:24, Amit Vijairania wrote:

> Hello!
>
> Does RADOS Gateway supports or integrates with OpenStack (Grizzly)
> Authentication (Keystone PKI)?
>
> Can RADOS Gateway use PKI tokens to conduct user token verification
> without explicit calls to Keystone.
>
> Thanks!
> Amit
>
> Amit Vijairania  |  978.319.3684
> --*--
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Rados gw upload problems

[Darren Birkett]

Hi Warren,

Try using the ceph specific fastcgi module as detailed here:

http://ceph.com/docs/next/radosgw/manual-install/

And see if that helps.

There was a similar discussion on the list previously:

http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-March/000360.html


Thanks
Darren



On 4 October 2013 05:12, Warren Wang  wrote:

> Hi all, I'm having a problem uploading through the Rados GW.  I'm getting
> the following error, and searches haven't lead me to a solution.
>
> [Fri Oct 04 04:05:11 2013] [error] [client xxx.xxx.xxx.xxx] chunked
> Transfer-Encoding forbidden: /swift/v1/wwang-container/test
>
> FastCGI version:
> ii  libapache2-mod-fastcgi
> 2.4.7~0910052141-1 amd64Apache 2 FastCGI module
> for long-running CGI scripts
>
> Auth works properly through keystone.  Getting hung up on this final part.
>
> Thanks for any help,
> Warren
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph access using curl

[Darren Birkett]

Try using passing '--debug' to the swift command. It should output the 
equivalent curl command for you to use.

- Darren

"Snider, Tim"  wrote:
>I'm having pilot error with getting the path correct using curl.
>Bucket listing using "radosgw-admin bucket list" works as does the
>swift API.
>Can someone point out my (obvious) error?
>
>Bucket list works:
>root@controller21:/home/ceph/my-cluster#  radosgw-admin bucket list
>2013-10-04 11:28:13.144065 7fe5f26fe780  0 WARNING: cannot read region
>map
>[
>"xxx"]
>
>Swift API to Ceph works
>swift -V 1.0 -A
>http://ictp-R2C4-Controller21.ict.englab.netapp.com/auth -U rados:swift
>-K "xxx"  stat
>   Account: v1
>Containers: 1
>   Objects: 1
> Bytes: 604
>  Vary: Accept-Encoding
>Server: Apache/2.2.22 (Ubuntu)
>X-Account-Bytes-Used-Actual: 4096
>Content-Type: text/plain; charset=utf-8
>root@controller21:/home/ceph/my-cluster# swift -V 1.0 -A
>http://ictp-R2C4-Controller21.ict.englab.netapp.com/auth -U rados:swift
>-K "77iJvemrxWvYk47HW7pxsL+eHdA53AtLl2T0OyuG"  list
>xxx
>
>However I can't get the correct http path. Variations of auth/v1.0,
>auth/v1 ... don't work either:
>root@controller21:/home/ceph/my-cluster# curl -v -H 'X-Storage-User:
>rados:swift' -H 'X-Storage-Pass: xxx' 
>http://ictp-R2C4-Controller21.ict.englab.netapp.com/auth
>* About to connect() to ictp-R2C4-Controller21.ict.englab.netapp.com
>port 80 (#0)
>*   Trying 10.113.193.189... connected
>> GET /auth HTTP/1.1
>> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0
>OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
>> Host: ictp-R2C4-Controller21.ict.englab.netapp.com
>> Accept: */*
>> X-Storage-User: rados:swift
>> X-Storage-Pass: 77iJvemrxWvYk47HW7pxsL+eHdA53AtLl2T0OyuG
>>
>< HTTP/1.1 403 Forbidden
>< Date: Fri, 04 Oct 2013 18:30:01 GMT
>< Server: Apache/2.2.22 (Ubuntu)
>< Accept-Ranges: bytes
>< Content-Length: 23
>< Content-Type: application/json
><
>* Connection #0 to host ictp-R2C4-Controller21.ict.englab.netapp.com
>left intact
>* Closing connection #0
>{"Code":"AccessDenied"}root@controller21:/home/ceph/my-cluster#
>
>Thanks,
>Tim
>
>
>
>
>___
>ceph-users mailing list
>ceph-users@lists.ceph.com
>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] radosgw keystone authtoken caching still not working with 0.67.4

[Darren Birkett]

Hi All,

In our prior tests with 0.67.3, keystone authtoken caching was broken
causing dreadful performance - see
http://www.spinics.net/lists/ceph-users/msg04531.html

We upgraded to release 0.67.4 as we wanted to test the apparent fix to
authtoken caching that was included in the release notes.  Unfortunately,
we still seem to be seeing the same issues - keystone is hit for every
single operation, causing a huge bottleneck in performance.

Aside from upgrading packages and restarting all daemons, are there some
options we are missing from the below to get things to work properly:

:/etc/ceph/ceph.conf
[client.radosgw.gateway]

rgw_keystone_url = http://x.x.x.x:35357
rgw_keystone_admin_token = xxx
rgw_keystone_accepted_roles = admin, Member
rgw_keystone_token_cache_size = 1



Thanks
Darren
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw keystone authtoken caching still not working with 0.67.4

[Darren Birkett]

Hi All,

What's the best way to try and track down why this isn't working for us?
 It doesn't seem that there are any other options I can provide.  Lack of
(working) radosgw integration with keystone would be a huge blocker for us
being able to adopt ceph as part of our product set.

Thanks,
Darren


On 7 October 2013 14:28, Darren Birkett  wrote:

> Hi All,
>
> In our prior tests with 0.67.3, keystone authtoken caching was broken
> causing dreadful performance - see
> http://www.spinics.net/lists/ceph-users/msg04531.html
>
> We upgraded to release 0.67.4 as we wanted to test the apparent fix to
> authtoken caching that was included in the release notes.  Unfortunately,
> we still seem to be seeing the same issues - keystone is hit for every
> single operation, causing a huge bottleneck in performance.
>
> Aside from upgrading packages and restarting all daemons, are there some
> options we are missing from the below to get things to work properly:
>
> :/etc/ceph/ceph.conf
> [client.radosgw.gateway]
> 
> rgw_keystone_url = http://x.x.x.x:35357
> rgw_keystone_admin_token = xxx
> rgw_keystone_accepted_roles = admin, Member
> rgw_keystone_token_cache_size = 1
> 
>
>
> Thanks
> Darren
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw keystone authtoken caching still not working with 0.67.4

[Darren Birkett]

Is anyone else using keystone authentication with radosgw?  Anyone having
any luck getting the authtoken caching working?

- Darren


On 8 October 2013 10:17, Darren Birkett  wrote:

> Hi All,
>
> What's the best way to try and track down why this isn't working for us?
>  It doesn't seem that there are any other options I can provide.  Lack of
> (working) radosgw integration with keystone would be a huge blocker for us
> being able to adopt ceph as part of our product set.
>
> Thanks,
> Darren
>
>
> On 7 October 2013 14:28, Darren Birkett  wrote:
>
>> Hi All,
>>
>> In our prior tests with 0.67.3, keystone authtoken caching was broken
>> causing dreadful performance - see
>> http://www.spinics.net/lists/ceph-users/msg04531.html
>>
>> We upgraded to release 0.67.4 as we wanted to test the apparent fix to
>> authtoken caching that was included in the release notes.  Unfortunately,
>> we still seem to be seeing the same issues - keystone is hit for every
>> single operation, causing a huge bottleneck in performance.
>>
>> Aside from upgrading packages and restarting all daemons, are there some
>> options we are missing from the below to get things to work properly:
>>
>> :/etc/ceph/ceph.conf
>> [client.radosgw.gateway]
>> 
>> rgw_keystone_url = http://x.x.x.x:35357
>> rgw_keystone_admin_token = xxx
>> rgw_keystone_accepted_roles = admin, Member
>> rgw_keystone_token_cache_size = 1
>> 
>>
>>
>> Thanks
>> Darren
>>
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephforum.com

[Darren Birkett]

Hi,

I'd have to say in general I agree with the other responders.  Not really
for reasons of preferring a ML over a forum necessarily, but just because
the ML already exists.  One of the biggest challenges for anyone new coming
in to an open source project such as ceph is availability of information
and documentation.  Having this information in as few places as possible,
rather than sprawling over a lot of different formats and locations, makes
it easier to find what you need and to know where to go when you want to
ask a question.

- Darren


On 11 October 2013 00:16, Joao Eduardo Luis  wrote:

> On 10/10/2013 09:55 PM, Wido den Hollander wrote:
>
>> On 10/10/2013 10:49 PM, ja...@peacon.co.uk wrote:
>>
>>> Hello!
>>>
>>> Anyone else think a web forum for ceph could work?  I'm thinking simple
>>> vbulletin or phpBB site.
>>>
>>> To me it seems this would increase accessibility to the great info
>>> (&minds) on here... but obviously it would need those great minds to
>>> work :)
>>>
>>>
>> Well, I'm not sure. Forums are nice, but for technical discussions they
>> most of the time don't work that well.
>>
>> The problem imho usually is that they distract a lot from the technical
>> discussion with all the footers, banners, smilies, etc, etc.
>>
>> Another thing is that most people in the community already have a hard
>> time keeping up the the dev and users mailinglist, so adding another
>> channel of information would make it even harder to keep up.
>>
>> Following up on that you get the problem that you suddenly have multiple
>> channels:
>> - mailinglists
>> - irc
>> - forum
>>
>> So information becomes decentralized and harder to find for people.
>>
>> I'd personally prefer to stick to the mailinglist and IRC.
>>
>
> I'm with Wido.
>
> For a user-facing experience, where you ask a question about something
> ailing you and get an answer, stackoverflow (or the likes) work pretty well
> as it is, with the added benefit that the right answers get upvoted.  I
> however am not sure if it would work that well for Ceph, where answers may
> not be that straightforward.
>
> For technical discussions, I believe the lists tend to be the best format
> to address them.  Besides, as someone who already follows both ceph-users
> and ceph-devel, both irc channels, and the tracker, I feel that following
> an additional forum as well would impose an extra overhead to how we
> interact.  And we should keep in mind that most questions that would end up
> being asked in the forums would have already been answered on the mailing
> lists (which are archived btw), or would end up duplicating things that are
> under current discussion.
>
>
>   -Joao
>
> --
> Joao Eduardo Luis
> Software Engineer | http://inktank.com | http://ceph.com
>
> __**_
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/**listinfo.cgi/ceph-users-ceph.**com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Disk Density Considerations

[Darren Birkett]

Hi,

I understand from various reading and research that there are a number of
things to consider when deciding how many disks one wants to put into a
single chassis:

1. Higher density means higher failure domain (more data to re-replicate if
you lose a node)
2. More disks means more CPU/memory horsepower to handle the number of OSDs
3. Network becomes a bottleneck with too many OSDs per node
4. ...

We are looking at building high density nodes for small scale 'starter'
deployments for our customers (maybe 4 or 5 nodes).  High density in this
case could mean a 2u chassis with 2x external 45 disk JBOD containers
attached.  That's 90 3TB disks/OSDs to be managed by a single node.  That's
about 243TB of potential usable space, and so (assuming up to 75% fillage)
maybe 182TB of potential data 'loss' in the event of a node failure.  On an
uncongested, unused, 10Gbps network, my back-of-a-beer-mat calculations say
that would take about 45 hours to get the cluster back into an undegraded
state - that is the requisite number of copies of all objects.

Assuming that you can shove in a pair of hex core hyperthreaded processors,
you're probably OK with number 2.  If you're already considering 10GbE
networking for the storage network, there's probably not much you can do
about 3 unless you want to spend a lot more money (and the reason we're
going so dense is to keep this as a cheap option).  So the main thing would
seem to be a real fear of 'losing' so much data in the event of a node
failure.  Who wants to wait 45 hours (probably much longer assuming the
cluster remains live and has production traffic traversing that networl)
for the cluster to self-heal?

But surely this fear is based on an assumption that in that time, you've
not identified and replaced the failed chassis.  That you would sit for 2-3
days and just leave the cluster to catch up, and not actually address the
broken node.  Given good data centre processes, a good stock of spare
parts, isn't it more likely that you'd have replaced that node and got
things back up and running in a mater of hours?  In all likelyhood, a node
crash/failure is not likely to have taken out all, or maybe any, of the
disks, and a new chassis can just have the JBODs plugged back in and away
you go?

I'm sure I'm missing some other pieces, but if you're comfortable with your
hardware replacement processes, doesn't number 1 become a non-fear really?
I understand that in some ways it goes against the concept of ceph being
self healing, and that in an ideal world you'd have lots of lower density
nodes to limit your failure domain, but when being driven by cost isn't
this an OK way to look at things?  What other glaringly obvious
considerations am I missing with this approach?

Darren
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Disk Density Considerations

[Darren Birkett]

On 6 November 2013 14:08, Andrey Korolyov  wrote:

> > We are looking at building high density nodes for small scale 'starter'
> > deployments for our customers (maybe 4 or 5 nodes).  High density in this
> > case could mean a 2u chassis with 2x external 45 disk JBOD containers
> > attached.  That's 90 3TB disks/OSDs to be managed by a single node.
>  That's
> > about 243TB of potential usable space, and so (assuming up to 75%
> fillage)
> > maybe 182TB of potential data 'loss' in the event of a node failure.  On
> an
> > uncongested, unused, 10Gbps network, my back-of-a-beer-mat calculations
> say
> > that would take about 45 hours to get the cluster back into an undegraded
> > state - that is the requisite number of copies of all objects.
> >
>
> For such large number of disks you should consider that the cache
> amortization will not take any place even if you are using 1GB
> controller(s) - only tiered cache can be an option. Also recovery will
> take much more time even if you have a room for client I/O in the
> calculations because raw disks have very limited IOPS capacity and
> recovery will either take a much longer than such expectations at a
> glance or affect regular operations. For S3/Swift it may be acceptable
> but for VM images it does not.


Sure, but my argument was that you are never likely to actually let that
entire recovery operation complete - you're going to replace the hardware
and plug the disks back in and let them catch up by log replay/backfill.
 Assuming you don't ever actually expect to really lose all data on 90
disks in one go...

By tiered caching, do you mean using something like flashcache or bcache?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] qemu-kvm packages for centos

[Darren Birkett]

Hi List,

Any chance the following will be updated with the latest packages for
dumpling/emperor:

http://ceph.com/packages/qemu-kvm/centos/x86_64/

Using CentOS 6.4 and dumpling with OpenStack Havana, I am unable to boot
from rbd volumes until I install an rbd-ified qemu-kvm.  I have grabbed the
latest (cuttlefish) and it seems to work ok, but I can't be 100% sure.

I noted the following ticket where the original packaging work was done:

http://tracker.ceph.com/issues/4834

...but nothing seems to have been done since.

Also, is this the official place for that package to live?  I'm in the
process of working with the ceph chef cookbooks, as well as our own
openstack cookbooks, and want to make sure the package is pulled from the
right place.

Thanks
Darren
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com