Re: [ceph-users] Planning for many small files

2013-03-17 Thread Gregory Farnum
On Sunday, March 17, 2013 at 5:49 PM, Yehuda Sadeh wrote: > Not at the moment. We had some discussions about "blind" buckets, it's > definitely on our mind, but we're not there yet. > > Yehuda > > On Sun, Mar 17, 2013 at 3:09 PM, Rustam Aliyev (mailto:rustam.li...@code.az)> wrote: > > Thanks for

Re: [ceph-users] Planning for many small files

2013-03-17 Thread Yehuda Sadeh
Not at the moment. We had some discussions about "blind" buckets, it's definitely on our mind, but we're not there yet. Yehuda On Sun, Mar 17, 2013 at 3:09 PM, Rustam Aliyev wrote: > Thanks for detailed explanation. > > Is there any way to disable bucket indexes? We already store index in our >

Re: [ceph-users] Planning for many small files

2013-03-17 Thread Rustam Aliyev
Thanks for detailed explanation. Is there any way to disable bucket indexes? We already store index in our Cassandra cluster and need RADOS only to store objects. We don't plan to do any listing operations, only PUT and GET. On 17/03/2013 16:24, Gregory Farnum wrote: RADOS doesn't store a l

Re: [ceph-users] Planning for many small files

2013-03-17 Thread Gregory Farnum
RADOS doesn't store a list of objects. The RADOS Gateway uses a separate data format on top of objects stored in RADOS, and it keeps a per-user list of buckets and a per-bucket index of objects as "omap" objects in the OSDs (which ultimately end up in a leveldb store). A bucket index is currentl

Re: [ceph-users] Planning for many small files

2013-03-11 Thread Rustam Aliyev
Thanks Sam, That's great. I'm trying to understand a bit of RADOS internals and I went through architecture wiki. Yet unclear about some points. Where does RADOS store the list of objects (object metadata)? According to RADOSGW docs S3 bucket listing is available, so it must be stored somewh

Re: [ceph-users] Planning for many small files

2013-03-11 Thread Sam Lang
On Fri, Mar 8, 2013 at 9:57 PM, Rustam Aliyev wrote: > Hi, > > We need to store ~500M of small files (<1MB) and we were looking to RadosGW > solution. We expect about 20 ops/sec (read+write). I'm trying to understand > how Monitoring nodes store Crush maps and what are the limitations. > > For ins

[ceph-users] Planning for many small files

2013-03-08 Thread Rustam Aliyev
Hi, We need to store ~500M of small files (<1MB) and we were looking to RadosGW solution. We expect about 20 ops/sec (read+write). I'm trying to understand how Monitoring nodes store Crush maps and what are the limitations. For instance, is there any recommended max number of objects per Mo