We (my customer ) are trying to test at Jewell now but I can say that the above behavior was also observed by my customer at Infernalis. After 300 million or so objects in a single bucket the cluster basically fell down as described above. Few hundred osds in this cluster. We are concerned that this may not be remedied by a hundreds of buckets approach as well. Testing continues. On Mon, May 23, 2016 at 7:35 PM Vickey Singh <vickey.singh22...@gmail.com> wrote:
> Hello Guys > > Is several millions of object with Ceph ( for RGW use case ) still an > issue ? Or is it fixed ? > > Thnx > Vickey > > On Thu, Jan 28, 2016 at 12:55 AM, Krzysztof Księżyk <kksie...@gmail.com> > wrote: > >> Stefan Rogge <stefan.ceph@...> writes: >> >> > >> > >> > Hi, >> > we are using the Ceph with RadosGW and S3 setting. >> > With more and more objects in the storage the writing speed slows down >> significantly. With 5 million object in the storage we had a writing speed >> of 10MS/s. With 10 million objects in the storage its only 5MB/s. >> > Is this a common issue? >> > Is the RadosGW suitable for a large amount of objects or would you >> recommend to not use the RadosGW with these amount of objects? >> > >> > Thank you. >> > >> > Stefan >> > >> > I found also a ticket at the ceph tracker with the same issue: >> > >> > >> http://tracker.ceph.com/projects/ceph/wiki/Rgw_-_bucket_index_scalability >> > >> > _______________________________________________ >> > ceph-users mailing list >> > ceph-users@... >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > >> >> Hi, >> >> I'm struggling with the same issue on Ceph 9.2.0. Unfortunately I wasn't >> aware of it and now the only way to improve things is create new bucket >> with bucket index shrading or change way our apps store data into buckets. >> And of course copy tons of data :( In my case also sth happened to leveldb >> files and now I cannot even run some radosgw-admin commands like: >> >> radosgw-admin bucket check -b .... >> >> what causes osd daemon flapping and process timeout messages in logs. PGS >> containing .rgw.bucket.index can't be even backfilled to other osd as >> osd >> process dies with messages: >> >> [...] >> > 2016-01-25 15:47:22.700737 7f79fc66d700 1 heartbeat_map is_healthy >> 'OSD::osd_op_tp thread 0x7f7992c86700' had suicide timed out after 150 >> > 2016-01-25 15:47:22.702619 7f79fc66d700 -1 common/HeartbeatMap.cc: In >> function 'bool ceph::HeartbeatMap::_check(const ceph::heartbeat_handle_d*, >> const char*, time_t)' thread 7f79fc66d700 time 2016-01-25 15:47:22.700751 >> > common/HeartbeatMap.cc: 81: FAILED assert(0 == "hit suicide timeout") >> > >> > ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299) >> > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char >> const*)+0x85) [0x7f7a019f4be5] >> > 2: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char >> const*, long)+0x2d9) [0x7f7a019343b9] >> > 3: (ceph::HeartbeatMap::is_healthy()+0xd6) [0x7f7a01934bf6] >> > 4: (ceph::HeartbeatMap::check_touch_file()+0x2c) [0x7f7a019353bc] >> > 5: (CephContextServiceThread::entry()+0x15b) [0x7f7a01a10dcb] >> > 6: (()+0x7df5) [0x7f79ffa8fdf5] >> > 7: (clone()+0x6d) [0x7f79fe3381ad] >> > >> > >> I don't know - maybe it's because number of leveldb files in omap folder >> (total 5.1GB). Read somewhere that things can be improved by setting >> 'leveldb_compression' to false and leveldb_compact_on_mount to true but I >> don't know if these options have any effect in 9.2.0 as they are not >> documented for this release. Tried with 'leveldb_compression' but without >> visible effect and wasn't brave enough with trying >> leveldb_compact_on_mount >> on production env. But setting it to true on my test 0.94.5 makes osd >> failing on restart. >> >> Kind regards - >> Krzysztof Księżyk >> >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
