Marcus, You may want to look at the bluestore_min_alloc_size setting as well as the respective bluestore_min_alloc_size_ssd and bluestore_min_alloc_size_hdd. By default bluestore sets a 64k block size for ssds. I'm also using ceph for small objects and I've see my OSD usage go down from 80% to 20% after setting the min alloc size to 4k.
Thanks, Pavel On Thu, Aug 3, 2017 at 3:59 PM, Gregory Farnum <gfar...@redhat.com> wrote: > Don't forget that at those sizes the internal journals and rocksdb size > tunings are likely to be a significant fixed cost. > > On Thu, Aug 3, 2017 at 3:13 AM Wido den Hollander <w...@42on.com> wrote: >> >> >> > Op 2 augustus 2017 om 17:55 schreef Marcus Haarmann >> > <marcus.haarm...@midoco.de>: >> > >> > >> > Hi, >> > we are doing some tests here with a Kraken setup using bluestore backend >> > (on Ubuntu 64 bit). >> > We are trying to store > 10 mio very small objects using RADOS. >> > (no fs, no rdb, only osd and monitors) >> > >> > The setup was done with ceph-deploy, using the standard bluestore >> > option, no separate devices >> > for wal. The test cluster spreads over 3 virtual machines, each with >> > 100GB storage für osd. >> > >> > We are now in the following situation (used pool is "test"): >> > rados df >> > POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRAED >> > RD_OPS RD WR_OPS WR >> > rbd 0 2 0 6 0 0 0 49452 39618k 855 12358k >> > test 17983M 595427 0 1786281 0 0 0 29 77824 596426 17985M >> > >> > total_objects 595429 >> > total_used 141G >> > total_avail 158G >> > total_space 299G >> > >> > ceph osd df >> > ID WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS >> > 0 0.09760 1.00000 102298M 50763M 51535M 49.62 1.00 72 >> > 1 0.09760 1.00000 102298M 50799M 51499M 49.66 1.00 72 >> > 2 0.09760 1.00000 102298M 50814M 51484M 49.67 1.00 72 >> > TOTAL 299G 148G 150G 49.65 >> > MIN/MAX VAR: 1.00/1.00 STDDEV: 0.02 >> > >> > As you can see, there are about 18GB data stored in ~595000 objects now. >> > The actual space consumption is about 150GB, which fills about half of >> > the storage. >> > >> >> Not really. Each OSD uses 50GB, but since you replicate 3 times (default) >> it's storing 150GB spread out over 3 OSDs. >> >> So your data is 18GB, but consumes 50GB. That's still ~2.5x which is a >> lot, but a lot less then 150GB. >> >> > Objects have been added with a test script using the rados command line >> > (put). >> > >> > Obviously, the stored objects are counted byte by byte in the rados df >> > command, >> > but the real space allocation is about factor 8. >> > >> >> As written above, it's ~2.5x, not 8x. >> >> > The stored objects are a mixture of 2kb, 10kb, 50kb, 100kb objects. >> > >> > Is there any recommended way to configure bluestore with a better >> > suitable >> > block size for those small objects ? I cannot find any configuration >> > option >> > which would allow modification of the internal block handling of >> > bluestore. >> > Is luminous an option which allows more specific configuration ? >> > >> >> Could you try this with the Luminous RC as well? I don't know the answer >> here, but since Kraken a LOT has been improved to BlueStore. >> >> Wido >> >> > Thank you all in advance for support. >> > >> > Marcus Haarmann >> > >> > _______________________________________________ >> > ceph-users mailing list >> > ceph-users@lists.ceph.com >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> _______________________________________________ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
