On 3/12/19 7:24 AM, Benjamin Zapiec wrote:
Hello,
i was wondering about ceph block.db to be nearly empty and I started
to investigate.
The recommendations from ceph are that block.db should be at least
4% the size of block. So my OSD configuration looks like this:
wal.db - not explicit specified
block.db - 250GB of SSD storage
block - 6TB
By default we currently use 4 256MB WAL buffers. 2GB should be enough,
though in most cases you are better off just leaving it on block.db as
you did below.
Since wal is written to block.db if not available i didn't configured
wal. With the size of 250GB we are slightly above 4%.
WAL will only use about 1GB of that FWIW
So everything should be "fine". But the block.db only contains
about 10GB of data.
If this is an RBD workload, that's quite possible as RBD tends to use
far less metadata than RGW.
If figured out that an object in block.db gets "amplified" so
the space consumption is much higher than the object itself
would need.
Data in the DB in general will suffer space amplification and it gets
worse the more levels in rocksdb you have as multiple levels may have
copies of the same data at different points in time. The bigger issue
is that currently an entire level has to fit on the DB device. IE if
level 0 takes 1GB, level 1 takes 10GB, level 2 takes 100GB, and level 3
takes 1000GB, you will only get 0, 1 and 2 on block.db with 250GB.
I'm using ceph as storage backend for openstack and raw images
with a size of 10GB and more are common. So if i understand
this correct i have to consider that a 10GB images may
consume 100GB of block.db.
The DB holds metadata for the images (and some metadata for bluestore).
This is going to be a very small fraction of the overall data size but
is really important. Whenever we do a write to an object we first try
to read some metadata about it (if it exists). Having those read
attempts happen quickly is really important to make sure that the write
happens quickly.
Beside the facts that the image may have a size of 100G and
they are only used for initial reads unitl all changed
blocks gets written to a SSD-only pool i was question me
if i need a block.db and if it would be better to
save the amount of SSD space used for block.db and just
create a 10GB wal.db?
See above. Also, rocksdb periodically has to compact data and with lots
of metadata (and as a result lots of levels) it can get pretty slow.
Having rocksdb on fast storage helps speed that process up and avoid
write stalls due to level0 compaction (higher level compaction can
happen in alternate threads).
Has anyone done this before? Anyone who had sufficient SSD space
but stick with wal.db to save SSD space?
If i'm correct the block.db will never be used for huge images.
And even though it may be used for one or two images does this make
sense? The images are used initially to read all unchanged blocks from
it. After a while each VM should access the images pool less and
less due to the changes made in the VM.
The DB is there primarily to store metadata. RBD doesn't use a lot of
space but may do a lot of reads from the DB if it can't keep all of the
bluestore onodes in it's own in-memory cache (the kv cache). RGW uses
the DB much more heavily and in some cases you may see 40-50% space
usage if you have tiny RGW objects (~4KB). See this spreadsheet for
more info:
https://drive.google.com/file/d/1Ews2WR-y5k3TMToAm0ZDsm7Gf_fwvyFw/view?usp=sharing
Mark
Any thoughts about this?
Best regards
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
