On Mon, Apr 2, 2018 at 8:21 AM Kevin Hrpcek <kevin.hrp...@ssec.wisc.edu> wrote:
> Hello, > > We use python librados bindings for object operations on our cluster. For > a long time we've been using 2 ec pools with k=4 m=1 and a fixed 4MB > read/write size with the python bindings. During preparations for migrating > all of our data to a k=6 m=2 pool we've discovered that ec pool alignment > size is dynamic and the librados bindings for python and go fail to write > objects because they are not aware of the the pool alignment size and > therefore cannot adjust the write block size to be a multiple of that. The > ec pool alignment size seems to be (k value * 4K) on new pools, but is only > 4K on old pools from the hammer days. We haven't been able to find much > useful documentation for this pool alignment setting other than the > librados docs (http://docs.ceph.com/docs/master/rados/api/librados) > rados_ioctx_pool_requires_alingment, rados_ioctx_pool_requires_alignment2, > rados_ioctx_pool_required_alignment, rados_ioctx_pool_required_alignment2. > After going through the rados binary source we found that the binary is > rounding the write op size for an ec pool to a multiple of the pool > alignment size (line ~1945 > https://github.com/ceph/ceph/blob/master/src/tools/rados/rados.cc#L1945). > The min write op size can be figured out by writing to an ec pool like this > to get the binary to round up and print it out `rados -b 1k -p $pool put > .....`. All of the support for being alignment aware is obviously available > but simply isn't available in the bindings, we've only tested python and > go. > > We've gone ahead and submitted a patch and pull request to the pycradox > project which seems to be what was merged into the ceph project for python > bindings https://github.com/sileht/pycradox/pull/4. It replicates getting > the alignment size of the pool in the python bindings so that we can then > calculate the proper op sizes for writing to a pool > > We find it hard to believe that we're the only ones to have run into this > problem when using the bindings. Have we missed something obvious for > cluster configuration? Or maybe we're just doing things different compared > to most users... Any insight would be appreciated as we'd prefer to use an > official solution rather than our bindings fix for long term use. > It's not impossible you're the only user both using the python bindings and targeting EC pools. Even now with overwrites they're limited in terms of object class and omap support, and I think all the direct-access users I've heard about required at least one of omap or overwrites. Just submit the patch to the Ceph github repo and it'll get fixed up! :) -Greg > > Tested on Luminous 12.2.2 and 12.2.4. > > Thanks, > Kevin > > -- > Kevin Hrpcek > Linux Systems Administrator > NASA SNPP Atmospheric SIPS > Space Science & Engineering Center > University of Wisconsin-Madison > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
