Based on a quick read of the code, compaction in bitcask is performed only on "readable" files, and the current active file for writing is excluded from that list. With default settings, that active file can grow to 2GB. So it is possible that if objects had been replaced/deleted many times within the active file, that space will not be recovered if all the replacements amount to < 2GB per vnode. So at these small data sizes - you may get a relatively significant discrepancy between an old and recovered node in terms of disk space usage.
On 8 August 2018 at 17:37, Martin Sumner <martin.sum...@adaptip.co.uk> wrote: > Sean, > > Some partial answers to your questions. > > I don't believe force-replace itself will sync anything up - it just > reassigns ownership (hence handoff happens very quickly). > > Read repair would synchronise a portion of the data. So if 10% of you > data is read regularly, this might explain some of what you see. > > AAE should also repair your data. But if nothing has happened for 4 days, > then that doesn't seem to be the case. It would be worth checking the > aae-status page (http://docs.basho.com/riak/kv/2.2.3/using/admin/riak- > admin/#aae-status) to confirm things are happening. > > I don't know if there are any minimum levels of data before bitcask will > perform compaction. There's nothing obvious in the code that wouldn't be > triggered way before 90%. I don't know if it will merge on the active file > (the one currently being written to), but that is 2GB max size (configured > through bitcask.max_file_size). > > When you say the size of the bitcask directory - is this the size shared > across all vnodes on the node? I guess if each vnode has a single file > <2GB, and there are multiple vnodes - something unexpected might happen > here? If bitcask does indeed not merge the file active for writing. > > In terms of distribution around the cluster, if you have an n_val of 3 you > should normally expect to see a relatively even distribution of the data on > failure (certainly not it all going to one). Worst case scenario is that 3 > nodes get all the load from that one failed node. > > When a vnode is inaccessible, 3 (assuming n=3) fallback vnodes are > selected to handle the load for that 1 vnode (as that vnode would normally > be in 3 preflists, and commonly a different node will be asked to start a > vnode for each preflist). > > > I will try and dig later into bitcask merge/compaction code, to see if I > spot anything else. > > Martin > > >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
