Well, compaction does not mean that a partition is unavailable -- that is to say, compaction happens in a non-blocking manner. So worst, case you'lll have disk-related latency hits for a given partition, but requests should still be getting served. Also, a single node only compacts a single partition at a time.
FWIW, in my own testing, with a 50/50 read/write mix, compaction (based on fragmentation %) typically doesn't happen that often, particularly when you have a cluster of machines. Hope that helps. D. On Fri, Jun 11, 2010 at 12:20 PM, Jason J. W. Williams < jasonjwwilli...@gmail.com> wrote: > Is it smart enough to coordinate with the other partitions to ensure > not more than 25% (just a plug number) of the partitions are > compacting at the same time? It would seem to me there's the > possibility for a performance drop if you had the perfect storm of too > many shards compacting at the same time. > > -J > > On Fri, Jun 11, 2010 at 4:54 AM, Justin Sheehy <jus...@basho.com> wrote: > > Hi, Germain. > > > > On Fri, Jun 11, 2010 at 11:07 AM, Germain Maurice > > <germain.maur...@linkfluence.net> wrote: > > > >> Because of its append-only nature, stale data are created, so, how does > >> Bitcask to remove stale data ? > > > > An excellent question, and one that we haven't yet written enough about. > > > >> With CouchDB the compaction process on our data never succeed, too much > >> data. > >> I really don't like to have to launch manually this kind of process. > > > > Bitcask's merging (compaction) process is automated and very tunable. > > These parameters are the most relevant in your bitcask section of > > app.config: > > > > (see the whole thing at > http://hg.basho.com/bitcask/src/tip/ebin/bitcask.app) > > > > %% Merge trigger variables. Files exceeding ANY of these > > %% values will cause bitcask:needs_merge/1 to return true. > > %% > > {frag_merge_trigger, 60}, % >= 60% fragmentation > > {dead_bytes_merge_trigger, 536870912}, % Dead bytes > 512 MB > > > > %% Merge thresholds. Files exceeding ANY of these values > > %% will be included in the list of files marked for merging > > %% by bitcask:needs_merge/1. > > %% > > {frag_threshold, 40}, % >= 40% fragmentation > > {dead_bytes_threshold, 134217728}, % Dead bytes > 128 MB > > {small_file_threshold, 10485760}, % File is < 10 MB > > > > Every few minutes, the Riak storage backend for a given partition will > > send a message to bitcask, requesting that it queue up a possible > > merge job. (only one partition will be in the merge process at once > > as a result of that queue) The bitcask application will examine that > > partition when that request reaches the front of the queue. If any of > > the trigger values have been exceeded, then all of the files in that > > partition which exceed any threshold values will be run through > > compaction. > > > > This allows you a great deal of flexibility in your demands, and also > > provides reasonable amortization of the cost since each partition is > > processed independently. > > > > -Justin > > > > _______________________________________________ > > riak-users mailing list > > riak-users@lists.basho.com > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
