Dan -
Q: What does the syntax: 16#80000000 represent in the max_file_size
parameter? It's supposed to be 2GB, but I can't see where that means 2GB
anywhere.
Even if that meant 16 files of 80MB each, that only comes out to slightly
over 1GB.
- Steve
--
Steve Webb - Senior System Administrator for gnip.com
http://twitter.com/GnipWebb
On Mon, 13 Jun 2011, Dan Reverri wrote:
Hi Steve,
The article points out that the active data file is not considered during
merge checks. Your 250-ish MB data file is the active file and not
considered during the merge check. The file will eventually role over to a
non-active file when it hits 2 GB in size. Once the file is not active it
will be considered during the merge check and merging will take place.
The 2 GB file size is configurable via the max_file_size parameter:
https://github.com/basho/bitcask/blob/master/ebin/bitcask.app#L22
Thanks,
Dan
Daniel Reverri
Developer Advocate
Basho Technologies, Inc.
d...@basho.com
On Mon, Jun 13, 2011 at 2:38 PM, Steve Webb <sw...@gnip.com> wrote:
Dan -
I've got dead_bytes_threshold=5242880 (5M) and
dead_bytes_merge_trigger=10242880. My bitcask *.data files are 250-ish MB
in size:
root@ha2:/data/riaksearch/bitcask/1027618338748291114361965898003636498195577569280#
ls -lah
total 771M
drwxr-xr-x 2 riak riak 4.0K 2011-06-12 01:08 .
drwxr-xr-x 34 riak riak 4.0K 2011-06-12 01:10 ..
-rw------- 1 riak riak 229M 2011-06-08 13:11 1307415077.bitcask.data
-rw-r--r-- 1 riak riak 4.3M 2011-06-08 13:11 1307415077.bitcask.hint
-rw------- 1 riak riak 276M 2011-06-10 13:30 1307562153.bitcask.data
-rw-r--r-- 1 riak riak 5.1M 2011-06-10 13:30 1307562153.bitcask.hint
-rw------- 1 riak riak 1.4M 2011-06-08 13:45 1307562333.bitcask.data
-rw-r--r-- 1 riak riak 27K 2011-06-08 13:45 1307562333.bitcask.hint
-rw------- 1 riak riak 246M 2011-06-13 15:34 1307862506.bitcask.data
-rw-r--r-- 1 riak riak 9.4M 2011-06-13 15:34 1307862506.bitcask.hint
-rw------- 1 riak riak 107 2011-06-12 01:08 bitcask.write.lock
I'm pretty sure that 50% or more of the data in these files should've
aged-off by now and the merge trigger should've happened. The article shows
why merges happen when a restart is done, but it doesn't really explain why
merges don't happen at normal runtime.
I really don't want to restart riak every day to merge files.
Q: What are some good trigger settings for my use case?
I want to collect and store 1 day worth of tweets from the twitter spritzer
feed and have the data files auto-merge once in a while (once a day or more
frequently) when they've gotten 10% of 'dead' data in them (aka, the tweets
expire after 1 day).
- Steve
--
Steve Webb - Senior System Administrator for gnip.com
http://twitter.com/GnipWebb
On Mon, 13 Jun 2011, Dan Reverri wrote:
Hi Steve,
This Knowledge Base article may be related:
https://help.basho.com/entries/20141178-why-does-it-seem-that-bitcask-merging-is-only-triggered-when-a-riak-node-is-restarted
Thanks,
Dan
Daniel Reverri
Developer Advocate
Basho Technologies, Inc.
d...@basho.com
On Mon, Jun 13, 2011 at 10:25 AM, Steve Webb <sw...@gnip.com> wrote:
Justin -
My current bitcask settings are:
%% Bitcask Config
{bitcask, [
{data_root, "/var/lib/riaksearch/bitcask" },
{dead_bytes_merge_trigger, 10242880 },
{dead_bytes_threshold, 5242880 },
{expiry_secs, 86400}
]},
My understanding of these settings mean that the data should auto-expire
after one day. Also, once each bitcask file in
.../riaksearch/bitcask/xxx/*.data once it has 10M of "dead" or expired
data
in it, should be merged, right?
I'm collecting the spritzer twitter stream and loading it into two
buckets
(one non-indexed bucket holds the full tweet, one indexed bucket holds
the
tweet string, id, date and username). I used to see about 10 GB of data
total, but it's growing and currently at 26GB of data total.
I'm seeing these in the logs:
INFO REPORT==== 13-Jun-2011::08:28:19 ===
Pid <0.6844.0> compacted 3 segments for 942232 bytes in 4.900694 seconds,
0.18 MB/sec
=INFO REPORT==== 13-Jun-2011::08:29:01 ===
Pid <0.6267.0> compacted 3 segments for 1721790 bytes in 9.690511
seconds,
0.17 MB/sec
=INFO REPORT==== 13-Jun-2011::08:31:23 ===
Pid <0.6924.0> compacted 3 segments for 6988416 bytes in 44.659753
seconds,
0.15 MB/sec
... but I'm not seeing any "merging" related entries.
- Steve
--
Steve Webb - Senior System Administrator for gnip.com
http://twitter.com/GnipWebb
On Wed, 8 Jun 2011, Justin Sheehy wrote:
Hi, Steve.
Check out this page:
http://wiki.basho.com/Bitcask-Configuration.html#Disk-Usage-and-Merging-Settings
Basically, a "merge trigger" must be met in order to have the merge
process occur. When it does occur, it will affect all existing files
that
meet a "merge threshold."
One note that is relevant for your specific use: the expiry_secs
parameter
will cause a given item to disappear from the client API immediately
after
expiry, and to be cleaned if it is in a file already being merged, but
will
not currently contribute toward merge triggers or thresholds on its own
if
not otherwise "dead".
-Justin
On Jun 7, 2011, at 4:29 PM, Steve Webb wrote:
Hello there.
I'm curious - I'm up to about 10GB of storage and I'm guessing that
I'll
be full in 3-4 more days of ingesting data. I have no idea if/when a
merge
will run to expire the older data.
I'm loading a 2-node (1GB mem, 20GB storage, vmware VMs) riaksearch
cluster with the spritzer twitter feed. I used the bitcask
'expiry_secs' to
expire data after 3 days. Q: Is there a method or command to force a
merge
at any time? Q: Is there a way to run a merge when the storage size
reaches
a specific threshold?
- Steve
--
Steve Webb - Senior System Administrator for gnip.com
http://twitter.com/GnipWebb
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
