Hi everyone,
Upgraded to 1.4.8 and got into some issues with AAE overloading our vnodes.
We’ve worked around that.
But now we’re seeing a merge issue manifest itself since we are into our
nightly merge window.
I see several vnodes on multiple nodes with similar in the logs:
2014-02-26 20:27:39.736 [info] <0.1040.0>
"/var/lib/riak/bitcask/639406966332270026714112114313373821099470487552"
needs_merge: []
2014-02-26 20:27:39.736 [info] <0.29858.20> Merged {[],[]} in 1.0e-6 seconds.
2014-02-26 20:30:39.739 [info] <0.1040.0>
"/var/lib/riak/bitcask/639406966332270026714112114313373821099470487552"
needs_merge: []
2014-02-26 20:30:39.739 [info] <0.32391.20> Merged {[],[]} in 2.0e-6 seconds.
2014-02-26 20:33:39.742 [info] <0.1040.0>
"/var/lib/riak/bitcask/639406966332270026714112114313373821099470487552"
needs_merge: []
2014-02-26 20:33:39.742 [info] <0.2113.21> Merged {[],[]} in 1.0e-6 seconds.
2014-02-26 20:36:39.744 [info] <0.1040.0>
"/var/lib/riak/bitcask/639406966332270026714112114313373821099470487552"
needs_merge: []
2014-02-26 20:36:39.744 [info] <0.4756.21> Merged {[],[]} in 1.0e-6 seconds.
2014-02-26 20:39:39.746 [info] <0.1040.0>
"/var/lib/riak/bitcask/639406966332270026714112114313373821099470487552"
needs_merge: []
2014-02-26 20:39:39.746 [info] <0.7288.21> Merged {[],[]} in 1.0e-6 seconds.
Looking at that vnodes bit cask folder:
root@app25u:/var/lib/riak/bitcask/639406966332270026714112114313373821099470487552#
ls -ltrh
total 7.3G
-rw-rw-r-- 1 riak riak 499M Jun 22 2013 2.bitcask.data
-rw-rw-r-- 1 riak riak 2.4M Jun 22 2013 2.bitcask.hint
-rw-rw-r-- 1 riak riak 43K Oct 3 11:25 9.bitcask.hint
-rw-rw-r-- 1 riak riak 15M Oct 3 11:25 9.bitcask.data
-rw-rw-r-- 1 riak riak 51M Oct 6 19:35 12.bitcask.data
-rw-rw-r-- 1 riak riak 129K Oct 6 20:49 12.bitcask.hint
-rw-r--r-- 1 riak riak 121M Dec 23 20:56 15.bitcask.data
-rw-r--r-- 1 riak riak 301K Dec 23 21:19 15.bitcask.hint
-rw-r--r-- 1 riak riak 804K Feb 5 12:41 17.bitcask.hint
-rw-r--r-- 1 riak riak 304M Feb 5 12:41 17.bitcask.data
-rw-r--r-- 1 riak riak 2.0G Feb 5 12:42 18.bitcask.data
-rw-r--r-- 1 riak riak 5.0M Feb 5 12:42 18.bitcask.hint
-rw-r--r-- 1 riak riak 2.0G Feb 5 12:43 20.bitcask.data
-rw-r--r-- 1 riak riak 5.1M Feb 5 12:43 20.bitcask.hint
-rw-r--r-- 1 riak riak 1.7G Feb 5 12:43 22.bitcask.data
-rw-r--r-- 1 riak riak 8.7M Feb 5 12:43 22.bitcask.hint
-rw-r--r-- 1 riak riak 495M Feb 23 17:30 77.bitcask.data
-rw-r--r-- 1 riak riak 1.5M Feb 23 17:34 77.bitcask.hint
-rw-r--r-- 1 riak riak 71M Feb 26 09:05 78.bitcask.data
-rw-r--r-- 1 riak riak 194K Feb 26 09:06 78.bitcask.hint
-rw-r--r-- 1 riak riak 13M Feb 26 18:05 80.bitcask.data
-rw-r--r-- 1 riak riak 33K Feb 26 18:05 80.bitcask.hint
-rw-r--r-- 1 riak riak 75K Feb 26 18:32 79.bitcask.hint
-rw-r--r-- 1 riak riak 28M Feb 26 18:32 79.bitcask.data
-rw------- 1 riak riak 93 Feb 26 18:32 bitcask.write.lock
-rw-r--r-- 1 riak riak 1.7K Feb 26 20:45 81.bitcask.hint
-rw-r--r-- 1 riak riak 667K Feb 26 20:45 81.bitcask.data
We’re starting riak via the /etc/init.d/riak script and have a
/etc/default/riak file with ulimit -n 65535 in it so open files should not be
an issue (and have not been previously with 1.4.7 and earlier).
Any suggestions?
Thanks,
John
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
