Re: Lots of bitcask files for a vnode, unable to merge

2011-07-01 Thread David Smith
On Fri, Jul 1, 2011 at 4:30 AM, Nico Meyer wrote: > > Bitcask doesn't handle corrupted files very well in all cases, which is > unfortunate. I patched our version of Riak/bitcask, to gracefully handle all Yes, we've recently corrected some of these edge cases here: https://github.com/basho/bitca

Re: Lots of bitcask files for a vnode, unable to merge

2011-07-01 Thread Nico Meyer
Hi Apyr, I have had this problem in the past. Most likely the 1309482913.bitcask.data is corrupted somehow. I mostly see this problem after a machine crash or disk problems, and in this case it is alway the end of the file that is corrupted or truncated (the last record that Riak was trying t

Re: Lots of bitcask files for a vnode, unable to merge

2011-06-30 Thread David Smith
The fix is quite easy -- you can safely delete any zero-length files (while the node is down!). How it got to that state is less clear. The "every three minutes" suggests that the vnode triggers a merge only to find there is nothing to actually merge. It is odd, however, that it doesn't clean up t

Lots of bitcask files for a vnode, unable to merge

2011-06-30 Thread Aphyr
One of the vnodes on one of my hosts has a *lot* of bitcask data/hint files, and makes a new one every 3 minutes. In the logs, I get =ERROR REPORT 30-Jun-2011::20:24:14 === Failed to merge ["/var/lib/riak/bitcask/794976964837219653749465284983368790965189869568", [], ...HUGE LIST OF DATA