Re: Issue with removing a node and adding it back

2015-03-30 Thread Robert Coli
On Fri, Mar 27, 2015 at 4:27 PM, Shiwen Cheng wrote: > Thanks Robert! > Yes I tried what you said: clean the data and re-bootstrap. But still it > failed, once at the point of 600GB transferred and once at 1.1TB :( > 1) figure out what is making your streams die (usually either flaky network (A

Re: Issue with removing a node and adding it back

2015-03-27 Thread Shiwen Cheng
Thanks Robert! Yes I tried what you said: clean the data and re-bootstrap. But still it failed, once at the point of 600GB transferred and once at 1.1TB :( But I could see following exceptions from time to time: = java.io.IOException: net.jpountz.lz4.LZ4Exception: Error decodin

Re: Issue with removing a node and adding it back

2015-03-26 Thread Robert Coli
On Thu, Mar 26, 2015 at 11:31 AM, Shiwen Cheng wrote: > I encountered an issue by removing and adding back a node. > You are encountering a failed/hung bootstrap, which probably has nothing to do with the node having been previously removenoded. Stop the node, wipe all the data on the node, inc

Issue with removing a node and adding it back

2015-03-26 Thread Shiwen Cheng
Hi all, I encountered an issue by removing and adding back a node. Here is how this issue came out: (1) We have four nodes cluster running, but there was a hard disk failure on one of the node. Since we need to replace the hard disk, I chose to use *removenode *to remove the failed node. (2) few d