I have 1 DC that was originally 3 nodes each set with a single token: '-9223372036854775808', '-3074457345618258603', '3074457345618258602'
I added two more nodes and ran nodetool move and nodetool cleanup one server at a time with these tokens: '-9223372036854775808', '-5534023222112865485', '-1844674407370955162', '1844674407370955161', '5534023222112865484' Everything looked good so I changed the replication factor for my keyspace from 1 to 2 and started running nodetool repair on each node. The first node ran for a while then threw an error: " Repair session 8d2a1190-25aa-11e4-8a15-ff681618d551 for range (1844674407370955161,5534023222112865484] failed with error org.apache.cassandra.exceptions.RepairException: [repair #8d2a1190-25aa-11e4-8a15-ff681618d551 on PLAGIARISM/STATS, (1844674407370955161,5534023222112865484]] Validation failed in /###.###.###" Since this was a temporary Column Family, I just dropped that CF and tried to run nodetool repair again. It ended up giving me the same error with a different CF. I tried nodetool cleanup and then nodetool repair again and it eventually crashed the node (Status: DN). When I restarted Cassandra, it still had the "initial_token" value as "-9223372036854775808'" but the default "num_tokens" value was 256. When I checked the status it showed that it had 256 tokens (while my other 4 nodes still had 1). Luckily, it chose 256 tokens that were in the existing token range for that server, so it had the same "owns" value. My question is three fold: 1) Is it better to use 256 vnodes and just move the other 4 servers to have 256 tokens as well or is it possible (or better) to change the tokens back to just a single token for the first server again? I did see warnings about not using "shuffle" and the new method is to create a new DC and move it over, but I don't have space to do this (the current DC is 31TB). I'm fine with 256 tokens that are in the original token range, so would it be ok to just not ever run the shuffle but add the 256 tokens to each server? If I should change it back to 1 token per server again, is it possible to do so w/o decommissioning and removing all existing data and restarting? With having such a large dataset, I feel that nodetool repair works better when there are more tokens (my theory is that its working in smaller chunks), is this a good reason to use 256 instead of one? 2) How can I fix the Repair Exception above? 3) Nodetool repair takes forever to run (5+ days). Is this because I have 1 token per node or is there a better way to run this? Should I set the start and end keys? I'm running Cassandra 2.0.2 Any help would be greatly appreciated. Thanks, Bryan
