On Tue, Jul 19, 2011 at 11:36 AM, Maxim Potekhin <potek...@bnl.gov> wrote:
> Thanks for pointer. As a comment to your disk speed comment -- > I had already installed SSDs on these nodes. > > Maxim > > > On 7/19/2011 9:41 AM, Edward Capriolo wrote: > > > > On Tue, Jul 19, 2011 at 7:24 AM, Maxim Potekhin <potek...@bnl.gov> wrote: > >> We have something of the order of 200GB load on each of 3 machines in a >> balanced cluster under 0.8.1. >> I started repair about 24hrs ago and did some moderate amount of inserts >> since then (a small fraction of >> data load). The repair still appears to be running. What could go wrong? >> >> Thanks, >> Maxim >> >> > > Repair calculates a merkle tree on each node and then transmits differences > to its neighbours. Because this process run on all your data it can take a > LONG time. > See http://wiki.apache.org/cassandra/LargeDataSetConsiderations > > Use 'nodetool compactionstats' and 'nodetool streams' to check the > progress. My rule of thumb is do not have more storage on a node then you > can compact in 3 hours. So for that you either need fast disks or more > nodes. > > > Well most SSD's are pretty fast. There is one more to consider. If Cassandra determines nodes are out of sync it has to transfer data across the network. If that is the case you have to look at 'nodetool streams' and determine how much data is being transferred between nodes. There are some open tickets where with larger tables repair is streaming more then it needs to. But even if the transfers are only 10% of your 200GB. Transferring 20 GB is not trivial. If you have multiple keyspaces and column families repair one at a time might make the process more manageable.
