Thanks for your help Peter. We gave up and rolled back to our mysql implementation (we did all writes to our old store in parallel so we did not lose anything). Problem was that every solution we came up with would require at least on major compaction before the new nodes could join and our cluster could not survive this (in terms of serving requests at reasonable latencies).
But thanks anyway, Daniel On Dec 9, 2010, at 8:25 PM, Peter Schuller wrote: >> Currently I am copying all data files (thats all existing data) from one >> node to the new nodes in hope that I could than manually assign them their >> new tokenrange (nodetool move) and do cleanup. > > Unless I'm misunderstanding you I believe you should be setting the > initial token. nodetool move would be for a node already in the ring. > And keep in mind that a nodetool move is currently a > decommission+bootstrap - so if you're teetering on the edge of > overload you will want to keep that in mind when moving a node to > avoid ending up in a worse situation as another node temporarily > receives more load than usual as a result of increased ring ownership. > >> Obviously I will try this tomorrow (it's been a long day) on a test system >> but any advice would be highly appreciated. > > One possibility if you have additional hardware to spare temporarily, > is to add more nodes than you actually need and then, once you are > significantly over capacity, you have the flexibility to move nodes > around to an optimum position and then decommission those machines > that were only "borrowed". I.e., initial bootstrap of nodes takes a > shorter amount of time because you're giving them less token space per > new node. And once all are in the ring, you're free to move things > around and then free up the hardware. > > (Another option may be to implement throttling of the anti-compaction > so that it runs very slowly during peak hours, but that requires > patching cassandra or else firewall/packet filtering fu and is > probably likely to be more risky than it's worth.) > > -- > / Peter Schuller
