After loosing one node we had to repair, CFs was on leveled compaction. For one CF each node had about 7GB of data. Running a repair without primary range switch ended up some nodes exhausted to about 60-100GB of 5MB sstables for that CF (a lot of files). After switching back from leveled to tiered we ended up completely blocked compactions on all nodes since this CF were compacting forever. On one node a major compaction for that CF is CPU bound and may run with unlimited compaction speed for 4-7 days at maximum 1MB/s rate, finally compacting to 3GB of data (some data is deleted by TTL, some merged).
What we did to speedup this process to return all exhausted nodes into normal state faster: We have created a 6 temporary virtual single Cassandra nodes with 2 CPU cores and 8GB RAM. Stopped completely a compaction for CF on a production node. Leveled sstables from this production node was divided into 6 ranges and copied into 6 temporary empty nodes. On each node we ran a major compaction to compact just 1/6 of data, about 10-14GB. It took 1-2 hours to compact them into 1GB of data. Then all 6 sstables was copied into one of 6 nodes for a major compaction, finally getting expected 3GB sstable. Stopping production node, deleting files that was copied, returning compacted (may need renaming) and node is back to normal. Using separate nodes we saved original production nodes time not to compact exhausted CF forever, blocking compactions for other CFs. With 6 separate nodes we have compacted 2 productions nodes a day, so maybe it took the same time, but production nodes were free for regular compactions for other CFs. After back to normal for our use case we stick to tiered compaction with a major compaction nightly. With our insertion/TTL deletion rates a leveled compaction is a nightmare, even if amount of data is not very huge, just a few GBs/node. 2012/3/13 Thorsten von Eicken <t...@rightscale.com> > On 3/12/2012 6:52 AM, Brandon Williams wrote: > > On Mon, Mar 12, 2012 at 4:44 AM, aaron morton <aa...@thelastpickle.com> > wrote: > >> I don't understand why I > >> don't get multiple concurrent compactions running, that's what would > >> make the biggest performance difference. > >> > >> concurrent_compactors > >> Controls how many concurrent compactions to run, by default it's the > number > >> of cores on the machine. > I'm on a quad-core machine so not setting concurrent_compactors should > not be a limiting factor... > > With leveled compaction, I don't think you get any concurrency because > > it has to compact an entire level, and it can't proceed to the next > > level without completing the one before it. > > > > In short, if you want maximum throughput, stick with size tiered. > I switched the CFs to tiered compaction and I still get no concurrency > for the same CF. I now have two compactions running concurrently but > always for different CFs. I've briefly seen a third for one of the small > CFs, so it's willing to run more than two concurrently. Looks like I > have to wait for a few days for all the compactions to complete. Talk > about compaction hell! > > > > > -Brandon > > >
