0.8 trunk: When playing back a fairly large chunk of hints, things basically locks up under load. The hints are never processed successfully. Lots of Mutations dropped.
One thing is that maybe the default 10k columns per send with 50ms delays is a bit on the aggressive side (10k*20 =200.000 columns in a second?), the other thing is that it seems like the whole memtable flushing locks up. I tried to increase number of memtable flushers and queue a bit (8 concurrent flushers) to make things work, but no luck. Pool Name Active Pending Completed ReadStage 0 0 1 RequestResponseStage 0 0 2236304 MutationStage 100 17564 4011533 ReadRepairStage 0 0 0 ReplicateOnWriteStage 0 0 0 GossipStage 0 0 2281 AntiEntropyStage 0 0 0 MigrationStage 0 0 0 MemtablePostFlusher 1 13 50 StreamStage 0 0 0 FlushWriter 8 14 73 MiscStage 0 0 0 FlushSorter 0 0 0 InternalResponseStage 0 0 0 HintedHandoff 1 8 3 A quick source code scan makes me believe that the MemtablePostFlusher should not normally use a lot of time, but it seem like it does so here. What may cause this? Terje
