I succeeded to insert 1 billion records into a single node cassandra, >> bin/stress -d cas01 -o insert -n 1000000000 -c 5 -S 34 -C5 -t 20 Inserts finished in about 14 hours at a speed of 20k/sec.
But when I added another node, tests always failed with UnavailableException in an hour. >> bin/stress -d cas01,cas02 -o insert -n 1000000000 -c 5 -S 34 -C5 -t 20 Writes speed is also 20k/sec because of the bottleneck in the client, so the pressure on each server node should be 50% of the single node test. Why couldn't they handle? By default, rf=1, consistency=ONE Some information that may be helpful, 1. no warn/error in log file, the cluster is still alive after those exception 2. the last logs on both nodes happen to be a compaction complete info 3. gossip log shows one node is dead and then up again in 3 seconds 4. I set hinted_handoff_enabled: false, but still see lots of handoff logs
