Jonathan Ellis <jbellis <at> gmail.com> writes: > > If you read the stack traces you pasted, the node in question ran out > of diskspace. When you have < 25% space free this is not surprising. > > But fundamentally you are missing something important from your story > here. Disk space doesn't just increase spontaneously with "absolutely > no activity."
I agree, it's not surprising for a node to run out of disk space when it is occupying 122GB (133GB including the still present compacted files) on a 160GB drive. What I don't understand is how 30GB of data grew into 122GB. I could send you the Cassandra log files from the time period after the 800,000 rows of writes complete. There was no activity other than Cassandra doing whatever work it does behind the scenes. The 800,000 unique key writes started at 12:15pm and completed at 1:47pm. These 8 vms were created solely for this test and I was the only one logged on. After the writes were done, Cassandra did what it does behind the scenes, there was no human intervention and no clients running/connecting to it. I didn't log in again to check on things until 11pm that night which is when I saw the uneven data distribution (122GB on one of the nodes). I actually have the log files from all 8 nodes if it helps to diagnose what activity was going on behind the scenes. I really need to understand how this happened. Thanks again for your help, Julie
