Yes. If you have exhausted all the options I think it will be good to see if this issue persists accross other nodes after you decommission that node.
If this is not production and issue is reproducible easily you can also try using strace with fork option to see if it gets killed at the same place. Are these vms? On Wed, Oct 12, 2011 at 12:41 PM, Ashley Martens <amart...@ngmoco.com> wrote: > We have 20 nodes in this cluster. > Yes, however are you recommending that I decommission the node? > > I noted the compaction because it is common for the last line in the log > file. For reference: > INFO [FlushWriter:12] 2011-10-12 18:10:09,823 Memtable.java (line 157) > Writing Memtable-HintsColumnFamily@ > 1313157506(0 bytes, 1 operations) > INFO [CompactionExecutor:1] 2011-10-12 18:10:09,823 CompactionManager.java > (line 395) Compacting > [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily-f-7958-Data.db'),SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily-f-7959-Data.db'),SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily-f-7960-Data.db'),SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily-f-7961-Data.db')] > INFO [FlushWriter:12] 2011-10-12 18:10:09,862 Memtable.java (line 172) > Completed flushing > /var/lib/cassandra/data/system/HintsColumnFamily-f-7962-Data.db (61 bytes) > Load, cpu and memory are nominal. The box is not stressed. iostat reports > low usage. > > On Wed, Oct 12, 2011 at 12:24 PM, Mohit Anchlia <mohitanch...@gmail.com> > wrote: >> >> You mentioned this happens only on one node? How many nodes do you >> have? Is it possible to turn off this node completely and run >> compactions on other nodes and see if this happens there too? >> >> Also, you mentioned this happens after compaction. Did you mean during >> compaction or right after it? What load, cpu, memory etc do you see >> during those times? >
