You are likely hitting the point where compaction is running all the time and consuming all the weak cloud io. Ebs is not suggested for performance you should use the ephermal drives.
On Friday, February 1, 2013, Marcelo Elias Del Valle wrote: > Hello, > > I am trying to figure out why the following behavior happened. Any > help would be highly appreciated. > This graph shows the server resources allocation of my single > cassandra machine (running at Amazon EC2): > http://mvalle.com/downloads/cassandra_host1.png > I ran a hadoop process that reads a CSV file and writtes data to > Cassandra. For about 1 h, the process ran fine, but taking about 100% of > CPU. After 1 h, my hadoop process started to have its connection attempts > refused by cassandra, as shown bellow. > Since them, it has been taking 100% of the machine IO. It has been 2 > h already since the IO is 100% on the machine running Cassandra. > I am running Cassandra under Amazon EBS, which is slow, but I didn't > think it would be that slow. Just wondering, is it normal for Cassandra to > use a high amount of CPU? I am guessing all the writes were going to the > memtables and when it was time to flush the server went down. > Makes sense? I am still learning Cassandra as it's the first time I > use it in production, so I am not sure if I am missing something really > basic here. > > 2013-02-01 16:44:43,741 ERROR com.s1mbi0se.dmp.input.service.InputService > (Thread-18): EXCEPTION:PoolTimeoutException: [host=(10.84.65.108):9160, > latency=5005(5005), attempts=1] Timed out waiting for connection > com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException: > PoolTimeoutException: [host=nosql1.s1mbi0se.com.br(10.84.65.108):9160, > latency=5005(5005), attempts=1] Timed out waiting for connection > at > com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.waitForConnection(SimpleHostConnectionPool.java:201) > at > com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.borrowConnection(SimpleHostConnectionPool.java:158) > at > com.netflix.astyanax.connectionpool.impl.RoundRobinExecuteWithFailover.borrowConnection(RoundRobinExecuteWithFailover.java:60) > at > com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:50) > at > com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:229) > at > com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1.execute(ThriftColumnFamilyQueryImpl.java:186) > at > com.s1mbi0se.dmp.input.service.InputService.searchUserByKey(InputService.java:700) > > ... > at > com.s1mbi0se.dmp.importer.map.ImporterMapper.map(ImporterMapper.java:20) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) > at > org.apache.hadoop.mapreduce.lib.map.MultithreadedMapper$MapRunner.run(MultithreadedMapper.java:268) > 2013-02-01 16:44:43,743 ERROR com.s1mbi0se.dmp.input.service.InputService > (Thread-15): EXCEPTION:PoolTimeoutException: > > > Best regards, > > -- > Marcelo Elias Del Valle > http://mvalle.com - @mvallebr >
