You are likely exhausting your heap space (probably still at the very small 1G default?), and maximizing the amount of resource consumption by using CL.ALL. Why are you using ALL?
On Tue, Jun 15, 2010 at 11:58 AM, Julie <julie.su...@nextcentury.com> wrote: > I am running a 10 node cassandra 0.6.1 cluster with a replication factor of 3. > > To populate the database to perform my read benchmarking, I have 8 > applications > using Thrift, each connecting to a different cassandra server and writing > 100,000 rows of data (100 KB each row), using a consistencyLevel of ALL. My > server nodes are ec2-smalls (1.7GB memory, 100GB disk). > > With consistency set to ALL, it takes 5-6 minutes for each app to write > 10,000 (100 KB) rows. When each of my 8 writing apps reaches about 90,000 > rows > written, I start seeing write timeouts but my app retries twice and all data > appears to get written. > > It sppears to take about 1hr 45min for all compacting to complete. > > Coinciding with my write timeouts, all 10 of my cassandra servers are getting > the following exception written to system.log: > > > INFO [FLUSH-WRITER-POOL:1] 2010-06-15 13:13:54,411 Memtable.java (line 162) > Completed flushing /var/lib/cassandra/data/Keyspace1/Standard1-359-Data.db > ERROR [MESSAGE-STREAMING-POOL:1] 2010-06-15 13:13:59,145 > DebuggableThreadPoolExecutor.java (line 101) Error in ThreadPoolExecutor > java.lang.RuntimeException: java.io.IOException: Value too large for defined > data type > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask > (ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.io.IOException: Value too large for defined data type > at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) > at > sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415) > at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516) > at > org.apache.cassandra.net.FileStreamTask.stream(FileStreamTask.java:95) > at > org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:63) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > ... 3 more > ERROR [MESSAGE-STREAMING-POOL:1] 2010-06-15 13:13:59,146 CassandraDaemon.java > (line 78) Fatal exception in thread Thread[MESSAGE-STREAMING-POOL:1,5,main] > java.lang.RuntimeException: java.io.IOException: Value too large for defined > data type > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask > (ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.io.IOException: Value too large for defined data type > at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) > at > sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415) > at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516) > at > org.apache.cassandra.net.FileStreamTask.stream(FileStreamTask.java:95) > at > org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:63) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > ... 3 more > > On 8 out of 10 servers, I see this just before the exception: > > INFO [AE-SERVICE-STAGE:1] 2010-06-15 13:41:36,292 StreamOut.java (line 66) > Sending a stream initiate message to /10.210.34.212 ... > ERROR [MESSAGE-STREAMING-POOL:1] 2010-06-15 13:43:32,956 > DebuggableThreadPoolExecutor.java (line 101) Error in ThreadPoolExecutor > > On the other 2 servers, I see the AE-SERVICE stream initiate message about 6-9 > minutes prior to the exception. > > Another thing that is odd is that even when the server nodes are quiescent > because compacting is complete, I am still seeing cpu usage stay at > about 40% . Even after several hours, no reading or writing to the database > and all compactions complete, the cpu usage is staying around 40%. > > Thank you for your help and advice, > Julie > >
