Look at the nodetool tpstats when you get the TimedOutException, to work out which nodes are backing up with pending messages. Then try to identify why. Check the server logs for GC, and the CPU and IO usage.
Somehow the cluster is getting overwhelmed and cannot respond. Either the clients have started whacking it harder than before, the environment (memory, io etc) has changed or cassandra cannot keep up with it's maintenance tasks any more (e.g. compacting lots of wide rows that hogs IO). I *think* the DEBUG message is a client disconnecting and it can be ignored. Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 4/01/2012, at 12:44 PM, Daning Wang wrote: > Hi All, > > We are getting TimedOutException() when inserting data into Cassandra, it was > working fine for a few months, but suddenly got this problem. I have increase > rpc_timout_in_ms to 30000, but it still timed out in 30 secs. > > I turned on debug, I saw many of this error in the log > > DEBUG [pool-2-thread-420] 2012-01-03 15:25:43,689 > CustomTThreadPoolServer.java (line 197) Thrift transport error occurred > during processing of message. > org.apache.thrift.transport.TTransportException: Cannot read. Remote side has > closed. Tried to read 4 bytes, but only got 0 bytes. (This is often > indicative of an internal error on the server side. Please ch > eck your server logs.) > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) > at > org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) > at > org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) > at > org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204) > at > org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2877) > at > org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > > > We are on 0.8.6. Any idea how to fix this? Your help is much appreciated. > > Daning >
