I'm having problems with Hadoop job failures on a Cassandra 1.2 cluster due to
Caused by: TimedOutException()
2013-06-24 11:29:11,953 INFO Driver - at
org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassandra.java:12932)
This is running on a 6-node cluster, RF=3. If I run the job with CL=ONE, it
usually runs pretty well, with an occasional timeout. But
if I run at CL=QUORUM, the number of timeouts is often enough to kill the job.
The table being read is effectively read-only when this job runs.
It has from 5 to 10 million rows, with each row having no more than 256
columns. Each column typically only has a few hundred bytes of data at most.
I've fiddled with the batch-range size and increasing the timeout, without a
lot of luck. I see some evidence of GC activity in the Cassandra logs, but
it's hard to see a clear correlation with the timeouts.
I could use some suggestions on an approach to pin down the root cause.
TIA
Brian
