Sounds like you need to configure Hadoop to not create a whole bunch of Map tasks at once
On Fri, May 7, 2010 at 3:47 AM, gabriele renzi <rff....@gmail.com> wrote: > Hi everyone, > > I am trying to develop a mapreduce job that does a simple > selection+filter on the rows in our store. > Of course it is mostly based on the WordCount example :) > > > Sadly, while it seems the app runs fine on a test keyspace with little > data, when run on a larger test index (but still on a single node) I > reliably see this error in the logs > > 10/05/06 16:37:58 WARN mapred.LocalJobRunner: job_local_0001 > java.lang.RuntimeException: TimedOutException() > at > org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.maybeInit(ColumnFamilyRecordReader.java:165) > at > org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:215) > at > org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:97) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130) > at > org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(ColumnFamilyRecordReader.java:91) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:423) > at > org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:176) > Caused by: TimedOutException() > at > org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassandra.java:11015) > at > org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:623) > at > org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:597) > at > org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.maybeInit(ColumnFamilyRecordReader.java:142) > ... 11 more > > and after that the job seems to finish "normally" but no results are produced. > > FWIW this is on 0.6.0 (we didn't move to 0.6.1 yet because, well, if > it ain't broke don't fix it). > > The single node has a data directory of about 127GB in two column > families, off which the one used in the mapred job is about 100GB. > The cassandra server is run with 6GB of heap on a box with 8GB > available and no swap enabled. read/write latency from cfstat are > > Read Latency: 0.8535837762577986 ms. > Write Latency: 0.028849603764075547 ms. > > row cache is not enabled, key cache percentage is default. Load on the > machine is basically zero when the job is not running. > > As my code is 99% that from the wordcount contrib, I shall notice that > In 0.6.1's contrib (and trunk) there is a RING_DELAY constant that we > can supposedly change, but it's apparently not used anywhere, but as I > said, running on a single node this should not be an issue anyway. > > Does anyone has suggestions or has seen this error before? On the > other hand, did people run this kind of jobs in similar conditions > flawlessly, so I can consider it just my problem? > > > Thanks in advance for any help. > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
