Doesn't seem to be Spark problem, assuming TDigest comes from mahout. Cheers
On Wed, Jul 8, 2015 at 7:49 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <[email protected]> wrote: > Same exception with different values of compression (10,100) > > var digest: TDigest = TDigest.createAvlTreeDigest(100) > > On Wed, Jul 8, 2015 at 6:50 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <[email protected]> wrote: > >> Any suggestions ? >> Code: >> >> val dimQuantiles = genericRecordsAndKeys >> >> .map { >> >> case (keyToOutput, rec) => >> >> var digest: TDigest = TDigest.createAvlTreeDigest(10000) >> >> val fpPaidGMB = rec.get("fpPaidGMB").asInstanceOf[Double] >> >> digest.add(fpPaidGMB) >> >> var bbuf: ByteBuffer = ByteBuffer.allocate(digest.byteSize()); >> >> digest.asBytes(bbuf); >> >> (keyToOutput.toString, bbuf.array()) >> >> } >> >> >> Trace: >> aborted due to stage failure: Task 4487 in stage 1.0 failed 4 times, most >> recent failure: Lost task 4487.3 in stage 1.0 (TID 281, >> phxaishdc9dn0111.phx.ebay.com): java.nio.BufferOverflowException >> at java.nio.Buffer.nextPutIndex(Buffer.java:519) >> at java.nio.HeapByteBuffer.putDouble(HeapByteBuffer.java:519) >> at com.tdunning.math.stats.AVLTreeDigest.asBytes(AVLTreeDigest.java:336) >> at >> com.ebay.ep.poc.spark.reporting.process.service.OutlierService$$anonfun$4.apply(OutlierService.scala:96) >> at >> com.ebay.ep.poc.spark.reporting.process.service.OutlierService$$anonfun$4.apply(OutlierService.scala:89) >> at >> org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:197) >> at >> org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:196) >> at >> org.apache.spark.util.collection.AppendOnlyMap.changeValue(AppendOnlyMap.scala:138) >> at >> org.apache.spark.util.collection.SizeTrackingAppendOnlyMap.changeValue(SizeTrackingAppendOnlyMap.scala:32) >> at >> org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:202) >> at >> org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:56) >> at >> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:70) >> at >> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) >> >> >> -- >> Deepak >> >> > > > -- > Deepak > >
