You could try increasing the driver memory. Also, can you be more specific about the data volume?
Thanks Best Regards On Mon, Feb 9, 2015 at 3:30 PM, Yifan LI <iamyifa...@gmail.com> wrote: > Hi, > > I just found the following errors during computation(graphx), anyone has > ideas on this? thanks so much! > > (I think the memory is sufficient, spark.executor.memory 30GB ) > > > 15/02/09 00:37:12 ERROR Executor: Exception in task 162.0 in stage 719.0 (TID > 7653) > java.lang.OutOfMemoryError: Java heap space > at > com.esotericsoftware.kryo.util.IdentityObjectIntMap.resize(IdentityObjectIntMap.java:410) > at > com.esotericsoftware.kryo.util.IdentityObjectIntMap.put(IdentityObjectIntMap.java:113) > at > com.esotericsoftware.kryo.util.MapReferenceResolver.addWrittenObject(MapReferenceResolver.java:23) > at com.esotericsoftware.kryo.Kryo.writeReferenceOrNull(Kryo.java:598) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:566) > at > com.twitter.chill.TraversableSerializer$$anonfun$write$1.apply(Traversable.scala:29) > at > com.twitter.chill.TraversableSerializer$$anonfun$write$1.apply(Traversable.scala:27) > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) > at > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226) > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) > at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) > at com.twitter.chill.TraversableSerializer.write(Traversable.scala:27) > at com.twitter.chill.TraversableSerializer.write(Traversable.scala:21) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) > at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:36) > at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) > at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:37) > at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) > at > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:318) > at > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:293) > at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501) > at > com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564) > at > com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) > at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:37) > at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) > at > org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:128) > at > org.apache.spark.serializer.SerializationStream.writeAll(Serializer.scala:110) > 15/02/09 00:37:12 ERROR SparkUncaughtExceptionHandler: Uncaught exception in > thread Thread[Executor task launch worker-15,5,main] > java.lang.OutOfMemoryError: Java heap space > at > com.esotericsoftware.kryo.util.IdentityObjectIntMap.resize(IdentityObjectIntMap.java:410) > at > com.esotericsoftware.kryo.util.IdentityObjectIntMap.put(IdentityObjectIntMap.java:113) > at > com.esotericsoftware.kryo.util.MapReferenceResolver.addWrittenObject(MapReferenceResolver.java:23) > at com.esotericsoftware.kryo.Kryo.writeReferenceOrNull(Kryo.java:598) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:566) > at > com.twitter.chill.TraversableSerializer$$anonfun$write$1.apply(Traversable.scala:29) > at > com.twitter.chill.TraversableSerializer$$anonfun$write$1.apply(Traversable.scala:27) > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) > at > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226) > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) > at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) > at com.twitter.chill.TraversableSerializer.write(Traversable.scala:27) > at com.twitter.chill.TraversableSerializer.write(Traversable.scala:21) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) > at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:36) > at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) > at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:37) > at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) > at > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:318) > at > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:293) > at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501) > at > com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564) > at > com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) > at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:37) > at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) > at > org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:128) > at > org.apache.spark.serializer.SerializationStream.writeAll(Serializer.scala:110) > > > > Best, > Yifan LI > > > > > >
