Yes, it works after I append the two properties in spark-defaults.conf. As I use python programing on spark platform,the python api does not have SparkConf api.
Thanks. 2015-03-25 21:07 GMT+08:00 Cheng Lian <[email protected]>: > Oh, just noticed that you were calling sc.setSystemProperty. Actually > you need to set this property in SparkConf or in spark-defaults.conf. And > there are two configurations related to Kryo buffer size, > > - spark.kryoserializer.buffer.mb, which is the initial size, and > - spark.kryoserializer.buffer.max.mb, which is the max buffer size. > > Make sure the 2nd one is larger (it seems that Kryo doesn’t check for it). > > Cheng > > On 3/25/15 7:31 PM, 李铖 wrote: > > Here is the full track > > 15/03/25 17:48:34 WARN TaskSetManager: Lost task 0.0 in stage 1.0 (TID > 1, cloud1): com.esotericsoftware.kryo.KryoException: Buffer overflow. > Available: 0, required: 39135 > at com.esotericsoftware.kryo.io.Output.require(Output.java:138) > at com.esotericsoftware.kryo.io.Output.writeBytes(Output.java:220) > at com.esotericsoftware.kryo.io.Output.writeBytes(Output.java:206) > at > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ByteArraySerializer.write(DefaultArraySerializers.java:29) > at > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ByteArraySerializer.write(DefaultArraySerializers.java:18) > at com.esotericsoftware.kryo.Kryo.writeObjectOrNull(Kryo.java:549) > at > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:312) > at > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:293) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) > at > org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:165) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:206) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > 2015-03-25 19:05 GMT+08:00 Cheng Lian <[email protected]>: > >> Could you please provide the full stack trace? >> >> >> On 3/25/15 6:26 PM, 李铖 wrote: >> >> It is ok when I do query data from a small hdfs file. >> But if the hdfs file is 152m,I got this exception. >> I try this code >> .'sc.setSystemProperty("spark.kryoserializer.buffer.mb",'256')'.error >> still. >> >> ``` >> com.esotericsoftware.kryo.KryoException: Buffer overflow. Available: 0, >> required: 39135 >> at com.esotericsoftware.kryo.io.Output.require(Output.java:138) >> at com.esotericsoftware.kryo.io.Output.writeBytes(Output.java:220) >> at com.esotericsoftware.kryo.io.Output.writeBytes(Output.java:206) >> at >> com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ByteArraySerializer.write(DefaultArraySerializers.java:29) >> at >> >> >> ``` >> >> >> > >
