I am running a simple rdd filter command. What does it mean?
Here is the full stack trace(and code below it):
com.esotericsoftware.kryo.KryoException: Buffer overflow. Available: 0,
required: 133
at com.esotericsoftware.kryo.io.Output.require(Output.java:138)
at
com.esotericsoftware.kryo.io.Output.writeString_slow(Output.java:420)
at com.esotericsoftware.kryo.io.Output.writeString(Output.java:326)
at
com.esotericsoftware.kryo.serializers.DefaultArraySerializers$StringArraySerializer.write(DefaultArraySerializers.java:274)
at
com.esotericsoftware.kryo.serializers.DefaultArraySerializers$StringArraySerializer.write(DefaultArraySerializers.java:262)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at
org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:138)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:197)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
*Here is the code of the main function:*
/String comparisonFieldIndexes = "16,18";
String segmentFieldIndexes = "14,15";
String comparisonFieldWeights = "50, 50";
String delimiter = ""+'\001';
PartitionDataOnColumn parOnCol = new PartitionDataOnColumn(70,
comparisonFieldIndexes, comparisonFieldWeights, segmentFieldIndexes,
delimiter);
JavaRDD<String> filtered_rdd = origRDD.filter(parOnCol.new
FilterEmptyFields(parOnCol.fieldIndexes, parOnCol.DELIMITER) );
parOnCol.printRDD(filtered_rdd);/
*Here is the FilterEmptyFields class:*
/public class FilterEmptyFields implements Function<String, Boolean>
{
final int[] nonEmptyFields;
final String DELIMITER;
public FilterEmptyFields(int[] nonEmptyFields, String
delimiter){
this.nonEmptyFields = nonEmptyFields;
this.DELIMITER = delimiter;
}
@Override
public Boolean call(String s){
String[] fields = s.split(DELIMITER);
for(int i=0; i<nonEmptyFields.length; i++){
if(fields[nonEmptyFields[i]] == null ||
fields[nonEmptyFields[i]].isEmpty()){
return false;
}
}
return true;
}
}</i>
Any suggestions guys?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/com-esotericsoftware-kryo-KryoException-Buffer-overflow-tp16947.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
