I have found the issue, but I think it is bug.
If I change my class to:
public class ModelSessionBuilder implements Serializable {
/**
*
*/
.
private Properties[] propertiesList;
private static final long serialVersionUID =
-8139500301736028670L;
}
The broadcast value has no issue. But in my original form, if I broadcast it
as array of my custom subclass of Properties, after broadcast, the
propertiesList array will be an array of empty PropertiesUtils objects
there (empty, not NULL), I am not sure why this happen (the code without any
problem when run with default java serializer). So I think this is a bug,
but I am not sure it is a bug of spark or a bug of Kryo.
Regards,
Shuai
From: Shuai Zheng [mailto:[email protected]]
Sent: Monday, April 06, 2015 5:34 PM
To: [email protected]
Subject: Broadcast value return empty after turn to
org.apache.spark.serializer.KryoSerializer
Hi All,
I have tested my code without problem on EMR yarn (spark 1.3.0) with default
serializer (java).
But when I switch to org.apache.spark.serializer.KryoSerializer, the
broadcast value doesn't give me right result (actually return me empty
custom class on inner object).
Basically I broadcast a builder object, which carry an array of
propertiesUtils object. The code should not have any logical issue because
it works on default java serializer. But when I turn to the
org.apache.spark.serializer.KryoSerializer, it looks like the Array doesn't
initialize, propertiesList will give a right size, but then all element in
the array is just a normal empty PropertiesUtils.
Do I miss anything when I use this KryoSerializer? I just put the two lines,
do I need to implement some special code to enable KryoSerializer, but I
search all places but can't find any places mention it.
sparkConf.set("spark.serializer",
"org.apache.spark.serializer.KryoSerializer");
sparkConf.registerKryoClasses(new Class[]{ModelSessionBuilder.class,
Constants.class, PropertiesUtils.class, ModelSession.class});
public class ModelSessionBuilder implements Serializable {
/**
*
*/
.
private PropertiesUtils[] propertiesList;
private static final long serialVersionUID =
-8139500301736028670L;
}
public class PropertiesUtils extends Properties {
/**
*
*/
private static final long serialVersionUID = -3684043338580885551L;
public PropertiesUtils(Properties prop) {
super(prop);
}
public PropertiesUtils() {
// TODO Auto-generated constructor stub
}
}
Regards,
Shuai
