I have a set of transformers (each with specific parameters) in spark
1.3.1. I have two versions, one that works and one that does not:
1.- working version
//featureprovidertransformer contains already a set of ml params
class DemographicTransformer(override val uid: String) extends
FeatureProviderTransformer {
def this() = this(Identifiable.randomUID("demo-transformer"))
override def copy(extra: ParamMap): DemographicTransformer =
defaultCopy(extra)
....
}
2.- not working version
class DemographicTransformer(override val uid: String) extends
FeatureProviderTransformer {
def this() = this(Identifiable.randomUID("demo-transformer"))
override def copy(extra: ParamMap): DemographicTransformer =
defaultCopy(extra)
*//add another transformer parameter*
final val anotherParam: Param[String] = new Param[String](this,
"anotherParam", "dummy parameter")
....
}
Somehow adding an *anotherParam* to my class make it fail, with the
following error:
[info] java.lang.NullPointerException:
[info] at
org.apache.spark.ml.param.Params$$anonfun$hasParam$1.apply(params.scala:408)
[info] at
org.apache.spark.ml.param.Params$$anonfun$hasParam$1.apply(params.scala:408)
[info] at
scala.collection.IndexedSeqOptimized$$anonfun$exists$1.apply(IndexedSeqOptimized.scala:40)
[info] at
scala.collection.IndexedSeqOptimized$$anonfun$exists$1.apply(IndexedSeqOptimized.scala:40)
[info] at
scala.collection.IndexedSeqOptimized$class.segmentLength(IndexedSeqOptimized.scala:189)
[info] at
scala.collection.mutable.ArrayOps$ofRef.segmentLength(ArrayOps.scala:108)
[info] at
scala.collection.GenSeqLike$class.prefixLength(GenSeqLike.scala:92)
[info] at
scala.collection.mutable.ArrayOps$ofRef.prefixLength(ArrayOps.scala:108)
[info] at
scala.collection.IndexedSeqOptimized$class.exists(IndexedSeqOptimized.scala:40)
[info] at
scala.collection.mutable.ArrayOps$ofRef.exists(ArrayOps.scala:108)
Debugging the params.scala class shows me that actually adding
*anotherParam* *replace all parameters by a single one called allParams.*
*Does anyone have any idea of what I may be doing wrong. My guess is that I
am doing something weird in my class hierarchy but can not figure out what.*
Thanks!
--
Cesar Flores
