Hi all!
A simplified code snippet of what my Spark pipeline written in Java does:
public class MyPojo implements Serializable {
... // some fields with Getter and Setter
}
a custom Aggregator (defined in the Driver class):
public static MyAggregator extends
org.apache.spark.sql.expressions.Aggregator<Row, MyPojo, MyPojo> { ... }
in my Driver I do:
Dataset<Row> inputDF = ... //some calculations before
inputDF.groupBy("col1", "col2", "col3").agg(new
MyAggregator().toColumn().name("aggregated");
When executing this part I get a CompileException complaining about an
unknown variable or type "MyPojo$.MODULE$". For me it looks like the
CodeGenerator generates code for Scala (since as far as I know .MODULE$
is a scala specific variable). I tried it with Spark 3.1.1 and Spark 3.0.1.
Does anyone have an idea what's going wrong here?
Best,
Rico.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
