i like the Aggregator a lot (org.apache.spark.sql.expressions.Aggregator), but i find the way to use it somewhat confusing. I am supposed to simply call aggregator.toColumn, but that doesn't allow me to specify which fields it operates on in a DataFrame.
i would basically like to do something like
dataFrame
.groupBy("k")
.agg(
myAggregator.on("v1", "v2").toColumn,
myOtherAggregator.on("v3", "v4").toColumn
)
