Niek Bartholomeus created SPARK-17872:
-----------------------------------------
Summary: aggregate function on dataset with tuples grouped by non
sequential fields
Key: SPARK-17872
URL: https://issues.apache.org/jira/browse/SPARK-17872
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.0.1
Reporter: Niek Bartholomeus
The following lines where the field index in the tuple used in an aggregate
function is lower than a field index used in the group by clause fails:
{code}
val testDS = Seq((1, 1, 1, 1)).toDS
// group by field one and three, aggregate on field 2:
testDS
.groupByKey { case (level1, level1FigureA, level2, level2FigureB) =>
(level1, level2) }
.agg((sum($"_2" * $"_4")).as[Double])
.collect
{code}
Error message:
{code}
org.apache.spark.sql.AnalysisException: Reference '_2' is ambiguous, could be:
_2#562, _2#569.;
at
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolve(LogicalPlan.scala:264)
at
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveChildren(LogicalPlan.scala:148)
at
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$9$$anonfun$applyOrElse$5$$anonfun$31.apply(Analyzer.scala:604)
at
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$9$$anonfun$applyOrElse$5$$anonfun$31.apply(Analyzer.scala:604)
at
org.apache.spark.sql.catalyst.analysis.package$.withPosition(package.scala:48)
at
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$9$$anonfun$applyOrElse$5.applyOrElse(Analyzer.scala:604)
at
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$9$$anonfun$applyOrElse$5.applyOrElse(Analyzer.scala:600)
at
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:301)
at
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:301)
at
org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:69)
{code}
While the following code - where the aggregate field indices are all higher
than the groupby field indices - works fine:
{code}
testDS
.map { case (level1, level1FigureA, level2, level2FigureB) => (level1,
level2, level1FigureA, level2FigureB) }
.groupByKey { case (level1, level2, level1FigureA, level2FigureB) =>
(level1, level2) }
.agg((sum($"_3" * $"_4")).as[Double])
.collect
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
