hvanhovell commented on code in PR #49111:
URL: https://github.com/apache/spark/pull/49111#discussion_r1940422165
##########
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala:
##########
@@ -460,27 +460,120 @@ private class KeyValueGroupedDatasetImpl[K, V, IK, IV](
}
override protected def aggUntyped(columns: TypedColumn[_, _]*): Dataset[_] =
{
- // TODO(SPARK-43415): For each column, apply the valueMap func first...
+ val originalDs = sparkSession.newDataset(ivEncoder, plan)
+
+ // Apply the value transformation, get a DS of two columns "iv" and "v".
+ // If any of "iv" or "v" consists of a single primitive field, wrap it
with a struct so it
+ // would not be flattened.
+ val valueTransformedDf = applyValueMapFunc(originalDs)
+
+ // Restore column/field names, if this KVGDS is from a DF.
+ val (valueTransformedDfWithSchema, ivFields, vFields) =
Review Comment:
I don't understand why we need to retain the names for multiple column
values? For a single column I understand that it is needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
