andygrove commented on code in PR #731:
URL: https://github.com/apache/datafusion-comet/pull/731#discussion_r1699068290
##########
spark/src/main/scala/org/apache/spark/sql/comet/CometRowToColumnarExec.scala:
##########
@@ -60,8 +62,17 @@ case class CometRowToColumnarExec(child: SparkPlan)
val timeZoneId = conf.sessionLocalTimeZone
val schema = child.schema
- child
- .execute()
+ val rdd: RDD[InternalRow] = if (child.supportsColumnar) {
+ child
+ .executeColumnar()
+ .mapPartitionsInternal { iter =>
+ iter.flatMap(_.rowIterator().asScala)
+ }
+ } else {
+ child.execute()
+ }
+
+ rdd
Review Comment:
This class is now capable of converting Spark rows or Spark columns to
Comet/Arrow columnar format. Perhaps we should consider renaming it. I'm not
sure what a good name would be. Some ideas:
- CometSparkToColumnarExec
- CometConvertFromSparkExec
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
