rishvin opened a new issue, #2214:
URL: https://github.com/apache/datafusion-comet/issues/2214

   ### Describe the bug
   
   I invoked the following in the spark-shell, version `3.5.6`.
   
   ```
   $SPARK_HOME/bin/spark-shell 
$COMET/spark/target/comet-spark-spark3.5_2.12-0.10.0-SNAPSHOT --conf 
spark.plugins=org.apache.spark.CometPlugin --conf spark.comet.enabled=true 
--conf spark.comet.exec.enabled=true
   ```
   
   ```
   import org.apache.spark.sql.types._
   import org.apache.spark.sql.functions._
   import org.apache.spark.sql.Row
   
   val schema = StructType(Seq(StructField("id", IntegerType, nullable = 
false), StructField("value", IntegerType, nullable = false)))
   val data = Seq(Row(1, 10), Row(2, 20), Row(3, 10), Row(4, 30), Row(5, 20), 
Row(6, 10))
   val df = spark.createDataFrame(spark.sparkContext.parallelize(data), schema)
   val out = "groupby"
   df.write.mode("overwrite").parquet(out)
   val parquetDF = spark.read.parquet(out)
   val grouped = parquetDF.groupBy("id").count()
   grouped.explain()
   ```
   
   And saw that the explain output has `CometHashAggregate` prefixed with `!`. 
   ```
   == Physical Plan ==
   AdaptiveSparkPlan isFinalPlan=false
   +- HashAggregate(keys=[id#137], functions=[count(1)])
      +- Exchange hashpartitioning(id#137, 4), ENSURE_REQUIREMENTS, 
[plan_id=420]
         +- !CometHashAggregate [id#137], Partial, [id#137], [partial_count(1)]
            +- CometNativeScan parquet [id#137] Batched: true, DataFilters: [], 
Format: CometParquet, Location: InMemoryFileIndex(1 
paths)[file:/home/testing/groupby], PartitionFilters: [], PushedFilters: [], 
ReadSchema: struct<id:int>
   ```
   
   
   
   **There was no correctness issues and no failures.**
   
   
   The spark codebase 
[here](https://github.com/apache/spark/blob/303c18c74664f161b9b969ac343784c088b47593/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala#L425)
 says - they use `"!" to indicate an invalid plan, and "'" to indicate an 
unresolved plan`, although, I am not fully sure, if  `!` is getting added from 
the same place in the code. 
   
   I just saw the comment in the spark-code and filed this issue to bring to 
notice.
   
   ### Steps to reproduce
   
   _No response_
   
   ### Expected behavior
   
   _No response_
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org

Reply via email to