parthchandra commented on code in PR #1575:
URL: https://github.com/apache/datafusion-comet/pull/1575#discussion_r2019608751
##########
spark/src/test/scala/org/apache/comet/parquet/ParquetReadSuite.scala:
##########
@@ -1460,6 +1460,25 @@ class ParquetReadV1Suite extends ParquetReadSuite with
AdaptiveSparkPlanHelper {
v1 = Some("parquet"))
}
}
+
+ test("test V1 parquet scan uses native_iceberg_compat -- case insensitive") {
+ withTempPath { path =>
+ spark.range(10).toDF("a").write.parquet(path.toString)
+ Seq(CometConf.SCAN_NATIVE_DATAFUSION,
CometConf.SCAN_NATIVE_ICEBERG_COMPAT).foreach(
+ scanMode => {
+ withSQLConf(CometConf.COMET_NATIVE_SCAN_IMPL.key -> scanMode) {
+ withTable("test") {
+ sql("create table test (A long) using parquet options (path '" +
path + "')")
+ val df = sql("select A from test where A > 5")
+ checkSparkAnswer(df)
+ // TODO: pushed down filters do not used schema adapter in
datafusion, will cause empty result
Review Comment:
I don't think we should fix this in the parquet reader (parquet by itself
does not specify whether the field names are case sensitive/insensitive).
Comet is the right place to fix this, I feel.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
