kazuyukitanimura commented on code in PR #1830:
URL: https://github.com/apache/datafusion-comet/pull/1830#discussion_r2175934790
##########
pom.xml:
##########
@@ -637,9 +637,9 @@ under the License.
<profile>
<id>scala-2.13</id>
<properties>
- <scala.version>2.13.14</scala.version>
+ <scala.version>2.13.16</scala.version>
<scala.binary.version>2.13</scala.binary.version>
- <semanticdb.version>4.9.5</semanticdb.version>
+ <semanticdb.version>4.13.6</semanticdb.version>
Review Comment:
Since we already updated the spark-4.0 profile, do we need this change?
##########
dev/diffs/4.0.0.diff:
##########
@@ -2150,7 +2517,17 @@ index 795e9f46a8d..1797084a941 100644
}
}
}
-@@ -1991,7 +2004,8 @@ abstract class ParquetFilterSuite extends QueryTest with
ParquetTest with Shared
+@@ -1750,7 +1762,8 @@ abstract class ParquetFilterSuite extends QueryTest with
ParquetTest with Shared
+ }
+ }
+
+- test("SPARK-17091: Convert IN predicate to Parquet filter push-down") {
++ test("SPARK-17091: Convert IN predicate to Parquet filter push-down",
++ IgnoreComet("IN predicate is not yet supported in Comet, see issue
#36")) {
Review Comment:
For other Spark version, looks like we are using
IgnoreCometNativeScan("Comet has different push-down behavior")
##########
dev/diffs/4.0.0.diff:
##########
@@ -3021,7 +3338,6 @@ index ed2e309fa07..a1fb4abe681 100644
+ conf
+ .set("spark.sql.extensions",
"org.apache.comet.CometSparkSessionExtensions")
+ .set("spark.comet.enabled", "true")
-+ .set("spark.comet.parquet.respectFilterPushdown", "true")
Review Comment:
Should we keep this? `.set("spark.comet.parquet.respectFilterPushdown",
"true")`
##########
pom.xml:
##########
@@ -1074,9 +1074,19 @@ under the License.
<ignoreClass>javax.annotation.meta.TypeQualifierNickname</ignoreClass>
</ignoreClasses>
</dependency>
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ <ignoreClasses>
+
<ignoreClass>com.google.thirdparty.publicsuffix.TrieParser</ignoreClass>
+
<ignoreClass>com.google.thirdparty.publicsuffix.PublicSuffixPatterns</ignoreClass>
+
<ignoreClass>com.google.thirdparty.publicsuffix.PublicSuffixType</ignoreClass>
Review Comment:
Is this a new conflict?
##########
dev/diffs/4.0.0.diff:
##########
@@ -159,6 +190,21 @@ index 698ca009b4f..57d774a3617 100644
-- Test tables
CREATE table explain_temp1 (key int, val int) USING PARQUET;
+diff --git
a/sql/core/src/test/resources/sql-tests/inputs/listagg-collations.sql
b/sql/core/src/test/resources/sql-tests/inputs/listagg-collations.sql
+index aa3d02dc2fb..c4f878d9908 100644
+--- a/sql/core/src/test/resources/sql-tests/inputs/listagg-collations.sql
++++ b/sql/core/src/test/resources/sql-tests/inputs/listagg-collations.sql
+@@ -5,7 +5,9 @@ WITH t(c1) AS (SELECT listagg(col1) WITHIN GROUP (ORDER BY
col1) FROM (VALUES ('
+ -- Test cases with utf8_lcase. Lower expression added for determinism
+ SELECT lower(listagg(c1) WITHIN GROUP (ORDER BY c1 COLLATE utf8_lcase)) FROM
(VALUES ('a'), ('A'), ('b'), ('B')) AS t(c1);
+ WITH t(c1) AS (SELECT lower(listagg(DISTINCT col1 COLLATE utf8_lcase)) FROM
(VALUES ('a'), ('A'), ('b'), ('B'))) SELECT len(c1), regexp_count(c1, 'a'),
regexp_count(c1, 'b') FROM t;
+-SELECT lower(listagg(DISTINCT c1 COLLATE utf8_lcase) WITHIN GROUP (ORDER BY
c1 COLLATE utf8_lcase)) FROM (VALUES ('a'), ('B'), ('b'), ('A')) AS t(c1);
++-- TODO https://github.com/apache/datafusion-comet/issues/1947
++-- TODO fix Comet for this query
++-- SELECT lower(listagg(DISTINCT c1 COLLATE utf8_lcase) WITHIN GROUP (ORDER
BY c1 COLLATE utf8_lcase)) FROM (VALUES ('a'), ('B'), ('b'), ('A')) AS t(c1);
Review Comment:
Should we disable comet instead of commenting out? due to this, the result
in `listagg-collations.sql.out` is also removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
