andygrove opened a new issue, #1974:
URL: https://github.com/apache/datafusion-comet/issues/1974
### What is the problem the feature request solves?
`COMET_ANSI_MODE_ENABLED` was added early on during testing with Spark 4.0.0
to ensure that queries would fall back to Spark by default when ANSI mode was
enabled. Since then, Comet has added ANSI support to many expressions, so we
should now aim to remove this legacy config.
Currently, we see the following `core-2` Spark SQL tests fail when this
configuration is removed:
```
2025-07-01T18:14:02.2231695Z [info] - SPARK-39166: Query context of binary
arithmetic should be serialized to executors when WSCG is off *** FAILED ***
(255 milliseconds)
2025-07-01T18:14:02.2283008Z [info] Expected exception
org.apache.spark.SparkException to be thrown, but no exception was thrown
(SQLQuerySuite.scala:4444)
2025-07-01T18:14:02.4710994Z [info] - SPARK-39175: Query context of Cast
should be serialized to executors when WSCG is off *** FAILED *** (230
milliseconds)
2025-07-01T18:14:02.4738565Z [info] "Job aborted due to stage failure:
Task 0 in stage 2399.0 failed 1 times, most recent failure: Lost task 0.0 in
stage 2399.0 (TID 2914) (c1c7cbad7c91 executor driver):
org.apache.comet.CometNativeException: [CAST_INVALID_INPUT] The value 'a' of
the type "STRING" cannot be cast to "INT" because it is malformed. Correct the
value as per the syntax, or change its target type. Use `try_cast` to tolerate
malformed input and return NULL instead. If necessary set
"spark.sql.ansi.enabled" to "false" to bypass this error.
2025-07-01T18:14:02.6819549Z [info] - SPARK-39190,SPARK-39208,SPARK-39210:
Query context of decimal overflow error should be serialized to executors when
WSCG is off *** FAILED *** (200 milliseconds)
2025-07-01T18:14:02.6828444Z [info] "Job aborted due to stage failure:
Task 0 in stage 2401.0 failed 1 times, most recent failure: Lost task 0.0 in
stage 2401.0 (TID 2917) (c1c7cbad7c91 executor driver):
org.apache.comet.CometNativeException: Invalid argument error:
170141183460469231731687303715884105727 is too large to store in a Decimal128
of precision 38. Max is 99999999999999999999999999999999999999
2025-07-01T18:16:29.8908057Z [info] - postgreSQL/int4.sql *** FAILED *** (3
seconds, 349 milliseconds)
2025-07-01T18:16:29.8911037Z [info] postgreSQL/int4.sql
2025-07-01T18:16:29.8912074Z [info] Expected "struct<[]>", but got
"struct<[five:string,f1:int,x:int]>" Schema did not match for query #21
2025-07-01T18:16:29.8912983Z [info] SELECT '' AS five, i.f1, i.f1 *
smallint('2') AS x FROM INT4_TBL i: -- !query
2025-07-01T18:16:29.8925605Z [info] SELECT '' AS five, i.f1, i.f1 *
smallint('2') AS x FROM INT4_TBL i
2025-07-01T18:16:29.8927367Z [info] -- !query schema
2025-07-01T18:16:29.8928392Z [info] struct<five:string,f1:int,x:int>
2025-07-01T18:16:29.8940049Z [info] -- !query output
2025-07-01T18:16:29.8948585Z [info] -123456 -246912
2025-07-01T18:16:29.8948935Z [info] -2147483647 2
2025-07-01T18:16:29.8949226Z [info] 0 0
2025-07-01T18:16:29.8949716Z [info] 123456 246912
2025-07-01T18:16:29.8950103Z [info] 2147483647 -2
(SQLQueryTestSuite.scala:491)
```
### Describe the potential solution
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
