ueshin opened a new pull request, #50779:
URL: https://github.com/apache/spark/pull/50779
### What changes were proposed in this pull request?
Prepares the test environment for pandas API on Spark with ANSI mode enabled.
- Remove forcibly disabling ANSI mode in tests
- Add a new option `compute.ansi_mode_support` to keep the current behavior
(default `False`)
- eventually it should be `True` by default
- Skip the failed tests affected by ANSI mode
- Make `pyspark-pandas` tests run in the nightly Non-ANSI test to also run
skipped tests with Non-ANSI mode
### Why are the changes needed?
Currently pandas API on Spark doesn't support ANSI mode and show warnings if
it's enabled.
```py
>>> import pyspark.pandas as ps
>>> ps.range(10)
...: PandasAPIOnSparkAdviceWarning: The config 'spark.sql.ansi.enabled' is
set to True. This can cause unexpected behavior from pandas API on Spark since
pandas API on Spark follows the behavior of pandas, not SQL.
warnings.warn(message, PandasAPIOnSparkAdviceWarning)
...
```
Now ANSI mode is enabled by default, pandas API on Spark should also support
it.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
The existing tests should pass.
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this
patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling
Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
