andygrove opened a new pull request, #368: URL: https://github.com/apache/datafusion-spark/pull/368
## Which issue does this PR close? - Relates to #7. ## Rationale for this change The PySpark runner reports 260 records in the `.slt` files whose expected result differs from Spark 4.2. Most are known bugs with an issue, some come from how the test is written, and some are limitations of the runner. They are reported together with any new difference, so a new difference is easy to miss and the runner always exits with status 1. ## What changes are included in this PR? - A comment directly above a record marks a known difference: `# xfail spark: <reason>`. It is a comment because the Rust sqllogictest parser rejects directives it does not know. - A marked record that differs from Spark counts as an expected failure. It is not reported and does not change the exit status. `--verbose` lists it with its reason. - A marked record that passes is reported as `marked xfail, but Spark agrees`, so the comment is removed once the difference is fixed. A marked record that the runner skips stays skipped. - The 260 records that differ today are marked. Every bug and literal-typing reason links an issue. The other reasons say whether the difference comes from the test, is intended, or is a runner limitation, and why. | Cause | Records | | --- | --- | | Bug, with a linked issue | 148 | | Literal typing, #97 | 63 | | Intended difference, such as `json_tuple` and `stack` returning a struct or list instead of rows | 29 | | Runner limitation | 14 | | Test written in a way Spark reads differently | 6 | Ten of the bug records had no issue, so this PR links new ones: #361, #362, #363, #364, #365, #366 and #367. The runner-limitation records come from translation gaps: 10 are a setting the runner does not pass on to Spark (`spark.map_key_dedup_policy`), 3 are DataFusion's `TIMESTAMP` literal, which is `TIMESTAMP_NTZ`, and 1 is `[]` indexing, which starts at 1 in DataFusion and at 0 in Spark. Those can be fixed in the runner in follow-up PRs, which removes their markers. ## Are these changes tested? Unit tests in `test_pyspark_slt_runner.py` cover parsing the comment, expected failures, a marked record that passes, and a marked record that is skipped. With PySpark 4.2.0, the full run now reports 2907 passing records, 260 expected failures and no differences. `cargo test -p datafusion-spark-sqllogictest` still parses every file. ## Are there any user-facing changes? No. This changes only the test files and the PySpark runner. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
