andygrove commented on issue #27103: URL: https://github.com/apache/datafusion/issues/27103#issuecomment-6840027198
The `datafusion-spark` code has now been deleted from the main repo. Since this issue is about how we test the crate to ensure Spark compatibility, here is a PR for a first step towards this. https://github.com/apache/datafusion-spark/pull/54 It adds a Python script that contributors can run locally. It runs the slt tests via Spark and compares the results with the expected values in the slt file. It is already catching real issues. ``` Spark 4.2.0: 2715 records in 245 file(s) 1652 pass 68 results differ 78 Spark raises an error 23 DataFusion expects an error, Spark returns a result 8 results match, column types differ 21 same numbers, printed with a different scale 865 skipped 584 Spark has no function `arrow_cast` 66 Spark cannot parse it (PARSE_SYNTAX_ERROR) 58 Spark has no function `spark_cast` 48 Spark has no function `arrow_typeof` 34 Spark has no function `bitwise_not` 18 Spark returned a DayTimeIntervalType value 13 Spark has no function `make_array` 10 a string literal contains a backslash, which Spark reads as an escape and DataFusion does not 9 Spark returned a CalendarIntervalType value 8 needs a table that Spark could not create 4 DESCRIBE output differs between engines 4 DECIMAL precision above 38 2 EXPLAIN output differs between engines 2 Spark has no function `unnest` 2 no Spark equivalent for datafusion.optimizer.max_passes ... 2 more reasons (use --verbose) ``` PTAL at the PR if this of interest. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
