goutamadwant opened a new pull request, #27213:
URL: https://github.com/apache/datafusion/pull/27213
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases. You can
link an issue to this PR using the GitHub syntax. For example `Closes #123`
indicates that this PR will close issue #123.
-->
- Closes #27185.
- Also addresses the decimal string-formatting portion of #26337; the
remaining formatter differences in that issue are outside this PR.
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in
the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your
changes and offer better suggestions for fixes.
Please explain the problem you are trying to solve in terms of the
user-visible
behavior, rather than the implementation.
For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of the
implementation. "COUNT(DISTINCT) returns wrong results when the column
contains
nulls" is the user-visible problem.
-->
Spark SQL uses INT for integer literals that fit and DECIMAL for unsuffixed
decimal literals. Using BIGINT and DOUBLE instead changes results:
`xxhash64(1)` hashes the wrong integer width, and decimal aggregates and
conditionals lose their decimal result types.
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here, but it is
sometimes worth providing a summary of the individual changes in this PR.
-->
- Carry the selected dialect into literal planning. In Spark mode, preserve
INT/BIGINT boundaries, explicit `L` suffixes, decimal precision and scale, and
scientific notation as DOUBLE.
- Widen mixed decimal/float scalar `VALUES` columns to DOUBLE, including NaN
and infinity, while preserving explicit destination schemas.
- Match Spark's decimal `format_string`/`printf` behavior: reject
unsupported numeric conversions for non-null decimals, preserve scale for
`%s`/`%S`, and handle typed nulls and boolean conversions.
- Accept the resulting literal types in `rint` and `generate_series`, and
update Spark fixtures. Fixtures specifically testing floating-point behavior
now use explicit DOUBLE inputs.
## What is the testing strategy for this PR?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
Briefly describe how this PR is tested, and point to the specific tests you
added. For example: 'This new feature is covered by the `sqllogictest` cases
added in `foo.slt`'.
If this PR does not add tests, explain why. For example, if the change is
already covered by existing tests, please mention it.
You should also check the `codecov` bot reply on this PR to confirm the
changed code is exercised.
-->
Added `spark/numeric_literals.slt` and `spark/literal_compatibility.slt`,
plus parser-option override and decimal-formatting unit tests. Coverage
includes all three issue examples, numeric boundaries, mixed VALUES row orders,
typed nulls, array evaluation, and generic-dialect controls. Reference cases
were checked against Spark 4.2.0 in both ANSI modes; the new compatibility
tests reproduced 18 failures before the corresponding fixes.
Local validation passed:
- Extended workspace suite: 13,563 tests passed, 8 existing tests ignored,
and all 548 SQLLogicTest files completed.
- Dedicated Spark suite: all 247 files passed.
- `cargo clippy --all-targets --all-features -- -D warnings`, `cargo fmt
--all -- --check`, and `git diff --check`.
- Rust documentation with warnings denied, generated-documentation checks,
license headers, spelling, dependency audit, and unused-dependency checks.
The extended workspace command was:
```sh
RUST_BACKTRACE=1 cargo test --profile ci \
--exclude datafusion-examples --exclude datafusion-benchmarks --exclude
datafusion-cli \
--workspace --lib --tests --bins \
--features
avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption
```
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
Yes. Spark-dialect numeric literal types and dependent query results now
follow Spark's rules. Numeric formatting of non-null decimal arguments now
errors where Spark errors.
`ParserOptions` gains a public `dialect` field. Downstream code using
exhaustive struct literals must provide that field or use
`..Default::default()`; this requires the `api change` label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
