alexandrefimov opened a new issue, #25043: URL: https://github.com/apache/datafusion/issues/25043
### Describe the bug The Substrait consumer maps standard decimal arithmetic to DataFusion operators whose derived precision and scale can differ from the referenced Substrait function definition. It accepts the plan and exposes the different type without reporting the mismatch. Reproduced on `main` at `1b6dc92a221b41088afa2753fe5526be6d057b7b`. Each plan references `extension:io.substrait:functions_arithmetic_decimal` and declares the return type given by its YAML formula. The add, multiply and divide definitions are identical in spec v0.87.0, v0.102.0 and v0.103.0. | Expression | Spec / declared output_type | DataFusion logical schema | | --- | --- | --- | | dec(10,2) + dec(5,1) | dec(11,2) | dec(11,2) | | dec(38,10) + dec(38,10) | dec(38,9) | dec(38,10) | | dec(10,2) * dec(5,1) | dec(16,3) | dec(16,3) | | dec(38,10) * dec(38,10) | dec(38,6) | dec(38,20) | | dec(10,2) / dec(5,1) | dec(21,8) | dec(15,6) | Inputs and declared outputs are required. The table focuses on precision and scale, with the two small add/multiply cases serving as controls. ### To reproduce From a DataFusion checkout: ```sh git clone https://github.com/alexandrefimov/substrait-conformance-cases conformance-cases git -C conformance-cases checkout f12ba6103217fbd50b95c407b5dc3e8fae1e05a0 mkdir -p conformance-inputs datafusion/substrait/examples cp conformance-cases/derived-schema/decimal_*.json conformance-inputs/ cp conformance-cases/probe/datafusion_corpus_probe.rs datafusion/substrait/examples/corpus_probe.rs SUBSTRAIT_CORPUS_DIR="$PWD/conformance-inputs" cargo run --locked -p datafusion-substrait --example corpus_probe ``` The [probe](https://github.com/alexandrefimov/substrait-conformance-cases/blob/f12ba6103217fbd50b95c407b5dc3e8fae1e05a0/probe/datafusion_corpus_probe.rs) registers a table with the exact declared input types. Its `DATAFUSION ACCEPTED` line reports the logical schema before execution. For example, [`decimal_divide`](https://github.com/alexandrefimov/substrait-conformance-cases/blob/f12ba6103217fbd50b95c407b5dc3e8fae1e05a0/derived-schema/decimal_divide.json) declares `decimal(21,8)` and reports `Decimal128(15, 6)`. ### Expected behavior The consumed expression should follow the return-type rule of the referenced [decimal function](https://github.com/substrait-io/substrait/blob/v0.103.0/extensions/functions_arithmetic_decimal.yaml), or reject a function contract it cannot implement. For division here, the YAML gives `scale = max(6, 2 + 5 + 1) = 8` and `precision = 10 - 2 + 5 + 8 = 21`. `from_scalar_function` resolves the function name to a native operator and constructs a `BinaryExpr` without checking its return type against the declaration. This is a Substrait conversion issue, not a request to change the semantics of ordinary DataFusion SQL arithmetic. The wide-add fixture also reports an execution overflow at scale 10. The core reproducer is the schema mismatch: these plans omit overflow options, so this report does not prescribe a rounding or overflow policy. In particular, the wide-multiply input value can overflow even with the correct declared type. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
