alexandrefimov opened a new issue, #25042: URL: https://github.com/apache/datafusion/issues/25042
### Describe the bug The Substrait consumer derives all three supported intersection schemas from the primary input's nullability. A field that the intersection makes required can therefore remain nullable in the logical output schema. Reproduced on `main` at `1b6dc92a221b41088afa2753fe5526be6d057b7b`. The [Substrait Set Operation rules](https://github.com/substrait-io/substrait/blob/v0.87.0/site/docs/relations/logical_relations.md#set-operation) define different nullability rules for each operation. With the spec's three input patterns (`R` = required, `N` = nullable), all fields having type `i64`: ```text primary: R R R R N N N N secondary 1: R R N N R R N N secondary 2: R N R N R N R N ``` the consumer returns: | Operation | Expected | Actual | | --- | --- | --- | | INTERSECTION_PRIMARY | R R R R R N N N | R R R R N N N N | | INTERSECTION_MULTISET | R R R R R R R N | R R R R N N N N | | INTERSECTION_MULTISET_ALL | R R R R R R R N | R R R R N N N N | ### To reproduce The plans and the table-registration probe are pinned in the [conformance corpus](https://github.com/alexandrefimov/substrait-conformance-cases/tree/f12ba6103217fbd50b95c407b5dc3e8fae1e05a0). From a DataFusion checkout: ```sh git clone https://github.com/alexandrefimov/substrait-conformance-cases conformance-cases git -C conformance-cases checkout f12ba6103217fbd50b95c407b5dc3e8fae1e05a0 mkdir -p conformance-inputs datafusion/substrait/examples cp conformance-cases/derived-schema/setop_*.json conformance-inputs/ cp conformance-cases/probe/datafusion_corpus_probe.rs datafusion/substrait/examples/corpus_probe.rs SUBSTRAIT_CORPUS_DIR="$PWD/conformance-inputs" cargo run --locked -p datafusion-substrait --example corpus_probe ``` The probe registers empty named tables with exactly those input schemas and prints the logical schema immediately after `from_substrait_plan`, before optimization or execution. The `setop_intersection_*` lines show the three mismatches. Union and primary-minus controls match their respective spec rules; `MINUS_MULTISET` is explicitly unsupported. ### Expected behavior For multiset intersections, a field is required if any input requires it. For primary intersection, it is nullable only when it is nullable in the primary input and in at least one secondary input. `from_set_rel` delegates to `LogicalPlanBuilder::intersect`, which uses a left semi join. Keeping the left field nullability there explains the observed schema. This report concerns the schema of consumed Substrait plans; it does not assert incorrect rows. The multiplicity bugs #12955 and #12956 are separate. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
