namanjain24-sudo commented on issue #25100: URL: https://github.com/apache/datafusion/issues/25100#issuecomment-5702430294
@alexandrefimov @alamb the test is in #25146 now, as f0f8429. It is in two halves, because producing the plan needs no JVM: - `datafusion/substrait/tests/cases/java_interop.rs` writes the plan for `SELECT count(i), sum(i), avg(i) FROM t`. It is `#[ignore]`d, so a normal `cargo test` is unchanged for anyone without Java. - `datafusion/substrait/java-interop/` is a small Maven project that converts that plan with substrait-java 0.103.0, runs it in Spark 3.5.4 over `t(i) = 1, 2, 3`, and checks both the aggregation mode and the rows. The two workarounds are applied on the Java side only, each named after the issue it stands in for, so the test fails for the reason it is about rather than an unrelated one: - #11545, the extension URN, which otherwise stops the plan first; - #25049, the unset `output_type`. That one can go once #25090 lands, and the test then needs only the URN workaround. What it catches, run both ways: | plan produced by | result | | --- | --- | | this branch | passes: `Complete`, and `count 3, sum 6, avg 2.0` | | `main` | fails: `expected: <[Complete, Complete, Complete]> but was: <[Final, Final, Final]>` | The second row is the point. A round trip cannot see this, because the consumer never reads `phase` and rebuilds a complete aggregation either way. On cost, since it pulls a Spark sized dependency set: the job runs only when `datafusion/substrait/**` or `rust.yml` changes, using `dorny/paths-filter`, and caches `~/.m2` through `actions/setup-java`. Locally the Maven half takes about 9 seconds with a warm cache, and the plan is written by a test that already exists in the crate. This is the first JVM job in the repo, so if you would rather it lived in its own workflow, or ran on a schedule instead of on Substrait changes, say so and I will move it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
