rich7420 opened a new pull request, #6497: URL: https://github.com/apache/datafusion-comet/pull/6497
## Which issue does this PR close? Related to #6424, which is already closed by #6339 on main and its release backport #6449. This PR adds permanent regression coverage for the typed Dataset.filter path. ## Rationale for this change A typed `Dataset.filter` predicate over a native aggregate can read sliced boolean inputs at the wrong offset through the codegen dispatcher, both retaining unwanted rows and dropping valid rows. The backport description verifies the exact Dataset path with a temporary test; this PR keeps that coverage in the repository. ## What changes are included in this PR? Add a `CometCodegenSuite` regression using its existing sliced-group fixture: 2,000 groups emitted in 100-row batches. Filter the typed `(Long, Boolean, Long)` Dataset by its boolean field. Compare all rows with Spark and the explicit expected rows, require native Partial/Final aggregates and a native filter, and assert dispatcher activity. The small fixture also exercises boolean bit offsets that are not byte-aligned. ## How are these changes tested? On Apple Silicon, JDK 21 and Spark 4.1.3: - Root-reactor `test-compile` and its Scala style/Spotless checks pass. - `./mvnw -B test -Dtest=none -Dsuites="org.apache.comet.CometCodegenSuite typed Dataset.filter"` passes with the freshly built matching release native library. One test ran; none canceled or ignored. - For the negative control, removed only `zero_offsets` from the dispatcher input export, rebuilt the native library, and ran the same test. It fails on differing rows. Restored the source and reran with the normal library: the test passes again. - `git diff --check` passes. No native implementation changes are included. Other Spark profiles and full Spark SQL/Iceberg suites were not run locally for this test-only change. CI is pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
