andygrove opened a new pull request, #6503:
URL: https://github.com/apache/datafusion-comet/pull/6503
## Which issue does this PR close?
Part of #5639.
Stacked on #6450. Until that merges, this PR also shows its commit. Only the
last two commits
belong here: the first only moves files, so blame follows them, and the
second wires up the crate.
## Rationale for this change
#5639 splits the native `core` crate into per-concern crates. Its step T4
moves the operators that
need nothing else from `core` into their own crate, which the epic's
dependency graph has depending
only on `common` and the expression crate. T3 finishes in #6450, so nothing
blocks this step.
## What changes are included in this PR?
- A new workspace crate, `datafusion-comet-operators` (`native/operators/`),
holds `ExpandExec`,
`ExplodeExec`, `CometFilterExec`, `PartitionedRankLimitExec` and
`SampleExec`, moved unchanged
with their unit tests. It depends on arrow, datafusion, futures,
`datafusion-comet-common` and
`datafusion-comet-spark-expr`, which is the epic's `operators -> common,
expr` edge. It is a
default workspace member, so CI's `cargo nextest run` covers it.
- The explode criterion bench moves with `ExplodeExec`, from `core/benches/`
to `operators/benches/`.
- The planner and `dynamic_filter` import the five types from the new crate,
and `core` keeps no
re-exports. `CometFilterExec` and the five accessors `core` calls
(`from_datafusion`, `input`,
`predicate`, `has_projection`, `with_execution_input`) become `pub`.
- The nested-nullability test fixture moves from a `#[cfg(test)]` module in
`core` to a hidden `pub`
module in `datafusion-comet-common`. The `ExpandExec` and
`ShuffleScanExec` tests that share it
are now in different crates, and `ShuffleScanExec` moves to the readers
crate in T6, which also
depends on `common`. The `test_common` module in `spark-expr` is the
precedent for a test helper
in a lower crate.
- `projection.rs` stays in `core`, because it only holds the projection
builder, which T7 moves to
the planner crate. The epic's `copy.rs` no longer exists (it was removed
in #6364).
- `dev/ci/compute-changes.py` routes `native/operators/benches/**` like the
other bench
directories, so a bench-only change still goes to the benchmark check
rather than the full build.
`check-ci-config.py` gains a routing case for it.
- `native/README.md` lists every workspace crate (it was missing `common`,
`jni-bridge` and
`shuffle`), and `adding_a_new_operator.md` points at the new crate.
git records the six moved files as renames. The only edits inside them are
the fixture import in
the `expand` tests, the `ExplodeExec` import in the bench, a doc path in
`explode.rs`, and the
visibility change in `filter.rs`.
## How are these changes tested?
This moves code without changing behavior, so there are no new tests. The 34
unit tests in the
moved files now run as part of `datafusion-comet-operators`.
- `cargo clippy --all-targets --workspace -- -D warnings`, `cargo fmt
--check` and `cargo machete`
are clean, and `python3 dev/ci/check-ci-config.py` passes.
- `cargo nextest run` over the default workspace passes (1862 tests, 34 of
them in the new crate).
`cargo test -p datafusion-comet-operators` also passes on its own, so the
crate builds without
features unified from the rest of the workspace. The explode bench runs in
test mode.
- The JVM tests that exercise the moved operators pass against the rebuilt
`libcomet`, 75 in all:
`CometGenerateExecSuite` (`ExplodeExec`), the `window group limit` tests in
`CometWindowExecSuite` (`PartitionedRankLimitExec`), the `sample`, `expand
operator` and
`project + filter` tests in `CometExecSuite`, and the `join dynamic
filter` tests in
`CometJoinSuite` (`CometFilterExec`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
