pingzh opened a new pull request, #7325:
URL: https://github.com/apache/datafusion-comet/pull/7325
## Which issue does this PR close?
Closes #7324. Related to #5124; metric evaluation remains in Spark, so this
does not implement the fully native collector proposed there.
## Rationale for this change
A Parquet scan followed by `observe(count(lit(1)).as("rows"))` and a grouped
aggregation currently introduces a row boundary at the observation. Preserve
the observation results while passing Arrow batches between eligible native
operators.
## What changes are included in this PR?
- Add `CometCollectMetricsExec`, enabled by
`spark.comet.exec.collectMetrics.enabled`. Spark evaluates the metric
expressions over batch row views and retains ownership of accumulator
publication, observation discovery and resets. The original Arrow batch then
feeds the downstream native block.
- Require a materializing shuffle or broadcast exchange above the collector.
Retain Spark collection below limits, joins, partial iterator consumers and
nonpositive replacement sampling, and for direct actions or writes without such
an exchange, so counts match Spark's row consumption.
- Integrate native input boundaries, AQE revisits and stage reversion. Adopt
the first completed accumulator state without copying large aggregate buffers,
matching Spark.
- Document the supported plan shapes and register `CometCollectMetricsSuite`
in Linux and macOS CI, with Spark-version-specific aggregate test helpers.
## How are these changes tested?
The focused suite compares output rows and observations with Spark, covering
AQE on/off, empty inputs, repeated execution, aggregate-state publication,
limits, joins, iterator consumers, sampling, stage reversion and disabled
configuration.
Formatting (`./mvnw spotless:apply`), CI configuration validation (`python3
dev/ci/check-ci-config.py`) and `git diff --check` passed. Two independent
static reviews completed. Local test execution was skipped; strict Spark 3.5
compilation is in progress. CI should include `run-all-spark-profiles` and
`run-spark-4.1-tests` for version and Spark SQL coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
