andygrove opened a new issue, #5385: URL: https://github.com/apache/datafusion-comet/issues/5385
Triage pass over the open `requires-triage` queue, per the project [Bug Triage Guide](https://github.com/apache/datafusion-comet/blob/main/docs/source/contributor-guide/bug_triage.md). - Date: 2026-08-17 - Total issues processed: 33 (31 triaged, 2 skipped, 0 failed) - Type counts: 8 bugs, 23 enhancements - Priority counts applied: `priority:critical` 2, `priority:high` 0, `priority:medium` 3, `priority:low` 3 - Guide: [docs/source/contributor-guide/bug_triage.md](https://github.com/apache/datafusion-comet/blob/main/docs/source/contributor-guide/bug_triage.md) Labels have already been applied. A reviewer should spot-check the calls below and close this issue when satisfied; corrections should be made directly on the affected issue. Notes on label availability and pre-existing labels: - The guide lists `spark 4` as a pre-existing area indicator, but the repository only has `spark 4.0` / `spark 4.1` / `spark 4.2` / `spark 3.x`, so no `spark 4` label was applied in this pass. - `area:Iceberg` exists in the repository but is not in the guide's area table, so it was not applied to the Iceberg sort-merge follow-ups (#5337–#5343); `area:scan` was used instead. Worth deciding whether the guide should add it. - Where an issue already carried non-guide labels (`correctness`, `performance`, `test`, `array expressions`, `temporal expressions`, `user experience`), those were left untouched and are included in the area lists below for context. - Two enhancements arrived with a priority label already set by the reporter: #5383 (`priority:medium`) and #5346 (`priority:low`). The guide says only bugs receive a priority label. This pass did not add priority labels to enhancements and did not strip the pre-existing ones; a reviewer may want to remove them. ## Bugs ### priority:critical - to_time / try_to_time: native parser rejects 'T12' and '12:30:45.' which Spark accepts ([#5366](https://github.com/apache/datafusion-comet/issues/5366)) - Area labels: `area:expressions`, `correctness`, `temporal expressions` - Rationale: `try_to_time` returns `NULL` where Spark returns a value, and fractional digits 7–9 are truncated into a different stored nanosecond value, both with no error surfaced — the guide's "silent wrong results" case at step 1 of the decision tree. Escalated from the reporter's `priority:medium` (see escalations below). - Boolean-to-decimal cast produces invalid Decimal128 when 10^scale does not fit precision ([#5334](https://github.com/apache/datafusion-comet/issues/5334)) - Area labels: `area:expressions` - Rationale: `cast_boolean_to_decimal` returns `Ok` with an unscaled value outside the declared precision because `with_precision_and_scale` validates only type metadata, so Comet emits invalid decimal data as success where Spark errors or returns `NULL` — data corruption, `priority:critical` per the guide's first decision-tree question. ### priority:medium - Report native child-operator spill metrics in Spark task metrics for unified shuffle plans ([#5382](https://github.com/apache/datafusion-comet/issues/5382)) - Area labels: `area:shuffle` - Rationale: the task-metric bridge publishes spill counters only from the root `ShuffleWriterExec` node, so a task that spills 160 MiB reports 32 MiB — a functional defect in existing reporting rather than a new feature, and diagnostic-only, so `priority:medium` per step 3. Classified consistently with #5336, the same category of metric defect. - Task input metrics are unreliable when a native block mixes a native scan with a JVM input ([#5336](https://github.com/apache/datafusion-comet/issues/5336)) - Area labels: `area:scan` - Rationale: the completion listener is registered after the iterator's close listener, so on the busy-poll path the reported `bytes_scanned` is stale or zero; reporter's `priority:medium` confirmed — broken feature, no query-result impact. - `NegativeExpr::get_properties` reports child ordering and range unchanged ([#5330](https://github.com/apache/datafusion-comet/issues/5330)) - Area labels: `area:expressions` - Rationale: `with_order` is an identity function as used, so `-a` is reported with the child's unflipped ordering and un-negated range; the defect is demonstrated at unit level with no end-to-end query yet, so `priority:medium` per step 3 rather than critical (see escalations below). ### priority:low - Nine expression benchmark rows labelled "Comet" are measuring Spark ([#5372](https://github.com/apache/datafusion-comet/issues/5372)) - Area labels: `area:expressions`, `test`, `performance` - Rationale: the deliverable is corrected benchmark reporting, which the guide's step 4 places at `priority:low` as a tooling/test-only issue; the underlying `ShortType` fallback may be more than that (see escalations below). - ANSI `abs` integer overflow errors use Arrow-style type names instead of Spark's ([#5356](https://github.com/apache/datafusion-comet/issues/5356)) - Area labels: `area:expressions` - Rationale: the error class matches Spark (`ARITHMETIC_OVERFLOW`) and only the interpolated type name differs (`Int64` vs `long`), so nothing is broken beyond user-visible message text — cosmetic, step 4. - Improve CI to run enough tests when dependency upgraded ([#5333](https://github.com/apache/datafusion-comet/issues/5333)) - Area labels: `area:ci` - Rationale: kept as `bug` because the guide counts CI failures as bugs and this gap let #5289 break `main`, but priority is `priority:low` per step 4, whose examples are explicitly CI and build tooling; the breakage itself has been reverted (#5332). ## Enhancements - Reduce JNI round-trips in the unified memory pools (batching, hysteresis, cheaper call path) ([#5383](https://github.com/apache/datafusion-comet/issues/5383)) - Area labels: `area:ffi`, `performance` - Rationale: a performance optimization of a working code path — chunked acquire with retained slack and hysteresis — which the guide's type table places under `enhancement`. - Make native contrib scan dispatch generic (remove per-contrib arms from core Rust) ([#5379](https://github.com/apache/datafusion-comet/issues/5379)) - Area labels: `area:scan` - Rationale: source-level tidiness, replacing `#[cfg]`-gated per-contrib calls with a `type_url -> handler` registry; the issue itself notes a default build links zero contrib symbols, so nothing is broken. - Move contrib scan proto messages out of core operator.proto into contrib/<name>/proto ([#5378](https://github.com/apache/datafusion-comet/issues/5378)) - Area labels: `area:scan` - Rationale: a refactor that needs new per-contrib proto build infrastructure; the messages are inert in core today and the relocation is invisible on the wire, so there is no defect. - Use end to end microbenchmarks to prevent performance regressions between releases ([#5373](https://github.com/apache/datafusion-comet/issues/5373)) - Area labels: `area:ci` - Rationale: asks for benchmark documentation, a reproducible EC2 instance type, and eventual scheduled automation — new process and tooling, not a defect. - Investigate: scan + shuffle-write slower than Spark on deeply nested schemas ([#5355](https://github.com/apache/datafusion-comet/issues/5355)) - Area labels: `area:scan`, `area:shuffle`, `performance` - Rationale: classified as an enhancement because it is an explicitly unverified collection of static-analysis optimization candidates ("no benchmark numbers yet") rather than a confirmed regression from a prior Comet release; if a benchmark confirms Comet is slower than Spark on a real workload, this should be re-triaged as a `priority:medium` performance bug. - Run replace natively by default when the search string is a non-empty literal ([#5354](https://github.com/apache/datafusion-comet/issues/5354)) - Area labels: `area:expressions`, `performance` - Rationale: the current dispatcher path is correct; this asks to widen the native default for a provably safe subset, which is a performance optimization. - Native candidate assessment: upper/lower deferred on the lack of a per-batch defer-to-dispatcher mechanism ([#5353](https://github.com/apache/datafusion-comet/issues/5353)) - Area labels: `area:expressions`, `performance` - Rationale: records a missing capability — plan-time-only native/dispatcher selection with no way to defer a single batch — as a blocker for a class of optimizations; nothing behaves incorrectly today. - Support Apache Celeborn remote shuffle service ([#5352](https://github.com/apache/datafusion-comet/issues/5352)) - Area labels: `area:shuffle` - Rationale: new functionality that Comet does not have today, the guide's canonical `enhancement` case. - Run rlike natively by default for patterns that are provably Java-regex equivalent ([#5351](https://github.com/apache/datafusion-comet/issues/5351)) - Area labels: `area:expressions`, `performance` - Rationale: proposes a plan-time pattern analyzer so a safe subset can default to the existing native kernel; the conservative current behavior is not a defect. - Implement sequence natively for integral types instead of JVM codegen dispatch ([#5349](https://github.com/apache/datafusion-comet/issues/5349)) - Area labels: `area:expressions`, `array expressions`, `performance` - Rationale: the codegen dispatcher is correct; this asks for a native implementation to remove per-row allocations, so it is a performance optimization. - Implement unbase64 natively instead of JVM codegen dispatch ([#5347](https://github.com/apache/datafusion-comet/issues/5347)) - Area labels: `area:expressions`, `performance` - Rationale: `unbase64` is already `Compatible()` via codegen dispatch, so behavior matches Spark; the ask is a new native kernel for the decode direction. - Tag fallback reasons with structured values instead of pre-formatted strings ([#5346](https://github.com/apache/datafusion-comet/issues/5346)) - Area labels: none (`user experience`) - Rationale: a refactor of the fallback-reason representation across ~205 call sites to make reasons dedupable and queryable — an improvement to existing working reporting. - Bound memory when a partition has many files ([#5343](https://github.com/apache/datafusion-comet/issues/5343)) - Area labels: `area:scan` - Rationale: a scoped follow-up to #5323 proposing bounded-concurrency file reads in the sorted merge path; filed as planned work on a feature that just landed, not a reported failure. - Broaden storage-partitioned-join tests, then enable reportPartitioning by default ([#5342](https://github.com/apache/datafusion-comet/issues/5342)) - Area labels: `area:scan` - Rationale: adds test coverage as the precondition for flipping a config default — new coverage plus a behavior change, no existing defect. - Skip the merge wrapper for single-file partitions ([#5341](https://github.com/apache/datafusion-comet/issues/5341)) - Area labels: `area:scan` - Rationale: avoids a no-op `SortPreservingMergeExec` for single-file partitions; correct today, just slightly wasteful, so a performance optimization. - Keep read concurrency in the sorted merge path ([#5340](https://github.com/apache/datafusion-comet/issues/5340)) - Area labels: `area:scan` - Rationale: asks to benchmark the many-small-files case and add read-ahead if it regresses — prospective optimization work, with the regression not yet established. - Support transformed sort orders (bucket / truncate / etc.) ([#5339](https://github.com/apache/datafusion-comet/issues/5339)) - Area labels: `area:scan` - Rationale: extends ordering reporting from identity sort keys to Iceberg transforms; the current narrow scope is documented as deliberate and always correct. - Merge on sort keys even when they are not selected ([#5338](https://github.com/apache/datafusion-comet/issues/5338)) - Area labels: `area:scan` - Rationale: widens when the sorted merge can be used by reading unprojected sort columns; skipping the merge today is a correct-but-conservative fallback. - Sort-aware Iceberg reads: per-partition streaming merge + report ordering and grouping (v1) ([#5337](https://github.com/apache/datafusion-comet/issues/5337)) - Area labels: `area:scan` - Rationale: the umbrella issue for a new capability — preserving Iceberg sort order and reporting key-grouped partitioning — with #5338–#5343 as its scoped follow-ups. - Add mode to run Comet planning but execute with Spark, so users can assess potential Comet coverage ([#5335](https://github.com/apache/datafusion-comet/issues/5335)) - Area labels: none - Rationale: proposes a new `spark.comet.planOnly.enabled` dry-run mode so users can estimate coverage without execution risk — new functionality and a new config. - Add a fallback-invariance suite: query outcome should not change when a compatible expression is forced from native to Spark ([#5328](https://github.com/apache/datafusion-comet/issues/5328)) - Area labels: `area:expressions` - Rationale: new test infrastructure sweeping the 293-expression registry for an invariant, replacing four hand-picked per-expression tests; classified as an enhancement even though it is expected to surface bugs, because the ask itself is coverage. - Comet 1.1.0 Release (September) ([#5327](https://github.com/apache/datafusion-comet/issues/5327)) - Area labels: none - Rationale: a release-planning tracking issue; `enhancement` is the closer of the two type labels since nothing is reported broken. - Could Rust 1.98 new floating point optimizations help Comet? ([#5326](https://github.com/apache/datafusion-comet/issues/5326)) - Area labels: none - Rationale: an open question about adopting new compiler float-math optimizations — exploratory performance work, no defect described. ## Escalations to consider - to_time / try_to_time: native parser rejects 'T12' and '12:30:45.' which Spark accepts ([#5366](https://github.com/apache/datafusion-comet/issues/5366)) - Escalated from the reporter's `priority:medium` to `priority:critical`. The `to_time` cases raise a visible error, which alone would sit at `priority:medium`, but `try_to_time` returns `NULL` for input Spark parses successfully and case 3 stores a different nanosecond value — both silent, which the guide's step 1 and the "correctness over crashes" principle put at critical. Narrow surface (Spark 4.1 only, unusual literals) is the counter-argument, but the guide's own critical examples include comparably narrow cast edge cases. Revert to `priority:medium` if the reviewer judges case 3 unobservable and the `try_to_time` `NULL` acceptable. - `NegativeExpr::get_properties` reports child ordering and range unchanged ([#5330](https://github.com/apache/datafusion-comet/issues/5330)) - Left at `priority:medium`, but this is the escalation trigger to watch: the wrong ordering is reported to the optimizer, so if a plan can be shown where DataFusion drops or keeps a sort based on it, the result is silently mis-ordered output and this becomes `priority:critical`. The issue currently demonstrates the defect only through an expression-property unit test. - Nine expression benchmark rows labelled "Comet" are measuring Spark ([#5372](https://github.com/apache/datafusion-comet/issues/5372)) - Left at `priority:low` as a benchmark-reporting defect, but the issue notes that Comet does support short-to-int and short-to-long casts, so if the `ShortType` projection failing to be replaced reproduces outside `CometCastNumericToNumericBenchmark`, it is a real coverage gap affecting user queries and warrants `priority:medium`. - Improve CI to run enough tests when dependency upgraded ([#5333](https://github.com/apache/datafusion-comet/issues/5333)) - Left at `priority:low`. The guide's escalation trigger — "a `priority:low` CI flake is blocking PR merges consistently" — applies if dependency bumps break `main` again; #5289 is one occurrence and was reverted. ## Skipped — needs more info - Bug triage results: 2026-08-11 ([#5325](https://github.com/apache/datafusion-comet/issues/5325)) - Not a bug report or a feature request but the summary issue from the previous triage pass, so neither type label applies and no priority is meaningful. `requires-triage` was left in place because the skill only removes it alongside a type label. The reviewer should close it once the 2026-08-11 calls have been spot-checked; it will otherwise reappear in every future pass. - Bug triage results: 2026-08-03 ([#5231](https://github.com/apache/datafusion-comet/issues/5231)) - Same as above, for the 2026-08-03 pass. This one has now been open across three triage passes, so it is worth either closing it or deciding that triage summary issues should be exempted from the `requires-triage` auto-label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
