qzyu999 opened a new pull request, #23656:
URL: https://github.com/apache/datafusion/pull/23656
## Which issue does this PR close?
Closes #23472.
## Rationale for this change
External table formats (Apache Iceberg, Delta Lake, Apache Hudi) require
per-file column statistics (column_sizes, null_counts, lower/upper bounds,
split_offsets) when committing written files to their catalogs. Currently,
`ParquetSink` computes and stores this metadata internally (`self.written`) but
the `DataSink` trait only returns a row count forcing external consumers to
either:
1. Bypass DataFusion's write pipeline entirely (use PyArrow/parquet-rs
directly)
2. Re-read Parquet footers after writing (additional I/O round-trips)
3. Downcast `DataSinkExec.sink()` to `ParquetSink` and call `written()`
(fragile, undocumented)
This PR adds a standard, non-breaking mechanism to expose per-file metadata
through the `DataSink` trait.
## What changes are included in this PR?
1. **New type** in `datafusion-datasource`:
- `FileWriteMetadata`: path + row_count + byte_size + optional
format-specific metadata bytes
2. **Trait extension** (non-breaking, default implementation):
- `DataSink::file_metadata()` `Vec<FileWriteMetadata>` (default: empty)
3. **ParquetSink implementation**:
- Overrides `file_metadata()` using the existing `self.written` HashMap
that already collects `ParquetMetaData` during writes zero additional I/O
4. **DataSinkExec convenience**:
- `file_metadata()` method that delegates to the inner sink, avoiding the
downcast dance
The typed `ParquetMetaData` remains accessible via the existing
`ParquetSink::written()` method for Rust consumers who need full column-level
statistics directly. The `format_metadata: Option<Bytes>` field on
`FileWriteMetadata` is reserved for future Thrift serialization when FFI
consumers (e.g. Python via PyO3) need it.
## Are these changes tested?
Yes 14 new tests across both crates:
**`datafusion-datasource` (7 tests):**
- `FileWriteMetadata` equality and clone behavior
- Default `file_metadata()` returns empty after `write_all`
- Overridden `file_metadata()` returns correct entries
- Idempotent behavior (calling twice returns same result)
- `DataSinkExec::file_metadata()` delegates correctly
**`datafusion-datasource-parquet` (7 tests):**
- Empty before write, populated after write
- Row count consistency (`write_all` count == sum of per-file row_counts)
- Consistent with existing `ParquetSink::written()` accessor
- Idempotent across multiple calls
- Partitioned writes produce one entry per partition
- E2E test through `DataSinkExec` wrapper with `Arc<dyn DataSink>` dispatch
All existing `parquet_sink_write*` tests continue to pass unchanged.
## Are there any user-facing changes?
New public API surface (additive only, no breaking changes):
- `DataSink::file_metadata()` default method
- `FileWriteMetadata` struct
- `DataSinkExec::file_metadata()` convenience method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
