andygrove opened a new issue, #6924: URL: https://github.com/apache/datafusion-comet/issues/6924
### What is the problem the feature request solves? After a native Iceberg write task, `IcebergReflection.rebuildDataFilesWithJavaMetrics` opens every data file the task wrote through the table's `FileIO` and reads its Parquet footer back, so that the committed metrics come from iceberg-java's `ParquetUtil.footerMetrics`. The native writer has just produced those exact bytes, so the re-read only adds cost: one ranged read per written file, done one file after another while the task assembles its commit message. The user guide notes it: "This costs one footer-sized ranged read per written file at write time." On a local disk this is cheap. After #6923, the whole metrics rebuild is under 3% of CPU even for a fanout write of 320 files in `CometIcebergEndToEndBenchmark` (#6910). On an object store, though, each footer read is a request round trip and a billed GET, so a fanout write that produces many files per task waits on many sequential requests. ### Describe the potential solution - Native side: capture each data file's footer as it is written. `CountingFileWrite` already sees every byte on its way to storage, so keeping the tail of the stream yields the footer; the trailing four-byte length and `PAR1` magic say how long it is. Return the footers with the task's output, for example as a third `BINARY` column framed like `written_file_locations` (see `build_output_schema`). - JVM side: have `rebuildDataFilesWithJavaMetrics` parse the returned bytes with parquet-mr's `ParquetMetadataConverter` instead of opening the file, then call `ParquetUtil.footerMetrics` as it does now. The metrics stay identical by construction, since the bytes are the file's own footer. - Keep the ranged read as a fallback for any file whose footer was not captured, for example one larger than the retained tail. ### Additional context Native side: `MeteredParquetWriter`, `CountedOutput`, `CountingFileWrite` and `build_output_schema` in `native/core/src/execution/operators/iceberg_write.rs`. JVM side: `CometIcebergWriteExec.drainNativePayload` and `doExecute`, and `IcebergReflection.rebuildDataFilesWithJavaMetrics` and `readParquetFooter`. The user guide's sentence about the ranged read in `docs/source/user-guide/latest/iceberg-writes.md` would change with it. Follow-up to #6912, whose fix (#6923) removed the per-file Hadoop configuration parse from this same re-read. Part of #6911. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
