xudong963 commented on PR #27027: URL: https://github.com/apache/datafusion/pull/27027#issuecomment-6620236689
> I am wondering if users of datafusion may have used that returned parquet metadata for creating external row group indexes so while internally its not used it could potentially be disruptive to users. Maybe it would be safer to keep existing functionality as is and add the ability to opt out of returning metadata? Good call, thanks @matthewmturner! I let llm re-check all github repos, lol, and there are real users of `written() `after all: InfluxDB 3 (parquet_file/src/serialize.rs), datafusion_iceberg (builds DataFiles from it) and LakeSoul (file_sink_writer.rs). All construct ParquetSink themselves and write one or a few files per sink, so the retention is harmless for them. I’ll switch this PR to keep `written()` as is and add an opt-out on the builder, turn it off in ParquetFormat::create_writer_physical_plan (nothing on the COPY/INSERT path reads it), and still fix the JoinSet buffering so that opting out retains nothing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
