Hi Team, TL;DR: I would like to introduce the possibility to parametrize the Iceberg table scans to include the metadata metrics only for specific columns.
We discussed this previously on the mailing list [1], but we did not finalize the direction there. *To recap* Currently there are two ways to affect which column metrics are returned from the metadata by the scan tasks: - Decide which column stats are collected/stored - write.metadata.metrics.* write configuration could be used - Decide if the scan should include the column stats - includeColumnStats - scan builder configuration In our experience this granularity is not enough if the table has many columns and multiple readers. Some readers might want to have a specific column statistics only, while other readers might need metrics for another column. *Current issue* As for the concrete case, I have a PR [1] under review to emit watermarks when reading an Iceberg table by the Flink Source. It would be good to provide a simple "watermarkFieldName" for the users, which would be used for generating the watermark. The issue is that for the simple implementation we would need the column metadata metrics to extract the watermark for the *FileScanTasks*. If we include all of the column statistics, then it could easily cause memory issues on the JobManager side. This is why we are reluctant to make this as a simple switch on the user side, and might opt for a more heavy extractor interface where the user has the responsibility to extract the watermark themselves. OTOH, if it would be possible to return column metrics only for the given column, the memory pressure would be more predictable, and we could have a more user-friendly solution. *Suggested solution* I would like to introduce a new method to the Scan class, like: *ThisT includeColumnStats(Collection<String> columns);* Using this configuration the users of the Scan task could decide which column metrics are retained during the planning process, and which column metrics are thrown away in the results. Would the community consider this as a valuable addition to the Scan API? Thanks, Peter - [1] https://lists.apache.org/thread/9rl8yq8ps3mfg91g1qvzvgd0tnkjvxgg - Scan statistics thread - [2] https://github.com/apache/iceberg/pull/8553 - Flink: Emit watermarks from the IcebergSource
