Thanks for reviving this thread, Reetika! When I took a look at this last year there was only a way to expose query-level metrics but the manifest file content cache is rather on a process-level. Hence I brought up this topic but apparently there was no interest coming from the community so I haven't investigated further. However, if there is an interest now, I think I can take another look to find a way to expose these metrics so that the clients could pick them up and use them for their purposes.
I'd also appreciate to hear the opinion from committers/PMC to see where the pitfalls are for instance. Cheers, Gabor On Sun, Nov 5, 2023 at 7:51 PM Reetika Agrawal <agrawal.reetika...@gmail.com> wrote: > Hi Iceberg Community, > > I was also exploring Manifest file caching feature which was introduced in > Iceberg https://github.com/apache/iceberg/pull/4518 for Presto execution > engine. And as I checked Cache Metrics are getting computed and recorded in > Iceberg library for same, but didn’t find a way to access these metrics on > execution engine side. To validate the cache usage we would require atleast > cache metrics access for this feature. Could you please let me know how > could I do that? > > Thanks, > Reetika > > On 2022/12/09 15:40:04 Gabor Kaszab wrote: > > Hey Iceberg Community, > > > > Scan metrics support was introduced in 1.1.0 that is pretty nice to > collect > > some per-query stats like 'total planning duration' and such. > > However, I haven't found a way to get process level metrics at the > moment. > > The particular thing I wanted to achieve is that there was a Manifest > file > > content cache <https://github.com/apache/iceberg/pull/4518> introduced > > recently where the stats are logged, but aren't exposed through the API. > > > > Such an API could be used to expose stats for various kinds of caches > (e.g. > > number of hits and misses), we could count the number of specific API > calls > > (like planFiles()) and I guess many more other useful stuff. > > > > With this mail I'd like to initiate a discussion around this topic. If > the > > community is open for such a change I could start working on an initial > > solution that would expose the Manifest file content cache stats but > could > > be extended later on with more metrics. > > > > Cheers, > > Gabor > > >
