Hi, I think it's reasonable to support it. I am thinking to take it one step further:
Client-side metrics, or job level metrics for filesystem could help us to monitor filesystem more precisely. Some metrics (like request rate , throughput, latency, retry count, etc) are useful to monitor the network or client problem of checkpointing or other access cases for a job. Some file systems like s3, s3-presto, gs, etc have supported enabling some metrics, these could be exported in these filesystems. I have created a ticket[1] for it and would like to help to complete it. Thanks for reporting this. It's appreciated if you could provide more details (like metrics you think useful) or help to push forward this feature. [1] https://issues.apache.org/jira/browse/FLINK-30450 On Fri, Dec 16, 2022 at 1:48 AM Anton Ippolitov via user < user@flink.apache.org> wrote: > Hi, > > We recently switched to the flink-s3-fs-presto library for checkpointing > in Flink 1.16.0 and we would like to get client-side metrics from the > Presto S3 client (request rate, throttling rate, etc). > > I can see that the upstream client from Presto 0.272 already comes with a > metric collector (PrestoS3FileSystemMetricCollector.java > <https://github.com/prestodb/presto/blob/0.272/presto-hive/src/main/java/com/facebook/presto/hive/s3/PrestoS3FileSystemMetricCollector.java>) > that records all sorts of interesting metrics in a PrestoS3FileSystemStats > <https://github.com/prestodb/presto/blob/0.272/presto-hive/src/main/java/com/facebook/presto/hive/s3/PrestoS3FileSystemStats.java> > object. > > If I understand correctly, in the Presto project those metrics are being > exposed via JMX in the HiveS3Module class here > <https://github.com/prestodb/presto/blob/d93d0d8b22756d85cfe3e5c62c7718d8f29dbe54/presto-hive/src/main/java/com/facebook/presto/hive/s3/HiveS3Module.java#L48-L49> > but > we don't use this class to instantiate the client in flink-s3-fs-presto. > > Has anyone managed to actually expose those metrics from a Flink > application? If not, what would be recommended way of doing it? > > Thank you! > -- Best, Hangxiang.
