Hi Yun, Thanks for getting back. We’re on a fork of Flink 1.9 (basically 1.9 with some backported fixes from 1.10 and a couple of minor patches) - https://github.com/criteo-forks/flink/tree/criteo-1.9 I’ll check the jira + fix and see if there’s something that was potentially missed.
-- Piyush From: Yun Tang <myas...@live.com> Date: Sunday, March 8, 2020 at 11:05 PM To: Piyush Narang <p.nar...@criteo.com>, user <user@flink.apache.org> Subject: Re: Understanding n LIST calls as part of checkpointing Hi Piyush Which version of Flink do you use? After Flink-1.5, Flink would not call any "List" operation on checkpoint side with FLINK-8540 [1]. The only left "List" operation would be used when reading files in file input format. In a nut shell, these "List" calls should not come from Flink if you're using Flink-1.5+ [1] https://issues.apache.org/jira/browse/FLINK-8540<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-8540&data=02%7C01%7Cp.narang%40criteo.com%7C9e22e8abdb364375bbb608d7c3d6b184%7C2a35d8fd574d48e3927c8c398e225a01%7C1%7C1%7C637193199179063184&sdata=2u90FMzKtc3AXqL4sjndkyjFhdAAswbhZyde0YOdwhQ%3D&reserved=0> Best Yun Tang ________________________________ From: Piyush Narang <p.nar...@criteo.com> Sent: Saturday, March 7, 2020 6:15 To: user <user@flink.apache.org> Subject: Understanding n LIST calls as part of checkpointing Hi folks, I was trying to debug a job which was taking 20-30s to checkpoint data to Azure FS (compared to typically < 5s) and as part of doing so, I noticed something that I was trying to figure out a bit better. Our checkpoint path is as follows: my_user/featureflow/foo-datacenter/cluster_name/my_flink_job/checkpoint/chk-1234 What I noticed was that while trying to take checkpoints (incremental using rocksDB) we make a number of List calls to Azure: my_user/featureflow/foo-datacenter/cluster_name/my_flink_job/checkpoint my_user/featureflow/foo-datacenter/cluster_name/my_flink_job my_user/featureflow/foo-datacenter/cluster_name my_user/featureflow/foo-datacenter my_user/featureflow my_user Each of these calls takes a few seconds and all of them seem to add up to make our checkpoint take time. The part I was hoping to understand on the Flink side was whether the behavior of making these List calls for each parent ‘directory’ / blob all the way to the top was normal / expected? We are exploring a couple of other angles on our end (potentially flattening the directory / blob structure to reduce the number of these calls, is the latency on the Azure side expected), but along with this I was hoping to understand if this behavior on the Flink side is expected / if there’s something which we could optimize as well. Thanks, -- Piyush
