Re: Understanding n LIST calls as part of checkpointing

2020-03-09 Thread Piyush Narang
. -- Piyush From: Yun Tang Date: Sunday, March 8, 2020 at 11:05 PM To: Piyush Narang , user Subject: Re: Understanding n LIST calls as part of checkpointing Hi Piyush Which version of Flink do you use? After Flink-1.5, Flink would not call any "List" operation on checkpoint side with FLI

Re: Understanding n LIST calls as part of checkpointing

2020-03-08 Thread Yun Tang
should not come from Flink if you're using Flink-1.5+ [1] https://issues.apache.org/jira/browse/FLINK-8540 Best Yun Tang From: Piyush Narang Sent: Saturday, March 7, 2020 6:15 To: user Subject: Understanding n LIST calls as part of checkpointing Hi fo

Understanding n LIST calls as part of checkpointing

2020-03-06 Thread Piyush Narang
Hi folks, I was trying to debug a job which was taking 20-30s to checkpoint data to Azure FS (compared to typically < 5s) and as part of doing so, I noticed something that I was trying to figure out a bit better. Our checkpoint path is as follows: my_user/featureflow/foo-datacenter/cluster_name