Re: Bandwidth throttling of checkpoints uploading to s3

Great, thank you. 12.07.2019, 21:31, "Yu Li" :Thanks for the information Pavel, good to know. And I've created FLINK-13251 to introduce the checkpoint bandwidth throttling feature, FYI. Best Regards,Yu On Sat, 13 Jul 2019 at 00:11, Павел Поцелуев wrote:We use FsStateBacke

Re: Bandwidth throttling of checkpoints uploading to s3

Thanks for the information Pavel, good to know. And I've created FLINK-13251 to introduce the checkpoint bandwidth throttling feature, FYI. Best Regards, Yu On Sat, 13 Jul 2019 at 00:11, Павел Поцелуев wrote: > >1. We use FsStateBackend

Re: Bandwidth throttling of checkpoints uploading to s3

We use FsStateBackend and state snapshot size is about 700 Mbyte.We are thinking about migration to RocksDBStateBackend and turning on incremental checkpoints.I think incremental size would be small in our current use case so incremental checkpoints can solve the problem.I think it is about 50 Mbit

Re: Bandwidth throttling of checkpoints uploading to s3

Hi Pavel, Currently there's no such throttling functionality in Flink and I think it's a valid requirement. But before opening a JIRA for this, please allow me to ask for more details to better understand your scenario: 1. What kind of state backend are you using? Since you observe high load to ne

Bandwidth throttling of checkpoints uploading to s3

Hello! We use flink with periodically checkpointing to s3 file system. And when flink uploads checkpoint to s3 it makes high load to the network. We have found in AWS CLI S3 configuration option called max_bandwidth which allows to limit rate in bytes per second. Is there a way to have the same fun