[
https://issues.apache.org/jira/browse/FLINK-9560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16508151#comment-16508151
]
ASF GitHub Bot commented on FLINK-9560:
---------------------------------------
GitHub user etiennecarriere opened a pull request:
https://github.com/apache/flink/pull/6149
[FLINK-9560] Add RateLimiting for FileSystem
## Contribution Checklist
- Make sure that the change passes the automated tests, i.e., `mvn clean
verify` passes. You can set up Travis CI to do that following [this
guide](http://flink.apache.org/contribute-code.html#best-practices).
- Once all items of the checklist are addressed, remove the above text
and this checklist, leaving only the filled out template below.
## What is the purpose of the change
* Adding rate limiting on FileSystem . Our first purpose is to limit the
bandwidth used by the writing of Checkpoint to object storage because it has
impact on the global throughput of flink
## Brief change log
Use the existing LimitedConnectionsFileSystem class to add some rate
limiting mechanisms to throttle the bandwidth needs. It use the RateLimiter
class provided by Guava
## Verifying this change
- Unit test for validation of the configuration parsing
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): Yes . Flink-core
now depends on shaded guava18
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: No
- The serializers: No
- The runtime per-record code paths (performance sensitive): No
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: No
- The file system connector: Yes
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? Ops documentation updated
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/DataDome/flink FLINK-9560
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6149.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6149
----
commit b6e22f3bb15652780efbe6375d6b24f5bba73f4e
Author: Etienne Carriere <etienne.carriere@...>
Date: 2018-06-11T12:14:54Z
[FLINK-9560] Add RateLimiting for FileSystem
Add the capacity to limit the bandwidth used by reading or writing to
Filesystem. It can be useful, for example, for limiting the bandwidth
used to write periodic checkpoint to object storage
----
> RateLimiting for FileSystem
> ---------------------------
>
> Key: FLINK-9560
> URL: https://issues.apache.org/jira/browse/FLINK-9560
> Project: Flink
> Issue Type: Improvement
> Components: FileSystem
> Affects Versions: 1.5.0
> Reporter: Etienne CARRIERE
> Priority: Major
>
> *Pain*: On our system, we see that during checkpoint , all the bandwidth is
> take to send the checkpoint to object storage (s3 in our case)
> *Proposal* : After the creation of some limitation on Filesystem (mostly
> number of connections with the tickets FLINK-8125/FLINK-8198/FLINK-9468), I
> propose to add ratelimiting "per Filesystem" .
> *Proposal of implementation* : Modify LimitedConnectionsFileSystem to add a
> ratelimiter on both Input and OutputStream.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
