Hi Alexander, The reference [3] mentioned in my previous email is your discussion thread on FLIP-238. We have noticed your idea of providing a new rate-limiter, which is great and matches our starting point. I was just about to chat with you on slack about this feature :)
I tend to propose a separate FLIP for the following reasons: 1. Source rate limiting is an independent feature and involves changes to the Source API. 2. Source rate limiting is a much more important feature. Projects such as Flink CDC can directly benefit from this feature, and a separate FLIP will help its implementation and application. 3. The function of the source rate-limiter may go further, it can provide not only limits on throughput but also limits on the number of requests and so on (perhaps your implementation is enough for now), which can better serve different types of sources. What do you think? Is it necessary to raise a FLIP separately? If necessary, you can also write the first version of the proposal for source rate-limiting, and then I will add some of my ideas :) Thanks, Steve Alexander Fedulov <alexan...@ververica.com> 于2022年7月14日周四 00:47写道： > > Hi Steve, > > in the FLIP-238 [1] I propose to also introduce a rate limiter with a > simple API at the level of SourceReader: > RateLimiter.java > <https://github.com/afedulov/flink/blob/FLINK-27919-generator-source/flink-core/src/main/java/org/apache/flink/api/common/io/ratelimiting/RateLimiter.java> > GuavaRateLimiter.java > <https://github.com/afedulov/flink/blob/FLINK-27919-generator-source/flink-core/src/main/java/org/apache/flink/api/common/io/ratelimiting/GuavaRateLimiter.java> > RateLimitedSourceReader.java > <https://github.com/afedulov/flink/blob/FLINK-27919-generator-source/flink-core/src/main/java/org/apache/flink/api/connector/source/lib/util/RateLimitedSourceReader.java> > DataGeneratorSourceV3.java#L140 > <https://github.com/afedulov/flink/blob/FLINK-27919-generator-source/flink-core/src/main/java/org/apache/flink/api/connector/source/lib/DataGeneratorSourceV3.java#L140> > > Did you plan to go further than this? > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-238%3A+Introduce+FLIP-27-based+Data+Generator+Source > > Best, > Alexander Fedulov > > On Wed, Jul 13, 2022 at 6:38 PM Steve Yurong Su <r...@apache.org> wrote: > > > Hi Jark, > > > > Thanks a lot :) > > > > Steve > > > > Jark Wu <imj...@gmail.com> 于2022年7月13日周三 22:48写道： > > > > > > Hi Steve, > > > > > > Welcome to the Flink community. I have granted the edit permission to > > you. > > > Looking forward to your FLIP. > > > > > > Best, > > > Jark > > > > > > On Wed, 13 Jul 2022 at 19:56, Steve Yurong Su <r...@apache.org> wrote: > > > > > > > Hi folks! > > > > > > > > My name is Steve Su, and I am currently a third-year master’s student > > > > at Tsinghua University. > > > > > > > > I am now participating in a Flink-related student project in OSPP > > > > (Open Source Promotion Plan[1]). The project aims to design and > > > > implement the generic rate limiter for the FLIP-27 source and apply > > > > the rate-limiting feature to the Flink CDC project. @Leonard Xu is my > > > > mentor in this project. > > > > > > > > We already had many discussions on source rate-limiting on the mailing > > > > list, such as [2], [3], and so on. Leonard and I had an offline > > > > discussion, and we thought that it would be great to have a generic > > > > rate limiter, so I’d like to raise a new FLIP to illustrate the design > > > > further. > > > > > > > > Please grant my account FLIP edit permissions. My apache confluence > > > > account id is `steveyurongsu`. :D > > > > > > > > Thanks, > > > > Steve Su > > > > --- > > > > [1] https://summer-ospp.ac.cn/homepagy > > > > [2]https://lists.apache.org/thread/ff6mcos8g4otnhjrp030lrcrf3omgfmf > > > > [3] https://lists.apache.org/thread/7gjxto1rmkpff4kl54j8nlg5db2rqhkt > > > > > >
