Re: Apache Flink - Throttling stream flow

2019-11-29 Thread M Singh
Thanks Rong for your references.  >From what I can see, the rate limiter is initialized statically.  But if the >load on downstream services varies, is there a way to update the rater limiter >at runtime ?  Please let me know if I missed anything. Thanks again for your advice.On Wednesday,

Re: Apache Flink - Throttling stream flow

2019-11-27 Thread Rong Rong
Hi Mans, is this what you are looking for [1][2]? -- Rong [1] https://issues.apache.org/jira/browse/FLINK-11501 [2] https://github.com/apache/flink/pull/7679 On Mon, Nov 25, 2019 at 3:29 AM M Singh wrote: > Thanks Ciazhi & Thomas for your responses. > > I read the throttling example but want

Re: Apache Flink - Throttling stream flow

2019-11-25 Thread M Singh
Thanks Ciazhi & Thomas for your responses. I read the throttling example but want to see if that work with a distributed broker like Kinesis and how to have throttling feedback to the Kinesis source so that it can vary the rate without interfering with watermarks, etc. Thanks again  Mans On

Re: Apache Flink - Throttling stream flow

2019-11-25 Thread Thomas Julian
related https://issues.apache.org/jira/browse/FLINK-13792 Regards, Julian. On Mon, 25 Nov 2019 15:25:14 +0530 Caizhi Weng wrote Hi, As far as I know, Flink currently doesn't have a built-in throttling function. You can write your own user-defined function to achieve

Re: Apache Flink - Throttling stream flow

2019-11-25 Thread Caizhi Weng
Hi, As far as I know, Flink currently doesn't have a built-in throttling function. You can write your own user-defined function to achieve this. Your function just gives out what it reads in and limits the speed it gives out records at the same time. If you're not familiar with user-defined funct

Apache Flink - Throttling stream flow

2019-11-24 Thread M Singh
Hi: I have an Flink streaming application that invokes  some other web services.  However the webservices have limited throughput.  So I wanted to find out if there any recommendations on how to throttle the Flink datastream so that they don't overload the downstrream services.  I am using Kines