Hi,
I would also go for Niels approach. If the mapper has the same parallelism
as the source and its right after it, it'll be chained to the source. The
throttling then happens with almost no overhead.
Regarding the ThrottledIterator: Afaik there is no iterator involved when
reading data out of th
There is a utility in flink-streaming-examples that might be useful, but is
generally the same idea that Niels suggests. [1]
[1]
https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/utils/ThrottledIterator.java
On Su
Simple idea: create a map function that only does "sleep 1/5 second" and
put that in your pipeline somewhere.
Niels
On 16 Apr 2016 22:38, "Chen Bekor" wrote:
> is there a way to consume a kafka stream using flink with a predefined
> rate limit (eg 5 events per second)
>
> we need this because w
is there a way to consume a kafka stream using flink with a predefined
rate limit (eg 5 events per second)
we need this because we need to control some 3rd party api rate limitations
so, even if we have a much larger throughput potential, we must control
the consumption rate in order not to over
