Hi Chen,
just a small comment regarding your proposition: this would work well when
one does a complete message passthrough. If there is some filtering in the
pipeline, which could be dependent on the incoming stream data itself, the
output throughput (the goal of the throttling) would be hard to
Hi Ray,
In a bit abstract point of view, you can always throttle source and get
proper sink throughput control.
One approach might be just override base KafkaFetcher and use shaded
guava rate limtier.
https://github.com/apache/flink/blob/59714b9d6addb1dbf2171cab937a0e3fec52f2b1/flink-connectors/f
I see. I appreciate your help, thank you so much!
Benchao Li 于2020年5月17日周日 下午2:48写道：
> Hi,
>
> > If I want to use the rate limiter in other connectors, such as Kafka
> sink, ES sink, I need to do some more work on these connectors.
> Yes, you can do this by changing Kafka/ES sink, actually, this
Hi,
> If I want to use the rate limiter in other connectors, such as Kafka
sink, ES sink, I need to do some more work on these connectors.
Yes, you can do this by changing Kafka/ES sink, actually, this is how we
did internally.
> I'd like to know if the community has a plan to make a lower-level
Hi Benchao
Thanks for your answer!
According to your answer, I found `GuavaFlinkConnectorRateLimiter` which is
the implementation of the `FlinkConnectorRateLimiter`.
If I want to use the rate limiter in other connectors, such as Kafka sink,
ES sink, I need to do some more work on these connector
AFAIK, `FlinkKafkaConsumer010#setRateLimiter` can configure the kafka
source to have a rate limiter.
(I assume you uses Kafka)
However it only exists in Kafka 0.10 DataStream Connector, not in other
versions nor table api.
王雷 于2020年5月14日周四 下午5:31写道：
> hi, All
>
> Does Flink support rate limitati
hi, All
Does Flink support rate limitation?
How to limit the rate when the external database connected by the sink
operator has throughput limitation.
Instead of passive back pressure after reaching the limit of the external
database, we want to limit rate actively.
Thanks
Ray
