[
https://issues.apache.org/jira/browse/FLINK-37918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hajimeni updated FLINK-37918:
-----------------------------
Affects Version/s: aws-connector-6.0.1
aws-connector-5.1.0
> Restore the ability to set an interval for GetRecords calls to Kinesis shards.
> ------------------------------------------------------------------------------
>
> Key: FLINK-37918
> URL: https://issues.apache.org/jira/browse/FLINK-37918
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kinesis
> Affects Versions: aws-connector-5.0.0, aws-connector-5.1.0,
> aws-connector-6.0.1
> Reporter: hajimeni
> Priority: Major
> Labels: pull-request-available
>
> h3. Background
> The previous Flink Kinesis connector (flink-connector-kinesis) provided a
> configuration parameter, SHARD_GETRECORDS_INTERVAL_MILLIS, which allowed
> users to set a specific interval between GetRecords calls for each shard.
> This functionality is absent in the new AWS Kinesis Streams connector
> (flink-connector-aws-kinesis-streams).
> h3. Problem
>
> The lack of a configurable interval for GetRecords calls in the new connector
> (KinesisStreamsSource) poses a significant challenge in scenarios with
> multiple consumers reading from the same Kinesis stream. Without the ability
> to increase the interval between GetRecords calls, consumers can easily
> exceed the AWS Kinesis limit of five GetRecords calls per second per shard.
> This leads to several issues: * Wasted API Calls and Increased Costs:
> Continuous, rapid calls that are likely to be throttled are inefficient and
> can lead to increased costs.
> * Operational Instability: In a multi-tenant or multi-application
> environment, the absence of this control makes it difficult to ensure stable
> and predictable data consumption across all consumers.
> * AWS documentation recommends adjusting the frequency of GetRecords calls
> to avoid these issues, especially when multiple consumers are involved. You
> can find this recommendation in the AWS Kinesis Developer Guide. (see:
> https://docs.aws.amazon.com/streams/latest/dev/kinesis-low-latency.html )
> h3. Feature Request
> We request the re-introduction of a configuration option, similar to
> SHARD_GETRECORDS_INTERVAL_MILLIS, in the flink-connector-aws-kinesis-streams
> connector. This would allow users to effectively manage the rate of
> GetRecords calls per shard, thereby preventing API throttling and ensuring
> the stability and efficiency of Flink applications that consume data from
> Kinesis Data Streams.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
