[
https://issues.apache.org/jira/browse/FLINK-9897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16562745#comment-16562745
]
ASF GitHub Bot commented on FLINK-9897:
---------------------------------------
glaksh100 edited a comment on issue #6408: [FLINK-9897][Kinesis Connector] Make
adaptive reads depend on run loop time instead of fetchintervalmillis
URL: https://github.com/apache/flink/pull/6408#issuecomment-409059048
@tzulitai Thanks for reviewing and apologies for the delay. I have addressed
all the comments.
I verified the most recent changes with a Flink job reading from a
back-logged Kinesis stream and writing to a no-op sink.
- Number of Kinesis shards - 512
- GetRecord I/O achieved ~= 1.04 Gb/sec
-
[numberOfAggregatedRecordsPerFetch](https://github.com/apache/flink/pull/6409)
~= `maxNumberOfRecordsPerFetch`
Let me know if it looks alright!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Further enhance adaptive reads in Kinesis Connector to depend on run loop time
> ------------------------------------------------------------------------------
>
> Key: FLINK-9897
> URL: https://issues.apache.org/jira/browse/FLINK-9897
> Project: Flink
> Issue Type: Improvement
> Components: Kinesis Connector
> Affects Versions: 1.4.2, 1.5.1
> Reporter: Lakshmi Rao
> Assignee: Lakshmi Rao
> Priority: Major
> Labels: pull-request-available
>
> In FLINK-9692, we introduced the ability for the shardConsumer to adaptively
> read more records based on the current average record size to optimize the 2
> Mb/sec shard limit. The feature maximizes maxNumberOfRecordsPerFetch of 5
> reads/sec (as prescribed by Kinesis limits). In the case where applications
> take more time to process records in the run loop, they are no longer able to
> read at a frequency of 5 reads/sec (even though their fetchIntervalMillis
> maybe set to 200 ms). In such a scenario, the maxNumberOfRecordsPerFetch
> should be calculated based on the time that the run loop actually takes as
> opposed to fetchIntervalMillis.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
