GitHub user jgrier opened a pull request:
https://github.com/apache/flink/pull/6290
[Flink-9691] [Kinesis Connector] Attempt to call getRecords() at correct
frequency.
## What is the purpose of the change
The purpose of this change is to make the Kinesis connector call
getRecords() on the interval specified by the SHARD_GETRECORDS_INTERVAL_MILLIS
configuration parameter. Without this change the Kinesis Connector sleeps()
for the given interval regardless of how long it takes to process the records.
This hurts badly when trying read through a backlog of data for example. With
this change we instead time the loop and sleep() for the specified interval
MINUS any time spent processing. This is more in line with what user's expect
and makes the connector perform much better when there is a backlog of data to
read through.
## Brief change log
Time the run loop and subtract that time from the time spent sleeping.
## Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/lyft/flink flink-9691-fix-runloop
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6290.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6290
----
commit 93028c19707776bc6af636e308f9b4763e2a727b
Author: Jamie Grier <jgrier@...>
Date: 2018-07-09T21:20:47Z
Modify runloop to try to track a particular getRecords() frequency.
commit 580c4001bc60cf4d92815047697a47bc6de96cf8
Author: Jamie Grier <jgrier@...>
Date: 2018-07-09T21:41:59Z
Fix indentation.
----
---
