Shikhar Bhushan created KAFKA-4161:
--------------------------------------
Summary: Allow connectors to request flush via the context
Key: KAFKA-4161
URL: https://issues.apache.org/jira/browse/KAFKA-4161
Project: Kafka
Issue Type: New Feature
Components: KafkaConnect
Reporter: Shikhar Bhushan
Assignee: Ewen Cheslack-Postava
It is desirable to have, in addition to the time-based flush interval, volume
or size-based commits. E.g. a sink connector which is buffering in terms of
number of records may want to request a flush when the buffer is full, or when
sufficient amount of data has been buffered in a file.
Having a method like say {{requestFlush()}} on the {{SinkTaskContext}} would
allow for connectors to have flexible policies around flushes. This would be in
addition to the time interval based flushes that are controlled with
{{offset.flush.interval.ms}}, for which the clock should be reset when any kind
of flush happens.
We should probably also support requesting flushes via the
{{SourceTaskContext}} for consistency though a use-case doesn't come to mind
off the bat.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
