[
https://issues.apache.org/jira/browse/KAFKA-2480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794612#comment-16794612
]
Ewen Cheslack-Postava commented on KAFKA-2480:
----------------------------------------------
[~olkuznsmith] This doesn't seem to have anything to do with this JIRA, but the
behavior is intended. If you want the behavior you're describing, it's easy to
follow up after poll returns by sleeping as long as you need to in order to
block until you hit your business-logic-timeout. Or, if you want to continue
fetching, just wrap the poll calls in a loop and accumulate the results (but be
sure to have some cap, or else you're bound to OOM). Vast majority of users
would not want this as it just blocks process and limits throughput,
potentially to the point that you can't ever catch up. The interface is
designed to be low-level enough to implement any of these patterns, and it is a
common pattern (e.g., going way back to `select`).
> Handle non-CopycatExceptions from SinkTasks
> -------------------------------------------
>
> Key: KAFKA-2480
> URL: https://issues.apache.org/jira/browse/KAFKA-2480
> Project: Kafka
> Issue Type: Sub-task
> Components: KafkaConnect
> Reporter: Ewen Cheslack-Postava
> Assignee: Ewen Cheslack-Postava
> Priority: Major
> Fix For: 0.9.0.0
>
>
> Currently we catch Throwable in WorkerSinkTask, but we just log the
> exception. This can lead to data loss because it indicates the messages in
> the {{put(records)}} call probably were not handled properly. We need to
> decide what the policy for handling these types of exceptions should be --
> try repeating the same records again, risking duplication? or skip them,
> risking loss? or kill the task immediately and require intervention since
> it's unclear what happened?
> SourceTasks don't have the same concern -- they can throw other exceptions
> and as long as we catch them, it is up to the connector to ensure that it
> does not lose data as a result.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
