Hi Scott, It seems that your flush takes longer time than consumer.session.timeout.ms. The consumers used in SinkTasks for a SinkConnector are in the same consumer group. In case that your flush method takes longer than the consumer.session.timeout.ms, the consumer for a SinkTask may be kicked out by the coordinator.
In this case, you may want to increase the consumer.session.timeout.ms or have some timeout mechanism in the implementation of the flush method to return the control back to the framework so that it can send heartbeat to the coordinator. Thanks, Liquan On Fri, Apr 15, 2016 at 9:56 AM, Scott Reynolds <sreyno...@twilio.com> wrote: > List, > > We are struggling with Kafka Connect settings. The process start up and > handle a bunch of messages and flush. Then slowly the Group coordinator > removes them. > > This is has to be a interplay between Connect's flush interval and the call > to poll for each of these tasks. Here is my current settings that I think > are relevant. > > Any insights someone could share with us ? > > # on shutdown wait this long for the tasks to finish their flush. > task.shutdown.graceful.timeout.ms=600000 > > # Flush records to s3 every 1/2 hour > offset.flush.interval.ms=1800000 > > # Max time to wait for flushing to finish. Wait at *most* this long every > offset.flush.interval.ms. > offset.flush.timeout.ms=600000 > > # Take your time on session timeouts. We do a lot of work. These control > the length of time a lock on a TopicPartition can be held > # by the coordinator broker. > session.timeout.ms=180000 > request.timeout.ms=190000 > consumer.session.timeout.ms=180000 > consumer.request.timeout.ms=190000 > -- Liquan Pei Software Engineer, Confluent Inc
