List, We are struggling with Kafka Connect settings. The process start up and handle a bunch of messages and flush. Then slowly the Group coordinator removes them.
This is has to be a interplay between Connect's flush interval and the call to poll for each of these tasks. Here is my current settings that I think are relevant. Any insights someone could share with us ? # on shutdown wait this long for the tasks to finish their flush. task.shutdown.graceful.timeout.ms=600000 # Flush records to s3 every 1/2 hour offset.flush.interval.ms=1800000 # Max time to wait for flushing to finish. Wait at *most* this long every offset.flush.interval.ms. offset.flush.timeout.ms=600000 # Take your time on session timeouts. We do a lot of work. These control the length of time a lock on a TopicPartition can be held # by the coordinator broker. session.timeout.ms=180000 request.timeout.ms=190000 consumer.session.timeout.ms=180000 consumer.request.timeout.ms=190000
