I am running a connect consumer that receives JSON records and indexes into elasticsearch. The consumer is pushing out 300 messages/s into the a topic with a single partition. The connect job is configured with 1 task. (This is all for testing).
What I see is that push is called about every 10s with about 1500 records. It takes about 1.5 seconds of wall time to complete the indexing of those records into elasticsearch. But then the task waits another 10s for the next batch from kafka connect. Is there some kind of consumer throttling happening? I cannot find any settings that would tell connect to deliver messages faster or in larger batches. I can of course run with more partitions and more tasks, but still, kafka connect should be able to deliver messages to the task orders of magnitude faster than elasticsearch can index them.
