Re: Kafka Connect: To much restarting with a SourceConnector with dynamic set of tasks

2017-05-23 Thread Per Steffensen
Thanks a lot for responding, Randall! See my comments below. Regards, Per Steffensen On 22/05/17 22:36, Randall Hauch wrote: You're not doing anything wrong, but I suspect you're requesting task reconfiguration more frequently than was originally envisioned, which means that the current implem

Re: Kafka Connect: To much restarting with a SourceConnector with dynamic set of tasks

2017-05-22 Thread Randall Hauch
You're not doing anything wrong, but I suspect you're requesting task reconfiguration more frequently than was originally envisioned, which means that the current implementation is not as optimal for your case. I'm not sure how much effort is required to implement this new behavior. The logic for

Kafka Connect: To much restarting with a SourceConnector with dynamic set of tasks

2017-05-16 Thread Per Steffensen
Hi Kafka (Connect) 0.10.2.1 I am writing my own SourceConnector. It will communicate with a remote server, and continuously calculate the set of tasks that has to be running. Each task also makes a connection to the remote server from which it will get its data to forward to Kafka. When the