Hi Kafka (Connect) 0.10.2.1
I am writing my own SourceConnector. It will communicate with a remote server, and continuously calculate the set of tasks that has to be running. Each task also makes a connection to the remote server from which it will get its data to forward to Kafka.
When the SourceConnector realizes that the set of tasks has to be modified, it makes sure taskConfigs-method will return config for the new complete set of tasks (likely including tasks that already existed before, probably some new tasks, and maybe some of the existing tasks will no longer be included). After that the SourceConnector calls context.requestTaskReconfiguration. This results in the current instance of my SourceConnector and all existing/running tasks gets stopped, a new instance of my SourceConnector gets created and all tasks (those that existed before and new ones) are started.
It all works nicely, but because my SourceConnector and my SourceTasks has to (re)establish connection and (re)initialize the streaming of data, and because my set of tasks changes fairly often, and because it very very often contains tasks that were also in the set before the change, I end up having lots of stop/start of tasks that really just ought to continue running.
Any plans on making this more delta-ish, so that when doing a requestTaskReconfiguration * Only tasks that were not already in the task-config-set before the requestTaskConfiguration are started * Only tasks that were in the task-config-set before the requestTaskConfiguration, but not in the set after, are stopped * Tasks that are in the task-config-set both before and after requestTaskConfiguration, are just allowed to keep running, without restarting * Not so important: Do not create a new instance of the SourceConnector, just because it has a changed task-config-set
Or am I doing something wrong in my SourceConnector? Are there a different way that I should maintain a dynamic set of tasks?
Thanks!!! Regards, Per Steffensen
