Re: Kafka Connect connector monitoring tasks via offset storage reader

2016-05-24 Thread Randall Hauch
On May 24, 2016 at 4:28:41 PM, Liquan Pei (liquan...@gmail.com) wrote: Hi Randall,  This is interesting. Essentially we can track the progress by getting data  from offset storage. What are the use cases you have in mind that uses the  offsets of source partitions? I can imagine that comparing

Re: Kafka Connect connector monitoring tasks via offset storage reader

2016-05-24 Thread Liquan Pei
Hi Randall, This is interesting. Essentially we can track the progress by getting data from offset storage. What are the use cases you have in mind that uses the offsets of source partitions? I can imagine that comparing the source offsets for the new data and already delivered data and make some

Kafka Connect connector monitoring tasks via offset storage reader

2016-05-24 Thread Randall Hauch
I have a need for one of my SourceConnector implementations to configure a bunch of tasks and, when those are all “done”, request a task reconfiguration so that it can run a single task. Think: many tasks to make snapshot of database tables, then when those are completed reconfigure itself so th