Thanks Jamie. This is helpful. Do we need to start the job again if cluster is restarted due to maintenance or upgrades ?
Sent from my iPhone > On Oct 24, 2019, at 11:58 AM, Jamie <jamied...@aol.co.uk.invalid> wrote: > > Hi Asmath > When you create a connector the request is forwarded onto the rest api of the > worker which is the leader, it uses a Kafka producer to update the topic > which stores your connector config. The other workers will continuously > monitor this config topic and know if it changes then then they need to > perform a rebalance since a new connector has been added. When your workers > start they will read the config from this toopic and keep it in memory so > they know the current configuration of the connectors. This topic and the > other internal topics used for Kafka connect should use compaction so that > they aren't removed after the retention time, the most recent version of each > connectors config remains. > For alerting specifically on connector status you could use the REST API to > get the status of each connector and it's tasks at regular intervals and > alert if the status is failed on any (or define threshold of an acceptable > number of failures ) > Hope this helps, > Jamie > > > -----Original Message----- > From: KhajaAsmath Mohammed <mdkhajaasm...@gmail.com> > To: users <users@kafka.apache.org> > Sent: Thu, Oct 24, 2019 03:52 PM > Subject: Monitor Kafka connect jobs > > > Hi, > > > We are using kafka connect in production and I have few questions about it. > when we submit kafka connect job using rest api . Job gets continously > running in the background and due to some issues, lets say we restarted > kafka cluster. do we need to start manually all the jobs again? > > > Is there a way to monitor these jobs using tools. I know we can use connect > UI but in case if we have more than 1000 jobs it would become more complex. > > > I am also looking for trigger mechanism to send out email or alert support > team if the connector was killed due to some reasons. > > > Thanks, > > Asmath
