On Wed, Jun 29, 2016 at 9:44 AM, Sumit Arora <[email protected] > wrote:
> Hello, > > We are currently building our data-pipeline using Confluent and as part of > this implementation, we have written couple of Kafka Connect Sink > Connectors for Azure and MS SQL server. To provide some more context, we > are planning to use SQL Server's Change Data Capture feature to track > changes happening on individual tables in the source database and then use > JDBC connector's "QUERY" mode to fetch the changed data and push it to > Kafka topics. We had to go down this route because we cannot guarantee the > presence of Identity or Timestamp columns in our source tables so we are > relying on the timestamp columns present in the system tables created by > Change Data Capture (trans_begin_time) and use it along with "QUERY" mode > in JDBC connector. However, this design requires us to run one JDBC > connector instance per table. > > On the consumer side, once the data is on Kafka, we have written an Azure > connector to push the data from Kafka topics onto Azure Blob Storage and > another one to push data to MS SQL server. > > Now coming to the actual question :) , the above design leads to a number > of Kafka Connect workers, connectors and tasks running on our distributed > cluster. We are planning to run multiple workers per host and have them > share the tasks from our 3 connectors (JDBC, Azure and SQL Server). How can > we monitor these connectors and tasks in a production environment? Currently your best bet is polling the REST API. There's a status API: http://docs.confluent.io/3.0.0/connect/userguide.html#get--connectors-(string-name)-status which will tell you the state of each task in a connector. (Note that this is an 0.10.0.0 feature.) There are also underlying producers & consumers which are named in a consistent way and have all the normal metrics available if you want to monitor via JMX as well. > How can we make sure that workers are restarted once they are killed? Connect is agnostic to the deployment strategy, so it doesn't get involved in process management at all. You should use whatever mechanism you normally use to monitor processes (supervisord, a cluster manager like Mesos or Kubernetes, etc). > How can we identify the failing tasks on different workers? The status API will give you this info. Are there any best practices on monitoring Kafka Connect and managing > workers/connectors/tasks? > Just the ones mentioned above. We'd like to improve the metrics available via JMX/MetricsReporter, but the underlying producer/consumer metrics are what's available for now. > We are close to finishing our development and deploying this to test and > production environments and it is very important that we figure out a way > to monitor this set-up. > > Thanks, > Sumit > P.S. We are looking at Command Central as our long-term monitoring > solution but need a solution for the short-term. > FYI, this actually works automatically if you configure things right -- see etc/schema-registry/connect-avro-*.properties for the two config settings that you should enable on your Connect workers so they report monitoring info that Control Center will use to track data delivery for Connect. > > > CONFIDENTIALITY NOTICE: This e-mail message including attachments, if any, > is intended for the person or entity to which it is addressed and may > contain confidential, privileged, and/or proprietary material. Any > unauthorized review, use, disclosure or distribution is prohibited. If you > are not the intended recipient, please contact the sender by reply e-mail > and destroy all copies of the original message. > -- Thanks, Ewen
