Ewen Cheslack-Postava created KAFKA-5061:
--------------------------------------------
Summary: client.id should be set for Connect producers/consumers
Key: KAFKA-5061
URL: https://issues.apache.org/jira/browse/KAFKA-5061
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Affects Versions: 0.10.2.1
Reporter: Ewen Cheslack-Postava
In order to properly monitor individual tasks using the producer and consumer
metrics, we need to have the framework disambiguate them. Currently when we
create producers
(https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java#L362)
and create consumers
(https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L371-L394)
the client ID is not being set. You can override it for the entire worker via
worker-level producer/consumer overrides, but you can't get per-task metrics.
There are a couple of things we might want to consider doing here:
1. Provide default client IDs based on the worker group ID + task ID (providing
uniqueness for multiple connect clusters up to the scope of the Kafka cluster
they are operating on). This seems ideal since it's a good default; however it
is a public-facing change and may need a KIP. Normally I would be less worried
about this, but some folks may be relying on picking up metrics without this
being set, in which case such a change would break their monitoring.
2. Allow overriding client.id on a per-connector basis. I'm not sure if this
will really be useful or not -- it lets you differentiate between metrics for
different connectors' tasks, but within a connector, all metrics would go to a
single client.id. On the other hand, this makes the tasks act as a single group
from the perspective of broker handling of client IDs.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
