[
https://issues.apache.org/jira/browse/FLINK-12548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041991#comment-17041991
]
Leonid Ilyevsky commented on FLINK-12548:
-----------------------------------------
[~sjwiesman] Seth,
Yes, I meant multiple slots. Minor correction to what I said in my first post:
the client.id does not have to be unique, it may be the same for all those
instances in the slots, because Kafka server additionally generates CONSUMER-ID
by adding uuid string to the client.id.
You said "My biggest concern is reporting back information that might diverge
from what is in Flink". Actually, you already are reporting the most important
part - your last offset, the only two little pieces missing are client.id and
host.
This is not super critical, but it does break some tools, that for some reason
care about client.id. For example,
[https://github.com/lightbend/kafka-lag-exporter] does not report the lag of
the consumers running under Flink, and I believe it is because of missing
client.id and host. You may say, let them fix it - I agree, but I also believe
you should fix your side too.
Also, it would be useful to see client.id and host in kafka-consumer-groups
reports.
So basically this issue is about consumers monitoring.
Thanks,
Leonid
> FlinkKafkaConsumer issues configuring underlying KafkaConsumer
> --------------------------------------------------------------
>
> Key: FLINK-12548
> URL: https://issues.apache.org/jira/browse/FLINK-12548
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
> Affects Versions: 1.8.0
> Reporter: Leonid Ilyevsky
> Assignee: Hugo Da Cruz Louro
> Priority: Major
> Attachments: KafkaError.txt
>
>
> I observe two issues that are possibly related, or at least belong to the
> same part of the code.
> I am running multiple instances of pipelines (by setting parallelism) under
> Flink cluster.
> First problem I see is the "Error registering AppInfo mbean", see attached
> stack trace [^KafkaError.txt] . From the trace I understand that it tries to
> use consumer group id as the bean name; obviously this fails as I have
> multiple instances using the same group id under the same task manager.
> Second problem is, the client id is not set at all (I do provide the
> "client.id" property when invoking FlinkKafkaConsumer constructor). This
> creates problems tracking consumers on Kafka side. On the other hand, I
> obviously do not want to set the same client id on all instances - preferably
> Flink should add maybe some UUID suffix to it or something to make it unique.
> The same unique name could be used for that mbean, this way solving the first
> problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
