[
https://issues.apache.org/jira/browse/KAFKA-13196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jozef Vilcek updated KAFKA-13196:
---------------------------------
Description:
I am using MirrorMaker 2.0 and running it via [
MirrorMaker.java|https://github.com/apache/kafka/blob/2.7/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorMaker.java]
class. This method will start up `DistributedHerder` and will use
non-functional `advertisedUrl`, and therefore workers can not talk to each
other and coordinate.
After upgrading from version `2.4.0` to `2.7.1` I did notice that when I am
starting up mirror maker ti does not always start tasks - just connector is
executing. Doing some amount of stop and start it will eventually start tasks
too.
After a bit of digging I did noticed that in attempt to configure connector's
task, code ends up in this [1] branch, where configure request is being
forwarded to the leader. From some reason, task configuration is not done on
leader. However, MirrorMaker does not pack RestServer and therefore that
request will never succeed.
I am not sure what is going no or why it does seem to work better on 2.4.0. I
noticed that connector start procedure did involve less callbacks on 2.4.0 in
connector start sequence [2].
Does this look like a bug? Or am I doing something wrong?
[1]
[https://github.com/apache/kafka/blob/2.7/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1494]
[2]
[https://github.com/apache/kafka/blob/2.4/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1236]
was:
I am using MirrorMaker 2.0 and running it via [
MirrorMaker.java|https://github.com/apache/kafka/blob/2.7/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorMaker.java]
class. This method will start up `DistributedHerder` and will use
non-functional `advertisedUrl`, and therefore workers can not talk to each
other and coordinate.
After upgrading from version `2.4.0` to `2.7.1` I did notice that when I am
starting up mirror maker ti does not always start tasks - just connector is
executing. Doing some amount of stop and start it will eventually start tasks
too.
After a bit of digging I did noticed that in attempt to configure connector's
task, code ends up in this [1] branch, where configure request is being
forwarded to the leader. From some reason, task configuration is not done on
leader. However, MirrorMaker does not pack RestServer and therefore that
request will never succeed.
I am not sure what is going no or why it does seem to work better on 2.4.0. I
noticed that connector start procedure did involve less callback on 2.4.0 in
connector start sequence [2].
I believe the root cause is change in startup procedure of connectors in
general. In vestion 2.4, when connector is started by the leader here [1] it
will immediately setup and submit configuration for connector's tasks. However,
in 2.7.1, is it more asynchronous. Connector is started here [2]
Does this look like a bug? Or am I doing something wrong?
[1]
https://github.com/apache/kafka/blob/2.7/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1494
[2]
https://github.com/apache/kafka/blob/2.4/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1236
> MirrorMaker 2 not always start tasks
> ------------------------------------
>
> Key: KAFKA-13196
> URL: https://issues.apache.org/jira/browse/KAFKA-13196
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
> Affects Versions: 2.7.1
> Reporter: Jozef Vilcek
> Priority: Major
>
> I am using MirrorMaker 2.0 and running it via [
> MirrorMaker.java|https://github.com/apache/kafka/blob/2.7/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorMaker.java]
> class. This method will start up `DistributedHerder` and will use
> non-functional `advertisedUrl`, and therefore workers can not talk to each
> other and coordinate.
> After upgrading from version `2.4.0` to `2.7.1` I did notice that when I am
> starting up mirror maker ti does not always start tasks - just connector is
> executing. Doing some amount of stop and start it will eventually start tasks
> too.
> After a bit of digging I did noticed that in attempt to configure connector's
> task, code ends up in this [1] branch, where configure request is being
> forwarded to the leader. From some reason, task configuration is not done on
> leader. However, MirrorMaker does not pack RestServer and therefore that
> request will never succeed.
> I am not sure what is going no or why it does seem to work better on 2.4.0. I
> noticed that connector start procedure did involve less callbacks on 2.4.0 in
> connector start sequence [2].
>
> Does this look like a bug? Or am I doing something wrong?
>
> [1]
> [https://github.com/apache/kafka/blob/2.7/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1494]
> [2]
> [https://github.com/apache/kafka/blob/2.4/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1236]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
