Hi, we're investigating an issue where occasionally config changes don't
propagate to connectors/tasks.
When this occurs, the only way to ensure that the configuration takes effect is
to resize the number of tasks back down to 1 and then resize back up to the
original number of tasks.
In search
once when generating task configurations.
>
> I hope this gives some more insight to the behavior you're seeing.
>
> Thanks,
> Greg Harris
>
> On Fri, Feb 3, 2023 at 7:36 AM Frank Grimes
> wrote:
>
>> Hi, we're investigating an issue where occasionally config
Hi Greg,
I actually just found the following comment on this PR for
https://issues.apache.org/jira/browse/KAFKA-13809:
https://github.com/apache/kafka/pull/12450
> we get the same behavior (KAFKA-9228 notwithstanding) by passing the original
>properties through to tasks transparently
It seems
So I've been looking into the codebase to familiarize myself with it.I'm
operating on the assumption that the connectors in question get stuck in an
inconsistent state which causes them to prune the new task configs from those
which are "broadcast" to the workers.I see on
KafkaConfigBackingSto
Another thought... if an API exists to list all connectors in such a state,
then at least some monitoring/alerting could be put in place, right?
I'm still having trouble understanding how the configs could match in the code
you highlighted when we change connector and/or task config values when no keys
are being pruned by the connector implementations in question.Would capturing a
new generation value within the config itself on every s
I don't think forcing the API users to introduce the nonce is desirable.For us,
it would mean reaching out to the Strimzi project to try to get that
implemented on their side, which I would imagine would be a proposal which
would meet some resistance.
Have you had any ideas on how this can be im
So we've just hit this issue again just with the MM2 connector and trying to
add a new mirrored topic.We're running MirrorMaker 2 in Strimzi. i.e.
"connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector"We
have 6 worker nodes.We changed the config to add a new mirror topic. i.e
Ah, it definitely seems like KIP-710 will address the issue we've been bitten
by most.We'll eagerly await the kafka-3.5.0 release and then see if enabling
'dedicated.mode.enable.internal.rest' is possible with Strimzi.
Thanks for the help and patience! :-)
