C0urante commented on PR #16001: URL: https://github.com/apache/kafka/pull/16001#issuecomment-2125789945
After some offline discussion with @gharris1727 we realized there was a potential for infinite rebalance loops with the previous implementation when config providers resolved values differently depending on the worker on which they were running. Assume some non-leader worker `W` hosts some connector `C`: 1. `W` computes the hash of a config-provider-resolved connector config 2. After observing that the hash differs from latest hash present in the config topic for that connector, `W` forwards a request to the leader to publish task configs for `C` to the config topic 4. The leader accepts this request, performs its own config provider resolution on the connector config, and writes a hash of that config-provider-resolved config to the config topic 5. After the ensuing rebalance, `W` sees a different hash in the config topic than the one that it computed in step 1, and the process repeats In order to address this, I've published a commit that causes non-leader workers to send their own connector config hash to the leader as an HTTP request header when forwarding task configs to the leader. I've also introduced a dedicated `ConfigHash` class that should make it significantly harder for config hash values to be leaked, including via log message. This class does not directly expose the underlying 32-bit integer and instead only permits indirect access to the value. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
