GitHub user rhauch opened a pull request:
https://github.com/apache/kafka/pull/3641
KAFKA-5704 Corrected Connect distributed startup behavior to allow older
brokers to auto-create topics
When a Connect distributed worker starts up talking with broker versions
0.10.1.0 and later, it will use the AdminClient to look for the internal topics
and attempt to create them if they are missing. Although the AdminClient was
added in 0.11.0.0, the AdminClient uses APIs to create topics that existed in
0.10.1.0 and later. This feature works as expected when Connect uses a broker
version 0.10.1.0 or later.
However, when a Connect distributed worker starts up using a broker older
than 0.10.1.0, the AdminClient is not able to find the required APIs and thus
will throw an UnsupportedVersionException. Unfortunately, this exception is not
caught and instead causes the Connect worker to fail even when the topics
already exist.
This change handles the UnsupportedVersionException by logging a debug
message and doing nothing. The existing producer logic will get information
about the topics, which will cause the broker to create them if they donât
exist and broker auto-creation of topics is enabled. This is the same behavior
that existed prior to 0.11.0.0, and so this change restores that behavior for
brokers older than 0.10.1.0.
This change also adds a system test that verifies Connect works with a
variety of brokers and is able to run source and sink connectors. The test
verifies that Connect can read from the internal topics when the connectors are
restarted.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rhauch/kafka kafka-5704
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/3641.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3641
----
commit 0d45fd113eeaff3844742181191db2cc508353fd
Author: Randall Hauch <[email protected]>
Date: 2017-08-07T19:32:29Z
KAFKA-5704 Corrected Connect distributed startup behavior to allow older
brokers to auto-create topics
When a Connect distributed worker starts up talking with broker versions
0.10.1.0 and later, it will use the AdminClient to look for the internal topics
and attempt to create them if they are missing. Although the AdminClient was
added in 0.11.0.0, the AdminClient uses APIs to create topics that existed in
0.10.1.0 and later. This feature works as expected when Connect uses a broker
version 0.10.1.0 or later.
However, when a Connect distributed worker starts up using a broker older
than 0.10.1.0, the AdminClient is not able to find the required APIs and thus
will throw an UnsupportedVersionException. Unfortunately, this exception is not
caught and instead causes the Connect worker to fail even when the topics
already exist.
This change handles the UnsupportedVersionException by logging a debug
message and doing nothing. The existing producer logic will get information
about the topics, which will cause the broker to create them if they donât
exist and broker auto-creation of topics is enabled. This is the same behavior
that existed prior to 0.11.0.0, and so this change restores that behavior for
brokers older than 0.10.1.0.
This change also adds a system test that verifies Connect works with a
variety of brokers and is able to run source and sink connectors. The test
verifies that Connect can read from the internal topics when the connectors are
restarted.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
