[
https://issues.apache.org/jira/browse/KAFKA-7509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673834#comment-16673834
]
Randall Hauch commented on KAFKA-7509:
--------------------------------------
Since changing the unused property log messages to debug was not the correct
change, I closed that PR and took yet another approach that just filters out
worker properties before passing them to the producer, consumer, and admin
clients. Previously, Connect would pass many of its worker configuration
properties into the producer, consumer, and admin client used for internal
topics, resulting in lots of log warnings about unused config properties.
With [this change|https://github.com/apache/kafka/pull/5876], Connect now
attempts to filter out the worker’s configuration properties that are not
applicable to the producer, consumer, or admin client used for internal topics.
(Connect is already including only producer and consumer properties when
creating those clients for connectors, since those properties are prefixed in
the worker config.)
For the most part, this is relatively straightforward, since there are some
top-level worker-specific properties that can be removed, and most
extension-specific properties have Connect-specific properties. Unfortunately,
the REST extension is the only type of connect extension that uses unprefixed
properties from the worker config, so any it is not possible to remove those
from the properties passed to the producer, consumer, and admin clients.
Hopefully, REST extensions are prevalant yet, and this will mean most users may
not see any warnings about unused properties in the producer, consumer, and
admin client.
Removing the Connect worker configs is one step. The other is for the producer
to remove the any properties that are specific to the consumer and admin
client. Likewise, for the consumer we have to remove any properties that are
specific to the producer and admin client, and for the admin client remove any
properties that are specific to the producer and consumer. Note that any
property that is unknown (e.g., properties for REST extension, interceptors,
metric reporters, serdes, partitioners, etc.) must be passed to the producer,
consumer, and admin client. All of these — except for the REST extension
properties — should indeed be used by the producer and consumer. But, since the
admin client only supports metric reporters, any properties for interceptors,
serdes, partitioners and REST extension will also be logged as unused. This is
about the best we can do at this point.
All of this filtering logic was added to the ConnectUtils class, allowing the
logic to be easily unit tested. All changes are limited to Kafka Connect, and
will work with all client and Connect extensions (passing them to the clients
if they are unknown).
This still does not go as far as I'd like. It'd be ideal if Connect could use a
RecordingMap to know which properties were used across _all_ of the clients and
the worker itself, and to then log _only those_ properties that were never used
at all. This is beyond the scope of this issue, however, and can be addressed
in a follow-on issue that will require changes to AbstractConfig.
> Kafka Connect logs unnecessary warnings about unused configurations
> -------------------------------------------------------------------
>
> Key: KAFKA-7509
> URL: https://issues.apache.org/jira/browse/KAFKA-7509
> Project: Kafka
> Issue Type: Improvement
> Components: clients, KafkaConnect
> Affects Versions: 0.10.2.0
> Reporter: Randall Hauch
> Assignee: Randall Hauch
> Priority: Major
>
> When running Connect, the logs contain quite a few warnings about "The
> configuration '{}' was supplied but isn't a known config." This occurs when
> Connect creates producers, consumers, and admin clients, because the
> AbstractConfig is logging unused configuration properties upon construction.
> It's complicated by the fact that the Producer, Consumer, and AdminClient all
> create their own AbstractConfig instances within the constructor, so we can't
> even call its {{ignore(String key)}} method.
> See also KAFKA-6793 for a similar issue with Streams.
> There are no arguments in the Producer, Consumer, or AdminClient constructors
> to control whether the configs log these warnings, so a simpler workaround
> is to only pass those configuration properties to the Producer, Consumer, and
> AdminClient that the ProducerConfig, ConsumerConfig, and AdminClientConfig
> configdefs know about.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
