[
https://issues.apache.org/jira/browse/KAFKA-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Randall Hauch reassigned KAFKA-10600:
-------------------------------------
Fix Version/s: 2.6.1
2.5.2
2.7.0
Reviewer: Konstantine Karantasis
Assignee: Randall Hauch
Resolution: Fixed
Connect should not always add an error to configuration values in validation
results that don't have a `ConfigKey` defined in the connector's `ConfigDef`,
and any errors on such configuration values included by the connector should be
counted in the total number of errors. Added more unit tests for
`AbstractHerder.generateResult(...)`.
> Connect adds error to property in validation result if connector does not
> define the property
> ---------------------------------------------------------------------------------------------
>
> Key: KAFKA-10600
> URL: https://issues.apache.org/jira/browse/KAFKA-10600
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
> Affects Versions: 0.10.0.0, 0.11.0.0, 1.0.0, 1.1.0, 2.0.0, 2.1.0, 2.2.0,
> 2.3.0, 2.4.0, 2.5.0, 2.6.0
> Reporter: Randall Hauch
> Assignee: Randall Hauch
> Priority: Major
> Fix For: 2.7.0, 2.5.2, 2.6.1
>
>
> Kafka Connect's {{AbstractHerder.generateResult(...)}} method is responsible
> for taking the result of a {{Connector.validate(...)}} call and constructing
> the {{ConfigInfos}} object that is then mapped to the JSON representation.
> As this method (see
> [code|https://github.com/apache/kafka/blob/1f8ac6e6fee3aa404fc1a4c01ac2e0c48429a306/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractHerder.java#L504-L507])
> iterates over the {{ConfigKey}} objects in the connector's {{ConfigDef}} and
> the {{ConfigValue}} objects returned by the {{Connector.validate(...)}}
> method, this method adds an error message to any {{ConfigValue}} whose
> {{configValue.name()}} does not correspond to a {{ConfigKey}} in the
> connector's {{ConfigDef}}.
> {code}
> if (!configKeys.containsKey(configName)) {
> configValue.addErrorMessage("Configuration is not defined: "
> + configName);
> configInfoList.add(new ConfigInfo(null,
> convertConfigValue(configValue, null)));
> }
> {code}
> Interestingly, these errors are not included in the total error count of the
> response. Is that intentional??
> This behavior does not allow connectors to report validation errors against
> extra properties not defined in the connector's {{ConfigDef}}.
> Consider a connector that allows arbitrary properties with some prefix (e.g.,
> {{connection.*}}) to be included and used in the connector properties. One
> example is to supply additional properties to a JDBC connection, where the
> connector may not be able to know these "additional properties" in advance
> because the connector either works with multiple JDBC drivers or the
> connection properties allowed by a JDBC driver are many and/or vary over
> different JDBC driver versions or server versions.
> Such "additional properties" are not prohibited by Connect API, yet if a
> connector implementation chooses to include any such additional properties in
> the {{Connector.validate(...)}} result (whether or not the corresponding
> {{ConfigValue}} has an error) then Connect will always add the following
> error to that property.
> {quote}
> Configuration is not defined: <additionalPropertyName>
> {quote}
> This code was in the 0.10.0.0 release of Kafka via the
> [PR|https://github.com/apache/kafka/pull/964] for KAFKA-3315, which is one of
> the tasks that implemented
> [KIP-26|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=58851767]
> for Kafka Connect (approved and partially added in 0.9.0.0). There is no
> mention of "validation" in KIP-26 nor any followup KIP (that I can find).
> I can kind of imagine the original thought process: any user-supplied
> property that is not defined by a {{ConfigDef}} is inherently an error.
> However, this assumption is not matched by any mention in the Connect API,
> documentation, or one of Connect's KIP.
> IMO, this is a bug in the {{AbstractHerder}} that over-constrains the
> connector properties to be only those defined in the connector's
> {{ConfigDef}}.
> Quite a few connectors already support additional properties, and it's
> perhaps only by chance that this happens to work:
> * If a connector does not override {{Connector.validate(...)}}, extra
> properties are not validated and therefore are not included in the resulting
> {{Config}} response with one {{ConfigValue}} per property defined in the
> connector's {{ConfigDef}}.
> * If a connector does override {{Connector.validate(...)}} and includes in
> the {{Config}} response a {{ConfigValue}} for the any additional properties,
> the {{AbstractHerder.generateResults(...)}} method does add the error but
> does not include this error in the error count, which is actually used to
> determine if there are any validation problems before starting/updating the
> connector.
> I propose that the {{AbstractHerder.generateResult(...)}} method be changed
> to not add any additional error message to the validation result, and to
> properly handle all {{ConfigValue}} objects regardless of whether there is a
> corresponding {{ConfigKey}} in the connector's {{ConfigDef}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
