[
https://issues.apache.org/jira/browse/KAFKA-13329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17421507#comment-17421507
]
Chris Egerton edited comment on KAFKA-13329 at 10/1/21, 4:02 PM:
-----------------------------------------------------------------
I agree that that'd be beneficial, although I think it'd require a KIP.
Thinking about Confluent's Avro converter, there are some preflight validations
that would be very useful but would require access to multiple properties at a
time (such as ensuring that the URL and credentials for Schema Registry are
valid), which is beyond the single-property-at-a-time scope of a {{ConfigDef}}
object.
If we're at the point of writing a KIP for this already, I wonder if we might
follow the same pattern for key/value converters as we do for connectors: a
[validate
method|https://github.com/apache/kafka/blob/5a5c05807ddc82b183f4489e81b9ad02fe83a0df/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L131-L146]
that, by default, simply delegates to a {{ConfigDef}} provided in a [config
method|https://github.com/apache/kafka/blob/5a5c05807ddc82b183f4489e81b9ad02fe83a0df/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L148-L152].
Obviously we'd have to make some small tweaks (default implementation for
{{config}} that returns {{null}}, permitting null {{ConfigDef}} objects in
{{validate}}), but overall it's been a very friendly API to use when writing
connectors (just took advantage of it in [a recent
PR|https://github.com/confluentinc/kafka-connect-bigquery/pull/153], in fact)
and I don't see why we wouldn't want to extend key converters, value
converters, and possibly even header converters, transformations, and
predicates in the same way.
was (Author: chrisegerton):
I agree that that'd be beneficial, although I think it'd require a KIP.
Thinking about Confluent's Avro converter, there are some useful preflight
validations that would be very useful but would require access to multiple
properties at a time (such as ensuring that the URL and credentials for Schema
Registry are valid), which is beyond the single-property-at-a-time scope of a
{{ConfigDef}} object.
If we're at the point of writing a KIP for this already, I wonder if we might
follow the same pattern for key/value converters as we do for connectors: a
[validate
method|https://github.com/apache/kafka/blob/5a5c05807ddc82b183f4489e81b9ad02fe83a0df/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L131-L146]
that, by default, simply delegates to a {{ConfigDef}} provided in a [config
method|https://github.com/apache/kafka/blob/5a5c05807ddc82b183f4489e81b9ad02fe83a0df/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L148-L152].
Obviously we'd have to make some small tweaks (default implementation for
{{config}} that returns {{null}}, permitting null {{ConfigDef}} objects in
{{validate}}), but overall it's been a very friendly API to use when writing
connectors (just took advantage of it in [a recent
PR|https://github.com/confluentinc/kafka-connect-bigquery/pull/153], in fact)
and I don't see why we wouldn't want to extend key converters, value
converters, and possibly even header converters, transformations, and
predicates in the same way.
> Connect does not perform preflight validation for per-connector key and value
> converters
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-13329
> URL: https://issues.apache.org/jira/browse/KAFKA-13329
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
> Reporter: Chris Egerton
> Assignee: Chris Egerton
> Priority: Major
>
> Users may specify a key and/or value converter class for their connector
> directly in the configuration for that connector. If this occurs, no
> preflight validation is performed to ensure that the specified converter is
> valid.
> Unfortunately, the [Converter
> interface|https://github.com/apache/kafka/blob/4eb386f6e060e12e1940c0d780987e3a7c438d74/connect/api/src/main/java/org/apache/kafka/connect/storage/Converter.java]
> does not require converters to expose a {{ConfigDef}} (unlike the
> [HeaderConverter
> interface|https://github.com/apache/kafka/blob/4eb386f6e060e12e1940c0d780987e3a7c438d74/connect/api/src/main/java/org/apache/kafka/connect/storage/HeaderConverter.java#L48-L52],
> which does have that requirement), so it's unlikely that the configuration
> properties of the converter itself can be validated.
> However, we can and should still validate that the converter class exists,
> can be instantiated (i.e., has a public, no-args constructor and is a
> concrete, non-abstract class), and implements the {{Converter}} interface.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
