Jurgis Pods created KAFKA-9744:
----------------------------------
Summary: SchemaProjector fails to handle backwards-compatible
schema change
Key: KAFKA-9744
URL: https://issues.apache.org/jira/browse/KAFKA-9744
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Affects Versions: 2.3.1
Reporter: Jurgis Pods
_Note_: This bug report is for CP 5.3.1 / Kafka 2.3.1, but it most likely
affects all versions.
We recently made a number of backwards-compatible changes to our Avro schemas
in the Confluent Schema Registry. Those changes were accepted as
backwards-compatible. However, when redeploying Kafka S3 connectors consuming
the relevant topics, we noticed two separate instances of failures in
SchemaProjector.project(), causing the connectors to crash und stop producing
data:
1) Changed namespace of record:
{code:java}
org.apache.kafka.connect.errors.SchemaProjectorException: Schema name mismatch.
source name: my.example.Record and target name: my.example.sub.Record {code}
A change of a record's namespace is compatible according to the Schema
Registry, but not for the Connect API. I would argue that the namespace/package
name should not affect compatibility, as it says nothing about the contained
data and its schema.
2) Change of type from 1-element union to primitive field:
{code:java}
Schema type mismatch. source type: STRUCT and target type: STRING {code}
This happened when changing the corresponding field's Avro schema from
{code:java}
name": "myfield", "type": ["string"] {code}
to
{code:java}
name": "myfield", "type": "string"{code}
In this case, I am less convinced that those two schemas should be compatible
(they are semantically identical - however, a Union is not a String). But it is
unfortunate that the Schema Registry sees the above change as compatible, while
the Connect API does not.
*Summary*:
We made two Avro schema changes which were accepted as compatible by the Schema
Registry, but were rejected at runtime by the Kafka S3 connectors. Would it be
possible to have a more consistent (and less restrictive) check in Connect API,
so that a schema change in the producer can be made more confidently, without
fear of breaking the consuming connectors?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
