This was fixed in https://github.com/apache/kafka/pull/18146

On Tue, Nov 12, 2024 at 4:41 PM Thomas Thornton <tthorn...@salesforce.com>
wrote:

> Hi we notice data loss i.e. dropped records when running Debezium on Kafka
> Connect with Apicurio Schema Registry. Specifically, multiple times we have
> observed that a single record is dropped when we get this exception (full
> stack trace
> <https://gist.github.com/twthorn/917bf3cc576f2b486dde04b16a60d681>).
>
>
> Failed to send HTTP request to endpoint:
> http://schema-registry.service.prod-us-east-1-dw1.consul:8080/apis/ccompat/v6/subjects/prod
> .<keyspace>.<table>-key/versions?normalize=false
>
>
> This exception is raised by the Kafka Connect worker, which receives it
> from the confluent schema registry client. This seems to be a network blip
> and after it doesn't have any errors and continues processing data without
> issue. But it will have data loss for one record that was received almost
> exactly one minute prior to when this exception is logged. We have observed
> the behavior with that same timeline occur on different days several weeks
> apart. We have these key Kafka config settings (see full configs here
> <https://gist.github.com/twthorn/78c2ac329a46ce1baa820753daad47dd>):
>
>
>  "producer.batch.size=524288"
>  "producer.linger.ms=100"
>  "producer.acks=-1"
>  "producer.compression.type=snappy"
>  "producer.buffer.memory=268435456"
>  "config.storage.replication.factor=4"
>  "offset.storage.replication.factor=4"
>  "status.storage.replication.factor=4"
>  "scheduled.rebalance.max.delay.ms=180000"
>
>
> Other version info:
>
>
>    - Kafka Version 3.8.1
>    - Confluent version (eg for kafka-schema-registry-client,
>    kafka-schema-registry-converter, etc ) 7.5.2
>    - Avro Version 1.11.4
>
> Questions we have:
>
>    - Are there any known issues with schema registry interacting with
>    Kafka Connect to cause data loss?
>    - If we drop a record does that mean that the offsets stored by the
>    Kafka Connect worker source task are incorrect? i.e., we are committing
>    offsets for data that we have not yet finished sending to Kafka
>    - Any recommended debug steps to root cause this issue?
>
> Thank you for the help
>
>

Reply via email to