This was fixed in https://github.com/apache/kafka/pull/18146
On Tue, Nov 12, 2024 at 4:41 PM Thomas Thornton <tthorn...@salesforce.com> wrote: > Hi we notice data loss i.e. dropped records when running Debezium on Kafka > Connect with Apicurio Schema Registry. Specifically, multiple times we have > observed that a single record is dropped when we get this exception (full > stack trace > <https://gist.github.com/twthorn/917bf3cc576f2b486dde04b16a60d681>). > > > Failed to send HTTP request to endpoint: > http://schema-registry.service.prod-us-east-1-dw1.consul:8080/apis/ccompat/v6/subjects/prod > .<keyspace>.<table>-key/versions?normalize=false > > > This exception is raised by the Kafka Connect worker, which receives it > from the confluent schema registry client. This seems to be a network blip > and after it doesn't have any errors and continues processing data without > issue. But it will have data loss for one record that was received almost > exactly one minute prior to when this exception is logged. We have observed > the behavior with that same timeline occur on different days several weeks > apart. We have these key Kafka config settings (see full configs here > <https://gist.github.com/twthorn/78c2ac329a46ce1baa820753daad47dd>): > > > "producer.batch.size=524288" > "producer.linger.ms=100" > "producer.acks=-1" > "producer.compression.type=snappy" > "producer.buffer.memory=268435456" > "config.storage.replication.factor=4" > "offset.storage.replication.factor=4" > "status.storage.replication.factor=4" > "scheduled.rebalance.max.delay.ms=180000" > > > Other version info: > > > - Kafka Version 3.8.1 > - Confluent version (eg for kafka-schema-registry-client, > kafka-schema-registry-converter, etc ) 7.5.2 > - Avro Version 1.11.4 > > Questions we have: > > - Are there any known issues with schema registry interacting with > Kafka Connect to cause data loss? > - If we drop a record does that mean that the offsets stored by the > Kafka Connect worker source task are incorrect? i.e., we are committing > offsets for data that we have not yet finished sending to Kafka > - Any recommended debug steps to root cause this issue? > > Thank you for the help > >
