Re: Idempotent producer and handling OutOfOrderSequenceException

2025-04-23 Thread Stig Rohde Døssing
Thanks. For us at least, there is no need to change the behavior, we just wanted to understand the reasoning behind the Javadoc. Den ons. 23. apr. 2025 kl. 20.36 skrev Matthias J. Sax : > I guess that make sense. Would need to dig into the code (not super > familiar with this part of the code bas

Re: Idempotent producer and handling OutOfOrderSequenceException

2025-04-23 Thread Matthias J. Sax
I guess that make sense. Would need to dig into the code (not super familiar with this part of the code base), to fully confirm. Not sure if it might not be possible to change though, if there is a demand for it. -- Might also be a question about how complex such a change would be. My read i

Re: Idempotent producer and handling OutOfOrderSequenceException

2025-04-21 Thread Stig Rohde Døssing
Thanks, this was very useful. > Well, the producer does not have x=1 in it's buffer any longer -- > otherwise, it our-of-sequence-error would not have be given to the > application, and the producer could just retry internally. Okay, so if I understand you, the reason the application has to repla

Re: Idempotent producer and handling OutOfOrderSequenceException

2025-04-20 Thread Matthias J. Sax
Thanks for providing more details. It makes sense that you need to ensure that an "insert" comes before an "update". If I understand you correctly, that's what idempotence guarantees. If the PUT is not appended to the log, it will be retried, but then thread 2 can't have observed it. Yes.

Re: Idempotent producer and handling OutOfOrderSequenceException

2025-04-17 Thread Stig Rohde Døssing
Thanks. What I'm looking for isn't that messages can't be interleaved, it's something like causal order. We're updating a database where every node has their own copy of the database state (similar to RocksDB), by sending changes to that database via Kafka. In our application thread 1 is creating

Re: Idempotent producer and handling OutOfOrderSequenceException

2025-04-16 Thread Matthias J. Sax
The behavior I'm looking for is that if thread 1 sends message x=1, and thread 2 sees message x=1 come through Kafka, thread 2 can't then write message x=2 and have that be clobbered because thread 1's producer is retrying the send since it didn't receive the ack in time. This is not how it work

Idempotent producer and handling OutOfOrderSequenceException

2025-04-15 Thread Stig Rohde Døssing
Hi, If I understand correctly, the idempotent producer should be able to guarantee that messages are not duplicated when the producer retries sends, and as of https://issues.apache.org/jira/browse/KAFKA-5494, it should allow for the producer to have multiple in flight requests at a time without r