@Becket It has been a pain in many cases that we do not know the number of > messages in a message set, not sure if the OffsetDelta field in the > wrapper > message will address this.
Interestingly, we had this in one of the design iterations, but we found in the prototype that we weren't really using it. Did you have a particular use case in mind? I share the intuition that it may be helpful to know, but I don't have a clear example in mind. In fact, in the initial version, we attempted to let the message set always represent a contiguous sequence of messages. In that case, the message set only needed a base offset and a count of the number of messages, and the individual messages no longer needed the offset delta. We ultimately abandoned that because we were uncomfortable with its impact on compaction. -Jason On Mon, Dec 12, 2016 at 5:55 PM, Guozhang Wang <wangg...@gmail.com> wrote: > Andrew, > > As I mentioned above, in Kafka durability is supported via data replication > instead of sync-flushing to disks. KIP-98 does not try to change that part > of the Kafka: if all your replicas are gone at the same time before the > data was ever flushed to disks, then your data is lost today, and it will > be still the case after KIP-98. > > As for atomicity, KIP-98 does provide all-or-nothing guarantee for writes > to multiple partitions, and it is based on its existing durability > guarantees. So it is possible that if your durability breaks, then > atomicity will be violated: some of the committed transaction's messages > could be lost if the above scenarios happen while others can be > successfully appended. My take is that, if you have concerns that Kafka's > replication mechanism i not good enough for your durability requirements as > of today, then you should have the same level of concerns with durability > if you want to use Kafka with KIP-98 as your transactional queuing system > as well. > > > Guozhang > > > On Mon, Dec 12, 2016 at 1:49 AM, Andrew Schofield < > andrew_schofi...@live.com > > wrote: > > > Guozhang, > > Exactly. This is the crux of the matter. Because it's async, the log is > > basically > > slightly out of date wrt to the run-time state and a failure of all > > replicas might > > take the data slightly back in time. > > > > Given this, do you think that KIP-98 gives an all-or-nothing, > > no-matter-what guarantee > > for Kafka transactions? I think the key is whether the data which is > > asynchronously > > flushed is guaranteed to be recovered atomically in all cases. > > Asynchronous but > > atomic would be good. > > > > Andrew Schofield > > IBM Watson and Cloud Platform > > > > > > > > > > From: Guozhang Wang <wangg...@gmail.com> > > > Sent: 09 December 2016 22:59 > > > To: dev@kafka.apache.org > > > Subject: Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional > > Messaging > > > > > > Onur, > > > > > > I understand your question now. So it is indeed possible that after > > > commitTxn() returned the messages could still be lost permanently if > all > > > replicas failed before the data was flushed to disk. This is the virtue > > of > > > Kafka's design to reply on replication (probably in memory) for high > > > availability, hence async flushing. This scenario already exist today > and > > > KIP-98 did not intend to change this factor in any ways. > > > > > > Guozhang > > > > > > > > > > > > > > > -- > -- Guozhang >
