Hi, thanks for the answer, it was helpful.
So, if there are several topic-partitions in a transaction, the reads
are eventually consistent; it is possible some message from that
transaction are not available yet, until some recovery/retry mechanism
is completed for the fail topic-partitions? If this is the case, what
kind of recovery/retry mechanism is implemented to deal with this, and
keep kafka transactions eventually consistent.
Thanks in advance,
On 12/19/18 2:17 PM, Andrew Schofield wrote:
Hi,
This is very similar to traditional two-phase commit. There are essentially
multiple logs
being used - one per TopicPartition involved and the overall transaction log.
At the point
where COMMIT is being written to the TopicPartitions, it is assumed that it
will be possible to
write all of these without error and it is assumed that it will be possible to
write COMMITTED
to the transaction log. Once a single COMMIT has been written, it's too late to
abort the
transaction and everything has to go forwards to complete the commit to end up
in a consistent
state.
When a consumer sees COMMIT for a transaction, it can deliver any messages it
has held
buffered for that transaction. It's not really sure whether COMMITTED was
written successfully
and fully replicated. It would generally be considered "good enough" to make
this assumption.
Andrew Schofield
IBM Event Streams
On 18/12/2018, 21:51, "Jose Raul Perez Rodriguez"
<joseraul.subscript...@gmail.com> wrote:
Hi all,
Reading this
<https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-98%2B-%2BExactly%2BOnce%2BDelivery%2Band%2BTransactional%2BMessaging&data=02%7C01%7C%7Cb1d4e3c2f6b7457112a108d66532fdf5%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636807667015030870&sdata=zcoqxussvy%2FbAUYzDg2Q2V%2BODTQfDGlFEfOqkxv42cM%3D&reserved=0>
document about transactions in Kafka, specifically epigraphs 5.2
WriteTxnMarkerRequest and 5.3 Writing the final Commit or Abort Message.
What I understand from *5.2* is that the coordinator sends a "write txt
market request" to each topic/partition leader in the transaction, then,
each leader broker in those topic-partition write a Commit or Abort
message to the log, then, using this information the Consumer for that
particular topic-partition decide to read the messages (in case of
Commit) or to drop the message (in case of Abort).
My doubt is; those messages that passed the "write txt market request"
phase for a particular topic-partition (the Commit cases) are just
delivered to the user?, or hided from the user until full transaction
confirms it Committed?, in the case is delivered to the user, there
could be inconsistent reads, because another topic-partition could fail
and then the full transaction needs to abort. On the other hand, if
those messages are not delivered to user until the Consumer reads a
Commit message for the full transaction *(5.3*), here is my second
question; how this works?, i.e how a Consumer is aware that a particular
message belonging to a transaction can be delivered to the user, i.e the
transaction that owns the succeed message?
I hope is clear this explanation, I have not found this question in any doc.
Thanks in advance,
