Hi,
There is a design document for transaction support linked at the bottom of
KIP-98 that you can read here
https://docs.google.com/document/d/11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8
.
That describes some of the recovery/retry mechanisms. The design relies on
partition availability
to make forward progress. If components restart, they carry on from where they
left off.
Andrew Schofield
IBM Event Streams
On 19/12/2018, 13:37, "Jose Raul Perez Rodriguez"
<joseraul.subscript...@gmail.com> wrote:
Hi, thanks for the answer, it was helpful.
So, if there are several topic-partitions in a transaction, the reads
are eventually consistent; it is possible some message from that
transaction are not available yet, until some recovery/retry mechanism
is completed for the fail topic-partitions? If this is the case, what
kind of recovery/retry mechanism is implemented to deal with this, and
keep kafka transactions eventually consistent.
Thanks in advance,
On 12/19/18 2:17 PM, Andrew Schofield wrote:
> Hi,
> This is very similar to traditional two-phase commit. There are
essentially multiple logs
> being used - one per TopicPartition involved and the overall transaction
log. At the point
> where COMMIT is being written to the TopicPartitions, it is assumed that
it will be possible to
> write all of these without error and it is assumed that it will be
possible to write COMMITTED
> to the transaction log. Once a single COMMIT has been written, it's too
late to abort the
> transaction and everything has to go forwards to complete the commit to
end up in a consistent
> state.
>
> When a consumer sees COMMIT for a transaction, it can deliver any
messages it has held
> buffered for that transaction. It's not really sure whether COMMITTED was
written successfully
> and fully replicated. It would generally be considered "good enough" to
make this assumption.
>
> Andrew Schofield
> IBM Event Streams
>
> On 18/12/2018, 21:51, "Jose Raul Perez Rodriguez"
<joseraul.subscript...@gmail.com> wrote:
>
> Hi all,
>
> Reading this
>
<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-98%2B-%2BExactly%2BOnce%2BDelivery%2Band%2BTransactional%2BMessaging&data=02%7C01%7C%7Cf25d15fb694e498608da08d665b71564%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636808234337757549&sdata=MtPOa%2Fi8%2B2dRmvMpwOPGAfXHvBvTZ6%2FMDqHtz7iX6vU%3D&reserved=0>
> document about transactions in Kafka, specifically epigraphs 5.2
> WriteTxnMarkerRequest and 5.3 Writing the final Commit or Abort
Message.
>
> What I understand from *5.2* is that the coordinator sends a "write
txt
> market request" to each topic/partition leader in the transaction,
then,
> each leader broker in those topic-partition write a Commit or Abort
> message to the log, then, using this information the Consumer for
that
> particular topic-partition decide to read the messages (in case of
> Commit) or to drop the message (in case of Abort).
>
> My doubt is; those messages that passed the "write txt market
request"
> phase for a particular topic-partition (the Commit cases) are just
> delivered to the user?, or hided from the user until full transaction
> confirms it Committed?, in the case is delivered to the user, there
> could be inconsistent reads, because another topic-partition could
fail
> and then the full transaction needs to abort. On the other hand, if
> those messages are not delivered to user until the Consumer reads a
> Commit message for the full transaction *(5.3*), here is my second
> question; how this works?, i.e how a Consumer is aware that a
particular
> message belonging to a transaction can be delivered to the user, i.e
the
> transaction that owns the succeed message?
>
> I hope is clear this explanation, I have not found this question in
any doc.
>
> Thanks in advance,
>
>
>
