[ 
https://issues.apache.org/jira/browse/KAFKA-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dong Lin updated KAFKA-1523:
----------------------------

    Description: 
* Entry point for transaction requests
* Appends transaction control records to the transaction journal
* Sends transaction control records to data brokers
* Responsible for expiring transactions
* Supports fail-over: for which it needs to maintain a transaction HW which is 
the offset of the BEGIN control record of the earliest pending transaction. It 
should checkpoint the HW periodically either to ZK/separate topic/offset commit.


We merge KAFKA-1565 transaction manager failover handling into this JIRA. 
Transaction manager should guarantee that, once a pre-commit/pre-abort request 
is acknowledged, commit/abort request will be delivered to partitions involved 
in the transaction.

This patch handles the following failover scenarios:
1) Transaction manager or its followers fail before txRequest is duplicated on 
local log and followers.
Solution: Transaction manager responds to request with error status. The 
producer keeps trying to commit.
2) The txPartition’s leader is not available.
Solution: Put txRequest on unSentTxRequestQueue. When metadataCache is updated, 
check and re-send txRequest from unSentTxRequestQueue if possible.
3) The txPartition’s leader fails when txRequest is in channel manager.
Solution: Retrieve all txRequests queued for transmission to this broker and 
put them on unSentTxRequestQueue.
4) Transaction manage does not receive success response from txPartition’s 
leaders within timeout period.
Solution: Transaction manager expires the txRequest and re-send it.
5) Transaction manager fails.
Solution: The new transaction manager reads transactionHW from zookeeper, and 
sends txRequest starting from the transactionHW.


This patch does not provide the following feature. These will be provided in 
separate patches.
1) Producer offset commit.
2) Transaction expiration.





  was:
* Entry point for transaction requests
* Appends transaction control records to the transaction journal
* Sends transaction control records to data brokers
* Responsible for expiring transactions
* Supports fail-over: for which it needs to maintain a transaction HW which is 
the offset of the BEGIN control record of the earliest pending transaction. It 
should checkpoint the HW periodically either to ZK/separate topic/offset commit.

The scope of this ticket will be the basic transaction coordinator 
functionality. E.g., we can just have a basic transaction manager that can 
begin/commit/abort transactions. No expiration, no fail-over. We will add 
failure handling in subsequent jiras.



> Implement transaction manager module
> ------------------------------------
>
>                 Key: KAFKA-1523
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1523
>             Project: Kafka
>          Issue Type: New Feature
>            Reporter: Joel Koshy
>            Assignee: Dong Lin
>              Labels: transactions
>         Attachments: KAFKA-1523_2014-07-17_20:12:55.patch, 
> KAFKA-1523_2014-07-22_16:45:42.patch, KAFKA-1523_2014-08-05_21:25:55.patch, 
> KAFKA-1523_2014-08-08_21:36:52.patch
>
>
> * Entry point for transaction requests
> * Appends transaction control records to the transaction journal
> * Sends transaction control records to data brokers
> * Responsible for expiring transactions
> * Supports fail-over: for which it needs to maintain a transaction HW which 
> is the offset of the BEGIN control record of the earliest pending 
> transaction. It should checkpoint the HW periodically either to ZK/separate 
> topic/offset commit.
> We merge KAFKA-1565 transaction manager failover handling into this JIRA. 
> Transaction manager should guarantee that, once a pre-commit/pre-abort 
> request is acknowledged, commit/abort request will be delivered to partitions 
> involved in the transaction.
> This patch handles the following failover scenarios:
> 1) Transaction manager or its followers fail before txRequest is duplicated 
> on local log and followers.
> Solution: Transaction manager responds to request with error status. The 
> producer keeps trying to commit.
> 2) The txPartition’s leader is not available.
> Solution: Put txRequest on unSentTxRequestQueue. When metadataCache is 
> updated, check and re-send txRequest from unSentTxRequestQueue if possible.
> 3) The txPartition’s leader fails when txRequest is in channel manager.
> Solution: Retrieve all txRequests queued for transmission to this broker and 
> put them on unSentTxRequestQueue.
> 4) Transaction manage does not receive success response from txPartition’s 
> leaders within timeout period.
> Solution: Transaction manager expires the txRequest and re-send it.
> 5) Transaction manager fails.
> Solution: The new transaction manager reads transactionHW from zookeeper, and 
> sends txRequest starting from the transactionHW.
> This patch does not provide the following feature. These will be provided in 
> separate patches.
> 1) Producer offset commit.
> 2) Transaction expiration.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Reply via email to