[
https://issues.apache.org/jira/browse/KAFKA-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16033956#comment-16033956
]
ASF GitHub Bot commented on KAFKA-5364:
---------------------------------------
GitHub user apurvam opened a pull request:
https://github.com/apache/kafka/pull/3202
KAFKA-5364: Don't fail producer if drained partition is not yet in
transaction
Due to the async nature of the producer, it is possible to attempt to drain
a messages whose partition hasn't been added to the transaction yet. Before
this patch, we considered this a fatal error. However, it is only in error if
the partition isn't in the queue to be sent to the coordinator.
This patch updates the logic so that we only fail the producer if the
partition would never be added to the transaction. If the partition of the
batch is yet to be added, we will simply wait for the partition to be added to
the transaction before sending the batch to the broker.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apurvam/kafka
KAFKA-5364-ensure-partitions-added-to-txn-before-send
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/3202.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3202
----
commit 7dd2f7f9c3c59c80ed0cac4965df819a8c7ddcbb
Author: Apurva Mehta <apu...@confluent.io>
Date: 2017-06-02T00:18:08Z
Fix logic in ensurePartitionAddedToTransaction to account for partitions
which would be added to the transaction.
----
> Producer attempts to send transactional messages before adding partitions to
> transaction
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-5364
> URL: https://issues.apache.org/jira/browse/KAFKA-5364
> Project: Kafka
> Issue Type: Sub-task
> Components: clients, core, producer
> Affects Versions: 0.11.0.0
> Reporter: Apurva Mehta
> Assignee: Apurva Mehta
> Priority: Blocker
> Labels: exactly-once
> Fix For: 0.11.0.0
>
>
> Due to a race condition between the sender thread and the producer.send(),
> the following is possible:
> # In KakfaProducer.doSend(), we add partitions to the transaction and then do
> accumulator.append.
> # In Sender.run(), we check whether there are transactional request. If there
> are, we send them and wait for the response.
> # If there aren't we drain the accumulator queue and send the produce
> requests.
> # The problem is that the sequence step 2, 1, 3 is entire possible. This
> means that we won't send the 'AddPartitions' request but yet try to send the
> produce data. Which results in a fatal error and requires the producer to
> close.
> The solution is that in the accumulator.drain, we should check again if there
> are pending add partitions requests, and if so, don't drain anything.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
