[
https://issues.apache.org/jira/browse/KAFKA-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101932#comment-16101932
]
Jiangjie Qin commented on KAFKA-5621:
-------------------------------------
[~ijuma] [~apurva] The expiration for messages in the accumulator was not for
memory footprint control, but for making progress when a partition is stuck.
For example, if the leader of a partition becomes unavailable for some reason,
the records in the accumulator cannot be sent. Retry only makes sense when the
producer can try. So we have to expire the records at some point when that
partition cannot make progress, whether it is expired after request_timeout or
retries * request_timeout could be discussed. But notice that some times client
will set the retries to be Integer.MAX_VALUE. This will also result in
unexpected behavior.
The reasons of having an explicit batch.expiry.ms are: 1) we have exposed the
concept of bathing to the users through batch.size and linger.ms. So users
should have already known the producer sends batches. No new concept is added.
2) If a record has been sitting in the record accumulator for more than
batch.expiry.ms, likely there is a very long queue or the producer cannot make
progress. So users may want to get an exception and do something. And this
expiration time is kind of an SLO and is not necessarily related to the
request_timeout * retries which is intended for the remote call. So decoupling
them would be useful.
> The producer should retry expired batches when retries are enabled
> ------------------------------------------------------------------
>
> Key: KAFKA-5621
> URL: https://issues.apache.org/jira/browse/KAFKA-5621
> Project: Kafka
> Issue Type: Bug
> Reporter: Apurva Mehta
> Fix For: 1.0.0
>
>
> Today, when a batch is expired in the accumulator, a {{TimeoutException}} is
> raised to the user.
> It might be better the producer to retry the expired batch rather up to the
> configured number of retries. This is more intuitive from the user's point of
> view.
> Further the proposed behavior makes it easier for applications like mirror
> maker to provide ordering guarantees even when batches expire. Today, they
> would resend the expired batch and it would get added to the back of the
> queue, causing the output ordering to be different from the input ordering.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
