[
https://issues.apache.org/jira/browse/KAFKA-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102096#comment-16102096
]
Apurva Mehta commented on KAFKA-5621:
-------------------------------------
> The expiration for messages in the accumulator was not for memory footprint
> control, but for making progress when a partition is stuck
This is a fair point. Since we have one queue per partition, there is no real
reason why a single stuck partition should affect the others. I think what
happens today is that the next {{producer.send}} to the stuck partition will
just block because there is no space remaining, and that affects everything
else. From this point of view, expiring accumulator batches does make sense.
Is that right?
On your second point of the concept of a batch being exposed, you are right, of
course. Tuning your producer batches is an important part of getting the best
performance out of kafka. We have the {{linger.ms}} and {{batch.size}} configs
to control this and these are fairly intuitive to use. As a user, you would
think "If I have an expected X byte/s steady state throughput and I want
batches of Y bytes, I should set linger to Zms to make sure I get my optimal
batching".
In this context, exposing the notion of a client side record accumulator with
batch expiry _is_ a new concept in my opinion.
> The producer should retry expired batches when retries are enabled
> ------------------------------------------------------------------
>
> Key: KAFKA-5621
> URL: https://issues.apache.org/jira/browse/KAFKA-5621
> Project: Kafka
> Issue Type: Bug
> Reporter: Apurva Mehta
> Fix For: 1.0.0
>
>
> Today, when a batch is expired in the accumulator, a {{TimeoutException}} is
> raised to the user.
> It might be better the producer to retry the expired batch rather up to the
> configured number of retries. This is more intuitive from the user's point of
> view.
> Further the proposed behavior makes it easier for applications like mirror
> maker to provide ordering guarantees even when batches expire. Today, they
> would resend the expired batch and it would get added to the back of the
> queue, causing the output ordering to be different from the input ordering.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
