[
https://issues.apache.org/jira/browse/KAFKA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022945#comment-16022945
]
Magnus Edenhill commented on KAFKA-4340:
----------------------------------------
While the idea behind this JIRA is good (as a means of optimization) I think it
might be troublesome in practice.
If a producer sends a batch of N messages, with one message being too old, the
entire batch will fail (errors are propagated per partition, not message) and
the producer can't really recover and retry gracefully to produce the
non-timedout messages.
This problem is not related to a specific client, but rather the nature of the
data being produced:
it will manifest itself with old timestamps, such as app-sourced timestamps, or
things like MM.
A better alternative would perhaps be to silently discard the message on the
broker instead (which is effectively the same as writing the message to log and
then immediately cleaning it before a consumer picks up the message).
> Change the default value of log.message.timestamp.difference.max.ms to the
> same as log.retention.ms
> ---------------------------------------------------------------------------------------------------
>
> Key: KAFKA-4340
> URL: https://issues.apache.org/jira/browse/KAFKA-4340
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.10.1.0
> Reporter: Jiangjie Qin
> Assignee: Jiangjie Qin
> Fix For: 0.11.0.0
>
>
> [~junrao] brought up the following scenario:
> If users are pumping data with timestamp already passed log.retention.ms into
> Kafka, the messages will be appended to the log but will be immediately
> rolled out by log retention thread when it kicks in and the messages will be
> deleted.
> To avoid this produce-and-deleted scenario, we can set the default value of
> log.message.timestamp.difference.max.ms to be the same as log.retention.ms.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
