[
https://issues.apache.org/jira/browse/KAFKA-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892046#comment-16892046
]
Bob Barrett commented on KAFKA-7190:
------------------------------------
The periodic producer expiration check expires producers if 1) there is no
ongoing transaction for the producer, and 2) the max timestamp of the last
batch written by that producer is more than `transactional.id.expiration.ms`
ago. The default value for `transactional.id.expiration.ms` is 7 days. If the
input topic only has messages older than 7 days, and the transformed records
produced to it also have a timestamp older than 7 days, then the producer will
be expired the first time that the expiration check runs without an ongoing
transaction. KIP-360 will indirectly address this problem by allowing the
producer to continue after receiving an UNKNOWN_PRODUCER_ID error, but a better
solution would probably be to set the producer state timestamp based on the
current time, not the batch timestamp, as [~rocketraman] suggested.
[~hachikuji] what do you think?
In the meantime, another workaround would be to set
`transactional.id.expiration.ms` to a larger number, which would allow the
transformed records to retain the default Streams timestamp behavior.
> Under low traffic conditions purging repartition topics cause WARN statements
> about UNKNOWN_PRODUCER_ID
> ---------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-7190
> URL: https://issues.apache.org/jira/browse/KAFKA-7190
> Project: Kafka
> Issue Type: Improvement
> Components: core, streams
> Affects Versions: 1.1.0, 1.1.1
> Reporter: Bill Bejeck
> Assignee: Guozhang Wang
> Priority: Major
>
> When a streams application has little traffic, then it is possible that
> consumer purging would delete
> even the last message sent by a producer (i.e., all the messages sent by
> this producer have been consumed and committed), and as a result, the broker
> would delete that producer's ID. The next time when this producer tries to
> send, it will get this UNKNOWN_PRODUCER_ID error code, but in this case,
> this error is retriable: the producer would just get a new producer id and
> retries, and then this time it will succeed.
>
> Possible fixes could be on the broker side, i.e., delaying the deletion of
> the produderIDs for a more extended period or on the streams side developing
> a more conservative approach to deleting offsets from repartition topics
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
