[
https://issues.apache.org/jira/browse/KAFKA-7480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743325#comment-16743325
]
Matthias J. Sax commented on KAFKA-7480:
----------------------------------------
{quote}Hi [~mjsax], just to confirm the scope of work here, after reading the
other ticket, it sounds like we just need to set the policy to "none", catch
the InvalidOffsetException, and in that case, seek to the beginning and rebuild
the store accordingly.
{quote}
Well, we want to make the behavior configurable.
{quote}But your latter comment confused me. What could be a more resilient
configuration than starting from the beginning and rebuilding the store?
{quote}
This was a little sloppy from my side – I meant to use `auto.offset.reset` –
note the the exception is only thrown if policy is none. This is of course a
simplification, as you mentioned correctly, we always need to set policy to
`none` and catch the exception and decide base on user specified
`auto.offset.reset` how to proceed: either rethrow if user specifies `none`, or
clean the state, seek-to-beginning, and recreate it.
Do avoid the tombstone issue, I would prefer to always wipe out the store for
this case though. Thus (C) is not a good option IMHO.
Not sure if we can pull off B – also, it's actually an further optimization
that we might want to track in it's own ticket – this ticket is about allowing
user to set a reset policy – options (B) would be an internal optimization that
is orthogonal to this ticket.
> GlobalThread should honor custom auto.offset.reset policy
> ---------------------------------------------------------
>
> Key: KAFKA-7480
> URL: https://issues.apache.org/jira/browse/KAFKA-7480
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Matthias J. Sax
> Priority: Major
> Labels: needs-kip
>
> With KAFKA-6121 we improved Kafka Streams resilience and correctness with
> regard to consumer auto.offset.reset and state cleanup.
> Back than, we decided to let GlobalStreamThread die and not handle
> InvalidOffsetException during regular processing, because this error
> indicates a fatal issue and the user should be notified about it. However, as
> reported on the user mailing list, the only thing a user can do is, to
> restart the application (and investigate the root cause). During restart, the
> state will be cleaned up and bootstrapped correctly.
> Thus, we might want to allow users to specify a more resilient configuration
> for this case and log an ERROR message if the error occurs. To ensure
> consistency, we might not allow to set reset policy "latest" though (need
> discussion)? By default, we can still keep "none" and fail.
> Note: `Topology.addGlobalStore` does not allow to set a reset policy. Thus,
> this might require a KIP to extend `Topology.addGlobalStore` accordingly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
