Just to close the loop on this for the mailing list: after discussion with Matthias Sax on Slack, I created this issue: https://issues.apache.org/jira/browse/KAFKA-8650.
Regards, Raman On Tue, Jul 9, 2019 at 12:43 PM Raman Gupta <rocketra...@gmail.com> wrote: > > I have a stream that is configured for exactly-once processing. I set > the "auto.offset.reset" of this stream to "none" for two reasons: > > 1) If an exactly-once stream loses its offset for some reason, using > either of earliest or latest is dangerous: "earliest" implies > reprocessing data, and "latest" may result in skipped messages. Using > "none" allows the ops team to investigate the situation and set the > offsets explicitly. > > 2) For a topic migration e.g. t-v1 -> t-v2, my exactly once stream > needs to stop reading from t-v1, and start reading from some > particular offset in t-v2 that corresponds to the message after the > last message the previous stream consumed in t-v1. To be safe, I want > my system to fail if the offset for t-v2 has not been set explicitly, > so again, neither "earliest" nor "latest" are an appropriate setting > here. > > However, when using "none", even if the offset has been set explicitly > via the application reset tool, the streams client stills throw an > exception on startup: `NoOffsetForPartitionException` for an internal > repartition topic. This seems like a nasty bug / leaky abstraction: > the consumer offsets, and policy on auto.offset.reset for an internal > topic should not be controlled or affected by the user-configured > policy for input topics. > > Or am I missing something here? > > Regards, > Raman
