Make totally sense and is a known issue: https://issues.apache.org/jira/browse/KAFKA-6711
Please follow up the ticket and/or PR -- not sure what the current status is. -Matthias On 4/17/18 11:16 PM, David Chu wrote: > I have a custom in-memory state store which I’d like to configure as a global > store for my Kafka Streams application. However, even though my state store > is not persistent (i.e. the implementation of StateStore.persistent() returns > false) it appears that the GlobalStateManager still continues to write offset > checkpoints to disk and then reads those checkpoints upon recovery of the > Kafka Streams application to adjust the offsets of the globalConsumer. > Because my state store is not persistent though, it does not preserve the > previous data which was read from the topic before the streams application > was restarted. Therefore, the fact that the globalConsumer has its offsets > set to the values read from the checkpoint file upon initialization causes my > state store to lose previously consumed data. To avoid this situation, it > would seem like non-persistent state stores should not have their offsets > checkpointed when they are used as a global store; this way the > globalConsumer would always consume events from the beginning of the topic. > This has the consequence of slowing down start-up time of the stream since it > will have to recover more messages from the topic but in my case the number > of messages on this topic is not expected to be large and I’m willing to > tolerate the slightly slower start-up times. Therefore, is it possible to > disable offset checkpointing for in-memory global stores? If not, is it > reasonable to request such a feature to be added? > > Thanks, > David >
signature.asc
Description: OpenPGP digital signature
