I have a custom in-memory state store which I’d like to configure as a global store for my Kafka Streams application. However, even though my state store is not persistent (i.e. the implementation of StateStore.persistent() returns false) it appears that the GlobalStateManager still continues to write offset checkpoints to disk and then reads those checkpoints upon recovery of the Kafka Streams application to adjust the offsets of the globalConsumer. Because my state store is not persistent though, it does not preserve the previous data which was read from the topic before the streams application was restarted. Therefore, the fact that the globalConsumer has its offsets set to the values read from the checkpoint file upon initialization causes my state store to lose previously consumed data. To avoid this situation, it would seem like non-persistent state stores should not have their offsets checkpointed when they are used as a global store; this way the globalConsumer would always consume events from the beginning of the topic. This has the consequence of slowing down start-up time of the stream since it will have to recover more messages from the topic but in my case the number of messages on this topic is not expected to be large and I’m willing to tolerate the slightly slower start-up times. Therefore, is it possible to disable offset checkpointing for in-memory global stores? If not, is it reasonable to request such a feature to be added?
Thanks, David
