I have a custom in-memory state store which I’d like to configure as a global 
store for my Kafka Streams application.  However, even though my state store is 
not persistent (i.e. the implementation of StateStore.persistent() returns 
false) it appears that the GlobalStateManager still continues to write offset 
checkpoints to disk and then reads those checkpoints upon recovery of the Kafka 
Streams application to adjust the offsets of the globalConsumer.  Because my 
state store is not persistent though, it does not preserve the previous data 
which was read from the topic before the streams application was restarted.  
Therefore, the fact that the globalConsumer has its offsets set to the values 
read from the checkpoint file upon initialization causes my state store to lose 
previously consumed data.  To avoid this situation, it would seem like 
non-persistent state stores should not have their offsets checkpointed when 
they are used as a global store; this way the globalConsumer would always 
consume events from the beginning of the topic.  This has the consequence of 
slowing down start-up time of the stream since it will have to recover more 
messages from the topic but in my case the number of messages on this topic is 
not expected to be large and I’m willing to tolerate the slightly slower 
start-up times.  Therefore, is it possible to disable offset checkpointing for 
in-memory global stores?  If not, is it reasonable to request such a feature to 
be added?

Thanks,
David 

Reply via email to