We've started experimenting with Kafka to see if it can be used to aggregate our application data. I think our use case is a match for Kafka streams, but we aren't sure if we are using the tool correctly. The proof of concept we've built seems to be working as designed, I'm not sure that we are using the APIs appropriately.
Our proof of concept is to use kafka streams to keep a running tally of information about a program, and writes that data to an output topic, e.g. { "numberActive": 0, "numberInactive": 0, "lastLogin": "01-01-1970T00:00:00Z" } Computing the tally is easy, it is essentially executing a compare and swap (CAS) operation based on the input topic & output field. The local state store is used to store of the most recent program for a given key. We join an input stream against the state store and run the CAS operation using a TransformSupplier, which explictly writes the data to the state store using context.put(...) context.commit(); Is this an appropriate use of the local state store? Is there another another approach to keeping a stateful running tally in a topic? thanks, Jason