Hi Yi, Your observation about accessing the state stores that are already there vs. keeping state outside of Kafka Streams is a good one. We are currently working on having the state stores accessible like you mention and should be able to share some design docs shortly.
Thanks Eno > On 19 Jun 2016, at 19:49, Yi Chen <y...@symphonycommerce.com> wrote: > > Hello, > > I am thinking of using the Kafka Steams feature to "unify" our real-time > and scheduled workflow. An example is that in our workflow with stages A--> > B --> C, the A --> B segment can be achieved in real-time, but B-->C > segment is usually a done with a scheduled job, running maybe once per hour > or once per 5 minutes, etc. > > I am hoping to model this using Kafka Streams. Each stage would be a topic: > the Kafka Streams will process real-time events in topic-A and send result > to topic-B. The challenge is when I process the events in topic-B, I want > to be able to process each event with a crontab-like schedule, so that if > the process is successful (by checking an external API) the event is send > to topic-C, otherwise, we will re-process the event again according to the > schedule. > > Can I use the RocksDB key/value state store to store the topic-B events > that failed to process, and have a scheduler (like quartz scheduler) to > iterate all events in the store and re-process again? I know I can always > keep the state outside of Kafka but I like that the state store is > fault-tolerant and can be rebuilt automatically if the instance fails. The > examples I found so far seems to imply that the state store is only > accessible from within a processor. > > Thanks, > Yi
