Yes! We just made KIP-67 available yesterday to start the discussions: https://cwiki.apache.org/confluence/display/KAFKA/KIP-67%3A+Queryable+state+for+Kafka+Streams <https://cwiki.apache.org/confluence/display/KAFKA/KIP-67:+Queryable+state+for+Kafka+Streams>
Any feedback is welcome, there is a mail thread in the dev mailing list. Thanks Eno > On 29 Jun 2016, at 15:52, Yi Chen <y...@symphonycommerce.com> wrote: > > This is awesome Eno! Would you mind sharing the JIRA ticket if you have one? > > On Sun, Jun 19, 2016 at 12:07 PM, Eno Thereska <eno.there...@gmail.com> > wrote: > >> Hi Yi, >> >> Your observation about accessing the state stores that are already there >> vs. keeping state outside of Kafka Streams is a good one. We are currently >> working on having the state stores accessible like you mention and should >> be able to share some design docs shortly. >> >> Thanks >> Eno >> >>> On 19 Jun 2016, at 19:49, Yi Chen <y...@symphonycommerce.com> wrote: >>> >>> Hello, >>> >>> I am thinking of using the Kafka Steams feature to "unify" our real-time >>> and scheduled workflow. An example is that in our workflow with stages >> A--> >>> B --> C, the A --> B segment can be achieved in real-time, but B-->C >>> segment is usually a done with a scheduled job, running maybe once per >> hour >>> or once per 5 minutes, etc. >>> >>> I am hoping to model this using Kafka Streams. Each stage would be a >> topic: >>> the Kafka Streams will process real-time events in topic-A and send >> result >>> to topic-B. The challenge is when I process the events in topic-B, I want >>> to be able to process each event with a crontab-like schedule, so that if >>> the process is successful (by checking an external API) the event is send >>> to topic-C, otherwise, we will re-process the event again according to >> the >>> schedule. >>> >>> Can I use the RocksDB key/value state store to store the topic-B events >>> that failed to process, and have a scheduler (like quartz scheduler) to >>> iterate all events in the store and re-process again? I know I can always >>> keep the state outside of Kafka but I like that the state store is >>> fault-tolerant and can be rebuilt automatically if the instance fails. >> The >>> examples I found so far seems to imply that the state store is only >>> accessible from within a processor. >>> >>> Thanks, >>> Yi >> >>
