Thank Eno, Yes, I am aware of that. It indeed looks like a very useful feature.
The result of the processing in kafka streams is only a small amount of data that is require by our service. Currently it make more sense for us to update the remote database were we have more data that our application require. Also, the data should be available in case of failures. The remote database data is replicated. AFAIK although RocksDb changelog is backed by kafka, if a node fail, the data will be unavailable until it will be replicated to a different node. On Sun, Mar 5, 2017 at 4:38 PM, Eno Thereska <eno.there...@gmail.com> wrote: > Hi Shimi, > > Could you tell us more about your scenario? Kafka Streams uses embedded > databases (RocksDb) to store it's state, so often you don't need to write > anything to an external database and you can query your streams state > directly from streams. Have a look at this blog if that matches your > scenario: https://www.confluent.io/blog/unifying-stream-processing- > and-interactive-queries-in-apache-kafka/ <https://www.confluent.io/ > blog/unifying-stream-processing-and-interactive-queries-in-apache-kafka/>. > > Cheers > Eno > > > On 5 Mar 2017, at 10:48, Shimi Kiviti <shim...@gmail.com> wrote: > > > > Hi Everyone, > > > > I was wondering about writing data to remote database. > > I see 4 possible options: > > > > 1. Read from a topic and write to the database. > > 2. Use kafka connect > > 3. Write from anywhere in kafka streams. > > 4. Register a CachedStateStore FlushListener that will send a batch of > > records when the store flush the records. > > > > Advantages of #4: > > > > - The code is decoupled from the processing code and easier to refactor > > in the future. > > - Remove the need for additional kafka topic. > > > > > > Thanks, > > > > Shimi > >
