Re: Kafka Streams & Distributed state question

2019-02-14 Thread Andrian Jardan
This is exactly what I was looking for, many thanks for the suggestion !! If there are any other solutions, would be happy to hear though. Retrying and querying other nodes sounds like an approach that would work, but a much better option would be to re-use the hashing algo kafka itself uses t

Re: Kafka Streams & Distributed state question

2019-02-13 Thread Bill Bejeck
Hi Andrian, There is an existing framework for Interactive Queries contributed by Lightbend - https://github.com/lightbend/kafka-streams-query HTH, Bill On Wed, Feb 13, 2019 at 10:14 AM Ryanne Dolan wrote: > Andrian, this looks useful: > > > https://docs.confluent.io/current/streams/develope

Re: Kafka Streams & Distributed state question

2019-02-13 Thread Ryanne Dolan
Andrian, this looks useful: https://docs.confluent.io/current/streams/developer-guide/interactive-queries.html The gist is you'd need to build your own routing and discovery layers. Also consider materializing your data to an external data store, e.g. Couchbase or Hive, which provides this funct

Kafka Streams & Distributed state question

2019-02-13 Thread Andrian Jardan
Hello dear Kafka community, We are planning to build a Kafka Streams application that will build a pretty big state (~100Gb) in real time from various sources. We later on need to query this state as fast as possible, and the main idea is to use the built-in in-memory RockDB and get the data vi