Hi Andrian, There is an existing framework for Interactive Queries contributed by Lightbend - https://github.com/lightbend/kafka-streams-query
HTH, Bill On Wed, Feb 13, 2019 at 10:14 AM Ryanne Dolan <ryannedo...@gmail.com> wrote: > Andrian, this looks useful: > > > https://docs.confluent.io/current/streams/developer-guide/interactive-queries.html > > The gist is you'd need to build your own routing and discovery layers. > > Also consider materializing your data to an external data store, e.g. > Couchbase or Hive, which provides this functionality already. > > Ryanne > > On Wed, Feb 13, 2019, 5:16 AM Andrian Jardan <andrianjar...@gmail.com > wrote: > > > Hello dear Kafka community, > > > > We are planning to build a Kafka Streams application that will build a > > pretty big state (~100Gb) in real time from various sources. > > > > We later on need to query this state as fast as possible, and the main > > idea is to use the built-in in-memory RockDB and get the data via, > > some sort of REST API we will build on top. The question is how do we > > identify where the data we need is, since the state will obviously > > not fit in the memory of a single instance and we need to scale somehow > in > > case this state keeps growing, and it will… > > > > Is there a way to identify where the data we care about resides, on what > > kafka streams app instance ? > > > > I tried to find the answer in the documentation, but was not able to > > figure it our unfortunately. > > > > Thank you in advance ! > > > > — > > Andrian Jardan > > Infrastructure and DevOps expert > > cell: +49 174 2815994 > > Skype: macrosdnb > > > > >
