If I understand this correctly: assuming I have a simple aggregator distributed across n-docker instances each instance will _also_ need to support some sort of communications process for allowing access to its statestore (last param from KStream.groupby.aggregate).
How would one go about substituting a separated db (EG redis) for the statestore? Some advantages to decoupling: - It would seem like having a centralized process like this would alleviate the need to execute multiple requests for a given kv pair (IE "who has this data?" and subsequent requests to retrieve it). - it would take some pressure off of each node to maintain a large disk store - Typical microservices would separate storing / retrieving data - It would raise some eyebrows if a spec called for a mysql/nosql instance to be installed with every docker container - The tombstoning facilities of redis or C* would lend themselves well to implementing a 'true' rolling aggregation I get that RocksDB has a small footprint but given the choice of implementing my own RPC / gossip-like process for data sharing and using a well tested one (ala C* or redis) I would almost always opt for the latter. (Footnote: Our implementations already heavily use redis/memcached for deduplication of kafka messages so it would seem a small step to use the same to store aggregation results.) Just my $0.02. I would love to hear why Im missing the 'big picture'. The kstreams architecture seems rife with potential. On Thu, Mar 23, 2017 at 3:17 PM, Matthias J. Sax <matth...@confluent.io> wrote: > The config does not "do" anything. It's metadata that get's broadcasted > to other Streams instances for IQ feature. > > See this blog post for more details: > https://www.confluent.io/blog/unifying-stream-processing- > and-interactive-queries-in-apache-kafka/ > > Happy to answer any follow up question. > > > -Matthias > > On 3/23/17 11:51 AM, Jon Yeargers wrote: > > What does this config param do? > > > > I see it referenced / used in some samples and here ( > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 67%3A+Queryable+state+for+Kafka+Streams > > ) > > > >
