Thank you both for the directions, I'll dive into these. Peter
On Jan 20, 2017 9:55 AM, "Michael Noll" <mich...@confluent.io> wrote: > As Eno said I'd use the interactive queries API for Q2. > > Demo apps: > - > https://github.com/confluentinc/examples/blob/3. > 1.x/kafka-streams/src/main/java/io/confluent/examples/ > streams/interactivequeries/kafkamusic/KafkaMusicExample.java > - > https://github.com/confluentinc/examples/blob/3. > 1.x/kafka-streams/src/main/java/io/confluent/examples/ > streams/interactivequeries/WordCountInteractiveQueriesExample.java > > Further docs: > http://docs.confluent.io/current/streams/developer- > guide.html#interactive-queries > > (FYI: We have begun moving the info in these docs to the Apache Kafka docs, > too, but it will take a while.) > > -Michael > > > > > On Thu, Jan 19, 2017 at 5:23 PM, Eno Thereska <eno.there...@gmail.com> > wrote: > > > For Q2: one way to export the state on demand would be to use the > > Interactive Queries API (https://www.confluent.io/blog/unifying-stream- > > processing-and-interactive-queries-in-apache-kafka/ < > > https://www.confluent.io/blog/unifying-stream-processing- > and-interactive- > > queries-in-apache-kafka/>). That can be seen as a current, materialized > > view of the state. There is some example code in the blog. > > > > Eno > > > > > > > On 19 Jan 2017, at 16:11, Peter Kopias <kopias.pe...@gmail.com> wrote: > > > > > > Q1: Thank you, the branch() is what I'm looking for, I just missed it > > > somehow. > > > > > > Q2: > > > I receive something like "imageid,x,y" as key, and a color as value. > I > > > aggregate this to something like average color for example. > > > So technically I do not have images, I have colored pixels with 3 > > > dimensions one being the image... > > > And then my newcomer user wants to join the fun, so we'd need to serve > > > him an image with the latest current state (all pixels of imageid=x), > and > > > of course everything that comes later (updates on the output of the > > > aggregate, but filtered to that imageid). > > > > > > Problem is that how to get the stream api node to "export" part of it's > > > current state on demand. (All imageid=x keys with values). > > > > > > There could be a "request topic" that I could join together with the > > > aggregated ktable maybe? > > > > > > Other problem is following the updates without getting the details of > > > other images (99% of records are not interesting for the specific > user). > > > > > > Thanks, > > > > > > Peter > > > > > > > > > > > > On Thu, Jan 19, 2017 at 4:55 PM, Eno Thereska <eno.there...@gmail.com> > > > wrote: > > > > > >> Hi Peter, > > >> > > >> About Q1: The DSL has the "branch" API, where one stream is branched > to > > >> several streams, based on a predicate. I think that could help. > > >> > > >> About Q2: I'm not entirely sure I understand the problem space. What > is > > >> the definition of a "full image"? > > >> > > >> Thanks > > >> Eno > > >>> On 19 Jan 2017, at 12:07, Peter Kopias <kopias.pe...@gmail.com> > wrote: > > >>> > > >>> Greetings Everyone, > > >>> > > >>> I'm just getting into the kafka world with a sample project, and I've > > got > > >>> two conceptional issues, you might have a trivial answer already at > > hand > > >> to. > > >>> > > >>> Scenario: multiuser painting webapp, with N user working on M images > > >>> simultaneously. The "brush" events go to one single kafka topic, > in a > > >>> format: imageid,x,y -> brushevent , that I aggregate to imageid,x,y > > >>> > > >>> Q1: > > >>> It would be nice to separate the stream to M output topics, so that > > would > > >>> work nice as "partitioning", and also we could just subscribe to > update > > >>> events of a specific image maybe. How can I fan out the records to > > >>> different (maybe not yet existing) topics by using DSL? > > >>> > > >>> Is that a good idea? (If I can solve every processing in a common > > >>> processing graph that would be the best, but then I'd need some high > > >>> performance solution of filtering out the noise, as the subscribers > are > > >>> only interested in a very small subset of the soup.) > > >>> > > >>> Q2: > > >>> - When a new user comes, I'd like give him the latest full image? > > >>> (I could do a "fullimages" output topic, but then also comes the > > problem > > >>> of serious overhead on each incoming update, and also the newcomer > > should > > >>> somehow only get the image he's interested in, not read all the > images, > > >> and > > >>> ignore the others.) > > >>> > > >>> I know I'm still new to this, but I'd like to learn the best > practices > > >> you > > >>> might already tried. > > >>> > > >>> Thank you, > > >>> > > >>> Peter > > >> > > >> > > > > >
