Sorry. I responded to the wrong message.... On 5 October 2016 at 09:40, Gary Ogden <gog...@gmail.com> wrote:
> Thanks Guozhang. > > So there's no way we could also use InternalTopicManager to specify the > number of partitions and RF? > > https://github.com/apache/kafka/blob/0.10.1/streams/src/ > main/java/org/apache/kafka/streams/processor/internals/ > InternalTopicManager.java > > > > On 4 October 2016 at 19:25, Guozhang Wang <wangg...@gmail.com> wrote: > >> Hello Gary, >> >> What you described should be workable with the lower-level Processor >> interface of Kafka Streams, i.e. dynamic aggregations based on the input >> data indicating changes to the JSON schemas. For detailed examples of how >> the Processor API works please read the corresponding sections on the web >> docs: >> >> http://docs.confluent.io/3.0.1/streams/developer-guide.html#processor-api >> >> >> Guozhang >> >> On Mon, Oct 3, 2016 at 6:51 AM, Gary Ogden <gog...@gmail.com> wrote: >> >> > I have a use case, and I'm wondering if it's possible to do this with >> > Kafka. >> > >> > Let's say we will have customers that will be uploading JSON to our >> system, >> > but that JSON layout will be different between each customer. They are >> able >> > to define the schema of the JSON being uploaded. >> > >> > They will then be able to define the fields in that JSON they want to >> > gather metrics on (sum, counts etc). >> > >> > Is there a way with Kafka streaming to dynamically read the >> configuration >> > for that customer and process the json and do counts and sums for the >> > fields they've defined. >> > >> > It's possible at any time they may want to modify the configuration for >> > their json as well. Stop counting one field, start counting another. >> > >> > They will also want to do some inferences as well. IE, if this >> particular >> > JSON is uploaded with a field in it, then check to see if another json >> was >> > uploaded within 8 hours. >> > >> > Is it possible for Kafka streaming to be this dynamic? >> > >> >> >> >> -- >> -- Guozhang >> > >
