If that is what it meant for DAG processing (we still need to confirm with Sameer), then programming-wise I do not see what's the difference with Kafka Streams since inside Streams users is also just specifying the topology as a DAG:
https://kafka.apache.org/0110/documentation/streams/core-concepts#streams_topology What is even better is that for Streams since we use Kafka as intermeidate buffer between connected sub-topologies user do not need to worry about back-pressure at all: http://docs.confluent.io/current/streams/architecture.html#backpressure And flexibility-wise, as you mention "it is a bit more flexible than Kafka Streams", I also cannot agree with you that it is the case, since with Kafka Streams threading model people can easily have multiple tasks representing different connected parts (i.e. sub-topologies) of the DAG which are then hosted by different threads executing on their own pace concurrently: https://kafka.apache.org/0110/documentation/streams/architecture#streams_architecture_threads Again, this is because different threads never need to talk to each other, but they just read / write data from / to Kafka topics which are then the persistent buffer of the intermeidate streams, no synchronization between threads are needed. Guozhang On Mon, Jul 17, 2017 at 10:38 AM, David Garcia <dav...@spiceworks.com> wrote: > On that note, akka streams has Kafka integration. We use it heavily and > it is quite a bit more flexible than K-Streams (which we also use…but for > simpler applications) Akka-streams-Kafka is particularly good for > asynchronous processing: http://doc.akka.io/docs/akka- > stream-kafka/current/home.html > > -David > > On 7/17/17, 12:35 PM, "David Garcia" <dav...@spiceworks.com> wrote: > > I think he means something like Akka Streams: > http://doc.akka.io/docs/akka/2.5.2/java/stream/stream-graphs.html > > Directed Acyclic Graphs are trivial to construct in Akka Streams and > use back-pressure to preclude memory issues. > > -David > > On 7/17/17, 12:20 PM, "Guozhang Wang" <wangg...@gmail.com> wrote: > > Sameer, > > Could you elaborate a bit more what do you mean by "DAG > processing"? > > > Guozhang > > > On Sun, Jul 16, 2017 at 11:58 PM, Sameer Kumar < > sam.kum.w...@gmail.com> > wrote: > > > Currently, we don't have DAG processing in Kafka Streams. Having > a DAG has > > its own share of advantages in that, it can optimize code on its > own and > > come up with a optimized execution plan. > > > > Are we exploring in this direction, do we have this in our > current roadmap. > > > > -Sameer. > > > > > > -- > -- Guozhang > > > > > -- -- Guozhang
