I did a bit of research on that matter recently, the comparison is between Spark Structured Streaming(SSS) and Kafka Streams,
Both are relatively new (~1y) and trying to solve similar problems, however if you go with Spark, you have to go with a cluster, if your environment already have a cluster, then it's good. However our team doesn't do any Spark, so the initial cost would be very high. On the other hand, Kafka Streams is a java library, since we have a service framework, doing stream inside a service is super easy. However for some reason, people see SSS is more mature and Kafka Streams is not so mature (like Beta). But old fashion stream is both mature enough (in my opinion), I didn't see any difference in DStream(Spark) and KStream(Kafka) DataFrame (Structured Streaming) and KTable, I found it quite different. Kafka's model is more like a change log, that means you need to see the latest entry to make a final decision. I would call this as 'Update' model, whereas Spark does 'Append' model and it doesn't support 'Update' model yet. (it's coming to 2.2) http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#output-modes I wanted to have 'Append' model with Kafka, but it seems it's not easy thing to do, also Kafka Streams uses an internal topic to keep state changes for fail-over scenario, but I'm dealing with a lots of tiny information and I have a big concern about the size of the state store / topic, so my decision is that I'm going with my own handling of Kafka API .. If you do stateless operation and don't have a spark cluster, yeah Kafka Streams is perfect. If you do stateful complicated operation and happen to have a spark cluster, give Spark a try else you have to write a code which is optimized for your use case thanks -Kohki On Fri, Feb 24, 2017 at 6:22 PM, Tianji Li <skyah...@gmail.com> wrote: > Hi there, > > Can anyone give a good explanation in what cases Kafka Streams is > preferred, and in what cases Sparking Streaming is better? > > Thanks > Tianji > -- Kohki Nishio
