This is probably a very subjective question, but nevertheless here are my reasons for choosing Flink over KStreams or even Spark.
a) KStreams couples you tightly to Kafka, and I personally don't want my stream processing engine to be married to my message bus. There are other (even better alternatives) to Kafka given your requirements, so I thought it best to keep these two separate. b) Fault Tolerance on KStreams relies on intermittent Kafka topics. The increased network and I/O for messages to/from Kafka for bookkeeping was to high for my use cases. The concept of local state and asynchronous barrier snapshotting are key principles as to why Flink does better in this area. You can't just bolt that on to a framework that didnt get those concepts right from the beginning. c) Lack of fine grained control over operator parallelism. A graph may have different computation hot spots where it maybe worth fanning out to process over multiple nodes in parallel. In KStreams as well as Spark you can't really do that. In Kstreams the parallelism is tightly bound to the number of partitions. This means that your entire processing graph needs to align well with the way you've partitioned your data in order to maximize throughput. You cant (or maybe even don't) want to do that. d) The API lends itself well to describing DAGs. I found Spark and KStreams not so intuitive in this regard. HTH Tim On Mon, May 20, 2019, 5:30 PM Peter Groesbeck <peter.groesb...@gmail.com> wrote: > Hi folks, > > I'm hoping to get some deeper clarification on which framework, Flink or > KStreams, to use in a given scenario. I've read over the following blog > article which I think sets a great baseline understanding of the > differences between those frameworks but I would like to get some outside > opinions: > > https://www.confluent.io/blog/apache-flink-apache-kafka-streams-comparison-guideline-users/ > > My understanding of this article is that KStreams works well as an > embedded library in a microservice, API layer, or as a standalone > application at a company with centralized deployment infrastructure, such > as a shared Kubernetes cluster. > > In this case, there is discussion around deploying KStreams as a > standalone application stack backed by EC2 or ECS, and whether or not Flink > is better suited to serve as the data transformation layer. We already do > run Flink applications on EMR. > > The point against Flink is that it requires a cluster whereas KStreams > does not, and can be deployed on ECS or EC2. We do not have a centralized > deployment team, and will still have to maintain either the CNF > Stack/AutoScaling Group or EMR Cluster ourselves. > > What are some of the advantages of using Flink over KStreams standalone? > > The Job management UI is one that comes to mind, and another are some of > the more advanced API options such as CEP. But I would really love to hear > the opinions of people who are familiar with both. In what scenarios would > you choose one over the other? Is it advisable or even preferable to > attempt to deploy KStreams as it's own stack and avoid the complexity of > maintaining a cluster? > > Thanks, > Peter >
