The different versions of the connector correspond to different versions of Kafka. If you are using Kafka 0.8 use 0.8 connector, etc. Versions of the connector after 0.10 support exactly once delivery, versions prior to that only offer at least once delivery.
Kafka supports distributed processing through deployment of multiple brokers. Each topic can be divided into partitions and those allocated to brokers to get distributed processing on a cluster. I have not used its stream processing API yet, but I assume it runs within the brokers working on partitions of a topic. Cluster deployment for Flink and Kafka can be as a stand alone cluster (manual deployment to a bunch of machines, via custom built AMI, etc), or through a cluster manager like Mesos, Yarn, Kubernetes which will manage the work performed on the machines in a cluster. I have not seen any good tutorials on multi-machine deployments, there are a few suggesting how to do so with kubernetes for Flink only, but none I have found for flink+kafka. For my proof of concept it was just easier to manually build out 4 machines. The install for kafka and flink are simple and getting java and maven on a base ubuntu image on AWS is quick, so I can manually build out a machine in about 5 minutes. Michael > On Apr 22, 2018, at 2:22 AM, m@xi <makisnt...@gmail.com> wrote: > > Hi Piotr! > > In this page of the documentation [1] I can see the different versions of > Kafka Connectors, but I am now learning about Kafka so some help would be > valuable. > > 1 -- Are 0.8, 0.9, 0.11 etc different version of the same thing or do they > same thing? I mean does 0.11 offers everything the 0.8 already has? > > 2 -- I would like to use Kafka Streams API in my flink cluster [2], which is > used for standalone clusters if I am not mistaken, i.e. one node only by > default. > > 3 -- Can you give some hints and explain briefly about the cluster > deployment with many machines? I mean what is Yarn, Mesos etc. I think they > are "coordinators" of the cluster. But now that I would like to test my > algorithm on a real cluster with several machines I would like some hints on > which one should I use. What about Kubernetes and Docker [3] ? > > Thanks a lot in advance! > > Best, > Max > > [1] -- > https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/connectors/kafka.html > > [2] -- https://kafka.apache.org/documentation/streams/ > > [3] -- > https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/deployment/cluster_setup.html > > > > -- > Sent from: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
