________________________________
From: thomas.k...@t-systems.com <thomas.k...@t-systems.com>
Sent: Monday, May 14, 2018 12:13 PM
To: users@kafka.apache.org
Subject: WG: Exactly Once, Producer configuration
Hi,
the following article describes how to configure Apache Kafka system to enforce
exactly once approach:
https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/
[https://www.confluent.io/wp-content/uploads/image2.png]<https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/>
Exactly-once Semantics is Possible: Here's How Apache
...<https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/>
www.confluent.io
Exactly-once is a hard problem to solve, but we've done it. Available now in
Apache Kafka 0.11, exactly-once semantics.
One recommendation states that parameter transactional.id should be set in
producer.properties file which usually exists only once per broker.
Now I have the following questions. Our system consists of 4 modules which
interact in a sequential way by exchanging messages using Kafka topics
A --> B --> C --> D
I.e. modules A, B and C act as message producers and modules B, C and D act as
message consumers (and all modules do some application specific useful
processing).
1) Do we need separate producer/consumer configurations for these modules
and if so where and how do we have to configure them (as producer configuration
files are related to the number of brokers and not to the number of modules)?
1a)the relationship between producer/consumer isnt necessarily a straight line
1a)similar-tasked producers can implement multiple partitions for one produce
1a)Consumers can pull data from a topic from either leader or follower broker
as illustrated here
https://www.tutorialspoint.com/apache_kafka/apache_kafka_quick_guide.htm
[https://www.tutorialspoint.com/apache_kafka/images/point_to_point_messaging_system.jpg]<https://www.tutorialspoint.com/apache_kafka/apache_kafka_quick_guide.htm>
Apache Kafka Quick Guide - Current Affairs 2018, Apache
...<https://www.tutorialspoint.com/apache_kafka/apache_kafka_quick_guide.htm>
www.tutorialspoint.com
Apache Kafka Quick Guide - Learn Apache kafka starting from the Introduction,
Fundamentals, Cluster Architecture, Workflow, Installation Steps, Basic
Operations, Simple Producer Example, Consumer Group Example, Integration with
Storm, Integration with Spark, Real Time Application(Twitter), Tools,
Applications.
2) Does transcational.id "only" cover technical aspects of Apache Kafka or
does it cover application aspects as well (i.e. is my assumption which led to
question 1 correct)?
3) In case it is sufficient to configure transactional.id parameter in
producer.properties files and we have more than one broker (and with that more
than one producer.properties file). Do the values of the parameter
transactional.id have to be equal or do there have to be different (pairwise
disjoint) values?
I would be glad to get answers to these questions.
Kind regards,
Thomas Kass
