In relation to producer idempotency Kafka documentation says:
... Since each new instance of a producer is assigned a new, unique, PID, we can only guarantee idempotent production within a single producer session. Does it mean that when we build a producer sourcing the data from whatever source (MySql for example) we need to take care explicitly of this problem and put in place effective counter measures, like checking what was the last message actually produced successfully on Kafka ? What if we use transactions, even in the case we don't need really need transactional logic on writing messages to different partitions in an all or nothing fashion ? Would using transactions in our producer get rid of the idempotency issue ? Thanks!
