RE: Kafka connect FieldPartitioner with scheduled rotation

2019-02-05 Thread Manu Jacob
Hi Pere, I am using 3.3.0. By the way, I could resolve this issue by setting the “timezone” property on the connector. It is a bit strange as I thought that value is used only by Time based partitioner(which could have defaulted to UTC) and not a requirement for scheduled rotation. Not sure if

Minimizing global store restoration time

2019-02-05 Thread Taylor P
Hi, I am having issues with the global store taking a very long time to restore during startup of a Kafka Streams 2.0.1 application. The global store is backed by a RocksDB persistent store and is added to the Streams topology in the following manner: https://pastebin.com/raw/VJutDyYe The global s

Kafka exactly-once with multiple producers

2019-02-05 Thread Tim Jiang
Hi, I'm read this about how Kafka transaction works: https://www.confluent.io/blog/transactions-apache-kafka/ Please help me understand what happens when there are multiple producers are writing to the same topic concurrently? Should all of them use the same transaction id, and if so, does Kafka gu

Re: Kafka exactly-once with multiple producers

2019-02-05 Thread Matthias J. Sax
Each producer will need to use it's own `transactional.id`. Otherwise, one producer would fence-off and "block" the other. Both producers can start transactions independently from each other, and also commit independently (or abort, or a mix of commit/abort between both). Messages of both producer