Hi team, This is our current use case: Assume there is a topic with multiple partitions. 1 Create a connector first and create multiple streams from the connector for a topic. 2 Create multiple threads, one for each stream. You can assume the thread's job is to save the message into the database. 3 When it is time to commit offsets, all threads have to synchronize on a barrier before committing the offsets. This is to ensure no message loss in case of process crash.
As all threads need to synchronize before committing, it is not efficient. This is a workaround: 1 We can create multiple connectors. From each connector create only one stream. 2 Use a single thread for a stream. In this case, the connector in each thread can commit freely without any dependence on the other threads. Is this the right way to go? Will it introduce any dead lock when multiple connectors commit at the same time? It would be great to allow committing on a per stream basis. Regards, Libo
