Hello everyone,
thanks for this KIP. It's indeed a long standing feature request to
support multi-partition TDD, so I am very happy to see this KIP (sorry
for ignoring it for way too long...)
Couple of question:
MJS1: The KIP states:
Each time a input topic is created with a partition number higher than the max
partition number already defined, the Tasks and GlobalTasks will be updated.
This sounds quite complex? Atm, because TTD is single-partitioned, one
can call `createInputTopic()` and `createOutputTopic()` at any time, and
pipe record through the topology interleaved. Because there is only a
single partition/task, it does not make things complicated.
However, with multiple partitions and task, I am worried that we might
introduce unnecessary complexity (potentially in both the TDD impl, as
well as for the user). Could it make sense to add a dedicated setup
phase? Ie, all input/output topic must be created before we allow to
pipe record? The control flow would be something like:
TopologyTestDriver driver = new TopologyTestDriver(...);
driver.createInputTopic(...)
driver.createOutputTopic(...)
// created TestInputTopics / TestOutputTopics cannot be used yet
driver.init(); // sets up the driver and creates tasks
// now TestInputTopic / TestOutputTopic can be used
MJS2: I am somewhat concerned about the number of overloads we are
adding to `TestInputTopic` and `TestOutputTopic`.
Would it be simpler to instead modify `TestRecord` and allow to set the
partition there, to avoid all the overloads to `pipeInput`? I would
assume that most tests would rely on `DefaultPartitioner` anyway, and it
seems ok to force the usage of `TestRecord` if a partitions must be set
explicitly?
For `TestOutputTopic` the partition number of each record would be set
on the `TestRecord` again (for `readRecord() and `readRecordsToList()`.
We could also introduce `TestOutputTopicPartition` and add
`TestOutputTopic.partition(int)` to allow reading data from a specific
partition?
-Matthias
On 11/12/25 11:48 PM, Sebastien Viale wrote:
Hi Everyone,
I would like to start a discussion on KIP-1238: Multipartition for
TopologyTestDriver in Kafka Streams
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1238%3A+Multipartition+for+TopologyTestDriver+in+Kafka+Streams
This KIP proposes to introduce multi-partition support in the
TopologyTestDriver, enabling more accurate and convenient stream testing while
improving automated unit test coverage.
Regards,
Julien & Adam & Marie-Laure & Sébastien
