Re: [DISCUSS] KIP-594: Expose output topic names from TopologyTestDriver

2020-04-23 Thread Matthias J. Sax
Thanks Andi! I buy your argument. For exposing topics names via `TopologyDescription` I create a JIRA for tracking: https://issues.apache.org/jira/browse/KAFKA-9913 -Matthias On 4/20/20 8:07 AM, Andy Coates wrote: > Hey all, > >> One other point, I’m actually mildly concerned about what you sa

Re: [DISCUSS] KIP-594: Expose output topic names from TopologyTestDriver

2020-04-20 Thread Andy Coates
Hey all, > One other point, I’m actually mildly concerned about what you say regarding the outputs to the repartition and changelog topics being captured and verified later. I would consider the data in these topics to be a private interface, and would strongly discourage user code to depend on it

Re: [DISCUSS] KIP-594: Expose output topic names from TopologyTestDriver

2020-04-16 Thread John Roesler
Thanks Matthias, It sounds like your proposal would be instead to add methods to TopologyDescription: getOuputTopicNames(), getInternalTopicNames(), and getInputTopicNames(). This sounds good to me. For one thing, it nicely resolved the ambiguity with the method name. What do you think about

Re: [DISCUSS] KIP-594: Expose output topic names from TopologyTestDriver

2020-04-16 Thread Matthias J. Sax
John, the "dynamic routing" case is certainly interesting. However, your argument about "simulating brokers" and verify which topics are created does not really hold up IMHO, because the names of all internal topics are contained in the TopologyDescription, too. To be fair, the `TopologyDescripti

Re: [DISCUSS] KIP-594: Expose output topic names from TopologyTestDriver

2020-04-15 Thread John Roesler
Hi Andy, Thanks for the KIP! To Matthias’s concern, I have to agree that the motivation doesn’t build a particularly strong case for the KIP, and people often look back to these documents to understand why something was done, and done a particular way. So, it would be nice to flesh it out a bi

Re: [DISCUSS] KIP-594: Expose output topic names from TopologyTestDriver

2020-04-14 Thread Matthias J. Sax
Andy, thanks for the KIP. The motivation is a little unclear to me: > This information allows all the outputs of a test run to be captured without > prior knowledge of the output topics. Given that the TTD users writes the `Topology` themselves, they should always have prior knowledge what outp

[DISCUSS] KIP-594: Expose output topic names from TopologyTestDriver

2020-04-14 Thread Andy Coates
Hey all, I would like to start off the discussion for KIP-594: https://cwiki.apache.org/confluence/display/KAFKA/KIP-594%3A+Expose+output+topic+names+from+TopologyTestDriver This KIP proposes to expose the names of the topics a topology produces records during a test run from the TopologyTestDrive