Re: [DISCUSS] KIP-451: Make TopologyTestDriver output iterable

2019-04-25 Thread Patrik Kleindl
Hi It's fine by me, I'll consider doing the PR a good exercise ;-) I'll discard it for the moment, if there should be any hold-up for KIP-456 to make it into the 2.3 release then we could still revive this one because it is mostly done already. best regards and thanks to Jukka for his great work! P

Re: [DISCUSS] KIP-451: Make TopologyTestDriver output iterable

2019-04-25 Thread Matthias J. Sax
Thanks for the discussion! After skipping over KIP-456, I was also wondering what the overlap of both KIPs are, and if KIP-456 might actually subsume KIP-451. If we all agree on this, I would recommend to discard KIP-451 in favor of KIP-456. I'll also follow up on KIP-456 DISCUSS thread to get s

Re: [DISCUSS] KIP-451: Make TopologyTestDriver output iterable

2019-04-25 Thread Patrik Kleindl
Hi As discussed, if the preferred option is to consume the records always I will change both methods in KIP-451 accordingly and also switch them to return a List. This would be a bit redundant with Jukkas proposal in KIP-456 so the question is if KIP-451 should be scraped in favor of KIP-456 which

Re: [DISCUSS] KIP-451: Make TopologyTestDriver output iterable

2019-04-24 Thread Jukka Karvanen
Hi, I played around with Patrick's KAFKA-8200 branch and I tested it with combined with my draft version of KIP-456. Some comments: These two version of iterableOutput methods are working now differently, if you reuse same fetched Iterable object after piping in new inputs. Version without serde

Re: [DISCUSS] KIP-451: Make TopologyTestDriver output iterable

2019-04-19 Thread Patrik Kleindl
Hi Matthias Seems I got a bit ahead of myself. With option C my aim was a simple alternative which gives back all output records that have happened up to this point (and which have not been removed by calls to readOutput). Based on that the user can decide how to step through or compare the records

Re: [DISCUSS] KIP-451: Make TopologyTestDriver output iterable

2019-04-18 Thread Matthias J. Sax
I am not sure if (C) is the best option to pick. What is the reasoning to suggest (C) over the other options? It seems that users cannot clear buffered output using option (C). This might it make difficult to write tests. The original Jira tickets suggest: > which returns either an iterator or

Re: [DISCUSS] KIP-451: Make TopologyTestDriver output iterable

2019-04-18 Thread Patrik Kleindl
Hi John Thanks for your feedback It's C, it does not consume the messages in contrast to the readOutput. Is it a requirement to do so? That's why I picked a different name so the difference is more noticeable. I will add that to the JavaDoc. I see your point regarding future changes, that's why I

Re: [DISCUSS] KIP-451: Make TopologyTestDriver output iterable

2019-04-18 Thread John Roesler
Hi, Patrik, Thanks for this proposal! I have one question, which I didn't see addressed by the KIP. Currently, when you call `readOutput`, it consumes the result (removes it from the test driver's output). Does your proposed method: A: consume the whole output stream for that topic "atomically" w

Re: [DISCUSS] KIP-451: Make TopologyTestDriver output iterable

2019-04-17 Thread Patrik Kleindl
Hi all Unless someone has objections I will start a VOTE thread tomorrow. The KIP adds two methods to the TopologyTestDriver and has no conflicts for existing users. PR https://github.com/apache/kafka/pull/6556 is already being reviewed. Side-note: https://cwiki.apache.org/confluence/display/KAFK

Re: [DISCUSS] KIP-451: Make TopologyTestDriver output iterable

2019-04-11 Thread Patrik Kleindl
Hi Matthias Thanks for the questions. Regarding the return type: Iterable offers the option of being used in a foreach loop directly and it gives you access to the .iterator method, too. (ref: https://www.techiedelight.com/differences-between-iterator-and-iterable-in-java/ ) To return a List obj

Re: [DISCUSS] KIP-451: Make TopologyTestDriver output iterable

2019-04-11 Thread Matthias J. Sax
Thanks for the KIP! Overall, this makes sense and can simplify testing. What I am wondering is, why you suggest to return an `Iterable`? Maybe returning an `Iterator` would make more sense? Or a List? Note that the order of emits matters, thus returning a generic `Collection` would not seem to be

[DISCUSS] KIP-451: Make TopologyTestDriver output iterable

2019-04-11 Thread Patrik Kleindl
Hi everyone, I would like to start the discussion on this small enhancement of the TopologyTestDriver. https://cwiki.apache.org/confluence/display/KAFKA/KIP-451%3A+Make+TopologyTestDriver+output+iterable Pull request is available at https://github.com/apache/kafka/pull/6556 Any feedback is welc