Hey Sang, unfortunately we couldn't make it in 2.6. Do you still plan to work on this KIP?
On Thu, May 14, 2020 at 6:49 PM Boyang Chen <reluctanthero...@gmail.com> wrote: > Hey Sang, seems this thread has been quiet, are you still working on this > KIP? > > On Sat, Mar 7, 2020 at 3:54 PM Matthias J. Sax <matth...@confluent.io> > wrote: > >> Thanks for the KIP Sang! >> >> I have a couple of more comments about the wiki page: >> >> (1) The "Public Interface" section should only list the new stuff. This >> KIP does not change anything with regard to the existing options >> `--input-topic` or `--intermediate-topic` and thus it's just "noise" to >> have them in this section. Only list the new option >> `allInputTopicsOption`. >> >> (2) Don't post code, ie, the implementation of private methods. KIPs >> should only describe public interface changes. >> >> (3) The KIP should describe that we intend to use >> `describeConsumerGroups` calls to discover the topic names -- atm, it's >> unclear from the KIP how the new feature actually works. >> >> (4) If the new flag is used, we will discover input and intermediate >> topics. Hence, the name is miss leading. We could call it >> `--all-user-topics` and explain in the description that "user topics" >> are input and intermediate topics for this case (in general, also output >> topics are "user topics" but there is nothing to be done for output >> topics). Thoughts? >> >> >> -Matthias >> >> On 1/27/20 6:35 AM, Sang wn Lee wrote: >> > thank you John Roesle >> > >> > It is a good idea >> > "—all-input-topics" >> > >> > I agree with you >> > >> > I'll update right away >> > >> > >> > On 2020/01/24 14:14:17, "John Roesler" <vvcep...@apache.org> wrote: >> >> Hi all, thanks for the explanation. I was also not sure how the kip >> would be possible to implement. >> >> >> >> No that it does seem plausible, my only feedback is that the command >> line option could align better with the existing one. That is, the existing >> option is called “—input-topics”, so it seems like the new one should be >> called “—all-input-topics”. >> >> >> >> Thanks, >> >> John >> >> >> >> On Fri, Jan 24, 2020, at 01:42, Boyang Chen wrote: >> >>> Thanks Sophie for the explanation! I read Sang's PR and basically he >> did >> >>> exactly what you proposed (check it here >> >>> <https://github.com/apache/kafka/pull/7948/files> in case I'm wrong). >> >>> >> >>> I think Sophie's response answers Gwen's question already, while in >> the >> >>> meantime for a KIP itself we are not required to mention all the >> internal >> >>> details about how to make the changes happen (like how to actually >> get the >> >>> external topics), considering the change scope is pretty small as >> well. But >> >>> again, it would do no harm if we mention it inside Proposed Change >> session >> >>> specifically so that people won't get confused about how. >> >>> >> >>> >> >>> On Thu, Jan 23, 2020 at 8:26 PM Sophie Blee-Goldman < >> sop...@confluent.io> >> >>> wrote: >> >>> >> >>>> Hi all, >> >>>> >> >>>> I think what Gwen is trying to ask (correct me if I'm wrong) is how >> we can >> >>>> infer which topics are associated with >> >>>> Streams from the admin client's topic list. I agree that this >> doesn't seem >> >>>> possible, since as she pointed out the >> >>>> topics list (or even description) lacks the specific information we >> need. >> >>>> >> >>>> What we could do instead is use the admin client's >> >>>> `describeConsumerGroups` API to get the information >> >>>> on the Streams app's consumer group specifically -- note that the >> Streams >> >>>> application.id config is also used >> >>>> as the consumer group id, so each app forms a group to read from the >> input >> >>>> topics. We could compile a list >> >>>> of these topics just by looking at each member's assignment (and >> even check >> >>>> for a StreamsPartitionAssignor >> >>>> to verify that this is indeed a Streams app group, if we're being >> >>>> paranoid). >> >>>> >> >>>> The reset tool actually already gets the consumer group description, >> in >> >>>> order to validate there are no active >> >>>> consumers in the group. We may as well grab the list of topics from >> it >> >>>> while it's there. Or did you have something >> >>>> else in mind? >> >>>> >> >>>> On Sat, Jan 18, 2020 at 6:17 PM Sang wn Lee <ssangdd...@gmail.com> >> wrote: >> >>>> >> >>>>> Thank you >> >>>>> >> >>>>> I understand you >> >>>>> >> >>>>> 1. admin client has topic list >> >>>>> 2. applicationId can only have one stream, so It won't be a problem! >> >>>>> 3. For example, --input-topic [reg] >> >>>>> Allowing reg solves some inconvenience >> >>>>> >> >>>>> >> >>>>> On 2020/01/18 18:15:23, Gwen Shapira <g...@confluent.io> wrote: >> >>>>>> I am not sure I follow. Afaik: >> >>>>>> >> >>>>>> 1. Topics don't include client ID information >> >>>>>> 2. Even if you did, the same ID could be used for topics that are >> not >> >>>>> Kafka >> >>>>>> Streams input >> >>>>>> >> >>>>>> The regex idea sounds doable, but I'm not sure it solves much? >> >>>>>> >> >>>>>> >> >>>>>> On Sat, Jan 18, 2020, 7:12 AM Sang wn Lee <ssangdd...@gmail.com> >> >>>> wrote: >> >>>>>> >> >>>>>>> Thank you >> >>>>>>> Gwen Shapira! >> >>>>>>> We'll add a flag to clear all topics by clientId >> >>>>>>> It is ‘reset-all-external-topics’ >> >>>>>>> >> >>>>>>> I also want to use regex on the input topic flag to clear all >> >>>> matching >> >>>>>>> topics. >> >>>>>>> >> >>>>>>> On 2020/01/17 19:29:09, Gwen Shapira <g...@confluent.io> wrote: >> >>>>>>>> Seem like a very nice improvement to me. But I have to admit >> that I >> >>>>>>>> don't understand how this will how - how could you infer the >> input >> >>>>>>>> topics? >> >>>>>>>> >> >>>>>>>> On Thu, Jan 16, 2020 at 10:03 AM Sang wn Lee < >> ssangdd...@gmail.com >> >>>>> >> >>>>>>> wrote: >> >>>>>>>>> >> >>>>>>>>> Hello, >> >>>>>>>>> >> >>>>>>>>> Starting this thread to discuss KIP-560: >> >>>>>>>>> wiki link : >> >>>>>>>>> >> >>>>>>> >> >>>>> >> >>>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-560%3A+Auto+infer+external+topic+partitions+in+stream+reset+tool >> >>>>>>>>> >> >>>>>>>>> I'm newbie >> >>>>>>>>> I would like to receive feedback on the following features! >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>>> >> >>>>> >> >>>> >> >>> >> >> >> >>
