Hi Viktor, Sounds good. If you want to propose a way of improving the metadata protocol so that "[deleted]" could be supported, you could probably create that KIP in parallel.
The last KIP in that area that I can remember is KIP-142, which didn't get adopted (yet?) https://cwiki.apache.org/confluence/display/KAFKA/KIP-142%3A+Add+ListTopicsRequest+to+efficiently+list+all+the+topics+in+a+cluster There have been other discussions though. In general there are a lot of features that would be nice to have in the metadata protocol (pagniation, regexes, skip stuff we don't need). best, Colin On Tue, Oct 16, 2018, at 10:11, Viktor Somogyi-Vass wrote: > Hi Colin, > > Thanks, it makes sense and simplifies this KIP tremendously. I'll move this > section to the rejected alternatives with a note that KIP-142 will have > this feature. > On a similar note: is there a KIP for describe topics protocol or have you > been thinking about it? I guess there it's the same problem, we often don't > want to forward the entire metadata. > > Viktor > > On Fri, Oct 12, 2018 at 12:03 PM Colin McCabe <cmcc...@apache.org> wrote: > > > Hi Viktor, > > > > Thanks for bumping this thread. > > > > I think we should just focus on transitioning the TopicCommand to use > > AdminClient, and talk about protocol changes in a separate KIP. Protocol > > changes often involve a lot of discussion. This does mean that we couldn't > > implement the "list topics under deletion" feature when using AdminClient > > at the moment. We could add a note to the tool output indicating this. > > > > We should move the protocol discussion to a separate thread. Probably > > also look at KIP-142 as well. > > > > best, > > Colin > > > > > > On Tue, Oct 9, 2018, at 07:45, Viktor Somogyi-Vass wrote: > > > Hi All, > > > > > > Would like to bump this as the conversation sank a little bit, but more > > > importantly I'd like to validate my plans/ideas on extending the Metadata > > > protocol. I was thinking about two other alternatives, namely: > > > 1. Create a ListTopicUnderDeletion protocol. This however would be > > > unnecessary: it'd have one very narrow functionality which we can't > > extend. > > > I'd make sense to have a list topics or describe topics protocol where we > > > can list/describe topics under deletion but for normal listing/describing > > > we already use the metadata, so it would be a duplication of > > functionality. > > > 2. DeleteTopicsResponse could return the topics under deletion if the > > > request's argument list is empty which might make sense at the first > > look, > > > but actually we'd mix the query functionality with the delete > > functionality > > > which is counterintuitive. > > > > > > Even though most clients won't need these "limbo" topics (which are under > > > deletion) in the foreseeable future, it can be considered as part of the > > > cluster state or metadata and to me it makes sense. Also it doesn't have > > a > > > big overhead in the response size as typically users don't delete topics > > > too often as far as I experienced. > > > > > > I'd be happy to receive some ideas/feedback on this. > > > > > > Cheers, > > > Viktor > > > > > > > > > On Fri, Sep 28, 2018 at 4:51 PM Viktor Somogyi-Vass < > > viktorsomo...@gmail.com> > > > wrote: > > > > > > > Hi All, > > > > > > > > I made an update to the KIP. Just in short: > > > > Currently KafkaAdminClient.describeTopics() and > > > > KafkaAdminClient.listTopics() uses the Metadata protocol to acquire > > topic > > > > information. The returned response however won't contain the topics > > that > > > > are under deletion but couldn't complete yet (for instance because of > > some > > > > replicas offline), therefore it is not possible to implement the > > current > > > > command's "marked for deletion" feature. To get around this I > > introduced > > > > some changes in the Metadata protocol. > > > > > > > > Thanks, > > > > Viktor > > > > > > > > On Fri, Sep 28, 2018 at 4:48 PM Viktor Somogyi-Vass < > > > > viktorsomo...@gmail.com> wrote: > > > > > > > >> Hi Mickael, > > > >> > > > >> Thanks for the feedback, I also think that many customers wanted this > > for > > > >> a long time. > > > >> > > > >> Cheers, > > > >> Viktor > > > >> > > > >> On Fri, Sep 28, 2018 at 11:45 AM Mickael Maison < > > mickael.mai...@gmail.com> > > > >> wrote: > > > >> > > > >>> Hi Viktor, > > > >>> Thanks for taking this task! > > > >>> This is a very nice change as it will allow users to use this tool in > > > >>> many Cloud environments where direct zookeeper access is not > > > >>> available. > > > >>> > > > >>> > > > >>> On Thu, Sep 27, 2018 at 10:34 AM Viktor Somogyi-Vass > > > >>> <viktorsomo...@gmail.com> wrote: > > > >>> > > > > >>> > Hi All, > > > >>> > > > > >>> > This is the continuation of the old KIP-375 with the same title: > > > >>> > > > > >>> > > https://lists.apache.org/thread.html/dc71d08de8cd2f082765be22c9f88bc9f8b39bb8e0929a3a4394e9da@%3Cdev.kafka.apache.org%3E > > > >>> > > > > >>> > The problem there was that two KIPs were created around the same > > time > > > >>> and I > > > >>> > chose to reorganize mine a bit and give it a new number to avoid > > > >>> > duplication. > > > >>> > > > > >>> > The KIP summary here once again: > > > >>> > > > > >>> > I wrote up a relatively simple KIP about improving the Kafka > > protocol > > > >>> and > > > >>> > the TopicCommand tool to support the new Java based AdminClient and > > > >>> > hopefully to deprecate the Zookeeper side of it. > > > >>> > > > > >>> > I would be happy to receive some opinions about this. In general I > > > >>> think > > > >>> > this would be an important addition as this is one of the few left > > but > > > >>> > important tools that still uses direct Zookeeper connection. > > > >>> > > > > >>> > Here is the link for the KIP: > > > >>> > > > > >>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-377%3A+TopicCommand+to+use+AdminClient > > > >>> > > > > >>> > Cheers, > > > >>> > Viktor > > > >>> > > > >> > >
