Hi Sönke, In KAFKA-1774 I have created a Kafka Shell java tool that unfortunately didn't get much attention so far from the creators of the jira. It works similarly to what Colin mentioned, like "kafka.sh topics create -n my-topic -p 3 -r 3" or has long names like "kafka.sh topics create --name my-topic --partitions 3 --replicas 3". The bootstrap server everywhere defaults to :9092 or reads up the configs from a path so in many scenarios you can omit it, also it uses the admin client of course and all in all provides a much better experience than the current tools. It has interactive mode and help too. Wanted to implement "code completion" too but for that I'd have to exercise the code a little bit more :). It is currently based on a quite old trunk but if you think it's interesting I can rebase it and we can continue with raising a KIP. https://github.com/viktorsomogyi/kafka/tree/kafka_shell
Viktor On Tue, May 7, 2019 at 11:10 AM Sönke Liebau <soenke.lie...@opencore.com.invalid> wrote: > Hi Colin, > > thanks! > While I personally don't like the current command line tools I > realistically think we'll be stuck with them for a while yet, so a cleanup > might make sense. > So I'll start looking into that. > > Regarding a central entrypoint, that would indeed be brilliant and I'd love > to work on that, but I currently have enough other open things to look at, > so I won't draw that one to myself as well for now. > > But I'll keep it in mind for when some time frees up :) > > Best regards, > Sönke > > > > Colin McCabe <cmcc...@apache.org> schrieb am Di., 7. Mai 2019, 00:56: > > > On Mon, May 6, 2019, at 10:21, Sönke Liebau wrote: > > > Hi Colin, > > > > > > it was my intention to keep the structure of the commands mostly intact > > > while doing the refactoring - if that is possible, have not really > > checked > > > yet to be honest. > > > > > > But what I wanted to try and do is recreate the current parsing with > > > argparse as much as possible. And in the process simply adding > synonyms, > > > for example make the kafka-console-producer understand a > > > bootstrap-parameter in addition to broker-list. > > > There is a bit of custom logic about which parameters go together etc. > in > > > the current classes, so output may look different here and there, but > in > > > principle I do believe that it should be possible to recreate the > current > > > structure. > > > > Sounds like a good idea. Thanks for the clarification. > > > > > > > > If there is an appetite for a new, hadoop-like entrypoint anyway, then > > all > > > of this might be "wasted" effort, or rather effort better spent though, > > you > > > are right. > > > > I don't think anyone is working on a new entry point right now -- or if > > they are, they haven't said anything yet :) > > > > I just wanted to mention it as a possible approach in case you wanted to > > do a bigger project. > > > > best, > > Colin > > > > > > > > Best regards, > > > Sönke > > > > > > > > > > > > On Mon, May 6, 2019 at 7:13 PM Colin McCabe <cmcc...@apache.org> > wrote: > > > > > > > Hi Sönke, > > > > > > > > #2 is a bit tough because people have come to rely on the way the > > commands > > > > are structured right now. > > > > > > > > If we want to make big changes, it might be easier just to create a > > > > separate tool and deprecate the old one(s). One thing we've talked > > about > > > > doing in the past is creating a single entry point for all the tool > > > > functionality, kind of like hadoop did with the "hadoop" command Or > > git > > > > with the "git" command, etc. Then we could deprecate the standalone > > > > commands and remove them after enough time had passed-- kind of like > > the > > > > old consumer. > > > > > > > > On the other hand, a more incremental change would be standardizing > > flags > > > > a bit. So for example, at least setting it up so that there is a > > standard > > > > way of supplying bootstrap brokers, etc. We could keep the old flags > > > > around for a while as variants to ease the transition. > > > > > > > > best, > > > > Colin > > > > > > > > > > > > On Sun, May 5, 2019, at 00:54, Sönke Liebau wrote: > > > > > Hi Colin, > > > > > > > > > > I totally agree! Especially the differently named bootstrap server > > > > options > > > > > have been annoying me a long time. > > > > > > > > > > I'd propose a two-step approach: > > > > > 1. Add new default options objects similar to CommandLineUtils and > > > > > CommandDefaultOptions (based on argparse4j) but in the clients > > project, > > > > as > > > > > this is referenced by all command line tools as far as I can tell > > > > > 2. Refactor tools one by one to use these new helper classes (and > > thus > > > > > argparse) and add standardized synonyms for parameters as necessary > > > > > > > > > > I think for step 1 we can get away with no KIP, as this doesn't > > change > > > > any > > > > > public interfaces or behavior. > > > > > Step 2 probably needs a KIP as we are adding new parameters? We can > > pick > > > > up > > > > > KIP-14 again for that I think. A lot of work has been done on that > > > > already. > > > > > > > > > > Does that sound useful to everybody? > > > > > > > > > > Best regards, > > > > > Sönke > > > > > > > > > > > > > > > On Thu, Apr 18, 2019 at 1:44 AM Colin McCabe <cmcc...@apache.org> > > wrote: > > > > > > > > > > > If we are going to standardize on one argument parsing library, > it > > > > should > > > > > > certainly be argparse4j, I think. > > > > > > argparse4j is simply a better argument parsing library with > > support > > > > for > > > > > > more features. One example is mutually exclusive options. > > argparse4j > > > > > > supports this with MutuallyExclusiveGroup. jopt doesn't support > > this, > > > > so > > > > > > when it is needed, we have to add extra code to manually check > that > > > > > > mutually exclusive options are not set. > > > > > > > > > > > > argparse4j also has subcommands. If you want something like "git > > add" > > > > > > with some set of flags, and "git remove" with another, you can do > > this > > > > with > > > > > > argparse4j, but not with jopt. This would be very helpful for > > > > clearing up > > > > > > confusion in a lot of our shell scripts which have accumulated > > dozens > > > > of > > > > > > arguments, most of which are only relevant to a very specific > > > > operation. > > > > > > But you just can't do it with jopt. > > > > > > > > > > > > Just to give an example, argparse4j with subcommands would allow > > you to > > > > > > run something like ./kafka-topics.sh list --help and get just > > options > > > > that > > > > > > were relevant for listing topics, not the full dozens of options > > that > > > > might > > > > > > relate to adding topics, removing them, etc. > > > > > > > > > > > > To be honest, though, what would help users the most is > > standardizing > > > > the > > > > > > option flags across tools. We should have a standard way of > > specifying > > > > > > bootstrap brokers, for example. (We can continue to support the > > old > > > > > > synonyms for a while, of course.) > > > > > > > > > > > > best, > > > > > > Colin > > > > > > > > > > > > > > > > > > On Wed, Apr 17, 2019, at 08:56, Guozhang Wang wrote: > > > > > > > I took another look at the PR itself and I think it would be > > great to > > > > > > have > > > > > > > this cleanup too -- I cannot remember at the beginning why we > > > > gradually > > > > > > > moved to different mechanism (argparse4j) for different cmds, > if > > > > there's > > > > > > no > > > > > > > rationales behind it we should just make them consistent. > > > > > > > > > > > > > > Thanks for driving this! > > > > > > > > > > > > > > Guozhang > > > > > > > > > > > > > > On Wed, Apr 17, 2019 at 7:19 AM Ryanne Dolan < > > ryannedo...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > > Sönke, I'd find this very helpful. It's annoying to keep > track > > of > > > > which > > > > > > > > commands use which form -- I always seem to guess wrong. > > > > > > > > > > > > > > > > Though I don't think there is any reason to deprecate > existing > > > > forms, > > > > > > e.g. > > > > > > > > consumer.config vs consumer-config. I think it's perfectly > > > > reasonable > > > > > > to > > > > > > > > have multiple spellings of the same arguments. I don't really > > see a > > > > > > > > downside to keeping the aliases around indefinitely. > > > > > > > > > > > > > > > > Ryanne > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Apr 17, 2019, 7:07 AM Sönke Liebau > > > > > > > > <soenke.lie...@opencore.com.invalid> wrote: > > > > > > > > > > > > > > > > > Hi everybody, > > > > > > > > > > > > > > > > > > Jason and I were recently discussing command line argument > > > > parsing on > > > > > > > > > KAFKA-8131 (or rather the related pull request) [1]. > > > > > > > > > > > > > > > > > > Command line tools and their arguments are somewhat diverse > > at > > > > the > > > > > > > > moment. > > > > > > > > > Most of the tools use joptsimple for argument parsing, some > > newer > > > > > > java > > > > > > > > > tools use argparse4j instead and some tools use nothing at > > all. > > > > > > > > > I've looked for a reason as to why there are two libraries > > being > > > > > > used, > > > > > > > > but > > > > > > > > > couldn't really find anything. Paolo brought up the same > > > > question on > > > > > > the > > > > > > > > > mailing list a while back [7], but got no response either. > > > > > > > > > Does anybody know why this is the case? > > > > > > > > > > > > > > > > > > This results in no central place to add universal > parameters > > like > > > > > > help > > > > > > > > and > > > > > > > > > version, as well as the help output looking different > between > > > > some > > > > > > of the > > > > > > > > > tools. > > > > > > > > > Also, there are a number of parameters that should be > > renamed to > > > > > > adhere > > > > > > > > to > > > > > > > > > defaults. > > > > > > > > > > > > > > > > > > There have been a few discussions and initiatives around > > this in > > > > the > > > > > > > > past. > > > > > > > > > Just of the top of my head (and a 5 minute jira search) > there > > > > are: > > > > > > > > > - KIP-14 [2] > > > > > > > > > - KAFKA-2111 [3] > > > > > > > > > - KIP-316 [4] > > > > > > > > > - KAFKA-1292 [5] > > > > > > > > > - KAFKA-3530 [6] > > > > > > > > > - and probably many more > > > > > > > > > > > > > > > > > > Would people generally be in favor of revisiting this > topic? > > > > > > > > > > > > > > > > > > What I'd propose to do is: > > > > > > > > > - comb through jira and KIPs, clean up old stuff and creae > a > > new > > > > > > umbrella > > > > > > > > > issue to track this (maybe reuse KIP-4 as well) > > > > > > > > > - agree on one library for parsing command line arguments > > (don't > > > > care > > > > > > > > which > > > > > > > > > one, but two is one too many I think) > > > > > > > > > - refactor tools to use one library and default way of > > argument > > > > > > parsing > > > > > > > > > with central help and version parameter > > > > > > > > > - add aliases for options that should be renamed according > to > > > > KIP-4 > > > > > > (and > > > > > > > > > maybe others) so that both new and old work for a while, > > > > deprecate > > > > > > old > > > > > > > > > parameters for a cycle or two and then remove them > > > > > > > > > > > > > > > > > > I'll shut up now and see if people would consider this > > useful or > > > > > > have any > > > > > > > > > other input :) > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > Sönke > > > > > > > > > > > > > > > > > > [1] > > > > https://github.com/apache/kafka/pull/6481#discussion_r273773003 > > > > > > > > > <https://issues.apache.org/jira/browse/KAFKA-8131> > > > > > > > > > [2] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-14+-+Tools+Standardization > > > > > > > > > [3] https://issues.apache.org/jira/browse/KAFKA-2111 > > > > > > > > > [4] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-316%3A+Command-line+overrides+for+ConnectDistributed+worker+properties > > > > > > > > > [5] https://issues.apache.org/jira/browse/KAFKA-1292 > > > > > > > > > [6] https://issues.apache.org/jira/browse/KAFKA-3530 > > > > > > > > > [7] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://sematext.com/opensee/m/Kafka/uyzND10ObP01p77VS?subj=From+Scala+to+Java+based+tools+joptsimple+vs+argparse4j > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Sönke Liebau > > > > > Partner > > > > > Tel. +49 179 7940878 > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - > Germany > > > > > > > > > > > > > > > > > > -- > > > Sönke Liebau > > > Partner > > > Tel. +49 179 7940878 > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany > > > > > >
