Hi Colin, it was my intention to keep the structure of the commands mostly intact while doing the refactoring - if that is possible, have not really checked yet to be honest.
But what I wanted to try and do is recreate the current parsing with argparse as much as possible. And in the process simply adding synonyms, for example make the kafka-console-producer understand a bootstrap-parameter in addition to broker-list. There is a bit of custom logic about which parameters go together etc. in the current classes, so output may look different here and there, but in principle I do believe that it should be possible to recreate the current structure. If there is an appetite for a new, hadoop-like entrypoint anyway, then all of this might be "wasted" effort, or rather effort better spent though, you are right. Best regards, Sönke On Mon, May 6, 2019 at 7:13 PM Colin McCabe <cmcc...@apache.org> wrote: > Hi Sönke, > > #2 is a bit tough because people have come to rely on the way the commands > are structured right now. > > If we want to make big changes, it might be easier just to create a > separate tool and deprecate the old one(s). One thing we've talked about > doing in the past is creating a single entry point for all the tool > functionality, kind of like hadoop did with the "hadoop" command Or git > with the "git" command, etc. Then we could deprecate the standalone > commands and remove them after enough time had passed-- kind of like the > old consumer. > > On the other hand, a more incremental change would be standardizing flags > a bit. So for example, at least setting it up so that there is a standard > way of supplying bootstrap brokers, etc. We could keep the old flags > around for a while as variants to ease the transition. > > best, > Colin > > > On Sun, May 5, 2019, at 00:54, Sönke Liebau wrote: > > Hi Colin, > > > > I totally agree! Especially the differently named bootstrap server > options > > have been annoying me a long time. > > > > I'd propose a two-step approach: > > 1. Add new default options objects similar to CommandLineUtils and > > CommandDefaultOptions (based on argparse4j) but in the clients project, > as > > this is referenced by all command line tools as far as I can tell > > 2. Refactor tools one by one to use these new helper classes (and thus > > argparse) and add standardized synonyms for parameters as necessary > > > > I think for step 1 we can get away with no KIP, as this doesn't change > any > > public interfaces or behavior. > > Step 2 probably needs a KIP as we are adding new parameters? We can pick > up > > KIP-14 again for that I think. A lot of work has been done on that > already. > > > > Does that sound useful to everybody? > > > > Best regards, > > Sönke > > > > > > On Thu, Apr 18, 2019 at 1:44 AM Colin McCabe <cmcc...@apache.org> wrote: > > > > > If we are going to standardize on one argument parsing library, it > should > > > certainly be argparse4j, I think. > > > argparse4j is simply a better argument parsing library with support > for > > > more features. One example is mutually exclusive options. argparse4j > > > supports this with MutuallyExclusiveGroup. jopt doesn't support this, > so > > > when it is needed, we have to add extra code to manually check that > > > mutually exclusive options are not set. > > > > > > argparse4j also has subcommands. If you want something like "git add" > > > with some set of flags, and "git remove" with another, you can do this > with > > > argparse4j, but not with jopt. This would be very helpful for > clearing up > > > confusion in a lot of our shell scripts which have accumulated dozens > of > > > arguments, most of which are only relevant to a very specific > operation. > > > But you just can't do it with jopt. > > > > > > Just to give an example, argparse4j with subcommands would allow you to > > > run something like ./kafka-topics.sh list --help and get just options > that > > > were relevant for listing topics, not the full dozens of options that > might > > > relate to adding topics, removing them, etc. > > > > > > To be honest, though, what would help users the most is standardizing > the > > > option flags across tools. We should have a standard way of specifying > > > bootstrap brokers, for example. (We can continue to support the old > > > synonyms for a while, of course.) > > > > > > best, > > > Colin > > > > > > > > > On Wed, Apr 17, 2019, at 08:56, Guozhang Wang wrote: > > > > I took another look at the PR itself and I think it would be great to > > > have > > > > this cleanup too -- I cannot remember at the beginning why we > gradually > > > > moved to different mechanism (argparse4j) for different cmds, if > there's > > > no > > > > rationales behind it we should just make them consistent. > > > > > > > > Thanks for driving this! > > > > > > > > Guozhang > > > > > > > > On Wed, Apr 17, 2019 at 7:19 AM Ryanne Dolan <ryannedo...@gmail.com> > > > wrote: > > > > > > > > > Sönke, I'd find this very helpful. It's annoying to keep track of > which > > > > > commands use which form -- I always seem to guess wrong. > > > > > > > > > > Though I don't think there is any reason to deprecate existing > forms, > > > e.g. > > > > > consumer.config vs consumer-config. I think it's perfectly > reasonable > > > to > > > > > have multiple spellings of the same arguments. I don't really see a > > > > > downside to keeping the aliases around indefinitely. > > > > > > > > > > Ryanne > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Apr 17, 2019, 7:07 AM Sönke Liebau > > > > > <soenke.lie...@opencore.com.invalid> wrote: > > > > > > > > > > > Hi everybody, > > > > > > > > > > > > Jason and I were recently discussing command line argument > parsing on > > > > > > KAFKA-8131 (or rather the related pull request) [1]. > > > > > > > > > > > > Command line tools and their arguments are somewhat diverse at > the > > > > > moment. > > > > > > Most of the tools use joptsimple for argument parsing, some newer > > > java > > > > > > tools use argparse4j instead and some tools use nothing at all. > > > > > > I've looked for a reason as to why there are two libraries being > > > used, > > > > > but > > > > > > couldn't really find anything. Paolo brought up the same > question on > > > the > > > > > > mailing list a while back [7], but got no response either. > > > > > > Does anybody know why this is the case? > > > > > > > > > > > > This results in no central place to add universal parameters like > > > help > > > > > and > > > > > > version, as well as the help output looking different between > some > > > of the > > > > > > tools. > > > > > > Also, there are a number of parameters that should be renamed to > > > adhere > > > > > to > > > > > > defaults. > > > > > > > > > > > > There have been a few discussions and initiatives around this in > the > > > > > past. > > > > > > Just of the top of my head (and a 5 minute jira search) there > are: > > > > > > - KIP-14 [2] > > > > > > - KAFKA-2111 [3] > > > > > > - KIP-316 [4] > > > > > > - KAFKA-1292 [5] > > > > > > - KAFKA-3530 [6] > > > > > > - and probably many more > > > > > > > > > > > > Would people generally be in favor of revisiting this topic? > > > > > > > > > > > > What I'd propose to do is: > > > > > > - comb through jira and KIPs, clean up old stuff and creae a new > > > umbrella > > > > > > issue to track this (maybe reuse KIP-4 as well) > > > > > > - agree on one library for parsing command line arguments (don't > care > > > > > which > > > > > > one, but two is one too many I think) > > > > > > - refactor tools to use one library and default way of argument > > > parsing > > > > > > with central help and version parameter > > > > > > - add aliases for options that should be renamed according to > KIP-4 > > > (and > > > > > > maybe others) so that both new and old work for a while, > deprecate > > > old > > > > > > parameters for a cycle or two and then remove them > > > > > > > > > > > > I'll shut up now and see if people would consider this useful or > > > have any > > > > > > other input :) > > > > > > > > > > > > Best regards, > > > > > > Sönke > > > > > > > > > > > > [1] > https://github.com/apache/kafka/pull/6481#discussion_r273773003 > > > > > > <https://issues.apache.org/jira/browse/KAFKA-8131> > > > > > > [2] > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-14+-+Tools+Standardization > > > > > > [3] https://issues.apache.org/jira/browse/KAFKA-2111 > > > > > > [4] > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-316%3A+Command-line+overrides+for+ConnectDistributed+worker+properties > > > > > > [5] https://issues.apache.org/jira/browse/KAFKA-1292 > > > > > > [6] https://issues.apache.org/jira/browse/KAFKA-3530 > > > > > > [7] > > > > > > > > > > > > > > > > > > > > > https://sematext.com/opensee/m/Kafka/uyzND10ObP01p77VS?subj=From+Scala+to+Java+based+tools+joptsimple+vs+argparse4j > > > > > > > > > > > > > > > > > > > > > > > -- > > > > -- Guozhang > > > > > > > > > > > > > -- > > Sönke Liebau > > Partner > > Tel. +49 179 7940878 > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany > > > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
