Hi Nicolò,
This is definitely a very awesome job, thank you for bringing Pulsar Shell to us! Also, I was wondering if we could integrate the pulsar shell entrypoint directly into the existing pulsar command[1]? (i.e, invoke ./bin/pulsar shell) So that we can reuse the current CLI instead of adding a new CLI into the bin folder. *Best,* *Max Xu* [1] https://github.com/apache/pulsar/blob/master/bin/pulsar On Mon, Jul 11, 2022 at 6:24 PM Nicolò Boschi <boschi1...@gmail.com> wrote: > Hi Asaf, > > Actually it's in the "Future works" of the PIP. > > I already implemented a PoC of it and you can see it here: > > https://github.com/nicoloboschi/pulsar/commit/f227be324c96daeaf89566260a63eb558357f34f > There are still some issues with some runtime config since GraalVM requires > to declare most of the reflection accesses you will do at runtime. And > Pulsar does a lot of reflections. > > Anyway, I don't think it is a different direction but instead a future > optimization and we should push to have it after the base Pulsar shell is > fully committed to the master branch. > > BR, > Nicolò Boschi > > > Il giorno lun 11 lug 2022 alle ore 12:18 tison <wander4...@gmail.com> ha > scritto: > > > Hi Asaf, > > > > Why not compile the related package with graal and submit a report on the > > master issue[1] :) > > > > AFAIK Pulsar didn't optimize for graal compilation so that you MAY meet > > some issues, and pulsar-shell actually depends on other core module so > that > > it's not an orphan issue to resolve. > > > > Best, > > tison. > > > > [1] https://github.com/apache/pulsar/issues/16250 > > > > > > Asaf Mesika <asaf.mes...@gmail.com> 于2022年7月11日周一 18:05写道： > > > > > Looks awesome. > > > I was wondering if you have thought about another direction which is > > > supplying a GraalVM compiled binary. This should take care of load > time. > > So > > > in effect, you can have a single command, which boots in microseconds, > > and > > > has shell auto-complete? > > > > > > On Mon, Jul 4, 2022 at 4:34 PM Hang Chen <chenh...@apache.org> wrote: > > > > > > > +1 Great work! > > > > > > > > Thanks, > > > > Hang > > > > > > > > PengHui Li <codelipeng...@gmail.com> 于2022年7月4日周一 19:36写道： > > > > > > > > > > + 1 > > > > > > > > > > Penghui > > > > > On Jul 4, 2022, 18:22 +0800, Dezhi Liu <dezhi...@apache.org>, > wrote: > > > > > > +1, Great Job. > > > > > > > > > > > > Thanks, > > > > > > Dezhi > > > > > > > > > > > > On 2022/06/28 08:06:01 Nicolò Boschi wrote: > > > > > > > Hi all, > > > > > > > > > > > > > > I opened a new PIP about Pulsar CLI tools. > > > > > > > Looking forward to seeing comments and suggestions. > > > > > > > > > > > > > > PIP: https://github.com/apache/pulsar/issues/16250 > > > > > > > > > > > > > > I posted a short video that shows how the new tool will work: > > > > > > > > > > > > > > > > > https://user-images.githubusercontent.com/23314389/176125261-35e123a1-1826-4553-b912-28d00914c0e4.mp4 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ## Motivation > > > > > > > > > > > > > > Currently Pulsar comes with a couple of utility scripts with > the > > > > goal of > > > > > > > managing an existing cluster, test behaviours and verify > > > > performances: > > > > > > > these tools are available as SH script inside the `bin` > > directory. > > > > > > > The `pulsar-admin` is the CLI tool supposed to help users and > > > > operators to > > > > > > > configure the system, operate over policies, install functions > > and > > > > much > > > > > > > else. > > > > > > > > > > > > > > This proposal basically aims to solve two different problems: > > > > > > > > > > > > > > 1. `pulsar-admin` is terribly slow. Every time the script is > > > > triggered, a > > > > > > > new JVM process is spawned. The JVM process creation is heavy > and > > > > most of > > > > > > > the time is spent by the JVM initialization process. A very > > common > > > > use case > > > > > > > for cluster operators is to create scripts with several > commands > > > > with the > > > > > > > goal of initialize the cluster, initialize a specific tenant > > > > (namespaces, > > > > > > > topics, policies, functions..); in this case, one JVM is > > > initialized > > > > for > > > > > > > each scripts leads to waste of time and resources. > > > > > > > > > > > > > > 2. User experience. The current design of the Pulsar CLIs can > be > > > > improved. > > > > > > > There are a couple of aspects that may be annoying for a user > and > > > can > > > > > > > discourage a user to use Pulsar. > > > > > > > 1. Poking around available commands and options in a CLI tool > > > > > > > (`pulsar-admin` for instance, but it's the same for > `pulsar-perf` > > > and > > > > > > > `pulsar-client`) is slow and hard. In order to discover > commands > > > and > > > > > > > options you need to use `-h` option and, since the performance > > > issue > > > > > > > pointed at 1., it can be annoying and time-consuming. > > Autocomplete > > > > feature > > > > > > > could be a real game-changer in this context. > > > > > > > 2. Different CLI tools. There are a couple of different shell > > > > scripts. > > > > > > > They have different goals and it's okay to keep them separated. > > > > However, > > > > > > > they raise a barrier for a non Pulsar expert that doesn't have > a > > > > convenient > > > > > > > entry-point. > > > > > > > > > > > > > > ## Goal > > > > > > > > > > > > > > Address all the issues in the previous section with a single > > > > solution. > > > > > > > > > > > > > > ## API Changes > > > > > > > > > > > > > > A new shell script `bin/pulsar-shell` will be introduced. > > > `bin/pulsar > > > > > > > shell` could be a valid alternative but it's not findable for a > > > > newbie user > > > > > > > since no direct file exists. > > > > > > > > > > > > > > ## Implementation > > > > > > > > > > > > > > ### Concepts > > > > > > > The new script `pulsar-shell` will differ from the existing for > > the > > > > > > > following reasons: > > > > > > > > > > > > > > 1. It's a shell. When you start it, it will wait for commands > to > > be > > > > > > > executed. After the command has been executed, despite its > > result, > > > > the > > > > > > > shell session will not be destroyed and it will wait for > another > > > > command. > > > > > > > 2. Unifies all the CLI scripts. In `pulsar-shell` you'll be > able > > to > > > > run all > > > > > > > the existing CLI commands. This will be done in a way that > when a > > > new > > > > > > > command/option is added, the pulsar-shell will be updated > > > > accordingly. > > > > > > > 3. It comes with sophisticated autocompletion and command > history > > > to > > > > highly > > > > > > > improve the UX. > > > > > > > 4. Performance. Since JVM is initiated once, it will gain on > > > > performance > > > > > > > thanks to the JVM warmup and internal libraries bootstraps. > > > > > > > 5. It will accept a file or a list of commands (parameter and > > > stdin) > > > > to > > > > > > > start a shell, run the commands and close the shell. We'll call > > it > > > > the > > > > > > > `non-interactive` mode and it will ease the cluster operations > > > > automations. > > > > > > > > > > > > > > Note that existing tools will not be removed/changed. > > > > > > > > > > > > > > ### Implementation > > > > > > > > > > > > > > The shell implementation will be developed in Java, using a > > > > well-known > > > > > > > library called [JLine 3](https://github.com/jline/jline3) for > > the > > > > shell > > > > > > > support. > > > > > > > There will be a new main class that will extends the existing > > class > > > > tools > > > > > > > (e.g. `PulsarAdminTool`) > > > > > > > > > > > > > > #### Configuration > > > > > > > The configuration file taken by default will be `client.conf`, > > like > > > > current > > > > > > > CLI tools. The env setup will be exactly the same as for > > > > `pulsar-admin` and > > > > > > > `pulsar-client`. > > > > > > > > > > > > > > #### Autocompletion > > > > > > > JLine3 has great support for autocompletion. The shell java > class > > > > will > > > > > > > translate current `JCommander` tools to the JLine3 completion > > API. > > > > This > > > > > > > will ensure that all APIs will be up-to-date and covered. > > > > > > > > > > > > > > #### History > > > > > > > JLine3 has built-in support for history, both in-memory and > > > > persisted (on > > > > > > > local FS). By default the history will be persisted in the user > > > home > > > > > > > directory. This feature can be turned off for security concerns > > > with > > > > a > > > > > > > configuration property. > > > > > > > > > > > > > > #### List of tools > > > > > > > The proposal is to get the following tools: > > > > > > > - bin/pulsar-admin > > > > > > > - bin/pulsar-client > > > > > > > - bin/pulsar-perf > > > > > > > > > > > > > > > > > > > > > ## Rejected Alternatives > > > > > > > > > > > > > > - Create a shell mode for `pulsar-admin`. This won't be a > > flexible > > > > solution > > > > > > > because then we may need the same for other tools. Also it > > doesn't > > > > cover > > > > > > > the CLI unification part. > > > > > > > - Create a brand-new shell mode. Not compatible with current > CLI > > > and > > > > > > > another tool to maintain. > > > > > > > - Use another shell library. There are a couple of valid > > > > alternatives to > > > > > > > JLine3 but it's wide-spread and the autocompletion API is much > > more > > > > > > > flexible than others since it's not done with annotations (the > > > shell > > > > needs > > > > > > > to translate current JCommander definitions from existing > tools) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Nicolò Boschi > > > > > > > > > > > > > > > > >
