Hi all, I opened a new PIP about Pulsar CLI tools. Looking forward to seeing comments and suggestions.
PIP: https://github.com/apache/pulsar/issues/16250 I posted a short video that shows how the new tool will work: https://user-images.githubusercontent.com/23314389/176125261-35e123a1-1826-4553-b912-28d00914c0e4.mp4 ## Motivation Currently Pulsar comes with a couple of utility scripts with the goal of managing an existing cluster, test behaviours and verify performances: these tools are available as SH script inside the `bin` directory. The `pulsar-admin` is the CLI tool supposed to help users and operators to configure the system, operate over policies, install functions and much else. This proposal basically aims to solve two different problems: 1. `pulsar-admin` is terribly slow. Every time the script is triggered, a new JVM process is spawned. The JVM process creation is heavy and most of the time is spent by the JVM initialization process. A very common use case for cluster operators is to create scripts with several commands with the goal of initialize the cluster, initialize a specific tenant (namespaces, topics, policies, functions..); in this case, one JVM is initialized for each scripts leads to waste of time and resources. 2. User experience. The current design of the Pulsar CLIs can be improved. There are a couple of aspects that may be annoying for a user and can discourage a user to use Pulsar. 1. Poking around available commands and options in a CLI tool (`pulsar-admin` for instance, but it's the same for `pulsar-perf` and `pulsar-client`) is slow and hard. In order to discover commands and options you need to use `-h` option and, since the performance issue pointed at 1., it can be annoying and time-consuming. Autocomplete feature could be a real game-changer in this context. 2. Different CLI tools. There are a couple of different shell scripts. They have different goals and it's okay to keep them separated. However, they raise a barrier for a non Pulsar expert that doesn't have a convenient entry-point. ## Goal Address all the issues in the previous section with a single solution. ## API Changes A new shell script `bin/pulsar-shell` will be introduced. `bin/pulsar shell` could be a valid alternative but it's not findable for a newbie user since no direct file exists. ## Implementation ### Concepts The new script `pulsar-shell` will differ from the existing for the following reasons: 1. It's a shell. When you start it, it will wait for commands to be executed. After the command has been executed, despite its result, the shell session will not be destroyed and it will wait for another command. 2. Unifies all the CLI scripts. In `pulsar-shell` you'll be able to run all the existing CLI commands. This will be done in a way that when a new command/option is added, the pulsar-shell will be updated accordingly. 3. It comes with sophisticated autocompletion and command history to highly improve the UX. 4. Performance. Since JVM is initiated once, it will gain on performance thanks to the JVM warmup and internal libraries bootstraps. 5. It will accept a file or a list of commands (parameter and stdin) to start a shell, run the commands and close the shell. We'll call it the `non-interactive` mode and it will ease the cluster operations automations. Note that existing tools will not be removed/changed. ### Implementation The shell implementation will be developed in Java, using a well-known library called [JLine 3](https://github.com/jline/jline3) for the shell support. There will be a new main class that will extends the existing class tools (e.g. `PulsarAdminTool`) #### Configuration The configuration file taken by default will be `client.conf`, like current CLI tools. The env setup will be exactly the same as for `pulsar-admin` and `pulsar-client`. #### Autocompletion JLine3 has great support for autocompletion. The shell java class will translate current `JCommander` tools to the JLine3 completion API. This will ensure that all APIs will be up-to-date and covered. #### History JLine3 has built-in support for history, both in-memory and persisted (on local FS). By default the history will be persisted in the user home directory. This feature can be turned off for security concerns with a configuration property. #### List of tools The proposal is to get the following tools: - bin/pulsar-admin - bin/pulsar-client - bin/pulsar-perf ## Rejected Alternatives - Create a shell mode for `pulsar-admin`. This won't be a flexible solution because then we may need the same for other tools. Also it doesn't cover the CLI unification part. - Create a brand-new shell mode. Not compatible with current CLI and another tool to maintain. - Use another shell library. There are a couple of valid alternatives to JLine3 but it's wide-spread and the autocompletion API is much more flexible than others since it's not done with annotations (the shell needs to translate current JCommander definitions from existing tools) Nicolò Boschi
