Hello, As Pulsar becomes increasingly popular, we will have to deal with a
larger userbase looking to deploy Pulsar in a wider array of use cases,
interfacing with a more diverse set of other components. To help with
this, we should create a plan as a project to help community members
publish and discover connectors beyond what the Pulsar PMC wants to
maintain.
Current plans include splitting connectors into separate repos (PIP 62) or
moving under the umbrella of the projects they integrate with (as per
conversations during the community meetings). This will definitely help
with the build times but may negatively affect the discoverability of the
connectors and ease of installation.
I think Pulsar can benefit from a simple package registry that (1) hosts a
list of free to use (apache or other approved license)
connectors/references to the binaries, and (2) provides a CLI (e.g. via
pulsar admin) to simplify discovery, download, and installation of the
connectors for the new users.
What do you think? Would you find something like this useful?
The implementation can be as simple as another GitHub repo with a
predefined structure like
{connector name}/{major version}/metadata
where metadata contains url to the nar, checksum, range of compatible
pulsar versions, contacts, license, short description, etc.
Plus the CLI that can search/list compatible connectors,
download/install/update the connector.
As prior art examples, one can refer to:
1.
brew (package manager for MacOS)
1.
Formulas/registry: https://github.com/Homebrew/homebrew-core
2.
brew itself https://github.com/Homebrew/brew
2.
Apache Solr
1.
https://solr.apache.org/guide/8_8/package-manager.html.
There is also the Helm chart repository from our cousins at the CNCF over
at the Artifact Hub <https://artifacthub.io/>.
I believe such a registry managed by the PMC will reduce the risk of
fragmentation of the ecosystem, improve discoverability, and allow simple
detection of not-up-to-date connectors for the new releases of Pulsar.
--
Andrey Yegorov
