Agreed that the main problem is about discovering the existing pre-built Pulsar connectors. I don't think the PMC should involve hosting and managing external connectors because it will put the PMC in the situation in handling licensing issues that I think we should avoid.
All the ASF accepted connectors are already in the main Pulsar repo. Even they are moved to the pulsar-connectors repo. They are managed and released as part of the Pulsar release. We have been improving how we generate documentation for Pulsar APIs and tools. What we can do is to extend existing tooling to automatically generate all the documentation for pre-built connectors. This tool can also generate a JSON file (or a list of JSON files) that contains all the metadata information for all the pre-built connectors. These JSON files can be put in a folder within the pulsar website and hosted as part of the Pulsar website. Then we can just add a command in the `pulsar-admin` tool to retrieve the JSON files and cache them locally. So pulsar-admin can search and download the connector accordingly. This approach ties very well into the current Pulsar release process without introducing additional maintenance overhead. Thanks, Sijie On Tue, May 25, 2021 at 11:59 AM Jerry Peng <jerry.boyang.p...@gmail.com> wrote: > Hello Andrey, > > Thank you for bringing this up! This is definitely an important issue! > > All of the connector binaries are already hosted on Maven central thus I > don't think hosting the binaries is an issue. Perhaps the key problem here > is about discovery. > > My thoughts: > > 1. We should document clearly on the Apache Pulsar website all the > connectors that we offer. > > https://pulsar.apache.org/docs/en/io-connectors/ > > Seems like we already do that? If not, we should make sure to keep this > list up to date. Maybe the list is not visible enough to new users. If > so, we should figure out how to advertise the connectors we already have in > a better fashion. > > 2. I do like the idea of having a tool that can perhaps search and install > connectors automatically for you. Perhaps this is a feature we can add to > the existing pulsar-admin CLI tool. This feature can search maven for > connector binaries and download / install them if instructed by the user. > > Best, > > Jerry > > On Tue, May 25, 2021 at 11:20 AM Andrey Yegorov < > andrey.yego...@datastax.com> > wrote: > > > Hello, As Pulsar becomes increasingly popular, we will have to deal with > a > > larger userbase looking to deploy Pulsar in a wider array of use cases, > > interfacing with a more diverse set of other components. To help with > > this, we should create a plan as a project to help community members > > publish and discover connectors beyond what the Pulsar PMC wants to > > maintain. > > > > Current plans include splitting connectors into separate repos (PIP 62) > or > > moving under the umbrella of the projects they integrate with (as per > > conversations during the community meetings). This will definitely help > > with the build times but may negatively affect the discoverability of the > > connectors and ease of installation. > > > > I think Pulsar can benefit from a simple package registry that (1) hosts > a > > list of free to use (apache or other approved license) > > connectors/references to the binaries, and (2) provides a CLI (e.g. via > > pulsar admin) to simplify discovery, download, and installation of the > > connectors for the new users. > > > > What do you think? Would you find something like this useful? > > > > The implementation can be as simple as another GitHub repo with a > > predefined structure like > > > > {connector name}/{major version}/metadata > > > > where metadata contains url to the nar, checksum, range of compatible > > pulsar versions, contacts, license, short description, etc. > > > > Plus the CLI that can search/list compatible connectors, > > download/install/update the connector. > > > > As prior art examples, one can refer to: > > > > 1. > > > > brew (package manager for MacOS) > > 1. > > > > Formulas/registry: https://github.com/Homebrew/homebrew-core > > 2. > > > > brew itself https://github.com/Homebrew/brew > > 2. > > > > Apache Solr > > 1. > > > > https://solr.apache.org/guide/8_8/package-manager.html. > > > > There is also the Helm chart repository from our cousins at the CNCF over > > at the Artifact Hub <https://artifacthub.io/>. > > > > I believe such a registry managed by the PMC will reduce the risk of > > fragmentation of the ecosystem, improve discoverability, and allow simple > > detection of not-up-to-date connectors for the new releases of Pulsar. > > > > -- > > Andrey Yegorov > > >
