FYI. We have completed transferring pulsar-manager repo to the ASF. It is at : https://github.com/apache/pulsar-manager
Thanks, Sijie On Mon, Sep 16, 2019 at 4:35 PM Sijie Guo <guosi...@gmail.com> wrote: > FYI. IP clearance is done in gene...@incubator.apache.org. I am moving > forward with transferring the pulsar-manager repo. > > INFRA Jira is created: https://issues.apache.org/jira/browse/INFRA-19030 > > - Sijie > > On Fri, Sep 6, 2019 at 5:29 AM Jia Zhai <zhaiji...@gmail.com> wrote: > >> +1 >> >> >> Best Regards. >> >> >> Jia Zhai >> >> Beijing, China >> >> Mobile: +86 15810491983 >> >> >> >> >> On Thu, Sep 5, 2019 at 3:34 AM Sijie Guo <guosi...@gmail.com> wrote: >> >> > Dave, thank you! That’s super helpful! >> > >> > Thanks, >> > Sijie >> > >> > On Wed, Sep 4, 2019 at 10:29 AM Dave Fisher <w...@apache.org> wrote: >> > >> > > >> > > >> > > > On Sep 4, 2019, at 10:10 AM, Sijie Guo <guosi...@gmail.com> wrote: >> > > > >> > > > Hi Dave, >> > > > >> > > > Thank you for the reminder. I was about to ping you yesterday about >> > this. >> > > > The IP clearance is the last piece that I wasn’t sure about. Just to >> > make >> > > > sure I understand correctly, so the Pulsar PMC has to fill out the >> IP >> > > > clearance template, check in the file to the SVN repo and start a >> > thread >> > > in >> > > > general@incubator to get the approval from IPMC. Is that correct? >> > > >> > > Yes. The approval is via LAZY CONSENSUS of 3 days. >> > > >> > > StreamNative ought to provide an SGA to secretary@ unless all of the >> IP >> > > is already covered by ICLAs. >> > > >> > > Infra can then help transfer the repository. >> > > >> > > Regards, >> > > Dave >> > > >> > > > >> > > > Thanks, >> > > > Sijie >> > > > >> > > > On Wed, Sep 4, 2019 at 9:41 AM Dave Fisher <w...@apache.org> wrote: >> > > > >> > > >> BTW - Once the VOTE is complete to accept the codebase there is >> some >> > > >> paperwork. >> > > >> >> > > >> http://incubator.apache.org/ip-clearance/ >> > > >> >> > > >> While this is the Incubator the Board has requested that the IPMC >> > track >> > > IP >> > > >> Clearance. Everything is in SVN and the pages are updated every >> > evening >> > > >> PDT. If help is needed then let me know. >> > > >> >> > > >> Regards, >> > > >> Dave >> > > >> >> > > >>> On Sep 4, 2019, at 9:01 AM, Enrico Olivelli <eolive...@gmail.com> >> > > wrote: >> > > >>> >> > > >>> For whom who is interested I am working on adding support for >> HerdDB >> > in >> > > >>> Pulsar Manager. >> > > >>> >> > > >>> The pull request is still very raw, but Pulsar Manager seems to >> work >> > > >>> https://github.com/streamnative/pulsar-manager/pull/183 >> > > >>> >> > > >>> >> > > >>> >> > > >>> Il giorno mar 3 set 2019 alle ore 08:37 Yuva raj < >> uvar...@gmail.com> >> > > ha >> > > >>> scritto: >> > > >>> >> > > >>>> On Fri, 30 Aug 2019 at 07:09, Sijie Guo <guosi...@gmail.com> >> wrote: >> > > >>>> >> > > >>>>>> I see there are dependencies on websockets, I would a prefer a >> > > simple >> > > >>>>> polling model of the http, also it's enable by default in >> pulsar. >> > > >>>>> >> > > >>>>> I don't think we depend don pulsar websocket. All are http >> restful >> > > >>>>> requests. >> > > >>>>> >> > > >>>>>> For the ui the default persistence should be sqlite. >> Potentially >> > > >>>> packaged >> > > >>>>> by default. >> > > >>>>> >> > > >>>>> Initially the default is sqlite. But sqlite doesn't work if >> there >> > > are a >> > > >>>> lot >> > > >>>>> of topic metrics. >> > > >>>>> We switched to MySQL as default for supporting production >> traffic. >> > We >> > > >> can >> > > >>>>> switch default to PostgresSQL (as pulsar dashboard). >> > > >>>>> >> > > >>>> +1 >> > > >>>> Yes, Switching to Postgres Would be great. Because Postgres >> license >> > is >> > > >> more >> > > >>>> liberal and works well with Apache license ecosystem. >> > > >>>> >> > > >>>>> >> > > >>>>> - Sijie >> > > >>>>> >> > > >>>>> On Thu, Aug 29, 2019 at 6:23 PM Ali Ahmed <alahmed...@gmail.com >> > >> > > >> wrote: >> > > >>>>> >> > > >>>>>> I see there are dependencies on websockets, I would a prefer a >> > > simple >> > > >>>>>> polling model of the http, also it's enable by default in >> pulsar. >> > > >>>>>> >> > > >>>>>> For the ui the default persistence should be sqlite. >> Potentially >> > > >>>> packaged >> > > >>>>>> by default. >> > > >>>>>> >> > > >>>>>> -Ali >> > > >>>>>> >> > > >>>>>> On Thu, Aug 29, 2019 at 2:47 PM Enrico Olivelli < >> > > eolive...@gmail.com> >> > > >>>>>> wrote: >> > > >>>>>> >> > > >>>>>>> Il gio 29 ago 2019, 23:28 Sijie Guo <guosi...@gmail.com> ha >> > > scritto: >> > > >>>>>>> >> > > >>>>>>>> That sounds an interesting idea! >> > > >>>>>>> >> > > >>>>>>> >> > > >>>>>>> Awesome >> > > >>>>>>> >> > > >>>>>>> Does HerdDB support JDBC? If so, it should >> > > >>>>>>>> be pretty straightforward to enable HerdDB. >> > > >>>>>>>> >> > > >>>>>>> >> > > >>>>>>> Sure, as far as I know, the JDBC client is the only client >> really >> > > >>>> used >> > > >>>>> in >> > > >>>>>>> production. >> > > >>>>>>> >> > > >>>>>>> In a replicated environment it uses Zookeeper for metadata and >> > > >>>> service >> > > >>>>>>> discovery and Bookkeeper as WAL and if you have a pulsar >> cluster >> > > you >> > > >>>>>>> already have both of them (zk cluster an bookies) >> > > >>>>>>> >> > > >>>>>>> I will take a look to how Pulsar Manager uses JDBC, maybe it >> > will >> > > >>>> be >> > > >>>>>> very >> > > >>>>>>> easy. >> > > >>>>>>> >> > > >>>>>>> I will be back with news >> > > >>>>>>> >> > > >>>>>>> >> > > >>>>>>> Enrico >> > > >>>>>>> >> > > >>>>>>> >> > > >>>>>>> >> > > >>>>>>> >> > > >>>>>>> >> > > >>>>>>>> Thanks, >> > > >>>>>>>> Sijie >> > > >>>>>>>> >> > > >>>>>>>> On Thu, Aug 29, 2019 at 12:08 PM Enrico Olivelli < >> > > >>>>> eolive...@gmail.com> >> > > >>>>>>>> wrote: >> > > >>>>>>>> >> > > >>>>>>>>> Maybe you can try to use HerdDB (1), it is a replicated SQL >> > > >>>>> database >> > > >>>>>>> that >> > > >>>>>>>>> can be run embedded in the JVM. >> > > >>>>>>>>> It is an SQL database that we developed in my company, in >> order >> > > >>>> to >> > > >>>>>>>>> explicitly replace MySQL. >> > > >>>>>>>>> It uses Bookkeeper to store the WAL and implement >> replication, >> > > >>>> but >> > > >>>>> it >> > > >>>>>>> can >> > > >>>>>>>>> run in standalone mode or in memory (for tests/dev). >> > > >>>>>>>>> >> > > >>>>>>>>> I can help in setting up a demo. >> > > >>>>>>>>> >> > > >>>>>>>>> Enrico >> > > >>>>>>>>> >> > > >>>>>>>>> (1) https://github.com/diennea/herddb >> > > >>>>>>>>> >> > > >>>>>>>>> Il gio 29 ago 2019, 16:59 Sijie Guo <guosi...@gmail.com> ha >> > > >>>>> scritto: >> > > >>>>>>>>> >> > > >>>>>>>>>> Hi Dave, >> > > >>>>>>>>>> >> > > >>>>>>>>>> It doesn't require MySQL. You can use any JDBC supported >> > > >>>>> databases. >> > > >>>>>>>>>> Initially we would use SQLite for development and change to >> > > >>>> MySQL >> > > >>>>>> for >> > > >>>>>>>>> more >> > > >>>>>>>>>> production-ready workloads. >> > > >>>>>>>>>> >> > > >>>>>>>>>> Hope this clarify your question. >> > > >>>>>>>>>> >> > > >>>>>>>>>> Thanks, >> > > >>>>>>>>>> Sijie >> > > >>>>>>>>>> >> > > >>>>>>>>>> On Thu, Aug 29, 2019 at 5:56 AM Dave Fisher < >> w...@apache.org> >> > > >>>>>> wrote: >> > > >>>>>>>>>> >> > > >>>>>>>>>>> Hi - >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> I see that MySQL is used. This would be a GPL dependency. >> > > >>>> Does >> > > >>>>>> the >> > > >>>>>>>>> Pulsar >> > > >>>>>>>>>>> Manager require MySQL, or can other Databases that are >> > > >>>>> compliant >> > > >>>>>>> with >> > > >>>>>>>>>>> Apache Release Policy be used instead? >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> Regards, >> > > >>>>>>>>>>> Dave >> > > >>>>>>>>>>> >> > > >>>>>>>>>>>> On Aug 29, 2019, at 7:49 AM, Guangning E < >> > > >>>>> eguangn...@gmail.com >> > > >>>>>>> >> > > >>>>>>>>> wrote: >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> Hi all, >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> We have developed a new Pulsar web UI - pulsar-manager, >> > > >>>>> aiming >> > > >>>>>> at >> > > >>>>>>>>>>>> supporting managing Pulsar clusters running in different >> > > >>>>>>>> environments >> > > >>>>>>>>>>>> (on-premise data centers, cloud, and so on). We’d like to >> > > >>>>>>>> contribute >> > > >>>>>>>>>> the >> > > >>>>>>>>>>>> project back to the Pulsar community. >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> --- >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> ## Motivation >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> Currently, Pulsar has a monitoring tool - [Pulsar >> > > >>>>> Dashboard]( [ >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>> >> > > >>>>>>>>>> >> > > >>>>>>>>> >> > > >>>>>>>> >> > > >>>>>>> >> > > >>>>>> >> > > >>>>> >> > > >>>> >> > > >> >> > > >> > >> http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/) >> > > >>>>>>>>>>>> ). >> > > >>>>>>>>>>>> However, it only focuses on the simple monitoring of >> > > >>>> Pulsar - >> > > >>>>>>>>>> collecting >> > > >>>>>>>>>>>> and displaying information such as statistics of tenants, >> > > >>>>>>>> namespaces, >> > > >>>>>>>>>>>> topics, subscriptions, and so on. It doesn’t provide any >> > > >>>>>>> management >> > > >>>>>>>>>>>> operations such as add, delete and update tenants, >> > > >>>>> namespaces, >> > > >>>>>>>>> topics, >> > > >>>>>>>>>>> and >> > > >>>>>>>>>>>> so on. When the scale of Pulsar cluster increases or the >> > > >>>>> number >> > > >>>>>>> of >> > > >>>>>>>>>>> clusters >> > > >>>>>>>>>>>> grows, using `pulsar-admin` to manage Pulsar can not >> > > >>>> satisfy >> > > >>>>>>>> demands. >> > > >>>>>>>>>>>> Therefore, Pulsar needs a simple and easy-to-use >> management >> > > >>>>>>> console >> > > >>>>>>>>> for >> > > >>>>>>>>>>>> administrators. >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> Pulsar Manager is a web-based GUI management and >> monitoring >> > > >>>>>> tool >> > > >>>>>>>> that >> > > >>>>>>>>>>> helps >> > > >>>>>>>>>>>> administrators and users manage and monitor tenants, >> > > >>>>>> namespaces, >> > > >>>>>>>>>> topics, >> > > >>>>>>>>>>>> subscriptions, brokers, clusters, and so on, and supports >> > > >>>>>> dynamic >> > > >>>>>>>>>>>> configuration of multiple environments. >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> ## Features >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> Pulsar manager provides two main features: management and >> > > >>>>>>>> monitoring. >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> ### Management >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> 1. Environment : (operations on the environment) >> > > >>>>>>>>>>>> Create / Delete / Update / List / Get >> > > >>>>>>>>>>>> 2. Cluster : (operations on clusters) >> > > >>>>>>>>>>>> Create / Delete / Update / List / Get >> > > >>>>>>>>>>>> 3. Brokers : (operations on brokers) >> > > >>>>>>>>>>>> Heartbeat / Unload >> > > >>>>>>>>>>>> 4. Tenants: >> > > >>>>>>>>>>>> Create / Delete / Update / List >> > > >>>>>>>>>>>> 5. Namespaces: >> > > >>>>>>>>>>>> Create / Delete / List >> > > >>>>>>>>>>>> Manage Namespace Policy >> > > >>>>>>>>>>>> Unload >> > > >>>>>>>>>>>> Operations on Namespace Bundles >> > > >>>>>>>>>>>> 6. Topics >> > > >>>>>>>>>>>> Create / Delete / List >> > > >>>>>>>>>>>> Unload / Terminate / Offload / Compact >> > > >>>>>>>>>>>> Topic Details >> > > >>>>>>>>>>>> 7. Subscriptions >> > > >>>>>>>>>>>> Create / Delete / List >> > > >>>>>>>>>>>> Reset Cursor / Skip / Clear backlog / Unsubscribe >> > > >>>>>>>>>>>> 8. Namespace Isolation Policies >> > > >>>>>>>>>>>> Create / Delete / Update / Get / List >> > > >>>>>>>>>>>> 9. Failure Domains >> > > >>>>>>>>>>>> Create / Delete / Update / Get / List >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> ### Monitoring: (display *aggregated* stats at different >> > > >>>>>> levels) >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> 1. Tenants >> > > >>>>>>>>>>>> List the total number of namespaces per tenant (both >> > > >>>>> aggregated >> > > >>>>>>> and >> > > >>>>>>>>>>>> per-cluster basis) >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> 2. Namespaces >> > > >>>>>>>>>>>> List the total number of topics per namespace >> > > >>>>>>>>>>>> List the aggregated stats per namespace (such as rate-in, >> > > >>>>>>> rate-out, >> > > >>>>>>>>>>>> throughput-in, and throughput-out) >> > > >>>>>>>>>>>> List the distributions of namespace bundles >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> 3. Topics >> > > >>>>>>>>>>>> List the total number of partitions per topic >> > > >>>>>>>>>>>> List the aggregated stats per topic (such as rate-in, >> > > >>>>> rate-out, >> > > >>>>>>>>>>>> throughput-in, and throughput-out) >> > > >>>>>>>>>>>> Detailed stats per topic partition >> > > >>>>>>>>>>>> Detailed stats of storage per topic partition >> > > >>>>>>>>>>>> Detailed stats of subscription per topic and per >> partition >> > > >>>>>>>>>>>> Detailed stats of producers >> > > >>>>>>>>>>>> Detailed stats of consumers >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> The detailed design proposal is in >> > > >>>>>>>>>>>> [ >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>> >> > > >>>>>>>>>> >> > > >>>>>>>>> >> > > >>>>>>>> >> > > >>>>>>> >> > > >>>>>> >> > > >>>>> >> > > >>>> >> > > >> >> > > >> > >> https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#) >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> Looking forward to any feedback. >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> Thanks, >> > > >>>>>>>>>>>> Guangning >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> >> > > >>>>>>>>>> >> > > >>>>>>>>> >> > > >>>>>>>> >> > > >>>>>>> >> > > >>>>>> >> > > >>>>> >> > > >>>> >> > > >>>> >> > > >>>> -- >> > > >>>> *Thanks* >> > > >>>> >> > > >>>> *Yuvaraj L* >> > > >>>> >> > > >> >> > > >> >> > > >> > > >> > >> >
