FYI. IP clearance is done in gene...@incubator.apache.org. I am moving forward with transferring the pulsar-manager repo.
INFRA Jira is created: https://issues.apache.org/jira/browse/INFRA-19030 - Sijie On Fri, Sep 6, 2019 at 5:29 AM Jia Zhai <zhaiji...@gmail.com> wrote: > +1 > > > Best Regards. > > > Jia Zhai > > Beijing, China > > Mobile: +86 15810491983 > > > > > On Thu, Sep 5, 2019 at 3:34 AM Sijie Guo <guosi...@gmail.com> wrote: > > > Dave, thank you! That’s super helpful! > > > > Thanks, > > Sijie > > > > On Wed, Sep 4, 2019 at 10:29 AM Dave Fisher <w...@apache.org> wrote: > > > > > > > > > > > > On Sep 4, 2019, at 10:10 AM, Sijie Guo <guosi...@gmail.com> wrote: > > > > > > > > Hi Dave, > > > > > > > > Thank you for the reminder. I was about to ping you yesterday about > > this. > > > > The IP clearance is the last piece that I wasn’t sure about. Just to > > make > > > > sure I understand correctly, so the Pulsar PMC has to fill out the IP > > > > clearance template, check in the file to the SVN repo and start a > > thread > > > in > > > > general@incubator to get the approval from IPMC. Is that correct? > > > > > > Yes. The approval is via LAZY CONSENSUS of 3 days. > > > > > > StreamNative ought to provide an SGA to secretary@ unless all of the > IP > > > is already covered by ICLAs. > > > > > > Infra can then help transfer the repository. > > > > > > Regards, > > > Dave > > > > > > > > > > > Thanks, > > > > Sijie > > > > > > > > On Wed, Sep 4, 2019 at 9:41 AM Dave Fisher <w...@apache.org> wrote: > > > > > > > >> BTW - Once the VOTE is complete to accept the codebase there is some > > > >> paperwork. > > > >> > > > >> http://incubator.apache.org/ip-clearance/ > > > >> > > > >> While this is the Incubator the Board has requested that the IPMC > > track > > > IP > > > >> Clearance. Everything is in SVN and the pages are updated every > > evening > > > >> PDT. If help is needed then let me know. > > > >> > > > >> Regards, > > > >> Dave > > > >> > > > >>> On Sep 4, 2019, at 9:01 AM, Enrico Olivelli <eolive...@gmail.com> > > > wrote: > > > >>> > > > >>> For whom who is interested I am working on adding support for > HerdDB > > in > > > >>> Pulsar Manager. > > > >>> > > > >>> The pull request is still very raw, but Pulsar Manager seems to > work > > > >>> https://github.com/streamnative/pulsar-manager/pull/183 > > > >>> > > > >>> > > > >>> > > > >>> Il giorno mar 3 set 2019 alle ore 08:37 Yuva raj < > uvar...@gmail.com> > > > ha > > > >>> scritto: > > > >>> > > > >>>> On Fri, 30 Aug 2019 at 07:09, Sijie Guo <guosi...@gmail.com> > wrote: > > > >>>> > > > >>>>>> I see there are dependencies on websockets, I would a prefer a > > > simple > > > >>>>> polling model of the http, also it's enable by default in pulsar. > > > >>>>> > > > >>>>> I don't think we depend don pulsar websocket. All are http > restful > > > >>>>> requests. > > > >>>>> > > > >>>>>> For the ui the default persistence should be sqlite. Potentially > > > >>>> packaged > > > >>>>> by default. > > > >>>>> > > > >>>>> Initially the default is sqlite. But sqlite doesn't work if there > > > are a > > > >>>> lot > > > >>>>> of topic metrics. > > > >>>>> We switched to MySQL as default for supporting production > traffic. > > We > > > >> can > > > >>>>> switch default to PostgresSQL (as pulsar dashboard). > > > >>>>> > > > >>>> +1 > > > >>>> Yes, Switching to Postgres Would be great. Because Postgres > license > > is > > > >> more > > > >>>> liberal and works well with Apache license ecosystem. > > > >>>> > > > >>>>> > > > >>>>> - Sijie > > > >>>>> > > > >>>>> On Thu, Aug 29, 2019 at 6:23 PM Ali Ahmed <alahmed...@gmail.com> > > > >> wrote: > > > >>>>> > > > >>>>>> I see there are dependencies on websockets, I would a prefer a > > > simple > > > >>>>>> polling model of the http, also it's enable by default in > pulsar. > > > >>>>>> > > > >>>>>> For the ui the default persistence should be sqlite. Potentially > > > >>>> packaged > > > >>>>>> by default. > > > >>>>>> > > > >>>>>> -Ali > > > >>>>>> > > > >>>>>> On Thu, Aug 29, 2019 at 2:47 PM Enrico Olivelli < > > > eolive...@gmail.com> > > > >>>>>> wrote: > > > >>>>>> > > > >>>>>>> Il gio 29 ago 2019, 23:28 Sijie Guo <guosi...@gmail.com> ha > > > scritto: > > > >>>>>>> > > > >>>>>>>> That sounds an interesting idea! > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> Awesome > > > >>>>>>> > > > >>>>>>> Does HerdDB support JDBC? If so, it should > > > >>>>>>>> be pretty straightforward to enable HerdDB. > > > >>>>>>>> > > > >>>>>>> > > > >>>>>>> Sure, as far as I know, the JDBC client is the only client > really > > > >>>> used > > > >>>>> in > > > >>>>>>> production. > > > >>>>>>> > > > >>>>>>> In a replicated environment it uses Zookeeper for metadata and > > > >>>> service > > > >>>>>>> discovery and Bookkeeper as WAL and if you have a pulsar > cluster > > > you > > > >>>>>>> already have both of them (zk cluster an bookies) > > > >>>>>>> > > > >>>>>>> I will take a look to how Pulsar Manager uses JDBC, maybe it > > will > > > >>>> be > > > >>>>>> very > > > >>>>>>> easy. > > > >>>>>>> > > > >>>>>>> I will be back with news > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> Enrico > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>>> Thanks, > > > >>>>>>>> Sijie > > > >>>>>>>> > > > >>>>>>>> On Thu, Aug 29, 2019 at 12:08 PM Enrico Olivelli < > > > >>>>> eolive...@gmail.com> > > > >>>>>>>> wrote: > > > >>>>>>>> > > > >>>>>>>>> Maybe you can try to use HerdDB (1), it is a replicated SQL > > > >>>>> database > > > >>>>>>> that > > > >>>>>>>>> can be run embedded in the JVM. > > > >>>>>>>>> It is an SQL database that we developed in my company, in > order > > > >>>> to > > > >>>>>>>>> explicitly replace MySQL. > > > >>>>>>>>> It uses Bookkeeper to store the WAL and implement > replication, > > > >>>> but > > > >>>>> it > > > >>>>>>> can > > > >>>>>>>>> run in standalone mode or in memory (for tests/dev). > > > >>>>>>>>> > > > >>>>>>>>> I can help in setting up a demo. > > > >>>>>>>>> > > > >>>>>>>>> Enrico > > > >>>>>>>>> > > > >>>>>>>>> (1) https://github.com/diennea/herddb > > > >>>>>>>>> > > > >>>>>>>>> Il gio 29 ago 2019, 16:59 Sijie Guo <guosi...@gmail.com> ha > > > >>>>> scritto: > > > >>>>>>>>> > > > >>>>>>>>>> Hi Dave, > > > >>>>>>>>>> > > > >>>>>>>>>> It doesn't require MySQL. You can use any JDBC supported > > > >>>>> databases. > > > >>>>>>>>>> Initially we would use SQLite for development and change to > > > >>>> MySQL > > > >>>>>> for > > > >>>>>>>>> more > > > >>>>>>>>>> production-ready workloads. > > > >>>>>>>>>> > > > >>>>>>>>>> Hope this clarify your question. > > > >>>>>>>>>> > > > >>>>>>>>>> Thanks, > > > >>>>>>>>>> Sijie > > > >>>>>>>>>> > > > >>>>>>>>>> On Thu, Aug 29, 2019 at 5:56 AM Dave Fisher < > w...@apache.org> > > > >>>>>> wrote: > > > >>>>>>>>>> > > > >>>>>>>>>>> Hi - > > > >>>>>>>>>>> > > > >>>>>>>>>>> I see that MySQL is used. This would be a GPL dependency. > > > >>>> Does > > > >>>>>> the > > > >>>>>>>>> Pulsar > > > >>>>>>>>>>> Manager require MySQL, or can other Databases that are > > > >>>>> compliant > > > >>>>>>> with > > > >>>>>>>>>>> Apache Release Policy be used instead? > > > >>>>>>>>>>> > > > >>>>>>>>>>> Regards, > > > >>>>>>>>>>> Dave > > > >>>>>>>>>>> > > > >>>>>>>>>>>> On Aug 29, 2019, at 7:49 AM, Guangning E < > > > >>>>> eguangn...@gmail.com > > > >>>>>>> > > > >>>>>>>>> wrote: > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Hi all, > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> We have developed a new Pulsar web UI - pulsar-manager, > > > >>>>> aiming > > > >>>>>> at > > > >>>>>>>>>>>> supporting managing Pulsar clusters running in different > > > >>>>>>>> environments > > > >>>>>>>>>>>> (on-premise data centers, cloud, and so on). We’d like to > > > >>>>>>>> contribute > > > >>>>>>>>>> the > > > >>>>>>>>>>>> project back to the Pulsar community. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> --- > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> ## Motivation > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Currently, Pulsar has a monitoring tool - [Pulsar > > > >>>>> Dashboard]( [ > > > >>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>> > > > >>>>>>> > > > >>>>>> > > > >>>>> > > > >>>> > > > >> > > > > > > http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/) > > > >>>>>>>>>>>> ). > > > >>>>>>>>>>>> However, it only focuses on the simple monitoring of > > > >>>> Pulsar - > > > >>>>>>>>>> collecting > > > >>>>>>>>>>>> and displaying information such as statistics of tenants, > > > >>>>>>>> namespaces, > > > >>>>>>>>>>>> topics, subscriptions, and so on. It doesn’t provide any > > > >>>>>>> management > > > >>>>>>>>>>>> operations such as add, delete and update tenants, > > > >>>>> namespaces, > > > >>>>>>>>> topics, > > > >>>>>>>>>>> and > > > >>>>>>>>>>>> so on. When the scale of Pulsar cluster increases or the > > > >>>>> number > > > >>>>>>> of > > > >>>>>>>>>>> clusters > > > >>>>>>>>>>>> grows, using `pulsar-admin` to manage Pulsar can not > > > >>>> satisfy > > > >>>>>>>> demands. > > > >>>>>>>>>>>> Therefore, Pulsar needs a simple and easy-to-use > management > > > >>>>>>> console > > > >>>>>>>>> for > > > >>>>>>>>>>>> administrators. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Pulsar Manager is a web-based GUI management and > monitoring > > > >>>>>> tool > > > >>>>>>>> that > > > >>>>>>>>>>> helps > > > >>>>>>>>>>>> administrators and users manage and monitor tenants, > > > >>>>>> namespaces, > > > >>>>>>>>>> topics, > > > >>>>>>>>>>>> subscriptions, brokers, clusters, and so on, and supports > > > >>>>>> dynamic > > > >>>>>>>>>>>> configuration of multiple environments. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> ## Features > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Pulsar manager provides two main features: management and > > > >>>>>>>> monitoring. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> ### Management > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> 1. Environment : (operations on the environment) > > > >>>>>>>>>>>> Create / Delete / Update / List / Get > > > >>>>>>>>>>>> 2. Cluster : (operations on clusters) > > > >>>>>>>>>>>> Create / Delete / Update / List / Get > > > >>>>>>>>>>>> 3. Brokers : (operations on brokers) > > > >>>>>>>>>>>> Heartbeat / Unload > > > >>>>>>>>>>>> 4. Tenants: > > > >>>>>>>>>>>> Create / Delete / Update / List > > > >>>>>>>>>>>> 5. Namespaces: > > > >>>>>>>>>>>> Create / Delete / List > > > >>>>>>>>>>>> Manage Namespace Policy > > > >>>>>>>>>>>> Unload > > > >>>>>>>>>>>> Operations on Namespace Bundles > > > >>>>>>>>>>>> 6. Topics > > > >>>>>>>>>>>> Create / Delete / List > > > >>>>>>>>>>>> Unload / Terminate / Offload / Compact > > > >>>>>>>>>>>> Topic Details > > > >>>>>>>>>>>> 7. Subscriptions > > > >>>>>>>>>>>> Create / Delete / List > > > >>>>>>>>>>>> Reset Cursor / Skip / Clear backlog / Unsubscribe > > > >>>>>>>>>>>> 8. Namespace Isolation Policies > > > >>>>>>>>>>>> Create / Delete / Update / Get / List > > > >>>>>>>>>>>> 9. Failure Domains > > > >>>>>>>>>>>> Create / Delete / Update / Get / List > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> ### Monitoring: (display *aggregated* stats at different > > > >>>>>> levels) > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> 1. Tenants > > > >>>>>>>>>>>> List the total number of namespaces per tenant (both > > > >>>>> aggregated > > > >>>>>>> and > > > >>>>>>>>>>>> per-cluster basis) > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> 2. Namespaces > > > >>>>>>>>>>>> List the total number of topics per namespace > > > >>>>>>>>>>>> List the aggregated stats per namespace (such as rate-in, > > > >>>>>>> rate-out, > > > >>>>>>>>>>>> throughput-in, and throughput-out) > > > >>>>>>>>>>>> List the distributions of namespace bundles > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> 3. Topics > > > >>>>>>>>>>>> List the total number of partitions per topic > > > >>>>>>>>>>>> List the aggregated stats per topic (such as rate-in, > > > >>>>> rate-out, > > > >>>>>>>>>>>> throughput-in, and throughput-out) > > > >>>>>>>>>>>> Detailed stats per topic partition > > > >>>>>>>>>>>> Detailed stats of storage per topic partition > > > >>>>>>>>>>>> Detailed stats of subscription per topic and per partition > > > >>>>>>>>>>>> Detailed stats of producers > > > >>>>>>>>>>>> Detailed stats of consumers > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> The detailed design proposal is in > > > >>>>>>>>>>>> [ > > > >>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>> > > > >>>>>>> > > > >>>>>> > > > >>>>> > > > >>>> > > > >> > > > > > > https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#) > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Looking forward to any feedback. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Thanks, > > > >>>>>>>>>>>> Guangning > > > >>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>> > > > >>>>>>> > > > >>>>>> > > > >>>>> > > > >>>> > > > >>>> > > > >>>> -- > > > >>>> *Thanks* > > > >>>> > > > >>>> *Yuvaraj L* > > > >>>> > > > >> > > > >> > > > > > > > > >
