Hi Dave, Thank you for the reminder. I was about to ping you yesterday about this. The IP clearance is the last piece that I wasn’t sure about. Just to make sure I understand correctly, so the Pulsar PMC has to fill out the IP clearance template, check in the file to the SVN repo and start a thread in general@incubator to get the approval from IPMC. Is that correct?
Thanks, Sijie On Wed, Sep 4, 2019 at 9:41 AM Dave Fisher <w...@apache.org> wrote: > BTW - Once the VOTE is complete to accept the codebase there is some > paperwork. > > http://incubator.apache.org/ip-clearance/ > > While this is the Incubator the Board has requested that the IPMC track IP > Clearance. Everything is in SVN and the pages are updated every evening > PDT. If help is needed then let me know. > > Regards, > Dave > > > On Sep 4, 2019, at 9:01 AM, Enrico Olivelli <eolive...@gmail.com> wrote: > > > > For whom who is interested I am working on adding support for HerdDB in > > Pulsar Manager. > > > > The pull request is still very raw, but Pulsar Manager seems to work > > https://github.com/streamnative/pulsar-manager/pull/183 > > > > > > > > Il giorno mar 3 set 2019 alle ore 08:37 Yuva raj <uvar...@gmail.com> ha > > scritto: > > > >> On Fri, 30 Aug 2019 at 07:09, Sijie Guo <guosi...@gmail.com> wrote: > >> > >>>> I see there are dependencies on websockets, I would a prefer a simple > >>> polling model of the http, also it's enable by default in pulsar. > >>> > >>> I don't think we depend don pulsar websocket. All are http restful > >>> requests. > >>> > >>>> For the ui the default persistence should be sqlite. Potentially > >> packaged > >>> by default. > >>> > >>> Initially the default is sqlite. But sqlite doesn't work if there are a > >> lot > >>> of topic metrics. > >>> We switched to MySQL as default for supporting production traffic. We > can > >>> switch default to PostgresSQL (as pulsar dashboard). > >>> > >> +1 > >> Yes, Switching to Postgres Would be great. Because Postgres license is > more > >> liberal and works well with Apache license ecosystem. > >> > >>> > >>> - Sijie > >>> > >>> On Thu, Aug 29, 2019 at 6:23 PM Ali Ahmed <alahmed...@gmail.com> > wrote: > >>> > >>>> I see there are dependencies on websockets, I would a prefer a simple > >>>> polling model of the http, also it's enable by default in pulsar. > >>>> > >>>> For the ui the default persistence should be sqlite. Potentially > >> packaged > >>>> by default. > >>>> > >>>> -Ali > >>>> > >>>> On Thu, Aug 29, 2019 at 2:47 PM Enrico Olivelli <eolive...@gmail.com> > >>>> wrote: > >>>> > >>>>> Il gio 29 ago 2019, 23:28 Sijie Guo <guosi...@gmail.com> ha scritto: > >>>>> > >>>>>> That sounds an interesting idea! > >>>>> > >>>>> > >>>>> Awesome > >>>>> > >>>>> Does HerdDB support JDBC? If so, it should > >>>>>> be pretty straightforward to enable HerdDB. > >>>>>> > >>>>> > >>>>> Sure, as far as I know, the JDBC client is the only client really > >> used > >>> in > >>>>> production. > >>>>> > >>>>> In a replicated environment it uses Zookeeper for metadata and > >> service > >>>>> discovery and Bookkeeper as WAL and if you have a pulsar cluster you > >>>>> already have both of them (zk cluster an bookies) > >>>>> > >>>>> I will take a look to how Pulsar Manager uses JDBC, maybe it will > >> be > >>>> very > >>>>> easy. > >>>>> > >>>>> I will be back with news > >>>>> > >>>>> > >>>>> Enrico > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>> Thanks, > >>>>>> Sijie > >>>>>> > >>>>>> On Thu, Aug 29, 2019 at 12:08 PM Enrico Olivelli < > >>> eolive...@gmail.com> > >>>>>> wrote: > >>>>>> > >>>>>>> Maybe you can try to use HerdDB (1), it is a replicated SQL > >>> database > >>>>> that > >>>>>>> can be run embedded in the JVM. > >>>>>>> It is an SQL database that we developed in my company, in order > >> to > >>>>>>> explicitly replace MySQL. > >>>>>>> It uses Bookkeeper to store the WAL and implement replication, > >> but > >>> it > >>>>> can > >>>>>>> run in standalone mode or in memory (for tests/dev). > >>>>>>> > >>>>>>> I can help in setting up a demo. > >>>>>>> > >>>>>>> Enrico > >>>>>>> > >>>>>>> (1) https://github.com/diennea/herddb > >>>>>>> > >>>>>>> Il gio 29 ago 2019, 16:59 Sijie Guo <guosi...@gmail.com> ha > >>> scritto: > >>>>>>> > >>>>>>>> Hi Dave, > >>>>>>>> > >>>>>>>> It doesn't require MySQL. You can use any JDBC supported > >>> databases. > >>>>>>>> Initially we would use SQLite for development and change to > >> MySQL > >>>> for > >>>>>>> more > >>>>>>>> production-ready workloads. > >>>>>>>> > >>>>>>>> Hope this clarify your question. > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Sijie > >>>>>>>> > >>>>>>>> On Thu, Aug 29, 2019 at 5:56 AM Dave Fisher <w...@apache.org> > >>>> wrote: > >>>>>>>> > >>>>>>>>> Hi - > >>>>>>>>> > >>>>>>>>> I see that MySQL is used. This would be a GPL dependency. > >> Does > >>>> the > >>>>>>> Pulsar > >>>>>>>>> Manager require MySQL, or can other Databases that are > >>> compliant > >>>>> with > >>>>>>>>> Apache Release Policy be used instead? > >>>>>>>>> > >>>>>>>>> Regards, > >>>>>>>>> Dave > >>>>>>>>> > >>>>>>>>>> On Aug 29, 2019, at 7:49 AM, Guangning E < > >>> eguangn...@gmail.com > >>>>> > >>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> Hi all, > >>>>>>>>>> > >>>>>>>>>> We have developed a new Pulsar web UI - pulsar-manager, > >>> aiming > >>>> at > >>>>>>>>>> supporting managing Pulsar clusters running in different > >>>>>> environments > >>>>>>>>>> (on-premise data centers, cloud, and so on). We’d like to > >>>>>> contribute > >>>>>>>> the > >>>>>>>>>> project back to the Pulsar community. > >>>>>>>>>> > >>>>>>>>>> --- > >>>>>>>>>> > >>>>>>>>>> ## Motivation > >>>>>>>>>> > >>>>>>>>>> Currently, Pulsar has a monitoring tool - [Pulsar > >>> Dashboard]( [ > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/) > >>>>>>>>>> ). > >>>>>>>>>> However, it only focuses on the simple monitoring of > >> Pulsar - > >>>>>>>> collecting > >>>>>>>>>> and displaying information such as statistics of tenants, > >>>>>> namespaces, > >>>>>>>>>> topics, subscriptions, and so on. It doesn’t provide any > >>>>> management > >>>>>>>>>> operations such as add, delete and update tenants, > >>> namespaces, > >>>>>>> topics, > >>>>>>>>> and > >>>>>>>>>> so on. When the scale of Pulsar cluster increases or the > >>> number > >>>>> of > >>>>>>>>> clusters > >>>>>>>>>> grows, using `pulsar-admin` to manage Pulsar can not > >> satisfy > >>>>>> demands. > >>>>>>>>>> Therefore, Pulsar needs a simple and easy-to-use management > >>>>> console > >>>>>>> for > >>>>>>>>>> administrators. > >>>>>>>>>> > >>>>>>>>>> Pulsar Manager is a web-based GUI management and monitoring > >>>> tool > >>>>>> that > >>>>>>>>> helps > >>>>>>>>>> administrators and users manage and monitor tenants, > >>>> namespaces, > >>>>>>>> topics, > >>>>>>>>>> subscriptions, brokers, clusters, and so on, and supports > >>>> dynamic > >>>>>>>>>> configuration of multiple environments. > >>>>>>>>>> > >>>>>>>>>> ## Features > >>>>>>>>>> > >>>>>>>>>> Pulsar manager provides two main features: management and > >>>>>> monitoring. > >>>>>>>>>> > >>>>>>>>>> ### Management > >>>>>>>>>> > >>>>>>>>>> 1. Environment : (operations on the environment) > >>>>>>>>>> Create / Delete / Update / List / Get > >>>>>>>>>> 2. Cluster : (operations on clusters) > >>>>>>>>>> Create / Delete / Update / List / Get > >>>>>>>>>> 3. Brokers : (operations on brokers) > >>>>>>>>>> Heartbeat / Unload > >>>>>>>>>> 4. Tenants: > >>>>>>>>>> Create / Delete / Update / List > >>>>>>>>>> 5. Namespaces: > >>>>>>>>>> Create / Delete / List > >>>>>>>>>> Manage Namespace Policy > >>>>>>>>>> Unload > >>>>>>>>>> Operations on Namespace Bundles > >>>>>>>>>> 6. Topics > >>>>>>>>>> Create / Delete / List > >>>>>>>>>> Unload / Terminate / Offload / Compact > >>>>>>>>>> Topic Details > >>>>>>>>>> 7. Subscriptions > >>>>>>>>>> Create / Delete / List > >>>>>>>>>> Reset Cursor / Skip / Clear backlog / Unsubscribe > >>>>>>>>>> 8. Namespace Isolation Policies > >>>>>>>>>> Create / Delete / Update / Get / List > >>>>>>>>>> 9. Failure Domains > >>>>>>>>>> Create / Delete / Update / Get / List > >>>>>>>>>> > >>>>>>>>>> ### Monitoring: (display *aggregated* stats at different > >>>> levels) > >>>>>>>>>> > >>>>>>>>>> 1. Tenants > >>>>>>>>>> List the total number of namespaces per tenant (both > >>> aggregated > >>>>> and > >>>>>>>>>> per-cluster basis) > >>>>>>>>>> > >>>>>>>>>> 2. Namespaces > >>>>>>>>>> List the total number of topics per namespace > >>>>>>>>>> List the aggregated stats per namespace (such as rate-in, > >>>>> rate-out, > >>>>>>>>>> throughput-in, and throughput-out) > >>>>>>>>>> List the distributions of namespace bundles > >>>>>>>>>> > >>>>>>>>>> 3. Topics > >>>>>>>>>> List the total number of partitions per topic > >>>>>>>>>> List the aggregated stats per topic (such as rate-in, > >>> rate-out, > >>>>>>>>>> throughput-in, and throughput-out) > >>>>>>>>>> Detailed stats per topic partition > >>>>>>>>>> Detailed stats of storage per topic partition > >>>>>>>>>> Detailed stats of subscription per topic and per partition > >>>>>>>>>> Detailed stats of producers > >>>>>>>>>> Detailed stats of consumers > >>>>>>>>>> > >>>>>>>>>> The detailed design proposal is in > >>>>>>>>>> [ > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#) > >>>>>>>>>> > >>>>>>>>>> Looking forward to any feedback. > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> Guangning > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > >> > >> -- > >> *Thanks* > >> > >> *Yuvaraj L* > >> > >
