Dave, thank you! That’s super helpful! Thanks, Sijie
On Wed, Sep 4, 2019 at 10:29 AM Dave Fisher <w...@apache.org> wrote: > > > > On Sep 4, 2019, at 10:10 AM, Sijie Guo <guosi...@gmail.com> wrote: > > > > Hi Dave, > > > > Thank you for the reminder. I was about to ping you yesterday about this. > > The IP clearance is the last piece that I wasn’t sure about. Just to make > > sure I understand correctly, so the Pulsar PMC has to fill out the IP > > clearance template, check in the file to the SVN repo and start a thread > in > > general@incubator to get the approval from IPMC. Is that correct? > > Yes. The approval is via LAZY CONSENSUS of 3 days. > > StreamNative ought to provide an SGA to secretary@ unless all of the IP > is already covered by ICLAs. > > Infra can then help transfer the repository. > > Regards, > Dave > > > > > Thanks, > > Sijie > > > > On Wed, Sep 4, 2019 at 9:41 AM Dave Fisher <w...@apache.org> wrote: > > > >> BTW - Once the VOTE is complete to accept the codebase there is some > >> paperwork. > >> > >> http://incubator.apache.org/ip-clearance/ > >> > >> While this is the Incubator the Board has requested that the IPMC track > IP > >> Clearance. Everything is in SVN and the pages are updated every evening > >> PDT. If help is needed then let me know. > >> > >> Regards, > >> Dave > >> > >>> On Sep 4, 2019, at 9:01 AM, Enrico Olivelli <eolive...@gmail.com> > wrote: > >>> > >>> For whom who is interested I am working on adding support for HerdDB in > >>> Pulsar Manager. > >>> > >>> The pull request is still very raw, but Pulsar Manager seems to work > >>> https://github.com/streamnative/pulsar-manager/pull/183 > >>> > >>> > >>> > >>> Il giorno mar 3 set 2019 alle ore 08:37 Yuva raj <uvar...@gmail.com> > ha > >>> scritto: > >>> > >>>> On Fri, 30 Aug 2019 at 07:09, Sijie Guo <guosi...@gmail.com> wrote: > >>>> > >>>>>> I see there are dependencies on websockets, I would a prefer a > simple > >>>>> polling model of the http, also it's enable by default in pulsar. > >>>>> > >>>>> I don't think we depend don pulsar websocket. All are http restful > >>>>> requests. > >>>>> > >>>>>> For the ui the default persistence should be sqlite. Potentially > >>>> packaged > >>>>> by default. > >>>>> > >>>>> Initially the default is sqlite. But sqlite doesn't work if there > are a > >>>> lot > >>>>> of topic metrics. > >>>>> We switched to MySQL as default for supporting production traffic. We > >> can > >>>>> switch default to PostgresSQL (as pulsar dashboard). > >>>>> > >>>> +1 > >>>> Yes, Switching to Postgres Would be great. Because Postgres license is > >> more > >>>> liberal and works well with Apache license ecosystem. > >>>> > >>>>> > >>>>> - Sijie > >>>>> > >>>>> On Thu, Aug 29, 2019 at 6:23 PM Ali Ahmed <alahmed...@gmail.com> > >> wrote: > >>>>> > >>>>>> I see there are dependencies on websockets, I would a prefer a > simple > >>>>>> polling model of the http, also it's enable by default in pulsar. > >>>>>> > >>>>>> For the ui the default persistence should be sqlite. Potentially > >>>> packaged > >>>>>> by default. > >>>>>> > >>>>>> -Ali > >>>>>> > >>>>>> On Thu, Aug 29, 2019 at 2:47 PM Enrico Olivelli < > eolive...@gmail.com> > >>>>>> wrote: > >>>>>> > >>>>>>> Il gio 29 ago 2019, 23:28 Sijie Guo <guosi...@gmail.com> ha > scritto: > >>>>>>> > >>>>>>>> That sounds an interesting idea! > >>>>>>> > >>>>>>> > >>>>>>> Awesome > >>>>>>> > >>>>>>> Does HerdDB support JDBC? If so, it should > >>>>>>>> be pretty straightforward to enable HerdDB. > >>>>>>>> > >>>>>>> > >>>>>>> Sure, as far as I know, the JDBC client is the only client really > >>>> used > >>>>> in > >>>>>>> production. > >>>>>>> > >>>>>>> In a replicated environment it uses Zookeeper for metadata and > >>>> service > >>>>>>> discovery and Bookkeeper as WAL and if you have a pulsar cluster > you > >>>>>>> already have both of them (zk cluster an bookies) > >>>>>>> > >>>>>>> I will take a look to how Pulsar Manager uses JDBC, maybe it will > >>>> be > >>>>>> very > >>>>>>> easy. > >>>>>>> > >>>>>>> I will be back with news > >>>>>>> > >>>>>>> > >>>>>>> Enrico > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Sijie > >>>>>>>> > >>>>>>>> On Thu, Aug 29, 2019 at 12:08 PM Enrico Olivelli < > >>>>> eolive...@gmail.com> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> Maybe you can try to use HerdDB (1), it is a replicated SQL > >>>>> database > >>>>>>> that > >>>>>>>>> can be run embedded in the JVM. > >>>>>>>>> It is an SQL database that we developed in my company, in order > >>>> to > >>>>>>>>> explicitly replace MySQL. > >>>>>>>>> It uses Bookkeeper to store the WAL and implement replication, > >>>> but > >>>>> it > >>>>>>> can > >>>>>>>>> run in standalone mode or in memory (for tests/dev). > >>>>>>>>> > >>>>>>>>> I can help in setting up a demo. > >>>>>>>>> > >>>>>>>>> Enrico > >>>>>>>>> > >>>>>>>>> (1) https://github.com/diennea/herddb > >>>>>>>>> > >>>>>>>>> Il gio 29 ago 2019, 16:59 Sijie Guo <guosi...@gmail.com> ha > >>>>> scritto: > >>>>>>>>> > >>>>>>>>>> Hi Dave, > >>>>>>>>>> > >>>>>>>>>> It doesn't require MySQL. You can use any JDBC supported > >>>>> databases. > >>>>>>>>>> Initially we would use SQLite for development and change to > >>>> MySQL > >>>>>> for > >>>>>>>>> more > >>>>>>>>>> production-ready workloads. > >>>>>>>>>> > >>>>>>>>>> Hope this clarify your question. > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> Sijie > >>>>>>>>>> > >>>>>>>>>> On Thu, Aug 29, 2019 at 5:56 AM Dave Fisher <w...@apache.org> > >>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> Hi - > >>>>>>>>>>> > >>>>>>>>>>> I see that MySQL is used. This would be a GPL dependency. > >>>> Does > >>>>>> the > >>>>>>>>> Pulsar > >>>>>>>>>>> Manager require MySQL, or can other Databases that are > >>>>> compliant > >>>>>>> with > >>>>>>>>>>> Apache Release Policy be used instead? > >>>>>>>>>>> > >>>>>>>>>>> Regards, > >>>>>>>>>>> Dave > >>>>>>>>>>> > >>>>>>>>>>>> On Aug 29, 2019, at 7:49 AM, Guangning E < > >>>>> eguangn...@gmail.com > >>>>>>> > >>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> Hi all, > >>>>>>>>>>>> > >>>>>>>>>>>> We have developed a new Pulsar web UI - pulsar-manager, > >>>>> aiming > >>>>>> at > >>>>>>>>>>>> supporting managing Pulsar clusters running in different > >>>>>>>> environments > >>>>>>>>>>>> (on-premise data centers, cloud, and so on). We’d like to > >>>>>>>> contribute > >>>>>>>>>> the > >>>>>>>>>>>> project back to the Pulsar community. > >>>>>>>>>>>> > >>>>>>>>>>>> --- > >>>>>>>>>>>> > >>>>>>>>>>>> ## Motivation > >>>>>>>>>>>> > >>>>>>>>>>>> Currently, Pulsar has a monitoring tool - [Pulsar > >>>>> Dashboard]( [ > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >> > http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/) > >>>>>>>>>>>> ). > >>>>>>>>>>>> However, it only focuses on the simple monitoring of > >>>> Pulsar - > >>>>>>>>>> collecting > >>>>>>>>>>>> and displaying information such as statistics of tenants, > >>>>>>>> namespaces, > >>>>>>>>>>>> topics, subscriptions, and so on. It doesn’t provide any > >>>>>>> management > >>>>>>>>>>>> operations such as add, delete and update tenants, > >>>>> namespaces, > >>>>>>>>> topics, > >>>>>>>>>>> and > >>>>>>>>>>>> so on. When the scale of Pulsar cluster increases or the > >>>>> number > >>>>>>> of > >>>>>>>>>>> clusters > >>>>>>>>>>>> grows, using `pulsar-admin` to manage Pulsar can not > >>>> satisfy > >>>>>>>> demands. > >>>>>>>>>>>> Therefore, Pulsar needs a simple and easy-to-use management > >>>>>>> console > >>>>>>>>> for > >>>>>>>>>>>> administrators. > >>>>>>>>>>>> > >>>>>>>>>>>> Pulsar Manager is a web-based GUI management and monitoring > >>>>>> tool > >>>>>>>> that > >>>>>>>>>>> helps > >>>>>>>>>>>> administrators and users manage and monitor tenants, > >>>>>> namespaces, > >>>>>>>>>> topics, > >>>>>>>>>>>> subscriptions, brokers, clusters, and so on, and supports > >>>>>> dynamic > >>>>>>>>>>>> configuration of multiple environments. > >>>>>>>>>>>> > >>>>>>>>>>>> ## Features > >>>>>>>>>>>> > >>>>>>>>>>>> Pulsar manager provides two main features: management and > >>>>>>>> monitoring. > >>>>>>>>>>>> > >>>>>>>>>>>> ### Management > >>>>>>>>>>>> > >>>>>>>>>>>> 1. Environment : (operations on the environment) > >>>>>>>>>>>> Create / Delete / Update / List / Get > >>>>>>>>>>>> 2. Cluster : (operations on clusters) > >>>>>>>>>>>> Create / Delete / Update / List / Get > >>>>>>>>>>>> 3. Brokers : (operations on brokers) > >>>>>>>>>>>> Heartbeat / Unload > >>>>>>>>>>>> 4. Tenants: > >>>>>>>>>>>> Create / Delete / Update / List > >>>>>>>>>>>> 5. Namespaces: > >>>>>>>>>>>> Create / Delete / List > >>>>>>>>>>>> Manage Namespace Policy > >>>>>>>>>>>> Unload > >>>>>>>>>>>> Operations on Namespace Bundles > >>>>>>>>>>>> 6. Topics > >>>>>>>>>>>> Create / Delete / List > >>>>>>>>>>>> Unload / Terminate / Offload / Compact > >>>>>>>>>>>> Topic Details > >>>>>>>>>>>> 7. Subscriptions > >>>>>>>>>>>> Create / Delete / List > >>>>>>>>>>>> Reset Cursor / Skip / Clear backlog / Unsubscribe > >>>>>>>>>>>> 8. Namespace Isolation Policies > >>>>>>>>>>>> Create / Delete / Update / Get / List > >>>>>>>>>>>> 9. Failure Domains > >>>>>>>>>>>> Create / Delete / Update / Get / List > >>>>>>>>>>>> > >>>>>>>>>>>> ### Monitoring: (display *aggregated* stats at different > >>>>>> levels) > >>>>>>>>>>>> > >>>>>>>>>>>> 1. Tenants > >>>>>>>>>>>> List the total number of namespaces per tenant (both > >>>>> aggregated > >>>>>>> and > >>>>>>>>>>>> per-cluster basis) > >>>>>>>>>>>> > >>>>>>>>>>>> 2. Namespaces > >>>>>>>>>>>> List the total number of topics per namespace > >>>>>>>>>>>> List the aggregated stats per namespace (such as rate-in, > >>>>>>> rate-out, > >>>>>>>>>>>> throughput-in, and throughput-out) > >>>>>>>>>>>> List the distributions of namespace bundles > >>>>>>>>>>>> > >>>>>>>>>>>> 3. Topics > >>>>>>>>>>>> List the total number of partitions per topic > >>>>>>>>>>>> List the aggregated stats per topic (such as rate-in, > >>>>> rate-out, > >>>>>>>>>>>> throughput-in, and throughput-out) > >>>>>>>>>>>> Detailed stats per topic partition > >>>>>>>>>>>> Detailed stats of storage per topic partition > >>>>>>>>>>>> Detailed stats of subscription per topic and per partition > >>>>>>>>>>>> Detailed stats of producers > >>>>>>>>>>>> Detailed stats of consumers > >>>>>>>>>>>> > >>>>>>>>>>>> The detailed design proposal is in > >>>>>>>>>>>> [ > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >> > https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#) > >>>>>>>>>>>> > >>>>>>>>>>>> Looking forward to any feedback. > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks, > >>>>>>>>>>>> Guangning > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>>> > >>>> -- > >>>> *Thanks* > >>>> > >>>> *Yuvaraj L* > >>>> > >> > >> > >
