+1
Best Regards. Jia Zhai Beijing, China Mobile: +86 15810491983 On Thu, Sep 5, 2019 at 3:34 AM Sijie Guo <guosi...@gmail.com> wrote: > Dave, thank you! That’s super helpful! > > Thanks, > Sijie > > On Wed, Sep 4, 2019 at 10:29 AM Dave Fisher <w...@apache.org> wrote: > > > > > > > > On Sep 4, 2019, at 10:10 AM, Sijie Guo <guosi...@gmail.com> wrote: > > > > > > Hi Dave, > > > > > > Thank you for the reminder. I was about to ping you yesterday about > this. > > > The IP clearance is the last piece that I wasn’t sure about. Just to > make > > > sure I understand correctly, so the Pulsar PMC has to fill out the IP > > > clearance template, check in the file to the SVN repo and start a > thread > > in > > > general@incubator to get the approval from IPMC. Is that correct? > > > > Yes. The approval is via LAZY CONSENSUS of 3 days. > > > > StreamNative ought to provide an SGA to secretary@ unless all of the IP > > is already covered by ICLAs. > > > > Infra can then help transfer the repository. > > > > Regards, > > Dave > > > > > > > > Thanks, > > > Sijie > > > > > > On Wed, Sep 4, 2019 at 9:41 AM Dave Fisher <w...@apache.org> wrote: > > > > > >> BTW - Once the VOTE is complete to accept the codebase there is some > > >> paperwork. > > >> > > >> http://incubator.apache.org/ip-clearance/ > > >> > > >> While this is the Incubator the Board has requested that the IPMC > track > > IP > > >> Clearance. Everything is in SVN and the pages are updated every > evening > > >> PDT. If help is needed then let me know. > > >> > > >> Regards, > > >> Dave > > >> > > >>> On Sep 4, 2019, at 9:01 AM, Enrico Olivelli <eolive...@gmail.com> > > wrote: > > >>> > > >>> For whom who is interested I am working on adding support for HerdDB > in > > >>> Pulsar Manager. > > >>> > > >>> The pull request is still very raw, but Pulsar Manager seems to work > > >>> https://github.com/streamnative/pulsar-manager/pull/183 > > >>> > > >>> > > >>> > > >>> Il giorno mar 3 set 2019 alle ore 08:37 Yuva raj <uvar...@gmail.com> > > ha > > >>> scritto: > > >>> > > >>>> On Fri, 30 Aug 2019 at 07:09, Sijie Guo <guosi...@gmail.com> wrote: > > >>>> > > >>>>>> I see there are dependencies on websockets, I would a prefer a > > simple > > >>>>> polling model of the http, also it's enable by default in pulsar. > > >>>>> > > >>>>> I don't think we depend don pulsar websocket. All are http restful > > >>>>> requests. > > >>>>> > > >>>>>> For the ui the default persistence should be sqlite. Potentially > > >>>> packaged > > >>>>> by default. > > >>>>> > > >>>>> Initially the default is sqlite. But sqlite doesn't work if there > > are a > > >>>> lot > > >>>>> of topic metrics. > > >>>>> We switched to MySQL as default for supporting production traffic. > We > > >> can > > >>>>> switch default to PostgresSQL (as pulsar dashboard). > > >>>>> > > >>>> +1 > > >>>> Yes, Switching to Postgres Would be great. Because Postgres license > is > > >> more > > >>>> liberal and works well with Apache license ecosystem. > > >>>> > > >>>>> > > >>>>> - Sijie > > >>>>> > > >>>>> On Thu, Aug 29, 2019 at 6:23 PM Ali Ahmed <alahmed...@gmail.com> > > >> wrote: > > >>>>> > > >>>>>> I see there are dependencies on websockets, I would a prefer a > > simple > > >>>>>> polling model of the http, also it's enable by default in pulsar. > > >>>>>> > > >>>>>> For the ui the default persistence should be sqlite. Potentially > > >>>> packaged > > >>>>>> by default. > > >>>>>> > > >>>>>> -Ali > > >>>>>> > > >>>>>> On Thu, Aug 29, 2019 at 2:47 PM Enrico Olivelli < > > eolive...@gmail.com> > > >>>>>> wrote: > > >>>>>> > > >>>>>>> Il gio 29 ago 2019, 23:28 Sijie Guo <guosi...@gmail.com> ha > > scritto: > > >>>>>>> > > >>>>>>>> That sounds an interesting idea! > > >>>>>>> > > >>>>>>> > > >>>>>>> Awesome > > >>>>>>> > > >>>>>>> Does HerdDB support JDBC? If so, it should > > >>>>>>>> be pretty straightforward to enable HerdDB. > > >>>>>>>> > > >>>>>>> > > >>>>>>> Sure, as far as I know, the JDBC client is the only client really > > >>>> used > > >>>>> in > > >>>>>>> production. > > >>>>>>> > > >>>>>>> In a replicated environment it uses Zookeeper for metadata and > > >>>> service > > >>>>>>> discovery and Bookkeeper as WAL and if you have a pulsar cluster > > you > > >>>>>>> already have both of them (zk cluster an bookies) > > >>>>>>> > > >>>>>>> I will take a look to how Pulsar Manager uses JDBC, maybe it > will > > >>>> be > > >>>>>> very > > >>>>>>> easy. > > >>>>>>> > > >>>>>>> I will be back with news > > >>>>>>> > > >>>>>>> > > >>>>>>> Enrico > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>>> Thanks, > > >>>>>>>> Sijie > > >>>>>>>> > > >>>>>>>> On Thu, Aug 29, 2019 at 12:08 PM Enrico Olivelli < > > >>>>> eolive...@gmail.com> > > >>>>>>>> wrote: > > >>>>>>>> > > >>>>>>>>> Maybe you can try to use HerdDB (1), it is a replicated SQL > > >>>>> database > > >>>>>>> that > > >>>>>>>>> can be run embedded in the JVM. > > >>>>>>>>> It is an SQL database that we developed in my company, in order > > >>>> to > > >>>>>>>>> explicitly replace MySQL. > > >>>>>>>>> It uses Bookkeeper to store the WAL and implement replication, > > >>>> but > > >>>>> it > > >>>>>>> can > > >>>>>>>>> run in standalone mode or in memory (for tests/dev). > > >>>>>>>>> > > >>>>>>>>> I can help in setting up a demo. > > >>>>>>>>> > > >>>>>>>>> Enrico > > >>>>>>>>> > > >>>>>>>>> (1) https://github.com/diennea/herddb > > >>>>>>>>> > > >>>>>>>>> Il gio 29 ago 2019, 16:59 Sijie Guo <guosi...@gmail.com> ha > > >>>>> scritto: > > >>>>>>>>> > > >>>>>>>>>> Hi Dave, > > >>>>>>>>>> > > >>>>>>>>>> It doesn't require MySQL. You can use any JDBC supported > > >>>>> databases. > > >>>>>>>>>> Initially we would use SQLite for development and change to > > >>>> MySQL > > >>>>>> for > > >>>>>>>>> more > > >>>>>>>>>> production-ready workloads. > > >>>>>>>>>> > > >>>>>>>>>> Hope this clarify your question. > > >>>>>>>>>> > > >>>>>>>>>> Thanks, > > >>>>>>>>>> Sijie > > >>>>>>>>>> > > >>>>>>>>>> On Thu, Aug 29, 2019 at 5:56 AM Dave Fisher <w...@apache.org> > > >>>>>> wrote: > > >>>>>>>>>> > > >>>>>>>>>>> Hi - > > >>>>>>>>>>> > > >>>>>>>>>>> I see that MySQL is used. This would be a GPL dependency. > > >>>> Does > > >>>>>> the > > >>>>>>>>> Pulsar > > >>>>>>>>>>> Manager require MySQL, or can other Databases that are > > >>>>> compliant > > >>>>>>> with > > >>>>>>>>>>> Apache Release Policy be used instead? > > >>>>>>>>>>> > > >>>>>>>>>>> Regards, > > >>>>>>>>>>> Dave > > >>>>>>>>>>> > > >>>>>>>>>>>> On Aug 29, 2019, at 7:49 AM, Guangning E < > > >>>>> eguangn...@gmail.com > > >>>>>>> > > >>>>>>>>> wrote: > > >>>>>>>>>>>> > > >>>>>>>>>>>> Hi all, > > >>>>>>>>>>>> > > >>>>>>>>>>>> We have developed a new Pulsar web UI - pulsar-manager, > > >>>>> aiming > > >>>>>> at > > >>>>>>>>>>>> supporting managing Pulsar clusters running in different > > >>>>>>>> environments > > >>>>>>>>>>>> (on-premise data centers, cloud, and so on). We’d like to > > >>>>>>>> contribute > > >>>>>>>>>> the > > >>>>>>>>>>>> project back to the Pulsar community. > > >>>>>>>>>>>> > > >>>>>>>>>>>> --- > > >>>>>>>>>>>> > > >>>>>>>>>>>> ## Motivation > > >>>>>>>>>>>> > > >>>>>>>>>>>> Currently, Pulsar has a monitoring tool - [Pulsar > > >>>>> Dashboard]( [ > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>>> > > >>>>> > > >>>> > > >> > > > http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/) > > >>>>>>>>>>>> ). > > >>>>>>>>>>>> However, it only focuses on the simple monitoring of > > >>>> Pulsar - > > >>>>>>>>>> collecting > > >>>>>>>>>>>> and displaying information such as statistics of tenants, > > >>>>>>>> namespaces, > > >>>>>>>>>>>> topics, subscriptions, and so on. It doesn’t provide any > > >>>>>>> management > > >>>>>>>>>>>> operations such as add, delete and update tenants, > > >>>>> namespaces, > > >>>>>>>>> topics, > > >>>>>>>>>>> and > > >>>>>>>>>>>> so on. When the scale of Pulsar cluster increases or the > > >>>>> number > > >>>>>>> of > > >>>>>>>>>>> clusters > > >>>>>>>>>>>> grows, using `pulsar-admin` to manage Pulsar can not > > >>>> satisfy > > >>>>>>>> demands. > > >>>>>>>>>>>> Therefore, Pulsar needs a simple and easy-to-use management > > >>>>>>> console > > >>>>>>>>> for > > >>>>>>>>>>>> administrators. > > >>>>>>>>>>>> > > >>>>>>>>>>>> Pulsar Manager is a web-based GUI management and monitoring > > >>>>>> tool > > >>>>>>>> that > > >>>>>>>>>>> helps > > >>>>>>>>>>>> administrators and users manage and monitor tenants, > > >>>>>> namespaces, > > >>>>>>>>>> topics, > > >>>>>>>>>>>> subscriptions, brokers, clusters, and so on, and supports > > >>>>>> dynamic > > >>>>>>>>>>>> configuration of multiple environments. > > >>>>>>>>>>>> > > >>>>>>>>>>>> ## Features > > >>>>>>>>>>>> > > >>>>>>>>>>>> Pulsar manager provides two main features: management and > > >>>>>>>> monitoring. > > >>>>>>>>>>>> > > >>>>>>>>>>>> ### Management > > >>>>>>>>>>>> > > >>>>>>>>>>>> 1. Environment : (operations on the environment) > > >>>>>>>>>>>> Create / Delete / Update / List / Get > > >>>>>>>>>>>> 2. Cluster : (operations on clusters) > > >>>>>>>>>>>> Create / Delete / Update / List / Get > > >>>>>>>>>>>> 3. Brokers : (operations on brokers) > > >>>>>>>>>>>> Heartbeat / Unload > > >>>>>>>>>>>> 4. Tenants: > > >>>>>>>>>>>> Create / Delete / Update / List > > >>>>>>>>>>>> 5. Namespaces: > > >>>>>>>>>>>> Create / Delete / List > > >>>>>>>>>>>> Manage Namespace Policy > > >>>>>>>>>>>> Unload > > >>>>>>>>>>>> Operations on Namespace Bundles > > >>>>>>>>>>>> 6. Topics > > >>>>>>>>>>>> Create / Delete / List > > >>>>>>>>>>>> Unload / Terminate / Offload / Compact > > >>>>>>>>>>>> Topic Details > > >>>>>>>>>>>> 7. Subscriptions > > >>>>>>>>>>>> Create / Delete / List > > >>>>>>>>>>>> Reset Cursor / Skip / Clear backlog / Unsubscribe > > >>>>>>>>>>>> 8. Namespace Isolation Policies > > >>>>>>>>>>>> Create / Delete / Update / Get / List > > >>>>>>>>>>>> 9. Failure Domains > > >>>>>>>>>>>> Create / Delete / Update / Get / List > > >>>>>>>>>>>> > > >>>>>>>>>>>> ### Monitoring: (display *aggregated* stats at different > > >>>>>> levels) > > >>>>>>>>>>>> > > >>>>>>>>>>>> 1. Tenants > > >>>>>>>>>>>> List the total number of namespaces per tenant (both > > >>>>> aggregated > > >>>>>>> and > > >>>>>>>>>>>> per-cluster basis) > > >>>>>>>>>>>> > > >>>>>>>>>>>> 2. Namespaces > > >>>>>>>>>>>> List the total number of topics per namespace > > >>>>>>>>>>>> List the aggregated stats per namespace (such as rate-in, > > >>>>>>> rate-out, > > >>>>>>>>>>>> throughput-in, and throughput-out) > > >>>>>>>>>>>> List the distributions of namespace bundles > > >>>>>>>>>>>> > > >>>>>>>>>>>> 3. Topics > > >>>>>>>>>>>> List the total number of partitions per topic > > >>>>>>>>>>>> List the aggregated stats per topic (such as rate-in, > > >>>>> rate-out, > > >>>>>>>>>>>> throughput-in, and throughput-out) > > >>>>>>>>>>>> Detailed stats per topic partition > > >>>>>>>>>>>> Detailed stats of storage per topic partition > > >>>>>>>>>>>> Detailed stats of subscription per topic and per partition > > >>>>>>>>>>>> Detailed stats of producers > > >>>>>>>>>>>> Detailed stats of consumers > > >>>>>>>>>>>> > > >>>>>>>>>>>> The detailed design proposal is in > > >>>>>>>>>>>> [ > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>>> > > >>>>> > > >>>> > > >> > > > https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#) > > >>>>>>>>>>>> > > >>>>>>>>>>>> Looking forward to any feedback. > > >>>>>>>>>>>> > > >>>>>>>>>>>> Thanks, > > >>>>>>>>>>>> Guangning > > >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>>> > > >>>> -- > > >>>> *Thanks* > > >>>> > > >>>> *Yuvaraj L* > > >>>> > > >> > > >> > > > > >
