> On Sep 4, 2019, at 10:10 AM, Sijie Guo <guosi...@gmail.com> wrote:
>
> Hi Dave,
>
> Thank you for the reminder. I was about to ping you yesterday about this.
> The IP clearance is the last piece that I wasn’t sure about. Just to make
> sure I understand correctly, so the Pulsar PMC has to fill out the IP
> clearance template, check in the file to the SVN repo and start a thread in
> general@incubator to get the approval from IPMC. Is that correct?
Yes. The approval is via LAZY CONSENSUS of 3 days.
StreamNative ought to provide an SGA to secretary@ unless all of the IP is
already covered by ICLAs.
Infra can then help transfer the repository.
Regards,
Dave
>
> Thanks,
> Sijie
>
> On Wed, Sep 4, 2019 at 9:41 AM Dave Fisher <w...@apache.org> wrote:
>
>> BTW - Once the VOTE is complete to accept the codebase there is some
>> paperwork.
>>
>> http://incubator.apache.org/ip-clearance/
>>
>> While this is the Incubator the Board has requested that the IPMC track IP
>> Clearance. Everything is in SVN and the pages are updated every evening
>> PDT. If help is needed then let me know.
>>
>> Regards,
>> Dave
>>
>>> On Sep 4, 2019, at 9:01 AM, Enrico Olivelli <eolive...@gmail.com> wrote:
>>>
>>> For whom who is interested I am working on adding support for HerdDB in
>>> Pulsar Manager.
>>>
>>> The pull request is still very raw, but Pulsar Manager seems to work
>>> https://github.com/streamnative/pulsar-manager/pull/183
>>>
>>>
>>>
>>> Il giorno mar 3 set 2019 alle ore 08:37 Yuva raj <uvar...@gmail.com> ha
>>> scritto:
>>>
>>>> On Fri, 30 Aug 2019 at 07:09, Sijie Guo <guosi...@gmail.com> wrote:
>>>>
>>>>>> I see there are dependencies on websockets, I would a prefer a simple
>>>>> polling model of the http, also it's enable by default in pulsar.
>>>>>
>>>>> I don't think we depend don pulsar websocket. All are http restful
>>>>> requests.
>>>>>
>>>>>> For the ui the default persistence should be sqlite. Potentially
>>>> packaged
>>>>> by default.
>>>>>
>>>>> Initially the default is sqlite. But sqlite doesn't work if there are a
>>>> lot
>>>>> of topic metrics.
>>>>> We switched to MySQL as default for supporting production traffic. We
>> can
>>>>> switch default to PostgresSQL (as pulsar dashboard).
>>>>>
>>>> +1
>>>> Yes, Switching to Postgres Would be great. Because Postgres license is
>> more
>>>> liberal and works well with Apache license ecosystem.
>>>>
>>>>>
>>>>> - Sijie
>>>>>
>>>>> On Thu, Aug 29, 2019 at 6:23 PM Ali Ahmed <alahmed...@gmail.com>
>> wrote:
>>>>>
>>>>>> I see there are dependencies on websockets, I would a prefer a simple
>>>>>> polling model of the http, also it's enable by default in pulsar.
>>>>>>
>>>>>> For the ui the default persistence should be sqlite. Potentially
>>>> packaged
>>>>>> by default.
>>>>>>
>>>>>> -Ali
>>>>>>
>>>>>> On Thu, Aug 29, 2019 at 2:47 PM Enrico Olivelli <eolive...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Il gio 29 ago 2019, 23:28 Sijie Guo <guosi...@gmail.com> ha scritto:
>>>>>>>
>>>>>>>> That sounds an interesting idea!
>>>>>>>
>>>>>>>
>>>>>>> Awesome
>>>>>>>
>>>>>>> Does HerdDB support JDBC? If so, it should
>>>>>>>> be pretty straightforward to enable HerdDB.
>>>>>>>>
>>>>>>>
>>>>>>> Sure, as far as I know, the JDBC client is the only client really
>>>> used
>>>>> in
>>>>>>> production.
>>>>>>>
>>>>>>> In a replicated environment it uses Zookeeper for metadata and
>>>> service
>>>>>>> discovery and Bookkeeper as WAL and if you have a pulsar cluster you
>>>>>>> already have both of them (zk cluster an bookies)
>>>>>>>
>>>>>>> I will take a look to how Pulsar Manager uses JDBC, maybe it will
>>>> be
>>>>>> very
>>>>>>> easy.
>>>>>>>
>>>>>>> I will be back with news
>>>>>>>
>>>>>>>
>>>>>>> Enrico
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Sijie
>>>>>>>>
>>>>>>>> On Thu, Aug 29, 2019 at 12:08 PM Enrico Olivelli <
>>>>> eolive...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Maybe you can try to use HerdDB (1), it is a replicated SQL
>>>>> database
>>>>>>> that
>>>>>>>>> can be run embedded in the JVM.
>>>>>>>>> It is an SQL database that we developed in my company, in order
>>>> to
>>>>>>>>> explicitly replace MySQL.
>>>>>>>>> It uses Bookkeeper to store the WAL and implement replication,
>>>> but
>>>>> it
>>>>>>> can
>>>>>>>>> run in standalone mode or in memory (for tests/dev).
>>>>>>>>>
>>>>>>>>> I can help in setting up a demo.
>>>>>>>>>
>>>>>>>>> Enrico
>>>>>>>>>
>>>>>>>>> (1) https://github.com/diennea/herddb
>>>>>>>>>
>>>>>>>>> Il gio 29 ago 2019, 16:59 Sijie Guo <guosi...@gmail.com> ha
>>>>> scritto:
>>>>>>>>>
>>>>>>>>>> Hi Dave,
>>>>>>>>>>
>>>>>>>>>> It doesn't require MySQL. You can use any JDBC supported
>>>>> databases.
>>>>>>>>>> Initially we would use SQLite for development and change to
>>>> MySQL
>>>>>> for
>>>>>>>>> more
>>>>>>>>>> production-ready workloads.
>>>>>>>>>>
>>>>>>>>>> Hope this clarify your question.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Sijie
>>>>>>>>>>
>>>>>>>>>> On Thu, Aug 29, 2019 at 5:56 AM Dave Fisher <w...@apache.org>
>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi -
>>>>>>>>>>>
>>>>>>>>>>> I see that MySQL is used. This would be a GPL dependency.
>>>> Does
>>>>>> the
>>>>>>>>> Pulsar
>>>>>>>>>>> Manager require MySQL, or can other Databases that are
>>>>> compliant
>>>>>>> with
>>>>>>>>>>> Apache Release Policy be used instead?
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Dave
>>>>>>>>>>>
>>>>>>>>>>>> On Aug 29, 2019, at 7:49 AM, Guangning E <
>>>>> eguangn...@gmail.com
>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>
>>>>>>>>>>>> We have developed a new Pulsar web UI - pulsar-manager,
>>>>> aiming
>>>>>> at
>>>>>>>>>>>> supporting managing Pulsar clusters running in different
>>>>>>>> environments
>>>>>>>>>>>> (on-premise data centers, cloud, and so on). We’d like to
>>>>>>>> contribute
>>>>>>>>>> the
>>>>>>>>>>>> project back to the Pulsar community.
>>>>>>>>>>>>
>>>>>>>>>>>> ---
>>>>>>>>>>>>
>>>>>>>>>>>> ## Motivation
>>>>>>>>>>>>
>>>>>>>>>>>> Currently, Pulsar has a monitoring tool - [Pulsar
>>>>> Dashboard]( [
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>> http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/)
>>>>>>>>>>>> ).
>>>>>>>>>>>> However, it only focuses on the simple monitoring of
>>>> Pulsar -
>>>>>>>>>> collecting
>>>>>>>>>>>> and displaying information such as statistics of tenants,
>>>>>>>> namespaces,
>>>>>>>>>>>> topics, subscriptions, and so on. It doesn’t provide any
>>>>>>> management
>>>>>>>>>>>> operations such as add, delete and update tenants,
>>>>> namespaces,
>>>>>>>>> topics,
>>>>>>>>>>> and
>>>>>>>>>>>> so on. When the scale of Pulsar cluster increases or the
>>>>> number
>>>>>>> of
>>>>>>>>>>> clusters
>>>>>>>>>>>> grows, using `pulsar-admin` to manage Pulsar can not
>>>> satisfy
>>>>>>>> demands.
>>>>>>>>>>>> Therefore, Pulsar needs a simple and easy-to-use management
>>>>>>> console
>>>>>>>>> for
>>>>>>>>>>>> administrators.
>>>>>>>>>>>>
>>>>>>>>>>>> Pulsar Manager is a web-based GUI management and monitoring
>>>>>> tool
>>>>>>>> that
>>>>>>>>>>> helps
>>>>>>>>>>>> administrators and users manage and monitor tenants,
>>>>>> namespaces,
>>>>>>>>>> topics,
>>>>>>>>>>>> subscriptions, brokers, clusters, and so on, and supports
>>>>>> dynamic
>>>>>>>>>>>> configuration of multiple environments.
>>>>>>>>>>>>
>>>>>>>>>>>> ## Features
>>>>>>>>>>>>
>>>>>>>>>>>> Pulsar manager provides two main features: management and
>>>>>>>> monitoring.
>>>>>>>>>>>>
>>>>>>>>>>>> ### Management
>>>>>>>>>>>>
>>>>>>>>>>>> 1. Environment : (operations on the environment)
>>>>>>>>>>>> Create / Delete / Update / List / Get
>>>>>>>>>>>> 2. Cluster : (operations on clusters)
>>>>>>>>>>>> Create / Delete / Update / List / Get
>>>>>>>>>>>> 3. Brokers : (operations on brokers)
>>>>>>>>>>>> Heartbeat / Unload
>>>>>>>>>>>> 4. Tenants:
>>>>>>>>>>>> Create / Delete / Update / List
>>>>>>>>>>>> 5. Namespaces:
>>>>>>>>>>>> Create / Delete / List
>>>>>>>>>>>> Manage Namespace Policy
>>>>>>>>>>>> Unload
>>>>>>>>>>>> Operations on Namespace Bundles
>>>>>>>>>>>> 6. Topics
>>>>>>>>>>>> Create / Delete / List
>>>>>>>>>>>> Unload / Terminate / Offload / Compact
>>>>>>>>>>>> Topic Details
>>>>>>>>>>>> 7. Subscriptions
>>>>>>>>>>>> Create / Delete / List
>>>>>>>>>>>> Reset Cursor / Skip / Clear backlog / Unsubscribe
>>>>>>>>>>>> 8. Namespace Isolation Policies
>>>>>>>>>>>> Create / Delete / Update / Get / List
>>>>>>>>>>>> 9. Failure Domains
>>>>>>>>>>>> Create / Delete / Update / Get / List
>>>>>>>>>>>>
>>>>>>>>>>>> ### Monitoring: (display *aggregated* stats at different
>>>>>> levels)
>>>>>>>>>>>>
>>>>>>>>>>>> 1. Tenants
>>>>>>>>>>>> List the total number of namespaces per tenant (both
>>>>> aggregated
>>>>>>> and
>>>>>>>>>>>> per-cluster basis)
>>>>>>>>>>>>
>>>>>>>>>>>> 2. Namespaces
>>>>>>>>>>>> List the total number of topics per namespace
>>>>>>>>>>>> List the aggregated stats per namespace (such as rate-in,
>>>>>>> rate-out,
>>>>>>>>>>>> throughput-in, and throughput-out)
>>>>>>>>>>>> List the distributions of namespace bundles
>>>>>>>>>>>>
>>>>>>>>>>>> 3. Topics
>>>>>>>>>>>> List the total number of partitions per topic
>>>>>>>>>>>> List the aggregated stats per topic (such as rate-in,
>>>>> rate-out,
>>>>>>>>>>>> throughput-in, and throughput-out)
>>>>>>>>>>>> Detailed stats per topic partition
>>>>>>>>>>>> Detailed stats of storage per topic partition
>>>>>>>>>>>> Detailed stats of subscription per topic and per partition
>>>>>>>>>>>> Detailed stats of producers
>>>>>>>>>>>> Detailed stats of consumers
>>>>>>>>>>>>
>>>>>>>>>>>> The detailed design proposal is in
>>>>>>>>>>>> [
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>> https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#)
>>>>>>>>>>>>
>>>>>>>>>>>> Looking forward to any feedback.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Guangning
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> *Thanks*
>>>>
>>>> *Yuvaraj L*
>>>>
>>
>>
