That looks good, Enrico. Thank you for driving the work. Sijie
On Wed, Sep 4, 2019 at 9:02 AM Enrico Olivelli <eolive...@gmail.com> wrote: > For whom who is interested I am working on adding support for HerdDB in > Pulsar Manager. > > The pull request is still very raw, but Pulsar Manager seems to work > https://github.com/streamnative/pulsar-manager/pull/183 > > > > Il giorno mar 3 set 2019 alle ore 08:37 Yuva raj <uvar...@gmail.com> ha > scritto: > > > On Fri, 30 Aug 2019 at 07:09, Sijie Guo <guosi...@gmail.com> wrote: > > > > > > I see there are dependencies on websockets, I would a prefer a simple > > > polling model of the http, also it's enable by default in pulsar. > > > > > > I don't think we depend don pulsar websocket. All are http restful > > > requests. > > > > > > > For the ui the default persistence should be sqlite. Potentially > > packaged > > > by default. > > > > > > Initially the default is sqlite. But sqlite doesn't work if there are a > > lot > > > of topic metrics. > > > We switched to MySQL as default for supporting production traffic. We > can > > > switch default to PostgresSQL (as pulsar dashboard). > > > > > +1 > > Yes, Switching to Postgres Would be great. Because Postgres license is > more > > liberal and works well with Apache license ecosystem. > > > > > > > > - Sijie > > > > > > On Thu, Aug 29, 2019 at 6:23 PM Ali Ahmed <alahmed...@gmail.com> > wrote: > > > > > > > I see there are dependencies on websockets, I would a prefer a simple > > > > polling model of the http, also it's enable by default in pulsar. > > > > > > > > For the ui the default persistence should be sqlite. Potentially > > packaged > > > > by default. > > > > > > > > -Ali > > > > > > > > On Thu, Aug 29, 2019 at 2:47 PM Enrico Olivelli <eolive...@gmail.com > > > > > > wrote: > > > > > > > > > Il gio 29 ago 2019, 23:28 Sijie Guo <guosi...@gmail.com> ha > scritto: > > > > > > > > > > > That sounds an interesting idea! > > > > > > > > > > > > > > > Awesome > > > > > > > > > > Does HerdDB support JDBC? If so, it should > > > > > > be pretty straightforward to enable HerdDB. > > > > > > > > > > > > > > > > Sure, as far as I know, the JDBC client is the only client really > > used > > > in > > > > > production. > > > > > > > > > > In a replicated environment it uses Zookeeper for metadata and > > service > > > > > discovery and Bookkeeper as WAL and if you have a pulsar cluster > you > > > > > already have both of them (zk cluster an bookies) > > > > > > > > > > I will take a look to how Pulsar Manager uses JDBC, maybe it will > > be > > > > very > > > > > easy. > > > > > > > > > > I will be back with news > > > > > > > > > > > > > > > Enrico > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > Sijie > > > > > > > > > > > > On Thu, Aug 29, 2019 at 12:08 PM Enrico Olivelli < > > > eolive...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > Maybe you can try to use HerdDB (1), it is a replicated SQL > > > database > > > > > that > > > > > > > can be run embedded in the JVM. > > > > > > > It is an SQL database that we developed in my company, in order > > to > > > > > > > explicitly replace MySQL. > > > > > > > It uses Bookkeeper to store the WAL and implement replication, > > but > > > it > > > > > can > > > > > > > run in standalone mode or in memory (for tests/dev). > > > > > > > > > > > > > > I can help in setting up a demo. > > > > > > > > > > > > > > Enrico > > > > > > > > > > > > > > (1) https://github.com/diennea/herddb > > > > > > > > > > > > > > Il gio 29 ago 2019, 16:59 Sijie Guo <guosi...@gmail.com> ha > > > scritto: > > > > > > > > > > > > > > > Hi Dave, > > > > > > > > > > > > > > > > It doesn't require MySQL. You can use any JDBC supported > > > databases. > > > > > > > > Initially we would use SQLite for development and change to > > MySQL > > > > for > > > > > > > more > > > > > > > > production-ready workloads. > > > > > > > > > > > > > > > > Hope this clarify your question. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Sijie > > > > > > > > > > > > > > > > On Thu, Aug 29, 2019 at 5:56 AM Dave Fisher <w...@apache.org > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi - > > > > > > > > > > > > > > > > > > I see that MySQL is used. This would be a GPL dependency. > > Does > > > > the > > > > > > > Pulsar > > > > > > > > > Manager require MySQL, or can other Databases that are > > > compliant > > > > > with > > > > > > > > > Apache Release Policy be used instead? > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > Dave > > > > > > > > > > > > > > > > > > > On Aug 29, 2019, at 7:49 AM, Guangning E < > > > eguangn...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > We have developed a new Pulsar web UI - pulsar-manager, > > > aiming > > > > at > > > > > > > > > > supporting managing Pulsar clusters running in different > > > > > > environments > > > > > > > > > > (on-premise data centers, cloud, and so on). We’d like to > > > > > > contribute > > > > > > > > the > > > > > > > > > > project back to the Pulsar community. > > > > > > > > > > > > > > > > > > > > --- > > > > > > > > > > > > > > > > > > > > ## Motivation > > > > > > > > > > > > > > > > > > > > Currently, Pulsar has a monitoring tool - [Pulsar > > > Dashboard]( [ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/) > > > > > > > > > > ). > > > > > > > > > > However, it only focuses on the simple monitoring of > > Pulsar - > > > > > > > > collecting > > > > > > > > > > and displaying information such as statistics of tenants, > > > > > > namespaces, > > > > > > > > > > topics, subscriptions, and so on. It doesn’t provide any > > > > > management > > > > > > > > > > operations such as add, delete and update tenants, > > > namespaces, > > > > > > > topics, > > > > > > > > > and > > > > > > > > > > so on. When the scale of Pulsar cluster increases or the > > > number > > > > > of > > > > > > > > > clusters > > > > > > > > > > grows, using `pulsar-admin` to manage Pulsar can not > > satisfy > > > > > > demands. > > > > > > > > > > Therefore, Pulsar needs a simple and easy-to-use > management > > > > > console > > > > > > > for > > > > > > > > > > administrators. > > > > > > > > > > > > > > > > > > > > Pulsar Manager is a web-based GUI management and > monitoring > > > > tool > > > > > > that > > > > > > > > > helps > > > > > > > > > > administrators and users manage and monitor tenants, > > > > namespaces, > > > > > > > > topics, > > > > > > > > > > subscriptions, brokers, clusters, and so on, and supports > > > > dynamic > > > > > > > > > > configuration of multiple environments. > > > > > > > > > > > > > > > > > > > > ## Features > > > > > > > > > > > > > > > > > > > > Pulsar manager provides two main features: management and > > > > > > monitoring. > > > > > > > > > > > > > > > > > > > > ### Management > > > > > > > > > > > > > > > > > > > > 1. Environment : (operations on the environment) > > > > > > > > > > Create / Delete / Update / List / Get > > > > > > > > > > 2. Cluster : (operations on clusters) > > > > > > > > > > Create / Delete / Update / List / Get > > > > > > > > > > 3. Brokers : (operations on brokers) > > > > > > > > > > Heartbeat / Unload > > > > > > > > > > 4. Tenants: > > > > > > > > > > Create / Delete / Update / List > > > > > > > > > > 5. Namespaces: > > > > > > > > > > Create / Delete / List > > > > > > > > > > Manage Namespace Policy > > > > > > > > > > Unload > > > > > > > > > > Operations on Namespace Bundles > > > > > > > > > > 6. Topics > > > > > > > > > > Create / Delete / List > > > > > > > > > > Unload / Terminate / Offload / Compact > > > > > > > > > > Topic Details > > > > > > > > > > 7. Subscriptions > > > > > > > > > > Create / Delete / List > > > > > > > > > > Reset Cursor / Skip / Clear backlog / Unsubscribe > > > > > > > > > > 8. Namespace Isolation Policies > > > > > > > > > > Create / Delete / Update / Get / List > > > > > > > > > > 9. Failure Domains > > > > > > > > > > Create / Delete / Update / Get / List > > > > > > > > > > > > > > > > > > > > ### Monitoring: (display *aggregated* stats at different > > > > levels) > > > > > > > > > > > > > > > > > > > > 1. Tenants > > > > > > > > > > List the total number of namespaces per tenant (both > > > aggregated > > > > > and > > > > > > > > > > per-cluster basis) > > > > > > > > > > > > > > > > > > > > 2. Namespaces > > > > > > > > > > List the total number of topics per namespace > > > > > > > > > > List the aggregated stats per namespace (such as rate-in, > > > > > rate-out, > > > > > > > > > > throughput-in, and throughput-out) > > > > > > > > > > List the distributions of namespace bundles > > > > > > > > > > > > > > > > > > > > 3. Topics > > > > > > > > > > List the total number of partitions per topic > > > > > > > > > > List the aggregated stats per topic (such as rate-in, > > > rate-out, > > > > > > > > > > throughput-in, and throughput-out) > > > > > > > > > > Detailed stats per topic partition > > > > > > > > > > Detailed stats of storage per topic partition > > > > > > > > > > Detailed stats of subscription per topic and per > partition > > > > > > > > > > Detailed stats of producers > > > > > > > > > > Detailed stats of consumers > > > > > > > > > > > > > > > > > > > > The detailed design proposal is in > > > > > > > > > > [ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#) > > > > > > > > > > > > > > > > > > > > Looking forward to any feedback. > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Guangning > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > *Thanks* > > > > *Yuvaraj L* > > >
