Re: [DISCUSS] Switch the client complier to JDK 11 or 17 from JDK 8

[mattisonchao]

-1

The reason is the same as Yunze mentioned. It's a kind of break for the user. 
We can't force our user upgrade the application JDK version by pulsar's bug. 
That will make our users frustrated.

Plus, the pulsar LTS is 3.0. I think it's better to keep the JDK 1.8 in this 
version. as users will most likely use this LTS version for long time.

Best,
Mattison
On Feb 16, 2023, 14:49 +0800, Yunze Xu , wrote:
> -1
>
> Upgrading the client is harder than upgrading the broker. Pulsar
> usually acts as a service to serve many lines of business and only a
> single team is responsible to maintain the server side. It would be
> hard to push all the client sides to upgrade the JDK.
>
> Instead of upgrading the minimum required JDK of the client, it's
> better to open a separate project if you think the client with JDK 17
> could bring significant performance and security improvements.
>
> Thanks,
> Yunze
>
> On Wed, Feb 15, 2023 at 1:08 AM Zixuan Liu  wrote:
> >
> > Hi all,
> >
> > We are using JDK 17 to compile all the components. For the main component,
> > the broker and proxy require JDK 17, and the client requires JDK 8.
> >
> > In Pulsar 3.x, we should keep up with modern JDKS for all components, and
> > over time, I believe many users have adopted JDK 11/17, which provides
> > significant performance and security improvements, see
> > https://learn.microsoft.com/en-us/java/openjdk/reasons-to-move-to-java-11
> >
> > Switching to JDK17 is a bit radical, and we still need to consider this.
> >
> > Thanks,
> > Zixuan

Re: [DISCUSS] Switch the client complier to JDK 11 or 17 from JDK 8

[Horizon]

-1. We could know the user dependency. Maybe the user dependent the third lib 
is not compatible with 11 or 17. It will introduce some problem for user. And 
the user need to pay more effort to test the project compatibility after 
upgrade the client.


>> Hi all,
>> 
>> We are using JDK 17 to compile all the components. For the main component,
>> the broker and proxy require JDK 17, and the client requires JDK 8.
>> 
>> In Pulsar 3.x, we should keep up with modern JDKS for all components, and
>> over time, I believe many users have adopted JDK 11/17, which provides
>> significant performance and security improvements, see
>> https://learn.microsoft.com/en-us/java/openjdk/reasons-to-move-to-java-11
>> 
>> Switching to JDK17 is a bit radical, and we still need to consider this.
>> 
>> Thanks,
>> Zixuan

Re: [Vote] PIP-245: Make subscriptions of non-persistent topic non-durable

[Jiuming Tao]

I’ve added the `Compatibility` selection into the PIP, please help review and 
vote the PIP

Thanks,
Tao Jiuming



> 2023年2月15日 14:58，Zike Yang  写道：
> 
> Hi, Jiuming
> 
>> bump
> 
> As for the discussion here[0], could you add a `Compatibility` section
> to talk about compatibility in more detail? WDYT?
> Then we could start the vote again.
> 
> [0] https://lists.apache.org/thread/2bjg39zh7z38bzbnqngbo5l4jzkjttrq
> 
> Thanks,
> Zike Yang
> 
> On Wed, Feb 15, 2023 at 1:34 PM Tao Jiuming  wrote:
>> 
>> 
>> bump
>> 
>> On 2023/02/13 06:56:09 Jiuming Tao wrote:
>>> Hi all,
>>> 
>>> I would like to start a VOTE on `PIP-245: Make subscriptions of 
>>> non-persistent topic non-durable`.
>>> 
>>> Motivation:
>>> 
>>> There are two types of subscriptions for a topic: Durable and Non-durable.
>>> 
>>> We create a Consumer with a Durable subscription and a Reader with a 
>>> Non-durable subscription.
>>> 
>>> But for NonPersistentTopic, creating a Durable subscription is meaningless, 
>>> NonPersistentSubscription doesn't have a ManagedCursor to persistent its 
>>> data. After its consumer disconnected, the subscription couldn't be removed 
>>> automatically if we didn't set the value of 
>>> subscriptionExpirationTimeMinutes greater than 0.
>>> 
>>> For subscriptionExpirationTimeMinutes, it controls the subscription 
>>> expiration of NonPersistentTopic and PersistentTopic, if we set the value 
>>> of subscriptionExpirationTimeMinutes greater than 0, it may lead to data 
>>> loss(The durable subscriptions of PersistentTopic also can be removed).
>>> 
>>> And the Non-durable subscriptions will be removed automatically after all 
>>> the consumers disconnected, it's the existing logic.
>>> 
>>> For the purpose of removing the subscriptions which have no active 
>>> consumers of NonPersistentTopic and the above reasons, we can make all the 
>>> subscriptions of a NonPersistentTopic Non-durable.
>>> 
>>> 
>>> 
>>> For more details, you can read: 
>>> https://github.com/apache/pulsar/issues/19448 
>>> 
>>> 
>>> And the discuss thread is available at: 
>>> https://lists.apache.org/thread/2ltmyglnb25jy8nk58twkwbglws43bst 
>>> 
>>> 
>>> Thanks,
>>> Tao Jiuming

Re: [DISCUSS] Switch the client complier to JDK 11 or 17 from JDK 8

[Enrico Olivelli]

-1
We should still stick to JDK8 for a while on the client side (and
shared modules).

I see that slowly the ecosystem is moving di JDK11, but it is not yet our time.
This year is Pulsar momentum, please do not add blockers to the adoption




Enrico

Il giorno gio 16 feb 2023 alle ore 09:39 Horizon
<1060026...@qq.com.invalid> ha scritto:
>
> -1. We could know the user dependency. Maybe the user dependent the third lib 
> is not compatible with 11 or 17. It will introduce some problem for user. And 
> the user need to pay more effort to test the project compatibility after 
> upgrade the client.
>
>
> >> Hi all,
> >>
> >> We are using JDK 17 to compile all the components. For the main component,
> >> the broker and proxy require JDK 17, and the client requires JDK 8.
> >>
> >> In Pulsar 3.x, we should keep up with modern JDKS for all components, and
> >> over time, I believe many users have adopted JDK 11/17, which provides
> >> significant performance and security improvements, see
> >> https://learn.microsoft.com/en-us/java/openjdk/reasons-to-move-to-java-11
> >>
> >> Switching to JDK17 is a bit radical, and we still need to consider this.
> >>
> >> Thanks,
> >> Zixuan
>

Re: Force redirect questions from Slack to GitHub Discussions or StackOverflow?

[Zike Yang]

+1, That sounds great to me.

> 2. You click the three dots button on his message -> “Convert to a GitHub 
> discussion”.

Is it a similar workflow for converting to a StackOverflow question?

BR,
Zike Yang

On Thu, Feb 16, 2023 at 12:14 PM  wrote:
>
> +1
>
> Since web pages are more easily and public.
>
> Best
> Mattison
> On Feb 16, 2023, 07:58 +0800, Christophe Bornet , 
> wrote:
> > +100
> > Also note that for good or bad reasons, the number of questions on
> > StaOverflow is often used as a metric for the popularity of a project.
> >
> > Le mer. 15 févr. 2023 à 13:12, Kiryl Valkovich 
> > a écrit :
> >
> > > Hi everyone! Enrico Olivelli asked me to repost it to the mailing list.
> > >
> > > --- Me
> > > I’m very worried that good answers from David Kjerrumgaard and others
> > > won’t be googleable because it’s in Slack. To make any search you need to
> > > pay a monthly subscription.
> > >
> > > In my opinion, it would be wiser to make StackOverflow, Discuss, or GitHub
> > > discussions a place for questions. And strictly redirect people who ask 
> > > any
> > > question in Slack to the right place.
> > > In GitHub discussions, you also can manage categories, as well as you can
> > > manage channels in Slack.
> > >
> > > Subscription to a specific category is coming soon.
> > > https://github.com/community/community/discussions/3951#discussioncomment-3879939
> > > The most important that it’s searchable.
> > >
> > > Examples:
> > > https://github.com/community/community/discussions
> > > https://github.com/apache/superset/discussions
> > >
> > > --- Later me to David Kjerrumgaard
> > > What do you think about the following workflow:
> > > 1. The user asks a new question in Slack.
> > > 2. You click the three dots button on his message -> “Convert to a GitHub
> > > discussion”.
> > > 3. The bot creates a new GitHub discussion in the “Q&A” category with the
> > > original message’s content.
> > > 4. The bot converts the original message to a thread and posts a message
> > > like:
> > > > > In order to keep history and make your question searchable by other
> > > Apache Pulsar users, it has been converted to a GitHub discussion.
> > > > > Further conversation is available by this link:
> > > https://github.com/apache/pulsar/discussions/18766.
> > > 5. You answer the question on GitHub.
> > >
> > > How it may work:
> > > https://api.slack.com/best-practices/blueprints/actions-and-dialogs
> > > I can try to implement it makes any sense for you. (edited)
> > >
> > > Full original thread:
> > > https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1676449603034309
> > >
> > >

Re: [DISCUSS] Switch the client complier to JDK 11 or 17 from JDK 8

[Zixuan Liu]

Hi all,

Thank you for your suggestion! Close this discussion now.

Thanks,
Zixuan

Enrico Olivelli  于2023年2月16日周四 18:19写道：

> -1
> We should still stick to JDK8 for a while on the client side (and
> shared modules).
>
> I see that slowly the ecosystem is moving di JDK11, but it is not yet our
> time.
> This year is Pulsar momentum, please do not add blockers to the adoption
>
>
>
>
> Enrico
>
> Il giorno gio 16 feb 2023 alle ore 09:39 Horizon
> <1060026...@qq.com.invalid> ha scritto:
> >
> > -1. We could know the user dependency. Maybe the user dependent the
> third lib is not compatible with 11 or 17. It will introduce some problem
> for user. And the user need to pay more effort to test the project
> compatibility after upgrade the client.
> >
> >
> > >> Hi all,
> > >>
> > >> We are using JDK 17 to compile all the components. For the main
> component,
> > >> the broker and proxy require JDK 17, and the client requires JDK 8.
> > >>
> > >> In Pulsar 3.x, we should keep up with modern JDKS for all components,
> and
> > >> over time, I believe many users have adopted JDK 11/17, which provides
> > >> significant performance and security improvements, see
> > >>
> https://learn.microsoft.com/en-us/java/openjdk/reasons-to-move-to-java-11
> > >>
> > >> Switching to JDK17 is a bit radical, and we still need to consider
> this.
> > >>
> > >> Thanks,
> > >> Zixuan
> >
>

Re: Force redirect questions from Slack to GitHub Discussions or StackOverflow?

[Kiryl Valkovich]

If there are no hidden obstacles, we can implement it.
StackOverflow supports question creation using REST API: 
https://api.stackexchange.com/docs/create-question

From: Zike Yang 
Date: Thursday, February 16, 2023 at 11:41 AM
To: dev@pulsar.apache.org 
Subject: Re: Force redirect questions from Slack to GitHub Discussions or 
StackOverflow?
+1, That sounds great to me.

> 2. You click the three dots button on his message -> “Convert to a GitHub 
> discussion”.

Is it a similar workflow for converting to a StackOverflow question?

BR,
Zike Yang

On Thu, Feb 16, 2023 at 12:14 PM  wrote:
>
> +1
>
> Since web pages are more easily and public.
>
> Best
> Mattison
> On Feb 16, 2023, 07:58 +0800, Christophe Bornet , 
> wrote:
> > +100
> > Also note that for good or bad reasons, the number of questions on
> > StaOverflow is often used as a metric for the popularity of a project.
> >
> > Le mer. 15 févr. 2023 à 13:12, Kiryl Valkovich 
> > a écrit :
> >
> > > Hi everyone! Enrico Olivelli asked me to repost it to the mailing list.
> > >
> > > --- Me
> > > I’m very worried that good answers from David Kjerrumgaard and others
> > > won’t be googleable because it’s in Slack. To make any search you need to
> > > pay a monthly subscription.
> > >
> > > In my opinion, it would be wiser to make StackOverflow, Discuss, or GitHub
> > > discussions a place for questions. And strictly redirect people who ask 
> > > any
> > > question in Slack to the right place.
> > > In GitHub discussions, you also can manage categories, as well as you can
> > > manage channels in Slack.
> > >
> > > Subscription to a specific category is coming soon.
> > > https://github.com/community/community/discussions/3951#discussioncomment-3879939
> > > The most important that it’s searchable.
> > >
> > > Examples:
> > > https://github.com/community/community/discussions
> > > https://github.com/apache/superset/discussions
> > >
> > > --- Later me to David Kjerrumgaard
> > > What do you think about the following workflow:
> > > 1. The user asks a new question in Slack.
> > > 2. You click the three dots button on his message -> “Convert to a GitHub
> > > discussion”.
> > > 3. The bot creates a new GitHub discussion in the “Q&A” category with the
> > > original message’s content.
> > > 4. The bot converts the original message to a thread and posts a message
> > > like:
> > > > > In order to keep history and make your question searchable by other
> > > Apache Pulsar users, it has been converted to a GitHub discussion.
> > > > > Further conversation is available by this link:
> > > https://github.com/apache/pulsar/discussions/18766.
> > > 5. You answer the question on GitHub.
> > >
> > > How it may work:
> > > https://api.slack.com/best-practices/blueprints/actions-and-dialogs
> > > I can try to implement it makes any sense for you. (edited)
> > >
> > > Full original thread:
> > > https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1676449603034309
> > >
> > >

Re: [DISCUSS] Using jetty-client instead of async-http-client

[Asaf Mesika]

On Tue, Feb 14, 2023 at 11:36 AM Zixuan Liu  wrote:

> Hi all,
>
> Our admin-client using the async-http-client [0] to request the web
> service, the async-http-client implements network request based on the
> netty, which has very good performance, but this project is not very
> active.
>

Netty - the most used network server in the world - is not very active?


>
> For security(library) reason or http feature(Follow up on future
> development), could we migrate to the jetty-client [1] from the
> async-http-client? The jetty project is very active, our web service is
> built based on the jetty-server, so I think use the jett-client is a good
> idea, but migrating this can be a lot of work.
>
> Please let me know what you think.
>
> [0] - https://github.com/AsyncHttpClient/async-http-client
> [1] - https://github.com/eclipse/jetty.project/tree/jetty-9.4.x
>
> Thanks,
> Zixuan
>

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

[Asaf Mesika]

On Tue, Feb 14, 2023 at 1:02 PM Yan Zhao  wrote:

> > Shouldn't you specify the changes you are going to make in
> > `PulsarApi.proto`?
> We didn't change the proto file, we use the Restful API(AdminAPI).
> > Also, shouldn't you change the wording in this sentence in the PIP to
> > clarify you will use AdminAPI / Pulsar RPC?
> We use AdminAPI, I will tune the pip.
>
>
>
> > > You need to specify in the PIP that you will publish one message for
> > deletion from BK, and one message for deletion of from offloader and
> > explain why. Also provide the context you just gave me about where you
> find
> > if it is actually offloaded or not.
> Ok.
>
> > Let me see if I understand correctly. Now the PIP says:
> >
> > private MLDataFormats.ManagedLedgerInfo.LedgerInfo context;
> >
> > and you say it should be:
> >
> > private MLDataFormats.ManagedLedgerInfo.LedgerInfo.OffloadContext
> > offloadContext;
> >
> > ?
> Yes.
>
>
> >
> > In that case I'm not sure I understand the need for
> >
> >   /**
> >  * Extent properties.
> >  */
> > private Map properties = new HashMap<>();
> >
>
> It can be removed.
>
> > Not just the title, your PIP needs to explain everything we wrote above
> > about trim.
>
> I know, and I will tune the PIP contents.
>
> > Ok, say you only delete from ZK in the first step. Still what I wrote
> above
> > still applies: You might have 2 messages trying to delete same ledger at
> > the same time. Shouldn't this have a lock to protect against it?
>
> We design that the consumer only handles one message at the same time,
> which can ensure sync.
>
If understood correctly, every broker will have a consumer right? You will
use a fail-over subscription? The retry-topic is consumed by the same
subscription, same consumer?


In this very long mailing list thread, we have mentioned many fixes to be
done. Can you ping in the mailing list once you have managed to fix it all?

Re: [DISCUSS] PIP-245: Subscriptions expiration for NonPersistentTopic only

[Asaf Mesika]

Regarding "+1 to throw this exception in future versions." - this means you
complete this issue once it throws an exception right?


On Wed, Feb 15, 2023 at 8:54 AM Zike Yang  wrote:

> > if we change the default
> subscription mode to be non-durable for non-persistent topics, then what
> will happen to users who previously had durable subscriptions on that
> topic? Are we ruining something if we create non-durable subscriptions
> instead?
>
> I think it doesn't matter. Currently, the durable subscription on
> non-persistent topics is actually non durable. It will be removed
> after restarting the broker. Users should not depend on this logic. It
> seems that it does not affect existing user behavior too much.
>
> > Can we add another phase to this PIP - say on version x we add WARN and
> override mode to non-durable, and on version x+1 we fail the client?
>
> +1. We can add a section called the `Compatibility` section to this
> PIP. +1 to throw this exception in future versions.
>
> Thanks,
> Zike Yang
>
>
> On Tue, Feb 14, 2023 at 5:11 PM Asaf Mesika  wrote:
> >
> > Can we add another phase to this PIP - say on version x we add WARN and
> > override mode to non-durable, and on version x+1 we fail the client?
> >
> > On Tue, Feb 14, 2023 at 9:48 AM Jiuming Tao
> 
> > wrote:
> >
> > > Hi Asaf,
> > >
> > > If we fail the client when users use Durable subscription to subscribe
> > > NonPersistentTopic, it will lead to break change.
> > >
> > > Users have to change their code, so we can change the
> `subscriptionMode`
> > > from `Durable` to `NonDurable` and print WARN log when users choose
> > > `Durable`, then, users don’t need to change their code and we will not
> > > introduce break change.
> > >
> > > The PIP will deprecate Durable subscriptions on NonPersistentTopic, all
> > > the subscriptions on NonPersistentTopic will be NonDurable.
> > >
> > >
> > > Thanks,
> > > Tao Jiuming
> > >
> > >
> > >
> > >
> > > > 2023年2月14日 03:24，Asaf Mesika  写道：
> > > >
> > > > Clarifying: as Pengu wrote in the pip comment - if we change the
> default
> > > > subscription mode to be non-durable for non-persistent topics, then
> what
> > > > will happen to users who previously had durable subscriptions on that
> > > > topic? Are we ruining something if we create non-durable
> subscriptions
> > > > instead?
> > > >
> > > >
> > > > On Mon, Feb 13, 2023 at 9:16 PM Asaf Mesika 
> > > wrote:
> > > >
> > > >> Can we fail the client if they choose NonDurable subscription mode
> on a
> > > >> non-persistent topic, instead of writing WARN log messages?
> > > >>
> > > >>
> > > >> On Mon, Feb 13, 2023 at 4:26 AM guo jiwei 
> wrote:
> > > >>
> > > >>> Agree，+1
> > > >>>
> > > >>>
> > > >>> Regards
> > > >>> Jiwei Guo (Tboy)
> > > >>>
> > > >>> On Thu, Feb 9, 2023 at 7:07 PM Jiuming Tao
> > > >>>  wrote:
> > > 
> > > 
> > >  It makes sense
> > > 
> > >  Thanks,
> > >  Tao Jiuming
> > > 
> > > 
> > > > 2023年2月9日 15:44，Baodi Shi  写道：
> > > >
> > > > Agree, and we can add verification on the client side. When the
> topic
> > > >>> is
> > > > non-persistent and uses durable to subscribe, print warns logs:
> > > > “non-persistent not support durable subscribe will use
> non-durable to
> > > > subscribe.”
> > > >
> > > >
> > > >
> > > > Thanks,
> > > > Baodi Shi
> > > >
> > > > 在 2023年2月9日 15:10:12 上，Jiuming Tao  >
> > > >>> 写道：
> > > >
> > > >> Agree. But we need to take care of the compatibility. How do we
> > > >>> handle
> > > >>
> > > >> this compatibility?
> > > >>
> > > >>
> > > >> How about set `isDurable = false` when create
> > > >>> `NonPersistentSubscription`?
> > > >>
> > > >> In `NonPersistentSubscription`, we can change the constructor
> from
> > > >> ```
> > > >> public NonPersistentSubscription(NonPersistentTopic topic,
> String
> > > >> subscriptionName, boolean isDurable,
> > > >> Map properties) {
> > > >> this.topic = topic;
> > > >> this.topicName = topic.getName();
> > > >> this.subName = subscriptionName;
> > > >> this.fullName =
> MoreObjects.toStringHelper(this).add("topic",
> > > >> topicName).add("name", subName).toString();
> > > >> IS_FENCED_UPDATER.set(this, FALSE);
> > > >> this.lastActive = System.currentTimeMillis();
> > > >> this.isDurable = isDurable;
> > > >> this.subscriptionProperties = properties != null
> > > >> ? Collections.unmodifiableMap(properties) :
> > > >> Collections.emptyMap();
> > > >> }
> > > >>
> > > >> ```
> > > >>
> > > >> to:
> > > >>
> > > >> ```
> > > >> public NonPersistentSubscription(NonPersistentTopic topic,
> String
> > > >> subscriptionName,
> > > >> Map properties) {
> > > >> this.topic = topic;
> > > >> this.topicName = topic.getName();
> > > >> this.subName = subscriptionName;
> > > >>  

Re: [DISCUSS] Using jetty-client instead of async-http-client

[Zixuan Liu]

I point to async-http-client.

Although there is a recent update.

Thanks,
Zixuan

Asaf Mesika  于2023年2月16日周四 21:19写道：

> On Tue, Feb 14, 2023 at 11:36 AM Zixuan Liu  wrote:
>
> > Hi all,
> >
> > Our admin-client using the async-http-client [0] to request the web
> > service, the async-http-client implements network request based on the
> > netty, which has very good performance, but this project is not very
> > active.
> >
>
> Netty - the most used network server in the world - is not very active?
>
>
> >
> > For security(library) reason or http feature(Follow up on future
> > development), could we migrate to the jetty-client [1] from the
> > async-http-client? The jetty project is very active, our web service is
> > built based on the jetty-server, so I think use the jett-client is a good
> > idea, but migrating this can be a lot of work.
> >
> > Please let me know what you think.
> >
> > [0] - https://github.com/AsyncHttpClient/async-http-client
> > [1] - https://github.com/eclipse/jetty.project/tree/jetty-9.4.x
> >
> > Thanks,
> > Zixuan
> >
>

Re: [DISCUSS] PIP-242 Topic name restrictions

[Asaf Mesika]

On Wed, Feb 15, 2023 at 4:36 PM  wrote:

> Hi, All
>
> First of all, I want to list all of the system topics as follows. That
> Yunze has mentioned before.
>
> Namespace level:
>
> • pulsar/system
> • transaction_coordinator_assign
> • __transaction_log_
> • resource-usage
> • pulsar/
> • healthcheck
>
> Topic level:
>
> • __change_events
> • __transaction_buffer_snapshot_segment
> • ...
>
> We can check the details here. [0]
>
> Secondly. I think we need focus on the system topic name prefix. we have
> some options as follows:
>
> • __SYSTEM__
> • __system__
>
>
> Both them make sense for me. As there are already two people prefer
> `__SYSTEM`. if no other people have concern, I will chose it.
>

I vote `__system__` since I think upper casing is not that common for
internal names (for me it's "shouting" and requires an extra key press to
write it).
Examples:

Cassandra System tables

:
* batches

Postgres database catalog
:
* pg_database


OpenSearch system indices

:
* .opendistro-alerting-config


So I think we should stick to `__system__`


>
> Best,
> Mattison
>
> -[0]
> https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/java/org/apache/pulsar/common/naming/SystemTopicNames.java
> On Feb 15, 2023, 11:36 +0800, Michael Marshall ,
> wrote:
> > I support this PIP, thank you for driving it forward Mattison.
> >
> > I support using the prefix __SYSTEM__ for system topics, and I agree
> > that DLQ and RETRY topics do not qualify as system topics because they
> > are produced to and consumed by regular clients.
> >
> > I think we might benefit from a generic definition for a system topic.
> > I provided a loose definition for system topics in this email [0]. I
> > copy it here:
> >
> > A system topic is a topic that is completely internal to Pulsar
> > components. Internally, it is a normal topic. It requires elevated
> > permission to produce/consume when authorization is enabled, even if
> > topic level policies are not enabled. Generic calls like
> > `clearNamespaceBacklog` should not affect system topics. Deleting a
> > namespace or tenant should delete the system topics within it.
> >
> > Thanks,
> > Michael
> >
> > [0] https://lists.apache.org/thread/sr01hvqmdrnk4lxwfwzcpg7y21psj6tt
> >
> >
> > On Tue, Feb 14, 2023 at 9:55 AM  wrote:
> > >
> > > > > Could you list all existing system topic names that are used so we
> canunderstand the rule better?
> > > Yes, sure. I will list it later.
> > > > > And I saw that this proposal only forbids the creation of
> thesetopics. What about writing messages to them? I think it's better notto
> allow Pulsar clients to write messages to them. The Pulsar clientshould
> configure something to get the permission to write messages tothem. To keep
> the compatibility, maybe we can make use of the protocolversion.
> > > Sure, I agree with your point, because the current proposal just want
> to have topic name restriction. maybe we can revise the system topic
> permission and etc in the next proposal. (relate to system topic)
> > >
> > > Best,
> > > Mattison
> > > On Feb 14, 2023, 21:22 +0800, Yunze Xu ,
> wrote:
> > > > > Could you list all existing system topic names that are used so we
> can
> > > > > understand the rule better? Such as the "-RETRY" and "-DLQ" topics
> > > > > mentioned above.
> > > > >
> > > > > And I saw that this proposal only forbids the creation of these
> > > > > topics. What about writing messages to them? I think it's better
> not
> > > > > to allow Pulsar clients to write messages to them. The Pulsar
> client
> > > > > should configure something to get the permission to write messages
> to
> > > > > them. To keep the compatibility, maybe we can make use of the
> protocol
> > > > > version.
> > > > >
> > > > > Thanks
> > > > > Yunze
> > > > >
> > > > > On Tue, Feb 14, 2023 at 5:38 PM Yubiao Feng
> > > > >  wrote:
> > > > > > >
> > > > > > > Hi Qiang
> > > > > > >
> > > > > > > ### System topic name
> > > > > > > '__system__' I think this format is clearer.
> > > > > > > Now the system automatically creates topics of type retry
> consumer letters
> > > > > > > and dead letters.
> > > > > > > These topics all end in uppercase letters, such as `-RETRY,`
> `-DLQ.`
> > > > > > > Is it better to define the system topic name in uppercase(
> > > > > > > '__SYSTEM__' )?
> > > > > > >
> > > > > > > ### Another comment
> > > > > > > If you now redesign the topic name restrictions, should we
> make `-RETRY`
> > > > > > > and `-DLQ` keywords?
> > > > > > >
> > > > > > > Thanks
> > > > > > > Yubiao Feng
> > > > > > >
> > > > > > > On Tue, Feb 14, 2023 at 5:06 PM Asaf Mesika <
> asaf.mes...@gmail.com> wrote:
> > > > > > >
> > > > > > > > > > > On Tue, Feb 14

Re: [Vote] PIP-245: Make subscriptions of non-persistent topic non-durable

[Asaf Mesika]

+1 (non-binding)


On Thu, Feb 16, 2023 at 11:11 AM Jiuming Tao 
wrote:

>
> I’ve added the `Compatibility` selection into the PIP, please help review
> and vote the PIP
>
> Thanks,
> Tao Jiuming
>
>
>
> > 2023年2月15日 14:58，Zike Yang  写道：
> >
> > Hi, Jiuming
> >
> >> bump
> >
> > As for the discussion here[0], could you add a `Compatibility` section
> > to talk about compatibility in more detail? WDYT?
> > Then we could start the vote again.
> >
> > [0] https://lists.apache.org/thread/2bjg39zh7z38bzbnqngbo5l4jzkjttrq
> >
> > Thanks,
> > Zike Yang
> >
> > On Wed, Feb 15, 2023 at 1:34 PM Tao Jiuming  wrote:
> >>
> >>
> >> bump
> >>
> >> On 2023/02/13 06:56:09 Jiuming Tao wrote:
> >>> Hi all,
> >>>
> >>> I would like to start a VOTE on `PIP-245: Make subscriptions of
> non-persistent topic non-durable`.
> >>>
> >>> Motivation:
> >>>
> >>> There are two types of subscriptions for a topic: Durable and
> Non-durable.
> >>>
> >>> We create a Consumer with a Durable subscription and a Reader with a
> Non-durable subscription.
> >>>
> >>> But for NonPersistentTopic, creating a Durable subscription is
> meaningless, NonPersistentSubscription doesn't have a ManagedCursor to
> persistent its data. After its consumer disconnected, the subscription
> couldn't be removed automatically if we didn't set the value of
> subscriptionExpirationTimeMinutes greater than 0.
> >>>
> >>> For subscriptionExpirationTimeMinutes, it controls the subscription
> expiration of NonPersistentTopic and PersistentTopic, if we set the value
> of subscriptionExpirationTimeMinutes greater than 0, it may lead to data
> loss(The durable subscriptions of PersistentTopic also can be removed).
> >>>
> >>> And the Non-durable subscriptions will be removed automatically after
> all the consumers disconnected, it's the existing logic.
> >>>
> >>> For the purpose of removing the subscriptions which have no active
> consumers of NonPersistentTopic and the above reasons, we can make all the
> subscriptions of a NonPersistentTopic Non-durable.
> >>>
> >>>
> >>>
> >>> For more details, you can read:
> https://github.com/apache/pulsar/issues/19448 <
> https://github.com/apache/pulsar/issues/19448>
> >>>
> >>> And the discuss thread is available at:
> https://lists.apache.org/thread/2ltmyglnb25jy8nk58twkwbglws43bst <
> https://lists.apache.org/thread/2ltmyglnb25jy8nk58twkwbglws43bst>
> >>>
> >>> Thanks,
> >>> Tao Jiuming
>
>

Re: Force redirect questions from Slack to GitHub Discussions or StackOverflow?

[Asaf Mesika]

+1
Funny thing is that I was thinking just this morning that Slack's biggest
downside is that it is not searchable, hence although you get that personal
touch by instant messaging people, and the answers can be quicker, the
unsearchable is a big issue.


On Thu, Feb 16, 2023 at 2:33 PM Kiryl Valkovich 
wrote:

> If there are no hidden obstacles, we can implement it.
> StackOverflow supports question creation using REST API:
> https://api.stackexchange.com/docs/create-question
>
> From: Zike Yang 
> Date: Thursday, February 16, 2023 at 11:41 AM
> To: dev@pulsar.apache.org 
> Subject: Re: Force redirect questions from Slack to GitHub Discussions or
> StackOverflow?
> +1, That sounds great to me.
>
> > 2. You click the three dots button on his message -> “Convert to a
> GitHub discussion”.
>
> Is it a similar workflow for converting to a StackOverflow question?
>
> BR,
> Zike Yang
>
> On Thu, Feb 16, 2023 at 12:14 PM  wrote:
> >
> > +1
> >
> > Since web pages are more easily and public.
> >
> > Best
> > Mattison
> > On Feb 16, 2023, 07:58 +0800, Christophe Bornet ,
> wrote:
> > > +100
> > > Also note that for good or bad reasons, the number of questions on
> > > StaOverflow is often used as a metric for the popularity of a project.
> > >
> > > Le mer. 15 févr. 2023 à 13:12, Kiryl Valkovich
> 
> > > a écrit :
> > >
> > > > Hi everyone! Enrico Olivelli asked me to repost it to the mailing
> list.
> > > >
> > > > --- Me
> > > > I’m very worried that good answers from David Kjerrumgaard and others
> > > > won’t be googleable because it’s in Slack. To make any search you
> need to
> > > > pay a monthly subscription.
> > > >
> > > > In my opinion, it would be wiser to make StackOverflow, Discuss, or
> GitHub
> > > > discussions a place for questions. And strictly redirect people who
> ask any
> > > > question in Slack to the right place.
> > > > In GitHub discussions, you also can manage categories, as well as
> you can
> > > > manage channels in Slack.
> > > >
> > > > Subscription to a specific category is coming soon.
> > > >
> https://github.com/community/community/discussions/3951#discussioncomment-3879939
> > > > The most important that it’s searchable.
> > > >
> > > > Examples:
> > > > https://github.com/community/community/discussions
> > > > https://github.com/apache/superset/discussions
> > > >
> > > > --- Later me to David Kjerrumgaard
> > > > What do you think about the following workflow:
> > > > 1. The user asks a new question in Slack.
> > > > 2. You click the three dots button on his message -> “Convert to a
> GitHub
> > > > discussion”.
> > > > 3. The bot creates a new GitHub discussion in the “Q&A” category
> with the
> > > > original message’s content.
> > > > 4. The bot converts the original message to a thread and posts a
> message
> > > > like:
> > > > > > In order to keep history and make your question searchable by
> other
> > > > Apache Pulsar users, it has been converted to a GitHub discussion.
> > > > > > Further conversation is available by this link:
> > > > https://github.com/apache/pulsar/discussions/18766.
> > > > 5. You answer the question on GitHub.
> > > >
> > > > How it may work:
> > > > https://api.slack.com/best-practices/blueprints/actions-and-dialogs
> > > > I can try to implement it makes any sense for you. (edited)
> > > >
> > > > Full original thread:
> > > > https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1676449603034309
> > > >
> > > >
>

Re: [DISCUSS] Using jetty-client instead of async-http-client

[Asaf Mesika]

Oh. I looked at the commits, it seems pretty active to me.


On Thu, Feb 16, 2023 at 3:31 PM Zixuan Liu  wrote:

> I point to async-http-client.
>
> Although there is a recent update.
>
> Thanks,
> Zixuan
>
> Asaf Mesika  于2023年2月16日周四 21:19写道：
>
> > On Tue, Feb 14, 2023 at 11:36 AM Zixuan Liu  wrote:
> >
> > > Hi all,
> > >
> > > Our admin-client using the async-http-client [0] to request the web
> > > service, the async-http-client implements network request based on the
> > > netty, which has very good performance, but this project is not very
> > > active.
> > >
> >
> > Netty - the most used network server in the world - is not very active?
> >
> >
> > >
> > > For security(library) reason or http feature(Follow up on future
> > > development), could we migrate to the jetty-client [1] from the
> > > async-http-client? The jetty project is very active, our web service is
> > > built based on the jetty-server, so I think use the jett-client is a
> good
> > > idea, but migrating this can be a lot of work.
> > >
> > > Please let me know what you think.
> > >
> > > [0] - https://github.com/AsyncHttpClient/async-http-client
> > > [1] - https://github.com/eclipse/jetty.project/tree/jetty-9.4.x
> > >
> > > Thanks,
> > > Zixuan
> > >
> >
>

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

[Yan Zhao]

> If understood correctly, every broker will have a consumer right? You will
> use a fail-over subscription? The retry-topic is consumed by the same
> subscription, same consumer?
Yes, you are right, there is the case you mention. The deletion is idempotent, 
I'm not sure if it's worth making it sync for it.

> In this very long mailing list thread, we have mentioned many fixes to be
> done. Can you ping in the mailing list once you have managed to fix it all?

That's fine, I will push the new pip in two days. After the new pip pushing, I 
will ping you.

Re: Force redirect questions from Slack to GitHub Discussions or StackOverflow?

[Kiryl Valkovich]

Played with Slack and StackOverflow APIs a bit.

The Slack API works as expected. After clicking the message button, it sends a 
message descriptor to my app where I can do anything with its content.

It’s possible to post messages via the StackOverflow API, but it’s unlikely 
that any Slack message can be converted to a StackOverflow question.

I encountered several types of validation errors:

-  This question body does not meet our quality standards. Please make 
sure that it completely describes your problem - including what you have 
already tried - and is written using proper grammar.

  *   Something similar to “This message looks like a duplicate of another 
message”.

I believe, GitHub Discussions don’t have such kind of “quality standards” 
validation.

From: Kiryl Valkovich 
Date: Thursday, February 16, 2023 at 1:33 PM
To: dev@pulsar.apache.org 
Subject: Re: Force redirect questions from Slack to GitHub Discussions or 
StackOverflow?
If there are no hidden obstacles, we can implement it.
StackOverflow supports question creation using REST API: 
https://api.stackexchange.com/docs/create-question

From: Zike Yang 
Date: Thursday, February 16, 2023 at 11:41 AM
To: dev@pulsar.apache.org 
Subject: Re: Force redirect questions from Slack to GitHub Discussions or 
StackOverflow?
+1, That sounds great to me.

> 2. You click the three dots button on his message -> “Convert to a GitHub 
> discussion”.

Is it a similar workflow for converting to a StackOverflow question?

BR,
Zike Yang

On Thu, Feb 16, 2023 at 12:14 PM  wrote:
>
> +1
>
> Since web pages are more easily and public.
>
> Best
> Mattison
> On Feb 16, 2023, 07:58 +0800, Christophe Bornet , 
> wrote:
> > +100
> > Also note that for good or bad reasons, the number of questions on
> > StaOverflow is often used as a metric for the popularity of a project.
> >
> > Le mer. 15 févr. 2023 à 13:12, Kiryl Valkovich 
> > a écrit :
> >
> > > Hi everyone! Enrico Olivelli asked me to repost it to the mailing list.
> > >
> > > --- Me
> > > I’m very worried that good answers from David Kjerrumgaard and others
> > > won’t be googleable because it’s in Slack. To make any search you need to
> > > pay a monthly subscription.
> > >
> > > In my opinion, it would be wiser to make StackOverflow, Discuss, or GitHub
> > > discussions a place for questions. And strictly redirect people who ask 
> > > any
> > > question in Slack to the right place.
> > > In GitHub discussions, you also can manage categories, as well as you can
> > > manage channels in Slack.
> > >
> > > Subscription to a specific category is coming soon.
> > > https://github.com/community/community/discussions/3951#discussioncomment-3879939
> > > The most important that it’s searchable.
> > >
> > > Examples:
> > > https://github.com/community/community/discussions
> > > https://github.com/apache/superset/discussions
> > >
> > > --- Later me to David Kjerrumgaard
> > > What do you think about the following workflow:
> > > 1. The user asks a new question in Slack.
> > > 2. You click the three dots button on his message -> “Convert to a GitHub
> > > discussion”.
> > > 3. The bot creates a new GitHub discussion in the “Q&A” category with the
> > > original message’s content.
> > > 4. The bot converts the original message to a thread and posts a message
> > > like:
> > > > > In order to keep history and make your question searchable by other
> > > Apache Pulsar users, it has been converted to a GitHub discussion.
> > > > > Further conversation is available by this link:
> > > https://github.com/apache/pulsar/discussions/18766.
> > > 5. You answer the question on GitHub.
> > >
> > > How it may work:
> > > https://api.slack.com/best-practices/blueprints/actions-and-dialogs
> > > I can try to implement it makes any sense for you. (edited)
> > >
> > > Full original thread:
> > > https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1676449603034309
> > >
> > >

Re: [Vote] PIP-245: Make subscriptions of non-persistent topic non-durable

[Baodi Shi]

 +1 (non-binding)

Thanks,
Baodi Shi


在 2023年2月16日 21:44:17 上，Asaf Mesika  写道：

> +1 (non-binding)
>
>
> On Thu, Feb 16, 2023 at 11:11 AM Jiuming Tao  >
> wrote:
>
>
> I’ve added the `Compatibility` selection into the PIP, please help review
>
> and vote the PIP
>
>
> Thanks,
>
> Tao Jiuming
>
>
>
>
> > 2023年2月15日 14:58，Zike Yang  写道：
>
> >
>
> > Hi, Jiuming
>
> >
>
> >> bump
>
> >
>
> > As for the discussion here[0], could you add a `Compatibility` section
>
> > to talk about compatibility in more detail? WDYT?
>
> > Then we could start the vote again.
>
> >
>
> > [0] https://lists.apache.org/thread/2bjg39zh7z38bzbnqngbo5l4jzkjttrq
>
> >
>
> > Thanks,
>
> > Zike Yang
>
> >
>
> > On Wed, Feb 15, 2023 at 1:34 PM Tao Jiuming  wrote:
>
> >>
>
> >>
>
> >> bump
>
> >>
>
> >> On 2023/02/13 06:56:09 Jiuming Tao wrote:
>
> >>> Hi all,
>
> >>>
>
> >>> I would like to start a VOTE on `PIP-245: Make subscriptions of
>
> non-persistent topic non-durable`.
>
> >>>
>
> >>> Motivation:
>
> >>>
>
> >>> There are two types of subscriptions for a topic: Durable and
>
> Non-durable.
>
> >>>
>
> >>> We create a Consumer with a Durable subscription and a Reader with a
>
> Non-durable subscription.
>
> >>>
>
> >>> But for NonPersistentTopic, creating a Durable subscription is
>
> meaningless, NonPersistentSubscription doesn't have a ManagedCursor to
>
> persistent its data. After its consumer disconnected, the subscription
>
> couldn't be removed automatically if we didn't set the value of
>
> subscriptionExpirationTimeMinutes greater than 0.
>
> >>>
>
> >>> For subscriptionExpirationTimeMinutes, it controls the subscription
>
> expiration of NonPersistentTopic and PersistentTopic, if we set the value
>
> of subscriptionExpirationTimeMinutes greater than 0, it may lead to data
>
> loss(The durable subscriptions of PersistentTopic also can be removed).
>
> >>>
>
> >>> And the Non-durable subscriptions will be removed automatically after
>
> all the consumers disconnected, it's the existing logic.
>
> >>>
>
> >>> For the purpose of removing the subscriptions which have no active
>
> consumers of NonPersistentTopic and the above reasons, we can make all the
>
> subscriptions of a NonPersistentTopic Non-durable.
>
> >>>
>
> >>>
>
> >>>
>
> >>> For more details, you can read:
>
> https://github.com/apache/pulsar/issues/19448 <
>
> https://github.com/apache/pulsar/issues/19448>
>
> >>>
>
> >>> And the discuss thread is available at:
>
> https://lists.apache.org/thread/2ltmyglnb25jy8nk58twkwbglws43bst <
>
> https://lists.apache.org/thread/2ltmyglnb25jy8nk58twkwbglws43bst>
>
> >>>
>
> >>> Thanks,
>
> >>> Tao Jiuming
>
>
>
>

Introducer Pulsar admin api to pulsar-client-go

[ZhangJian He]

I would like to express that the current Pulsar client for Go
(pulsar-client-go) is missing the pulsar Admin API. As such, I would like
to propose that we work towards adding this feature to pulsar-client-go.

I believe that this new feature would be a valuable addition to
pulsar-client-go, and I am excited to work to make it happen.

I have submitted a PR: https://github.com/apache/pulsar-client-go/pull/959
The full api is not currently available, but we are adding.

Below is a simple example about how to use

## usage

```go
package main

import (
"fmt"
"github.com/apache/pulsar-client-go/padmin"
)

func main() {
admin, err := padmin.NewDefaultPulsarAdmin()
if err != nil {
panic(err)
}
// get namespace topic list
topics, err := admin.PersistentTopics.ListNamespaceTopics("tenant",
"namespace")
if err != nil {
panic(err)
}
fmt.Println(topics)
}
```

Thanks
ZhangJian He

Re: Introducer Pulsar admin api to pulsar-client-go

[PengHui Li]

The pulsarctl is an admin CLI tool for Pulsar, written in go.
Does it match your requirement?

Penghui

> On Feb 17, 2023, at 09:47, ZhangJian He  wrote:
> 
> I would like to express that the current Pulsar client for Go
> (pulsar-client-go) is missing the pulsar Admin API. As such, I would like
> to propose that we work towards adding this feature to pulsar-client-go.
> 
> I believe that this new feature would be a valuable addition to
> pulsar-client-go, and I am excited to work to make it happen.
> 
> I have submitted a PR: https://github.com/apache/pulsar-client-go/pull/959
> The full api is not currently available, but we are adding.
> 
> Below is a simple example about how to use
> 
> ## usage
> 
> ```go
> package main
> 
> import (
> "fmt"
> "github.com/apache/pulsar-client-go/padmin"
> )
> 
> func main() {
> admin, err := padmin.NewDefaultPulsarAdmin()
> if err != nil {
> panic(err)
> }
> // get namespace topic list
> topics, err := admin.PersistentTopics.ListNamespaceTopics("tenant",
> "namespace")
> if err != nil {
> panic(err)
> }
> fmt.Println(topics)
> }
> ```
> 
> Thanks
> ZhangJian He

Re: Introducer Pulsar admin api to pulsar-client-go

[ZhangJian He]

Pulsarctl is not an apache official repository net, and given the
popularity of operators and Go-based dev-ops-tools, would it be possible to
consider merging some of the functionalities of pulsarctl into
pulsar-client-go

Thanks
ZhangJian He


On Fri, 17 Feb 2023 at 10:00, PengHui Li  wrote:

> The pulsarctl is an admin CLI tool for Pulsar, written in go.
> Does it match your requirement?
>
> Penghui
>
> > On Feb 17, 2023, at 09:47, ZhangJian He  wrote:
> >
> > I would like to express that the current Pulsar client for Go
> > (pulsar-client-go) is missing the pulsar Admin API. As such, I would like
> > to propose that we work towards adding this feature to pulsar-client-go.
> >
> > I believe that this new feature would be a valuable addition to
> > pulsar-client-go, and I am excited to work to make it happen.
> >
> > I have submitted a PR:
> https://github.com/apache/pulsar-client-go/pull/959
> > The full api is not currently available, but we are adding.
> >
> > Below is a simple example about how to use
> >
> > ## usage
> >
> > ```go
> > package main
> >
> > import (
> > "fmt"
> > "github.com/apache/pulsar-client-go/padmin"
> > )
> >
> > func main() {
> > admin, err := padmin.NewDefaultPulsarAdmin()
> > if err != nil {
> > panic(err)
> > }
> > // get namespace topic list
> > topics, err := admin.PersistentTopics.ListNamespaceTopics("tenant",
> > "namespace")
> > if err != nil {
> > panic(err)
> > }
> > fmt.Println(topics)
> > }
> > ```
> >
> > Thanks
> > ZhangJian He
>
>

Re: Introducer Pulsar admin api to pulsar-client-go

[Yunze Xu]

Just saw the mail after the PR.

> Pulsarctl is not an apache official repository net

It does not make sense to me. Why cannot add it as a dependency?

Thanks,
Yunze

On Fri, Feb 17, 2023 at 10:16 AM ZhangJian He  wrote:
>
> Pulsarctl is not an apache official repository net, and given the
> popularity of operators and Go-based dev-ops-tools, would it be possible to
> consider merging some of the functionalities of pulsarctl into
> pulsar-client-go
>
> Thanks
> ZhangJian He
>
>
> On Fri, 17 Feb 2023 at 10:00, PengHui Li  wrote:
>
> > The pulsarctl is an admin CLI tool for Pulsar, written in go.
> > Does it match your requirement?
> >
> > Penghui
> >
> > > On Feb 17, 2023, at 09:47, ZhangJian He  wrote:
> > >
> > > I would like to express that the current Pulsar client for Go
> > > (pulsar-client-go) is missing the pulsar Admin API. As such, I would like
> > > to propose that we work towards adding this feature to pulsar-client-go.
> > >
> > > I believe that this new feature would be a valuable addition to
> > > pulsar-client-go, and I am excited to work to make it happen.
> > >
> > > I have submitted a PR:
> > https://github.com/apache/pulsar-client-go/pull/959
> > > The full api is not currently available, but we are adding.
> > >
> > > Below is a simple example about how to use
> > >
> > > ## usage
> > >
> > > ```go
> > > package main
> > >
> > > import (
> > > "fmt"
> > > "github.com/apache/pulsar-client-go/padmin"
> > > )
> > >
> > > func main() {
> > > admin, err := padmin.NewDefaultPulsarAdmin()
> > > if err != nil {
> > > panic(err)
> > > }
> > > // get namespace topic list
> > > topics, err := admin.PersistentTopics.ListNamespaceTopics("tenant",
> > > "namespace")
> > > if err != nil {
> > > panic(err)
> > > }
> > > fmt.Println(topics)
> > > }
> > > ```
> > >
> > > Thanks
> > > ZhangJian He
> >
> >

Re: Introducer Pulsar admin api to pulsar-client-go

[ZhangJian He]

> It does not make sense to me. Why cannot add it as a dependency?

I am not saying it can't. Of course, adding pulsarctl as a dependency is
possible. I also recommended this repository in
https://github.com/apache/pulsar-client-go/issues/842. I think
incorporating the functionality of the HTTP admin into pulsar-client-go
might be even better. Developers can use pulsar-client-go to perform
administrative tasks without having to switch to a separate tool like
pulsarctl.

Thanks
ZhangJian He


On Fri, 17 Feb 2023 at 10:39, Yunze Xu  wrote:

> Just saw the mail after the PR.
>
> > Pulsarctl is not an apache official repository net
>
> It does not make sense to me. Why cannot add it as a dependency?
>
> Thanks,
> Yunze
>
> On Fri, Feb 17, 2023 at 10:16 AM ZhangJian He  wrote:
> >
> > Pulsarctl is not an apache official repository net, and given the
> > popularity of operators and Go-based dev-ops-tools, would it be possible
> to
> > consider merging some of the functionalities of pulsarctl into
> > pulsar-client-go
> >
> > Thanks
> > ZhangJian He
> >
> >
> > On Fri, 17 Feb 2023 at 10:00, PengHui Li 
> wrote:
> >
> > > The pulsarctl is an admin CLI tool for Pulsar, written in go.
> > > Does it match your requirement?
> > >
> > > Penghui
> > >
> > > > On Feb 17, 2023, at 09:47, ZhangJian He  wrote:
> > > >
> > > > I would like to express that the current Pulsar client for Go
> > > > (pulsar-client-go) is missing the pulsar Admin API. As such, I would
> like
> > > > to propose that we work towards adding this feature to
> pulsar-client-go.
> > > >
> > > > I believe that this new feature would be a valuable addition to
> > > > pulsar-client-go, and I am excited to work to make it happen.
> > > >
> > > > I have submitted a PR:
> > > https://github.com/apache/pulsar-client-go/pull/959
> > > > The full api is not currently available, but we are adding.
> > > >
> > > > Below is a simple example about how to use
> > > >
> > > > ## usage
> > > >
> > > > ```go
> > > > package main
> > > >
> > > > import (
> > > > "fmt"
> > > > "github.com/apache/pulsar-client-go/padmin"
> > > > )
> > > >
> > > > func main() {
> > > > admin, err := padmin.NewDefaultPulsarAdmin()
> > > > if err != nil {
> > > > panic(err)
> > > > }
> > > > // get namespace topic list
> > > > topics, err := admin.PersistentTopics.ListNamespaceTopics("tenant",
> > > > "namespace")
> > > > if err != nil {
> > > > panic(err)
> > > > }
> > > > fmt.Println(topics)
> > > > }
> > > > ```
> > > >
> > > > Thanks
> > > > ZhangJian He
> > >
> > >
>

Re: Introducer Pulsar admin api to pulsar-client-go

[Yunze Xu]

It's better to reuse existing code rather than reinventing the wheel.
I didn't mean using pulsarctl directly. We only need to wrap the
provided API from pulsarctl and provide our own public API in
pulsar-client-go.

Thanks,
Yunze

On Fri, Feb 17, 2023 at 10:48 AM ZhangJian He  wrote:
>
> > It does not make sense to me. Why cannot add it as a dependency?
>
> I am not saying it can't. Of course, adding pulsarctl as a dependency is
> possible. I also recommended this repository in
> https://github.com/apache/pulsar-client-go/issues/842. I think
> incorporating the functionality of the HTTP admin into pulsar-client-go
> might be even better. Developers can use pulsar-client-go to perform
> administrative tasks without having to switch to a separate tool like
> pulsarctl.
>
> Thanks
> ZhangJian He
>
>
> On Fri, 17 Feb 2023 at 10:39, Yunze Xu  wrote:
>
> > Just saw the mail after the PR.
> >
> > > Pulsarctl is not an apache official repository net
> >
> > It does not make sense to me. Why cannot add it as a dependency?
> >
> > Thanks,
> > Yunze
> >
> > On Fri, Feb 17, 2023 at 10:16 AM ZhangJian He  wrote:
> > >
> > > Pulsarctl is not an apache official repository net, and given the
> > > popularity of operators and Go-based dev-ops-tools, would it be possible
> > to
> > > consider merging some of the functionalities of pulsarctl into
> > > pulsar-client-go
> > >
> > > Thanks
> > > ZhangJian He
> > >
> > >
> > > On Fri, 17 Feb 2023 at 10:00, PengHui Li 
> > wrote:
> > >
> > > > The pulsarctl is an admin CLI tool for Pulsar, written in go.
> > > > Does it match your requirement?
> > > >
> > > > Penghui
> > > >
> > > > > On Feb 17, 2023, at 09:47, ZhangJian He  wrote:
> > > > >
> > > > > I would like to express that the current Pulsar client for Go
> > > > > (pulsar-client-go) is missing the pulsar Admin API. As such, I would
> > like
> > > > > to propose that we work towards adding this feature to
> > pulsar-client-go.
> > > > >
> > > > > I believe that this new feature would be a valuable addition to
> > > > > pulsar-client-go, and I am excited to work to make it happen.
> > > > >
> > > > > I have submitted a PR:
> > > > https://github.com/apache/pulsar-client-go/pull/959
> > > > > The full api is not currently available, but we are adding.
> > > > >
> > > > > Below is a simple example about how to use
> > > > >
> > > > > ## usage
> > > > >
> > > > > ```go
> > > > > package main
> > > > >
> > > > > import (
> > > > > "fmt"
> > > > > "github.com/apache/pulsar-client-go/padmin"
> > > > > )
> > > > >
> > > > > func main() {
> > > > > admin, err := padmin.NewDefaultPulsarAdmin()
> > > > > if err != nil {
> > > > > panic(err)
> > > > > }
> > > > > // get namespace topic list
> > > > > topics, err := admin.PersistentTopics.ListNamespaceTopics("tenant",
> > > > > "namespace")
> > > > > if err != nil {
> > > > > panic(err)
> > > > > }
> > > > > fmt.Println(topics)
> > > > > }
> > > > > ```
> > > > >
> > > > > Thanks
> > > > > ZhangJian He
> > > >
> > > >
> >

Re: Introducer Pulsar admin api to pulsar-client-go

[ZhangJian He]

> It's better to reuse existing code rather than reinventing the wheel.
No objection.
> I didn't mean using pulsarctl directly. We only need to wrap the
provided API from pulsarctl and provide our own public API in
pulsar-client-go.
It works, I have a point of concern, but for a small HTTP interaction, it's
probably too many layers of encapsulation.  Let's see what other people say

Thanks
ZhangJian He


On Fri, 17 Feb 2023 at 11:06, Yunze Xu  wrote:

> It's better to reuse existing code rather than reinventing the wheel.
> I didn't mean using pulsarctl directly. We only need to wrap the
> provided API from pulsarctl and provide our own public API in
> pulsar-client-go.
>
> Thanks,
> Yunze
>
> On Fri, Feb 17, 2023 at 10:48 AM ZhangJian He  wrote:
> >
> > > It does not make sense to me. Why cannot add it as a dependency?
> >
> > I am not saying it can't. Of course, adding pulsarctl as a dependency is
> > possible. I also recommended this repository in
> > https://github.com/apache/pulsar-client-go/issues/842. I think
> > incorporating the functionality of the HTTP admin into pulsar-client-go
> > might be even better. Developers can use pulsar-client-go to perform
> > administrative tasks without having to switch to a separate tool like
> > pulsarctl.
> >
> > Thanks
> > ZhangJian He
> >
> >
> > On Fri, 17 Feb 2023 at 10:39, Yunze Xu 
> wrote:
> >
> > > Just saw the mail after the PR.
> > >
> > > > Pulsarctl is not an apache official repository net
> > >
> > > It does not make sense to me. Why cannot add it as a dependency?
> > >
> > > Thanks,
> > > Yunze
> > >
> > > On Fri, Feb 17, 2023 at 10:16 AM ZhangJian He 
> wrote:
> > > >
> > > > Pulsarctl is not an apache official repository net, and given the
> > > > popularity of operators and Go-based dev-ops-tools, would it be
> possible
> > > to
> > > > consider merging some of the functionalities of pulsarctl into
> > > > pulsar-client-go
> > > >
> > > > Thanks
> > > > ZhangJian He
> > > >
> > > >
> > > > On Fri, 17 Feb 2023 at 10:00, PengHui Li 
> > > wrote:
> > > >
> > > > > The pulsarctl is an admin CLI tool for Pulsar, written in go.
> > > > > Does it match your requirement?
> > > > >
> > > > > Penghui
> > > > >
> > > > > > On Feb 17, 2023, at 09:47, ZhangJian He 
> wrote:
> > > > > >
> > > > > > I would like to express that the current Pulsar client for Go
> > > > > > (pulsar-client-go) is missing the pulsar Admin API. As such, I
> would
> > > like
> > > > > > to propose that we work towards adding this feature to
> > > pulsar-client-go.
> > > > > >
> > > > > > I believe that this new feature would be a valuable addition to
> > > > > > pulsar-client-go, and I am excited to work to make it happen.
> > > > > >
> > > > > > I have submitted a PR:
> > > > > https://github.com/apache/pulsar-client-go/pull/959
> > > > > > The full api is not currently available, but we are adding.
> > > > > >
> > > > > > Below is a simple example about how to use
> > > > > >
> > > > > > ## usage
> > > > > >
> > > > > > ```go
> > > > > > package main
> > > > > >
> > > > > > import (
> > > > > > "fmt"
> > > > > > "github.com/apache/pulsar-client-go/padmin"
> > > > > > )
> > > > > >
> > > > > > func main() {
> > > > > > admin, err := padmin.NewDefaultPulsarAdmin()
> > > > > > if err != nil {
> > > > > > panic(err)
> > > > > > }
> > > > > > // get namespace topic list
> > > > > > topics, err :=
> admin.PersistentTopics.ListNamespaceTopics("tenant",
> > > > > > "namespace")
> > > > > > if err != nil {
> > > > > > panic(err)
> > > > > > }
> > > > > > fmt.Println(topics)
> > > > > > }
> > > > > > ```
> > > > > >
> > > > > > Thanks
> > > > > > ZhangJian He
> > > > >
> > > > >
> > >
>

Re: Introducer Pulsar admin api to pulsar-client-go

[Michael Marshall]

I think it would be valuable to have admin clients in many languages.
Has anyone tried generating an admin client from our generated open
api spec?

I notice that the Java Client and the Java Admin Client are separate
dependencies. Is this boundary important to maintain for other
language admin clients?

Thanks,
Michael

On Thu, Feb 16, 2023 at 7:47 PM ZhangJian He  wrote:
>
> I would like to express that the current Pulsar client for Go
> (pulsar-client-go) is missing the pulsar Admin API. As such, I would like
> to propose that we work towards adding this feature to pulsar-client-go.
>
> I believe that this new feature would be a valuable addition to
> pulsar-client-go, and I am excited to work to make it happen.
>
> I have submitted a PR: https://github.com/apache/pulsar-client-go/pull/959
> The full api is not currently available, but we are adding.
>
> Below is a simple example about how to use
>
> ## usage
>
> ```go
> package main
>
> import (
> "fmt"
> "github.com/apache/pulsar-client-go/padmin"
> )
>
> func main() {
> admin, err := padmin.NewDefaultPulsarAdmin()
> if err != nil {
> panic(err)
> }
> // get namespace topic list
> topics, err := admin.PersistentTopics.ListNamespaceTopics("tenant",
> "namespace")
> if err != nil {
> panic(err)
> }
> fmt.Println(topics)
> }
> ```
>
> Thanks
> ZhangJian He

Re: Introducer Pulsar admin api to pulsar-client-go

[Yunze Xu]

> I have a point of concern, but for a small HTTP interaction, it's probably 
> too many layers of encapsulation.

It makes sense to me. Then I agree to go head for an independent API
implementation.

Thanks,
Yunze

On Fri, Feb 17, 2023 at 11:32 AM ZhangJian He  wrote:
>
> > It's better to reuse existing code rather than reinventing the wheel.
> No objection.
> > I didn't mean using pulsarctl directly. We only need to wrap the
> provided API from pulsarctl and provide our own public API in
> pulsar-client-go.
> It works, I have a point of concern, but for a small HTTP interaction, it's
> probably too many layers of encapsulation.  Let's see what other people say
>
> Thanks
> ZhangJian He
>
>
> On Fri, 17 Feb 2023 at 11:06, Yunze Xu  wrote:
>
> > It's better to reuse existing code rather than reinventing the wheel.
> > I didn't mean using pulsarctl directly. We only need to wrap the
> > provided API from pulsarctl and provide our own public API in
> > pulsar-client-go.
> >
> > Thanks,
> > Yunze
> >
> > On Fri, Feb 17, 2023 at 10:48 AM ZhangJian He  wrote:
> > >
> > > > It does not make sense to me. Why cannot add it as a dependency?
> > >
> > > I am not saying it can't. Of course, adding pulsarctl as a dependency is
> > > possible. I also recommended this repository in
> > > https://github.com/apache/pulsar-client-go/issues/842. I think
> > > incorporating the functionality of the HTTP admin into pulsar-client-go
> > > might be even better. Developers can use pulsar-client-go to perform
> > > administrative tasks without having to switch to a separate tool like
> > > pulsarctl.
> > >
> > > Thanks
> > > ZhangJian He
> > >
> > >
> > > On Fri, 17 Feb 2023 at 10:39, Yunze Xu 
> > wrote:
> > >
> > > > Just saw the mail after the PR.
> > > >
> > > > > Pulsarctl is not an apache official repository net
> > > >
> > > > It does not make sense to me. Why cannot add it as a dependency?
> > > >
> > > > Thanks,
> > > > Yunze
> > > >
> > > > On Fri, Feb 17, 2023 at 10:16 AM ZhangJian He 
> > wrote:
> > > > >
> > > > > Pulsarctl is not an apache official repository net, and given the
> > > > > popularity of operators and Go-based dev-ops-tools, would it be
> > possible
> > > > to
> > > > > consider merging some of the functionalities of pulsarctl into
> > > > > pulsar-client-go
> > > > >
> > > > > Thanks
> > > > > ZhangJian He
> > > > >
> > > > >
> > > > > On Fri, 17 Feb 2023 at 10:00, PengHui Li 
> > > > wrote:
> > > > >
> > > > > > The pulsarctl is an admin CLI tool for Pulsar, written in go.
> > > > > > Does it match your requirement?
> > > > > >
> > > > > > Penghui
> > > > > >
> > > > > > > On Feb 17, 2023, at 09:47, ZhangJian He 
> > wrote:
> > > > > > >
> > > > > > > I would like to express that the current Pulsar client for Go
> > > > > > > (pulsar-client-go) is missing the pulsar Admin API. As such, I
> > would
> > > > like
> > > > > > > to propose that we work towards adding this feature to
> > > > pulsar-client-go.
> > > > > > >
> > > > > > > I believe that this new feature would be a valuable addition to
> > > > > > > pulsar-client-go, and I am excited to work to make it happen.
> > > > > > >
> > > > > > > I have submitted a PR:
> > > > > > https://github.com/apache/pulsar-client-go/pull/959
> > > > > > > The full api is not currently available, but we are adding.
> > > > > > >
> > > > > > > Below is a simple example about how to use
> > > > > > >
> > > > > > > ## usage
> > > > > > >
> > > > > > > ```go
> > > > > > > package main
> > > > > > >
> > > > > > > import (
> > > > > > > "fmt"
> > > > > > > "github.com/apache/pulsar-client-go/padmin"
> > > > > > > )
> > > > > > >
> > > > > > > func main() {
> > > > > > > admin, err := padmin.NewDefaultPulsarAdmin()
> > > > > > > if err != nil {
> > > > > > > panic(err)
> > > > > > > }
> > > > > > > // get namespace topic list
> > > > > > > topics, err :=
> > admin.PersistentTopics.ListNamespaceTopics("tenant",
> > > > > > > "namespace")
> > > > > > > if err != nil {
> > > > > > > panic(err)
> > > > > > > }
> > > > > > > fmt.Println(topics)
> > > > > > > }
> > > > > > > ```
> > > > > > >
> > > > > > > Thanks
> > > > > > > ZhangJian He
> > > > > >
> > > > > >
> > > >
> >

Re: Introducer Pulsar admin api to pulsar-client-go

[Yunze Xu]

> I notice that the Java Client and the Java Admin Client are separate
dependencies. Is this boundary important to maintain for other
language admin clients?

IMO, separating them is better to maintain. I had an idea to implement
a pure C implementation of the Pulsar admin API. Only libcurl and
openssl dependencies are required. Compared with the C++ library, the
pure C library has smaller size, better ABI compatibility, and much
quicker compilation speed than the C++ library, which has some more
heavy dependencies like Boost and Protobuf.

Thanks,
Yunze

On Fri, Feb 17, 2023 at 11:58 AM Michael Marshall  wrote:
>
> I think it would be valuable to have admin clients in many languages.
> Has anyone tried generating an admin client from our generated open
> api spec?
>
> I notice that the Java Client and the Java Admin Client are separate
> dependencies. Is this boundary important to maintain for other
> language admin clients?
>
> Thanks,
> Michael
>
> On Thu, Feb 16, 2023 at 7:47 PM ZhangJian He  wrote:
> >
> > I would like to express that the current Pulsar client for Go
> > (pulsar-client-go) is missing the pulsar Admin API. As such, I would like
> > to propose that we work towards adding this feature to pulsar-client-go.
> >
> > I believe that this new feature would be a valuable addition to
> > pulsar-client-go, and I am excited to work to make it happen.
> >
> > I have submitted a PR: https://github.com/apache/pulsar-client-go/pull/959
> > The full api is not currently available, but we are adding.
> >
> > Below is a simple example about how to use
> >
> > ## usage
> >
> > ```go
> > package main
> >
> > import (
> > "fmt"
> > "github.com/apache/pulsar-client-go/padmin"
> > )
> >
> > func main() {
> > admin, err := padmin.NewDefaultPulsarAdmin()
> > if err != nil {
> > panic(err)
> > }
> > // get namespace topic list
> > topics, err := admin.PersistentTopics.ListNamespaceTopics("tenant",
> > "namespace")
> > if err != nil {
> > panic(err)
> > }
> > fmt.Println(topics)
> > }
> > ```
> >
> > Thanks
> > ZhangJian He

Re: Introducer Pulsar admin api to pulsar-client-go

[ZhangJian He]

Separating dependencies is better. For example, I think Pulsar-admin-go can
only have golang standard tls and http dependencies.
But it seems impossible to have two go modules when publishing packages
using github.

> Has anyone tried generating an admin client from our generated open
api spec?

I have attempted it, but it requires us to modify our Swagger file. Our
existing Swagger file can't generate HTTP clients directly. Perhaps we can
rewrite a unified and standardized Swagger file, and then generate all
code, including brokers, from there gradually.

Thanks
ZhangJian He


On Fri, 17 Feb 2023 at 12:37, Yunze Xu  wrote:

> > I notice that the Java Client and the Java Admin Client are separate
> dependencies. Is this boundary important to maintain for other
> language admin clients?
>
> IMO, separating them is better to maintain. I had an idea to implement
> a pure C implementation of the Pulsar admin API. Only libcurl and
> openssl dependencies are required. Compared with the C++ library, the
> pure C library has smaller size, better ABI compatibility, and much
> quicker compilation speed than the C++ library, which has some more
> heavy dependencies like Boost and Protobuf.
>
> Thanks,
> Yunze
>
> On Fri, Feb 17, 2023 at 11:58 AM Michael Marshall 
> wrote:
> >
> > I think it would be valuable to have admin clients in many languages.
> > Has anyone tried generating an admin client from our generated open
> > api spec?
> >
> > I notice that the Java Client and the Java Admin Client are separate
> > dependencies. Is this boundary important to maintain for other
> > language admin clients?
> >
> > Thanks,
> > Michael
> >
> > On Thu, Feb 16, 2023 at 7:47 PM ZhangJian He  wrote:
> > >
> > > I would like to express that the current Pulsar client for Go
> > > (pulsar-client-go) is missing the pulsar Admin API. As such, I would
> like
> > > to propose that we work towards adding this feature to
> pulsar-client-go.
> > >
> > > I believe that this new feature would be a valuable addition to
> > > pulsar-client-go, and I am excited to work to make it happen.
> > >
> > > I have submitted a PR:
> https://github.com/apache/pulsar-client-go/pull/959
> > > The full api is not currently available, but we are adding.
> > >
> > > Below is a simple example about how to use
> > >
> > > ## usage
> > >
> > > ```go
> > > package main
> > >
> > > import (
> > > "fmt"
> > > "github.com/apache/pulsar-client-go/padmin"
> > > )
> > >
> > > func main() {
> > > admin, err := padmin.NewDefaultPulsarAdmin()
> > > if err != nil {
> > > panic(err)
> > > }
> > > // get namespace topic list
> > > topics, err := admin.PersistentTopics.ListNamespaceTopics("tenant",
> > > "namespace")
> > > if err != nil {
> > > panic(err)
> > > }
> > > fmt.Println(topics)
> > > }
> > > ```
> > >
> > > Thanks
> > > ZhangJian He
>

Inconsistent GPG keys in dev and release repositories

[Yunze Xu]

Hi all,

I found the GPG keys, which are used in verifying the signatures of
release candidates, are much different in dev and release
repositories:
https://dist.apache.org/repos/dist/dev/pulsar/KEYS
https://dist.apache.org/repos/dist/release/pulsar/KEYS

>From here [1], it seems like we need to append the GPG key of a
committer into the release repo as well. But it seems that the KEYS
file in the release repo is never used. Should we make them
consistent? Or just remove the KEYS file in release repo?

[1] 
https://pulsar.apache.org/contribute/create-gpg-keys/#appending-the-key-to-keys-files

Re: Inconsistent GPG keys in dev and release repositories

[Zike Yang]

Hi, Yunze

I think the KEYS file in the release repo is necessary. They are both
used to verify the release file. Otherwise, the user will fail when
checking the GPG signature on the release file.

BR,
Zike Yang

On Fri, Feb 17, 2023 at 2:16 PM Yunze Xu  wrote:
>
> Hi all,
>
> I found the GPG keys, which are used in verifying the signatures of
> release candidates, are much different in dev and release
> repositories:
> https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> https://dist.apache.org/repos/dist/release/pulsar/KEYS
>
> From here [1], it seems like we need to append the GPG key of a
> committer into the release repo as well. But it seems that the KEYS
> file in the release repo is never used. Should we make them
> consistent? Or just remove the KEYS file in release repo?
>
> [1] 
> https://pulsar.apache.org/contribute/create-gpg-keys/#appending-the-key-to-keys-files

Re: [VOTE][PIP-240] A new API to unload subscriptions

[Yubiao Feng]

The PIP passes with 3 bindings +1: Bo Cong, Jiwei Guo, and Haiting Jiang. I
will start working, Please help to move to the wiki. Thanks
Yubiao Feng

On Tue, Feb 14, 2023 at 10:21 AM Haiting Jiang 
wrote:

> +1 binding
>
>
> Thanks
> Haiting
>
> On Mon, Feb 13, 2023 at 10:27 AM 丛搏  wrote:
> >
> > +1 (binding)
> >
> > Thanks,
> > Bo
> >
> > 易客 萧  于2023年2月13日周一 09:51写道：
> > >
> > > +1 (non-binding)
> > > 
> > > From: Yubiao Feng 
> > > Sent: Sunday, February 12, 2023 20:04
> > > To: dev@pulsar.apache.org 
> > > Subject: [VOTE][PIP-240] A new API to unload subscriptions
> > >
> > > Hi Pulsar Community
> > >
> > > I would like to start a VOTE on "A new API to unload subscriptions"
> > >
> > > The proposal can be read at *
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fissues%2F19187&data=05%7C01%7C%7Ca3616706ed51414081be08db0cf169b3%7C84df9e7fe9f640afb435%7C1%7C0%7C63811800349767%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=yY8VMC0%2BqCNr5JgmRDT6oLuz%2B8FiCf1vkMvIzpFimu8%3D&reserved=0
> > > <
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fissues%2F19187&data=05%7C01%7C%7Ca3616706ed51414081be08db0cf169b3%7C84df9e7fe9f640afb435%7C1%7C0%7C638118003222405993%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=eIy5wn26iW4CAXVRaWypGKa61IGWwoOMVgKMH3jHKLg%3D&reserved=0
> >*
> > >
> > > and the discussion thread is available at
> > >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2F55n75h9lyfscnc04wooz9p676cg44w79&data=05%7C01%7C%7Ca3616706ed51414081be08db0cf169b3%7C84df9e7fe9f640afb435%7C1%7C0%7C638118003222405993%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Xb5KQdHDG05qcb%2BnbV%2FZA3XjQ03RqJIUoR7d9Sl28KE%3D&reserved=0
> > >
> > > Voting will stay open for at least 48h.
> > >
> > > Thanks
> > > Yubiao Feng
>

Re: Introducer Pulsar admin api to pulsar-client-go

[Zixuan Liu]

Hi Zhangjian,

This is a good idea to write the admin client by golang, but I don't
suggest add the admin features to pulsar-go-client, it's better to use a
new repository to do that to  separate dependencies.

BTW, StreamNative has a pulsarctl [0] tool, which includes the admin api.

> > It's better to reuse existing code rather than reinventing the wheel.

I aggred this point. If possible, we can integrate the pulsarctl to this
new project.

[0] - https://github.com/streamnative/pulsarctl

Thanks,
Zixuan


ZhangJian He  于2023年2月17日周五 13:47写道：

> Separating dependencies is better. For example, I think Pulsar-admin-go can
> only have golang standard tls and http dependencies.
> But it seems impossible to have two go modules when publishing packages
> using github.
>
> > Has anyone tried generating an admin client from our generated open
> api spec?
>
> I have attempted it, but it requires us to modify our Swagger file. Our
> existing Swagger file can't generate HTTP clients directly. Perhaps we can
> rewrite a unified and standardized Swagger file, and then generate all
> code, including brokers, from there gradually.
>
> Thanks
> ZhangJian He
>
>
> On Fri, 17 Feb 2023 at 12:37, Yunze Xu 
> wrote:
>
> > > I notice that the Java Client and the Java Admin Client are separate
> > dependencies. Is this boundary important to maintain for other
> > language admin clients?
> >
> > IMO, separating them is better to maintain. I had an idea to implement
> > a pure C implementation of the Pulsar admin API. Only libcurl and
> > openssl dependencies are required. Compared with the C++ library, the
> > pure C library has smaller size, better ABI compatibility, and much
> > quicker compilation speed than the C++ library, which has some more
> > heavy dependencies like Boost and Protobuf.
> >
> > Thanks,
> > Yunze
> >
> > On Fri, Feb 17, 2023 at 11:58 AM Michael Marshall 
> > wrote:
> > >
> > > I think it would be valuable to have admin clients in many languages.
> > > Has anyone tried generating an admin client from our generated open
> > > api spec?
> > >
> > > I notice that the Java Client and the Java Admin Client are separate
> > > dependencies. Is this boundary important to maintain for other
> > > language admin clients?
> > >
> > > Thanks,
> > > Michael
> > >
> > > On Thu, Feb 16, 2023 at 7:47 PM ZhangJian He 
> wrote:
> > > >
> > > > I would like to express that the current Pulsar client for Go
> > > > (pulsar-client-go) is missing the pulsar Admin API. As such, I would
> > like
> > > > to propose that we work towards adding this feature to
> > pulsar-client-go.
> > > >
> > > > I believe that this new feature would be a valuable addition to
> > > > pulsar-client-go, and I am excited to work to make it happen.
> > > >
> > > > I have submitted a PR:
> > https://github.com/apache/pulsar-client-go/pull/959
> > > > The full api is not currently available, but we are adding.
> > > >
> > > > Below is a simple example about how to use
> > > >
> > > > ## usage
> > > >
> > > > ```go
> > > > package main
> > > >
> > > > import (
> > > > "fmt"
> > > > "github.com/apache/pulsar-client-go/padmin"
> > > > )
> > > >
> > > > func main() {
> > > > admin, err := padmin.NewDefaultPulsarAdmin()
> > > > if err != nil {
> > > > panic(err)
> > > > }
> > > > // get namespace topic list
> > > > topics, err := admin.PersistentTopics.ListNamespaceTopics("tenant",
> > > > "namespace")
> > > > if err != nil {
> > > > panic(err)
> > > > }
> > > > fmt.Println(topics)
> > > > }
> > > > ```
> > > >
> > > > Thanks
> > > > ZhangJian He
> >
>

Re: [DISCUSS] PIP-242 Topic name restrictions

[Enrico Olivelli]

Il giorno gio 16 feb 2023 alle ore 14:39 Asaf Mesika
 ha scritto:
>
> On Wed, Feb 15, 2023 at 4:36 PM  wrote:
>
> > Hi, All
> >
> > First of all, I want to list all of the system topics as follows. That
> > Yunze has mentioned before.
> >
> > Namespace level:
> >
> > • pulsar/system
> > • transaction_coordinator_assign
> > • __transaction_log_
> > • resource-usage
> > • pulsar/
> > • healthcheck
> >
> > Topic level:
> >
> > • __change_events
> > • __transaction_buffer_snapshot_segment
> > • ...
> >
> > We can check the details here. [0]
> >
> > Secondly. I think we need focus on the system topic name prefix. we have
> > some options as follows:
> >
> > • __SYSTEM__
> > • __system__
> >
> >
> > Both them make sense for me. As there are already two people prefer
> > `__SYSTEM`. if no other people have concern, I will chose it.
> >
>
> I vote `__system__` since I think upper casing is not that common for
> internal names (for me it's "shouting" and requires an extra key press to
> write it).
> Examples:
>
> Cassandra System tables
> 
> :
> * batches
>
> Postgres database catalog
> :
> * pg_database
> 
>
> OpenSearch system indices
> 
> :
> * .opendistro-alerting-config
>
>
> So I think we should stick to `__system__`

__system__ may work for me,
but how do we deal with existing clusters ?


Enrico

>
>
> >
> > Best,
> > Mattison
> >
> > -[0]
> > https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/java/org/apache/pulsar/common/naming/SystemTopicNames.java
> > On Feb 15, 2023, 11:36 +0800, Michael Marshall ,
> > wrote:
> > > I support this PIP, thank you for driving it forward Mattison.
> > >
> > > I support using the prefix __SYSTEM__ for system topics, and I agree
> > > that DLQ and RETRY topics do not qualify as system topics because they
> > > are produced to and consumed by regular clients.
> > >
> > > I think we might benefit from a generic definition for a system topic.
> > > I provided a loose definition for system topics in this email [0]. I
> > > copy it here:
> > >
> > > A system topic is a topic that is completely internal to Pulsar
> > > components. Internally, it is a normal topic. It requires elevated
> > > permission to produce/consume when authorization is enabled, even if
> > > topic level policies are not enabled. Generic calls like
> > > `clearNamespaceBacklog` should not affect system topics. Deleting a
> > > namespace or tenant should delete the system topics within it.
> > >
> > > Thanks,
> > > Michael
> > >
> > > [0] https://lists.apache.org/thread/sr01hvqmdrnk4lxwfwzcpg7y21psj6tt
> > >
> > >
> > > On Tue, Feb 14, 2023 at 9:55 AM  wrote:
> > > >
> > > > > > Could you list all existing system topic names that are used so we
> > canunderstand the rule better?
> > > > Yes, sure. I will list it later.
> > > > > > And I saw that this proposal only forbids the creation of
> > thesetopics. What about writing messages to them? I think it's better notto
> > allow Pulsar clients to write messages to them. The Pulsar clientshould
> > configure something to get the permission to write messages tothem. To keep
> > the compatibility, maybe we can make use of the protocolversion.
> > > > Sure, I agree with your point, because the current proposal just want
> > to have topic name restriction. maybe we can revise the system topic
> > permission and etc in the next proposal. (relate to system topic)
> > > >
> > > > Best,
> > > > Mattison
> > > > On Feb 14, 2023, 21:22 +0800, Yunze Xu ,
> > wrote:
> > > > > > Could you list all existing system topic names that are used so we
> > can
> > > > > > understand the rule better? Such as the "-RETRY" and "-DLQ" topics
> > > > > > mentioned above.
> > > > > >
> > > > > > And I saw that this proposal only forbids the creation of these
> > > > > > topics. What about writing messages to them? I think it's better
> > not
> > > > > > to allow Pulsar clients to write messages to them. The Pulsar
> > client
> > > > > > should configure something to get the permission to write messages
> > to
> > > > > > them. To keep the compatibility, maybe we can make use of the
> > protocol
> > > > > > version.
> > > > > >
> > > > > > Thanks
> > > > > > Yunze
> > > > > >
> > > > > > On Tue, Feb 14, 2023 at 5:38 PM Yubiao Feng
> > > > > >  wrote:
> > > > > > > >
> > > > > > > > Hi Qiang
> > > > > > > >
> > > > > > > > ### System topic name
> > > > > > > > '__system__' I think this format is clearer.
> > > > > > > > Now the system automatically creates topics of type retry
> > consumer letters
> > > > > > > > and dead letters.
> > > > > > > > These topics all end in uppercase letters, such as `-RETRY,`
> > `-DLQ.`
> > > > > > > > Is it better to define the system topic name in uppercas

Re: [DISCUSS] PIP-242 Topic name restrictions

[Enrico Olivelli]

Mattison,

Il giorno gio 16 feb 2023 alle ore 00:27  ha scritto:
>
> > I am sorry but I am not sure that this is enough to preventreads/writes 
> > from unallowed clients.
> IMO, We can consider the authorisation part in another PIP because We are 
> just focusing on adding the topic name constraint of topic creation.
>
> Maybe we can use another PIP to clearify all of system topic's behaviour, 
> like authorisation something.
> e.g. we just allow superusers to read/write the data to that system topic.
> > We should elaborate more on this topic on the PIP
> I will add the internal system topic creation logic in the PIP.
Why do you think that this is enough ?

I think that we are going off the initial scope of the PIP.
The initial problem is about preventing clients from creating topics
that contain the "-partition-" keyword.

I totally agree that there must be a clear way to distinguish topics
that are not meant to be accessed by "regular clients".

The answer is in Micheal's words: only super users are allowed to
access topics that are not meant to be accessed by clients.
Broker to Broker communications are always running with a "super user"
role, so it is not a problem.

BTW I wonder if it is better to narrow down the scope of the PIP and
go back to "-partition-"


Enrico


>
> Best,
> Mattison
> On Feb 16, 2023, 00:41 +0800, Enrico Olivelli , wrote:
> > Il giorno mer 15 feb 2023 alle ore 17:07  ha 
> > scritto:
> > >
> > > Hi Enrico
> > >
> > > I think it's a good question. We can introduce a new method in the 
> > > BrokerService to help brokers create the topic internally first(maybe 
> > > just metadata is enough), and then to use a pulsar client to connect to 
> > > it.
> >
> > I am sorry but I am not sure that this is enough to prevent
> > reads/writes from unallowed clients.
> > We should elaborate more on this topic on the PIP
> >
> > Enrico
> >
> > >
> > > WDYT?
> > >
> > >
> > > Best,
> > > Mattison
> > > On Feb 16, 2023, 00:01 +0800, Enrico Olivelli , 
> > > wrote:
> > > > > I have one question (apologies for the top posting).
> > > > >
> > > > > The Broker (and the other Pulsar components) use the regular Pulsar
> > > > > client to connect to "system topics"
> > > > > and in general they use the Pulsar wire protocol.
> > > > >
> > > > > The question is "how do you distinguish an internal component from a
> > > > > user component ?"
> > > > > How can you say that the broker is able to connect to a system topic
> > > > > and any other client cannot do it ?
> > > > >
> > > > > Enrico
> > > > >
> > > > > Il giorno mer 15 feb 2023 alle ore 15:38  ha 
> > > > > scritto:
> > > > > > >
> > > > > > > Hi Asaf
> > > > > > >
> > > > > > > There is a link to introduce the dynamic configuration.
> > > > > > > https://pulsar.apache.org/docs/2.10.x/admin-api-brokers/#dynamic-broker-configuration
> > > > > > >
> > > > > > > Best,
> > > > > > > Mattison
> > > > > > > On Feb 14, 2023, 17:06 +0800, Asaf Mesika 
> > > > > > > , wrote:
> > > > > > > > > > > On Tue, Feb 14, 2023 at 3:46 AM  
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > > > > Hi, Asaf
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Welcome to join this discussion.
> > > > > > > > > > > > > > > > > > > > > > > You mean that allows the *system* 
> > > > > > > > > > > > > > > > > > > > > > > to use it when it's a partitioned
> > > > > > > > > > > > > > > topic?
> > > > > > > > > > > > > > > Sorry, I didn't get your point. What do you mean 
> > > > > > > > > > > > > > > by *system*?
> > > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > This sentence was a reply to:
> > > > > > > > > > >
> > > > > > > > > > > 2. Make the `-partition-` string the keyword. That allows 
> > > > > > > > > > > the user to use
> > > > > > > > > > > > > > > it when it's a partitioned topic.
> > > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I wanted to say that this sentence should be:
> > > > > > > > > > > Make the `-partition-` string the keyword, that allows 
> > > > > > > > > > > the *system* to use
> > > > > > > > > > > it when it's a partitioned topic.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Why postfix of `__`?Why uppercase 
> > > > > > > > > > > > > > > > > > > > > > > ?Maybe `__system__`?
> > > > > > > > > > > > > > > Yes, That is a key point that I want to discuss 
> > > > > > > > > > > > > > > in this
> > > > > > > > > > > > > > > thread. `__system__` is good for me.
> > > > > > > > > > > > > > > > > > > > > > > Can you please elaborate what it 
> > > > > > > > > > > > > > > > > > > > > > > means to make it dynamic exactly?
> > > > > > > > > > > > > > > Sorry, I will refine it. it means we can update 
> > > > > > > > > > > > > > > this configuration
> > > > > > > > > > > > > > > dynamically. (using rest api or sth)
> > > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I'm unfamiliar with how P