Re: [DISCUSS] PIP-247: Notifications for partitions update

Thanks for the context Xiaoyu Hou and Asaf. I appreciate the efficiencies that we can gain by creating a specific implementation for the partitioned topic use case. I agree that this new notification system makes sense based on Pulsar's current features, and I have some implementation questions. >

Re: Please stop cherry-picking (breaking) changes to the released branches

If we are concerned about correctly ensuring only bug fixes go to release branches, I think we ought to discuss using a merge git strategy instead of a cherry pick git strategy. This change was discussed briefly on this mailing list at the end of 2021 [0]. I will make a brief argument for making th

Re: [VOTE] Pulsar Release 2.10.4 Candidate 1

Please ignore the previous email. This commit did not break CI. Instead, a very coincidental thing happened. 1. There may be problems with the maven server at that time. The three PRs mentioned at that time could not download the correct jar package, and the retry was invalid. 2. A flaky test `reco

Re: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema compatibility checks without using avro-protobuf

Great to see your comment, bo! 1. The first way. The protobuf website has a description of the rules, but no plans to implement them. https://protobuf.dev/programming-guides/proto/#updating 2. I think this PIP can be divided into two parts. (1) Add a flag(`ValidatorClassName`), load it into `Prot

Re: [DISCUSS] PIP-253: Expose producer metrics for deadLetterProducer and retryLetterProducer

Optional would work well, I don't believe that will make implementation easier. The ConsumerStats class will need a way of retrieving ProducerStats from the consumer at some point in the future, which isn't possible right now, since deadLetterProducer and retryLetterProducer are both private to Con

Re: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema compatibility checks without using avro-protobuf

Thanks for the advice, Asaf. 1. > For now, there is no official or third-party solution for ProtoBuf > compatibility. If in the future have better solutions of a third party or > the official, we develop new ProtobufNativeSchemaValidator and use, so add > a flag. Flag defaults not set, where the

Re: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema compatibility checks without using avro-protobuf

I think we have two ways to do that. First way: We need to advance the improvement of java in protobuf. Ask if they have plans to improve. Second way: the new PROTOBUF_NATIVE `SchemaCompatibilityCheck` should be implemented as a plugin, don't change any existing plugin logic (it's simple and alre

Re: [Discussion] Allowing configure if function consumer should skip to latest

Hi Penghui, Thanks for your question. One case is failure recovery for a windowing function. A windowing function will ack message until its window is emitted. If the window function fails due to issues such as OOM and restarts, it has a massive backlog to catch up. And the function will never

Re: [Vote] PIP-245: Make subscriptions of non-persistent topic non-durable

+1 (binding) Thanks, Bo guo jiwei 于2023年3月6日周一 19:10写道： > > +1 (binding) > > Regards > Jiwei Guo (Tboy) > > On Mon, Mar 6, 2023 at 9:59 AM Yunze Xu wrote: > > > > +1 (binding) > > > > Thanks, > > Yunze > > > > On Fri, Mar 3, 2023 at 11:46 AM PengHui Li wrote: > > > > > > +1 (binding) > > > > >

Re: [DISCUSS] new idea: reverse reading a topic

Hi Kannar, Just interested in what exactly your case. Why do you need to read messages in a reversed order? What is your case? Best, Yong On Mon, 6 Mar 2023 at 23:37, Alexandre DUVAL wrote: > Hi, > > I'm wondering if it is possible to introduce a new feature on Pulsar > which will enable user

Re: [DISCUSS] PMC/Committer Emiratus status

Tison, The suggestion was stated a bit differently: Quote: Rather, I would recommend a project-level “active/inactive” flag that PMC members can voluntarily apply to themselves. For example, do a PMC roll call on private@pulsar.a.o and ask whether current PMC members

Re: [DISCUSS] PIP-253: Expose producer metrics for deadLetterProducer and retryLetterProducer

I support exposing these stats, and I don't have a preference regarding where to put the method. > I believe the implementation will be harder that way Would it be sufficient to make the field return Optional or to return `null`? I don't think we have defined a convention stating we prefer to wra

Re: [DISCUSS] PIP-253: Expose producer metrics for deadLetterProducer and retryLetterProducer

I agree, adding it to the ConsumerStats interface makes more logical sense, but I believe the implementation will be harder that way, since the producers are lazily initialized. They won't be available when ConsumerStats is created, and there isn't currently a way to access them directly from the c

Re: [DISCUSS] PMC/Committer Emiratus status

I won't object if any PMC member or committer is willing to set their personal status; e.g., open to topics of a specific domain, "inactive" or "don't disturb" Best, tison. Asaf Mesika 于2023年3月7日周二 00:05写道： > Do other think it's a good thing to adopt P. Taylor Goetz idea of active > flag and t

Re: [DISCUSS] PMC/Committer Emiratus status

Do other think it's a good thing to adopt P. Taylor Goetz idea of active flag and the process suggested? On Mon, Mar 6, 2023 at 2:03 PM tison wrote: > Hi Yu, > > You can start by adding a page on the contribution guide or a table in the > README of the main repo/site repo to state that you're a

Re: [Discuss] PIP-248: Add backlog eviction metric

> > Pulsar has a feature called backlog quota (place link). You need to place a link :) Expose pulsar_storage_backlog_quota_count in the topic leve You already have "pulsar_storage_backlog_size", so why do you need this metric for? backlogQuotaLimitSize should be `backlogQuotaSizeBytes` backl

[DISCUSS] new idea: reverse reading a topic

Hi, I'm wondering if it is possible to introduce a new feature on Pulsar which will enable users to read topic from a defined MessageId to previous messages until the begin of the topic. I tried to use Pulsar SQL but it requires so much RAM even for little queries (due to Presto design). C

Re: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema compatibility checks without using avro-protobuf

Can you convert the code block which is actually a quote in the beginning of the PIP to something which doesn't require to scroll horizontally so much? Use https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syn

Re: [DISCUSS] PMC/Committer Emiratus status

Hi Yu, You can start by adding a page on the contribution guide or a table in the README of the main repo/site repo to state that you're an expert in the document domain. I have an in-house landscape about Pulsar modules, the broader ecosystem, and their active contributors/maintainers. Since peo

Re: [DISCUSS] Using bouncycastle fips instead bouncycastle non-fips

So it means the change is only on the client side, not the broker side? On Fri, Mar 3, 2023 at 11:42 AM Zixuan Liu wrote: > Hi all, > > We only use the BC to encrypt the message, not TLS, so I think we can > migrate to the BC-FIPS. > > If you think it's feasible, I'll try to do it, and if it do

Re: [DISCUSS] PMC/Committer Emiratus status

Hi Asaf, Thanks for bringing this up! If I may put my two pennies' worth: To be honest, this idea flashed across my mind previously. I talked about this to my colleague, and he was surprised that I was willing to be deprived of benefits (at that time, I was a PMC member already). PMC members ar

Re: [Discuss] PIP-248: Add backlog eviction metric

Hi Aasf, I've updated the PIP, PTAL Thanks, Tao Jiuming Asaf Mesika 于2023年3月5日周日 21:00写道： > On Thu, Mar 2, 2023 at 12:57 PM 太上玄元道君 wrote: > > > > I think you should fix this explanation: > > > > Thanks! I would like to copy the context you provide to the PIP > motivation, > > your description

Re: [Vote] PIP-245: Make subscriptions of non-persistent topic non-durable

+1 (binding) Regards Jiwei Guo (Tboy) On Mon, Mar 6, 2023 at 9:59 AM Yunze Xu wrote: > > +1 (binding) > > Thanks, > Yunze > > On Fri, Mar 3, 2023 at 11:46 AM PengHui Li wrote: > > > > +1 (binding) > > > > Penghui > > > > > On Feb 13, 2023, at 14:56, Jiuming Tao > > > wrote: > > > > > > Hi all

[ANNOUNCE] Apache Pulsar Node.js client 1.8.1 released

The Apache Pulsar team is proud to announce Apache Pulsar Node.js client version 1.8.1. Pulsar is a highly scalable, low latency messaging platform running on commodity hardware. It provides simple pub-sub semantics over topics, guaranteed at-least-once delivery of messages, automatic cursor manag