Re: Many test classes are using deprecated topic naming format V1 which includes the cluster name

>> but it would be useful to have more control so that V1 topic usage could be blocked completely for 4.0.x LTS when configured to do that. No, we can’t do that, as there are still legacy systems relying on V1 topics. Declaring end-of-support for V1 would block those systems from upgrading, effec

Re: [VOTE] PIP-412: Support setting messagePayloadProcessor in Pulsar Functions and Sinks

+1 (binding) Thanks, Rajan On Fri, Apr 4, 2025 at 5:14 PM PengHui Li wrote: > +1 (binding) > > - Penghui > > On Tue, Apr 1, 2025 at 2:10 AM Pengcheng Jiang > wrote: > > > Hi, All. > > > > I would like to start the voting thread for PIP-412: Support setting > > messagePayloadProcessor in Pulsa

[VOTE] PIP-395: Add Proxy configuration to support configurable response headers for http reverse-proxy

Hi, I would like to start voting thread for PIP-395 to add support for configurable response headers in http reverse-proxy PIP design PR: https://github.com/apache/pulsar/pull/23648 Thread: https://lists.apache.org/thread/s7zxrrcl59sbr06o52wj9oqmd6b9lhng Thanks, Rajan

[DISCUSS] PIP-395: Add Proxy configuration to support configurable response headers for http reverse-proxy

Hi, I have created a PIP to support configurable response headers for http reverse-proxy. PIP: https://github.com/apache/pulsar/pull/23648 Thanks, Rajan

Re: [VOTE] PIP-385 Add rate limit semantics to pulsar protocol and Java Client

+1 Thanks, Rajan On Sun, Nov 10, 2024 at 11:02 PM Girish Sharma wrote: > Hi Pulsar dev community, > > I would like to start the voting thread for PIP-385 Add rate limit > semantics to pulsar protocol and Java Client. > > Proposal PR: https://github.com/apache/pulsar/pull/23398 > Rendered PIP -

Re: [DISCUSS] PIP-385 Add rate limit semantics to pulsar protocol and Java client

> > > > > > > > >>> Well, even if we have throttle producer protocol, if client app > is > > keep > > > > >>> producing messages then client app will see high timeout and to > > fast > > > > fail > > > > &g

Re: [DISCUSS] Revert the PR 23395, which broke the behavior of schemas

eated a PR to resolve this disagreement with your suggestion: PR: https://github.com/apache/pulsar/pull/23428 Thanks, Rajan On Wed, Oct 9, 2024 at 4:02 AM Lari Hotari wrote: > Makes sense, just wondering if the behavior should only be enabled when > config.isSchemaLedgerForceRecovery()

Re: [DISCUSS] Revert the PR 23395, which broke the behavior of schemas

>> When the schemas of a topic are lost, all of the messages in the topic can not be consumed successfully, and producers can not publish messages anymore. Well, there are already many issues created related to the lost schema ledgers so, it is a very well known issue that the topic becomes unavai

Re: [DISCUSS] Core Implementation for PIP-379: Key_Shared Draining Hashes for Improved Message Ordering - Reviews Needed

Hi Lari, Thank you for your work but I have a simple question about why we should rush for specific PR and why it's necessary to keep a particular PR part of ongoing release. I know multiple PRs that tried to address critical bugs on this similar issue and they were again blocked without solid rea

Re: [DISCUSS] PIP-385 Add rate limit semantics to pulsar protocol and Java client

Hi Girish, I have gone through the proposal and you mentioned few problems as a motivation of this improvements >> Noisy neighbors - Even if one topic is exceeding the quota, since the entire channel read is paused, all topics sharing the same connect (for example - using the same java client obj

Re: [DISCUSS] PIP-381: Handle large PositionInfo state

Rajan On Tue, Oct 1, 2024 at 11:42 PM Lari Hotari wrote: > On 2024/09/27 19:18:03 Rajan Dhabalia wrote: > > Well, again PR#9292 already has an agreement to merge earlier as well and > > it was reviewed as well but it just was blocked for no reason. and > recently > > it wa

Re: [VOTE] PIP-384: ManagedLedger interface decoupling

+1 (binding) Thanks, Rajan On Tue, Oct 1, 2024 at 9:36 AM Lari Hotari wrote: > Hi, > > I'd like to start the voting thread for PIP-384: ManagedLedger interface > decoupling > > Proposal PR: https://github.com/apache/pulsar/pull/23363 > Discussion thread: > https://lists.apache.org/thread/rtnktr

Re: [DISCUSS] Revisit PIP voting max time

otherwise you must not cast your vote. > > I apologize if I have left some PRs in "Request changes" state for long > time, feel free to "Dismiss" my review if it is outdated > > my 2 cents > Enrico > > > > > > > Best, > > Dave > >

Re: [VOTE] PIP-381: Handle large PositionInfo state

Hi, I think I already voted in an incorrect email thread. +1 (binding). Thanks, Rajan On Mon, Sep 30, 2024 at 10:46 AM Andrey Yegorov wrote: > I am closing the voting process as *approved* with 2 binding votes and 1 > non-binding vote. > Thank you! > > On Wed, Sep 25, 2024 at 1:19 PM Apurva Tel

Re: [DISCUSS] PIP-384: ManagedLedger interface decoupling

+1 this will be also helpful to enhance various dispatch definitions at the broker. Thanks, Rajan On Fri, Sep 27, 2024 at 11:31 AM Lari Hotari wrote: > Dear Pulsar Community, > > I have created PIP-384: ManagedLedger interface decoupling, which > documents recent improvements to Pulsar's Manage

Re: [DISCUSS] PIP-381: Handle large PositionInfo state

date is in another thread [1]. > > -Lari > > 1 - https://lists.apache.org/thread/qy8xp2ht0htvctlx2cwgrq2ppnjcp4m3 > > On 2024/09/27 06:03:00 Rajan Dhabalia wrote: > > Hi Enrico, > > > > I have rebased https://github.com/apache/pulsar/pull/9292 PR again after > > you helped

Re: [DISCUSS] PIP-381: Handle large PositionInfo state

broker and PR#9292 was built by extending the same vision of below PRs but somehow it got delayed. https://github.com/apache/pulsar/pull/3818 https://github.com/apache/pulsar/pull/3819 Thanks, Rajan On Mon, Sep 23, 2024 at 6:59 PM Rajan Dhabalia wrote: > >> I am sorry I haven't follo

Re: [DISCUSS] PIP-381: Handle large PositionInfo state

lag that allows users to enable this feature > > > (keeping it disabled by default until there is higher demand). > > > Managed ledger and cursor complexities are well-known, so a smooth > opt-in > > > process is crucial for users to adopt new features gradually. > >

Re: [DISCUSS] PIP-381: Handle large PositionInfo state

Thanks, Rajan On Sat, Sep 21, 2024 at 4:18 AM Enrico Olivelli wrote: > Il Sab 21 Set 2024, 01:51 Rajan Dhabalia ha > scritto: > > > Hi Andrey, > > > > Thanks for submitting the PR as we have been facing this issue for a long > > time now and we also have PR whic

[DISCUSS] Revisit PIP voting max time

. Thanks, Rajan On Thu, Sep 5, 2024 at 8:08 AM Lari Hotari wrote: > +1 (binding) > > -Lari > > On 2024/09/04 04:38:01 Rajan Dhabalia wrote: > > Hi, > > > > I would like to start a voting thread for PIP-326 to support the admin > API > > to read sche

Re: [DISCUSS] PIP-381: Handle large PositionInfo state

Hi Andrey, Thanks for submitting the PR as we have been facing this issue for a long time now and we also have PR which solves this issue in a simple and a fundamental way with proven perf results as well. PR: https://github.com/apache/pulsar/pull/9292 But again I am not sure some folks blocked

[DISCUSS] Reduce PulsarClientException stacktrace to avoid heavy logging impact on broker and client

Hi, Pulsar broker can face high CPU and GC Impact due to change made in PIP-91 to log huge exception stacktrace for PulsarClientException. Detailed issue is described at : https://github.com/apache/pulsar/issues/23250. PulsarClientException tries to create and keep list of past Exception objects i

[VOTE] PIP-360 Add admin API to display Schema metadata

Hi, I would like to start a voting thread for PIP-326 to support the admin API to read schema metadata and display in readable format. PIP design PR: https://github.com/apache/pulsar/pull/22913 Thread: https://lists.apache.org/thread/8s8m6k7oprmkn3jpblgxqkdh6d8z43x2 Thanks, Rajan

[DISCUSS] PIP-360 Add admin API to display Schema metadata

Hi, I have created a PIP to support admin API to read schema metadata and display in readable format. PIP: https://github.com/apache/pulsar/pull/22913/files Thanks, Rajan

Re: [VOTE] PIP-327 Support force topic loading for unrecoverable errors

Let me try again to ping on this thread as it's already been more than 4 months for discussion and 2 weeks for the VOTE thread for such a small and necessary change. Thanks Rajan On Thu, Apr 25, 2024 at 12:12 AM 太上玄元道君 wrote: > +1 nonbinding > > > Rajan Dhabalia 于202

Re: [VOTE] PIP-327 Support force topic loading for unrecoverable errors

Hi, Can we please close this VOTE thread as it's been more than 8 days to this thread and 4 months since we created a PIP. Please let me know if we have any concern and can we close the VOTE so we can move forward with this PIP. Thanks, Rajan On Tue, Apr 16, 2024 at 7:09 PM Rajan Dha

Re: [VOTE] PIP-327 Support force topic loading for unrecoverable errors

f7zg7cdkns1om0q > > > Thanks, > sinan > > > Apurva Telang 于2024年4月17日 周三02:43写道： > > > +1 (non-binding) > > > > On Tue, Apr 16, 2024 at 10:11 AM Enrico Olivelli > > wrote: > > > > > +1 (binding) > > > > > >

Re: [DISCUSS] Solution after missing schema

e schema has been destroyed. > > > There is also content in PIP-327 that decides to skip processing when the > schema is lost. > > > Thanks, > sinan > > > Rajan Dhabalia 于2024年4月17日 周三02:34写道： > > > Hi, > > > > Broken schema ledgers and topic unav

Re: [DISCUSS] Solution after missing schema

Hi, Broken schema ledgers and topic unavailability are the issues we have been trying to resolve for multiple years now and We have been trying to add lot of patches to fix but we still find different usecases which can make topic unavailable when schema ledger is not recoverable (ledger is delete

[VOTE] PIP-327 Support force topic loading for unrecoverable errors

Hi, I would like to start voting thread for PIP-327 which also addresses issue # https://github.com/apache/pulsar/issues/21751 PIP design PR: https://github.com/apache/pulsar/pull/21752 Thread: https://lists.apache.org/thread/w7w91xztdyy07otw0dh71nl

Re: [DISCUSS] PIP-335: Oxia metadata support

+1 Thanks, Rajan On Wed, Jan 31, 2024 at 3:58 PM Matteo Merli wrote: > https://github.com/apache/pulsar/pull/22009 > > === > > # PIP-335: Supporty Oxia metadata store plugin > > # Motivation > > Oxia is a scalable metadata store and coordination system that can be used > as the core inf

Re: [E] Re: [DISCUSS] PIP-188(BlueGreenClusterMigration) Remove redirectedClusterURI in Pulsar Client

t;>> requirement. I was under the impression that the migration happens for all >>> topics from the old to the new cluster. >>> >>> For this requirement, I think we should keep redirectedClusterURI in >>> HandlerState. >>> >>> Thanks for this qu

Re: [DISCUSS] PIP-188(BlueGreenClusterMigration) Remove redirectedClusterURI in Pulsar Client

This might not work because same PulsarClient can be used to create multiple topics and producer/consumer and migration state can be different per topic and even for produce/consumer (of same topic) so, changing service url will not work for blue-green migration redirection for all topics served

Re: [DISCUSS] PIP-327 Support force topic loading for unrecoverable errors

n't see the topic load issues. The topic loading works fine, and the > > producer works fine. > > But the proposal said it would resolve the topic load issue, can you > > reproduce the topic load issue? > > > > Regards, > > Penghui > > > > > &

[DISCUSS] PIP-327 Support force topic loading for unrecoverable errors

Hi, We have an issue to fail loading topics in unrecoverable situation and impacting topic availability:: https://github.com/apache/pulsar/issues/21751 This PIP addresses the issue and allows brokers to handle such situations and maintain the topic availability: PIP: https://github.com/apache/pul

[VOTE] PIP-313 Support force unsubscribe using consumer api

Hi, I would like to start voting thread for PIP-313 which also addresses issue # https://github.com/apache/pulsar/issues/21451 PIP design PR: https://github.com/apache/pulsar/pull/21452 Thread: https://lists.apache.org/thread/hptx8z9mktn94gvqtt4547wzcfcgdsrv Thanks, Rajan

Re: Re: [Discuss] PIP-319 Unblock stuck Key_Shared subscription after consumer reconnect

t keys > processing. > > Do you have plans for Key_Shared dispatcher rework? I would be happy to > contribute to such a project if I can help somehow. > > > Best Regards, > > Nikolai > > > > On 2023/12/01 06:34:59 Rajan Dhabalia wrote: > > Hi Nikolai, >

Re: [Discuss] PIP-319 Unblock stuck Key_Shared subscription after consumer reconnect

Hi Nikolai, Thanks for bringing this to the attention. I have seen multiple PIP and bug fixes for the Key-Shared subscription and I feel Key_Shared subscription dispatcher is developed with multiple fundamental issues which required to introduce scheduled-task to unblock stuck reads, introducing s

Re: [DISCUSS] PIP-321 Split the responsibilities of namespace replication-clusters

Hi, I don't agree with your assumptions about allowed clusters and replication-cluster definition. Allowed clusters defined at tenant level will restrict tenants to create namespaces in regions/clusters where they are not allowed. and replication clusters will help tenant to set up full mesh repli

Re: [DISCUSS] PIP-310: Support custom publish rate limiters

Hi Lari/Girish, I am sorry for jumping late in the discussion but I would like to acknowledge the requirement of pluggable publish rate-limiter and I had also asked it during implementation of publish rate limiter as well. There are trade-offs between different rate-limiter implementations based o

Re: [Discuss] Disable `Rebase and Merge` on Pulsar Repo

I remember, It was always disabled since we open sourced Pulsar. Who enabled it and what was the reason for enabling that option? It's definitely not a good option and we should disable it asap. Thanks, Rajan On Sun, Nov 5, 2023 at 7:36 PM Heesung Sohn wrote: > Hi, > > I recently merged my PR w

Re: [DISCUSS] PIP-313 Support force unsubscribe using consumer api

connection. > > Regards, > Penghui > > On Fri, Oct 27, 2023 at 8:25 AM Rajan Dhabalia > wrote: > > > Hi, > > > > We have an issue created regarding unsubscription of shared subscription > in > > Pulsar spout at : https://github.com/apache/pulsar/

[DISCUSS] PIP-313 Support force unsubscribe using consumer api

Hi, We have an issue created regarding unsubscription of shared subscription in Pulsar spout at : https://github.com/apache/pulsar/issues/21451 This PIP addresses the issue and allows users to unsubscribe subscription with additional force option. PIP: https://github.com/apache/pulsar/pull/21452/

Re: [VOTE] PIP-271: Add broker health check status into prometheus metrics #20147

Thanks for adding it. +1 Thanks, Rajan On Thu, Oct 26, 2023 at 3:52 PM vineeth p wrote: > Broker metrics don't have anything to indicate the health of the broker (to > indicate if the broker is active). In Prometheus broker metrics which are > used for monitoring, it will be useful if metrics a

Re: [DISCUSS] PIP-296: Support storing broker internal client certificates in metadata store

ination cluster, are there any other recommendations you might have in > mind? > > Best, > Mattison > On 8 Sep 2023 at 16:47 +0800, Rajan Dhabalia , > wrote: > > Hi, > > > > Pulsar stores different types of metadata into a metadata store which > > conta

Re: [DISCUSS] PIP-296: Support storing broker internal client certificates in metadata store

Hi, Pulsar stores different types of metadata into a metadata store which contains tenant, namespaces and topic metadata. Metadata-store should store metadata definition and should avoid combining other non metadata related information especially certificates or keys like encryption/decryption key

Re: [VOTE] PIP-268: Add support of topic stats/stats-internal using

concerns and questions as it's been going for a long time and I would like to see it moving forward and would like to see any suggestions or feedback for API or any changes in the proposal. Thanks, Rajan On Wed, Aug 2, 2023 at 7:06 PM Rajan Dhabalia wrote: > >> If so, I think it could be

Re: [VOTE] PIP-268: Add support of topic stats/stats-internal using

f the users want to > remove it. > > I want to say that from the perspective of improving performance, > it may not be more convincing than the above reason. > > Thanks, > Penghui > > > On Mon, Jun 26, 2023 at 3:37 PM Rajan Dhabalia > wrote: > > > > I d

Re: [VOTE] PIP-268: Add support of topic stats/stats-internal using

n stats/internal-stats pojo-schema, and > accessing stats with consistent pojo definition. > > This is from the perspective of developers who are already very familiar > with Pulsar. > If you don't know it, just want to create a client based on Pulsar's API. > How to

Re: [VOTE] PIP-268: Add support of topic stats/stats-internal using

t; their own pojo based on > the REST API. > Well JSON would be the better option to avoid data conversion and copy for complex stats/internal-stats complex data-structures at broker and client side, maintaining consistency between stats/internal-stats pojo-schema, and accessing stats with consi

Re: [DISCUSS] PIP-267: Support multi-topic messageId deserialization to ack messages

rote: > On Tue, Jun 20, 2023 at 9:39 AM Rajan Dhabalia > wrote: > > > > So you say in that sentence that you will add the topic name into > > MessageIdData. MessageIdData is defined in PulsarApi.proto and is > > transferred over the wire, so how can you add the

Re: New pip process reminder

Hi, > (" a lazy majority of at least 3 binding +1s votes") I don't think it's fair at this stage where majority Pulsar committers are mostly part of one enterprise and only their PIP/PRs are moving forward and PR/PIP created by other community members get blocked or not reviewed without any major

[VOTE] PIP-268: Add support of topic stats/stats-internal using

Hi, I would like to start VOTE for : https://github.com/apache/pulsar/issues/20265 Thanks, Rajan

Re: [DISCUSS] PIP-268: Add support of topic stats/stats-internal using client api

ms. Thanks, Rajan On Mon, May 15, 2023 at 3:57 PM Rajan Dhabalia wrote: > Hi Penghui, > > > >> And for the Admin API, we have several supported params > Good point regarding the stats/internal-stats params. I think we can pass > option-map into below GET API for stats s

[VOTE] PIP-267: Support multi-topic messageId deserialization to ack messages

Hi. Pulsar api provides MessageId interface which is generally used by producer and consumer applications to manage topic offset. Sometimes, these applications would like to serialize and deserialize messageIds, specifically consumer app which would like to persist messageId and ack with those me

Re: [DISCUSS] PIP-267: Support multi-topic messageId deserialization to ack messages

oto and is transferred over the > wire, so how can you add the topic to this class without changing the wire > protocol? > > > > > > On Fri, Jun 16, 2023 at 10:47 PM Rajan Dhabalia > wrote: > > > Yes, the topic name will not be transferred and it's not part of

Re: [DISCUSS] PIP-267: Support multi-topic messageId deserialization to ack messages

me msgId is the data structure > the > > client transmit to the server, so that means you are transmitting topic > to > > the server? > > > > > > On Fri, May 12, 2023 at 7:45 AM Rajan Dhabalia > > wrote: > > > > > Thank you for sharing your

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

Hi, Are we proposing a change to break existing metrics compatibility (prometheus)? If that is the case then it's a big red flag as it will be a pain for any company to upgrade Pulsar as monitoring is THE most important part of the system and we don't even want to break compatibility for any small

Re: [DISCUSS] PIP-273: Enable hostname verification by default

Hi, I am not sure enabling hostname verification by default is a good idea because it will cause deployment failures for many users. Hostname verification requires x509 cert changes and wildcard SNName in some cases and it's not easy when broker hostname patterns don't match in x509 certs and there

Re: [DISCUSS] PIP-275: Introduce numWorkerThreadsForPersistentTopic to deprecate numWorkerThreadsForNonPersistentTopic in configuration

Hi, We already have a default number of threads for persistent topics but we have added a feature non-persistent topics and to isolate that path we introduced a number of worker threads which we can reduce or tune based on resources we would like to allocate for non-persistent topics. So, it reall

Re: [DISCUSS] PIP-271: Add broker health check status into prometheus metrics #20147

Thank you Vineeth for creating the PIP. This PIP will be useful to capture a broker's health as a part of metrics or dashboards and we can also set up various alerts on it. Generally we should check the status API of the broker for the HC/liveness probe in k8 instead of sanity because that can brin

Re: [DISCUSS] PIP-268: Add support of topic stats/stats-internal using client api

in Admin API should be > considered > to add to binary protocol? I mean, we'd better have a strategy to avoid > moving many > APIs to binary protocol. For example: > > - Tenant/Namespace level REST APIs, no > - Topic policies APIs, no > - Topic's internal info, maybe

Re: [DISCUSS] PIP-268: Add support of topic stats/stats-internal using client api

er with protocol > changes and client bindings. > > > Enrico > > Il giorno lun 15 mag 2023 alle ore 02:35 Rajan Dhabalia > ha scritto: > > > > There are multiple factors tcp would be better than http when you are > > considering performance and scale du

Re: [DISCUSS] PIP-268: Add support of topic stats/stats-internal using client api

binary TCP, if we end up writing > a JSON string, the only difference I see is the blocking I/O of writing the > response. > > WDYT? > > > On Fri, May 12, 2023 at 7:29 AM Rajan Dhabalia > wrote: > > > Communicating over binary protocol is more scalable

Re: [DISCUSS] PIP-267: Support multi-topic messageId deserialization to ack messages

about whether to add a new separate API. I'm concerned > > about whether existing applications would be affected, no matter if > > the existing implementation has the limitation. If yes, we should > > document it in the PIP so that users can know that. > > > > >

Re: [DISCUSS] PIP-268: Add support of topic stats/stats-internal using client api

rmance, and use accessibility in pulsar. Thanks, Rajan On Thu, May 11, 2023 at 6:24 AM Asaf Mesika wrote: > Before I dive into the PIP, I have several questions on the background > provided below: > > > On Tue, May 9, 2023 at 9:08 AM Rajan Dhabalia > wrote: > > > Hi, > >

Re: [DISCUSS] PIP-267: Support multi-topic messageId deserialization to ack messages

t; > P.S. it's better to add the API changes and potential breaking changes > in the proposal. > > Thanks, > Yunze > > On Tue, May 9, 2023 at 1:59 PM Rajan Dhabalia > wrote: > > > > Hi, > > > > Pulsar api provides MessageId interface which i

[DISCUSS] PIP-268: Add support of topic stats/stats-internal using client api

Hi, Right now, Pulsar provides the topic's stats and stats-internal over HTTP admin API, and this stats data is used by user applications and also by Pulsar internal components such as Pulsar-functions to derive the certain states of the applications. for example, there are use cases where the app

[DISCUSS] PIP-267: Support multi-topic messageId deserialization to ack messages

Hi, Pulsar api provides MessageId interface which is generally used by producer and consumer applications to manage topic offset. Sometimes, these applications would like to serialize and deserialize messageIds, specifically consumer app which would like to persist messageId and ack with those mes

Re: [DISCUSS] Is PIP required for small changes in metrics

Hi, Thank you Mattison for starting this thread because I am feeling some of the community members are making new contributors' life difficult and trying to enforce rules which were never discussed and discourage them from contributing to Pulsar. I really don't want to see the Pulsar community bec

Re: [Vote] PIP-245: Make subscriptions of non-persistent topic non-durable

> But for NonPersistentTopic, creating a Durable subscription is meaningless, NonPersistentSubscription doesn't have a ManagedCursor to persistent its data. After its consumer disconnected, the subscription couldn't be removed automatically if we didn't set the value of subscriptionExpirationTimeMi

Re: [DISCUSSION] Redesign the MessageId interface

Hi, I was reading the email thread why we want to change MessageId interface: https://lists.apache.org/thread/rdkqnkohbmkjjs61hvoqplhhngr0b0sd >> Currently we have the following 5 implementations of MessageId: >> These implementations are such a mess. For example, when users get a MessageId from `

Re: [DISCUSS] Planning for Apache Pulsar 3.0

we have any criteria on > what kinds of changes would necessitate cutting a new major release but > would still be considered acceptable by the community? > > -- > Devin Bost > Sent from mobile > Cell: 801-400-4602 > > On Sat, Oct 8, 2022, 2:14 PM Rajan Dhabalia wrote: > >

Re: [DISCUSS] Planning for Apache Pulsar 3.0

o. > > > > Rajan: That's the whole point of breaking it down. If you accumulate > > many "big" changes it introduces a lot of risk for instabilities and > > incompatibilities. Breaking it down in multiple steps helps to see the > > incremental changes and introd

Re: [DISCUSS] Planning for Apache Pulsar 3.0

Hi, Can we get the list of changes at one place which we are planning to get as part of 3.0. One thing I would like to see as a part of a major release, it CAN NOT impact existing usecases and users in any way which can force them to upgrade the client library. Applications using < 3.0 version sho

Re: [DISCUSS] Enable message deduplication for replicator by default

Message deduplication always comes with memory and CPU cost and making it default means charging this penalty to every user without having this requirement. Enabling by default means you are impacting every user who is not aware about this feature after upgrading the release. This is purely requir

Re: [DISCUSS] Move PIPs to the codebase?

Hi, >>> I think we can move all the PIPs to the codebase and the new proposal and proposal without any reviews should happen with a PR first. So that we can review and comment easily. I didn't understand this part. You want one to create a PR before submitting a proposal? That's clearly not a goo

Re: [DISCUSS] PIP-192 New Pulsar Broker Load Balancer

Hi, I have gone through PIP but I don't see some basic information as part of PIP: 1. current issue in current load balancer strategy 2. Are there any performance and feature gaps in the current load balancer? Please provide data or metrics to show the impact. 3. what exactly we are solving in new

Re: [DISCUSS] PIP-184: Cluster migration or Blue-Green cluster deployment support in Pulsar

nProgress" or "migrationStarted"? > >> It's just a state to mark that broker has considered it for migration similar to the Terminated state. we don't need extra state for completion because the broker is going to delete the topic once all subscribers reach the end o

Re: [DISCUSS] PIP-184: Cluster migration or Blue-Green cluster deployment support in Pulsar

t; > > On Tue, Jul 12, 2022 at 6:23 PM Rajan Dhabalia > wrote: > > > Hi, > > > > We have created PIP-184 which helps users to perform cluster migration > with > > Apache Pulsar. Cluster migration or Blue-Green cluster deployment is one > of > > the pr

Re: [DISCUSS] PIP-184: Cluster migration or Blue-Green cluster deployment support in Pulsar

manually dealt with) > >> Sure, I have updated PIP in the last section where it lists down the features this PIP supports. I haven't seen any limitation yet with this PIP but we can add if we see any concerns with the PIP. > -joe > > > On Wed, Jul 13, 2022 at 9:31 AM Raj

Re: [DISCUSS] PIP-184: Cluster migration or Blue-Green cluster deployment support in Pulsar

opic can now be deleted? > > > > "Broker sends topic migration message to client so, producer/consumer at > > client side can handle redirection accordingly" > > > > For producers, the message will be sent the moment the status of the > topic > > h

Re: [DISCUSS] PIP-184: Cluster migration or Blue-Green cluster deployment support in Pulsar

direction accordingly" > > For producers, the message will be sent the moment the status of the topic > has changed, so all messages from there on will be written to the new > cluster? > Yes. > For consumers, the message will be sent when there are no more messages to >

[DISCUSS] PIP-184: Cluster migration or Blue-Green cluster deployment support in Pulsar

Hi, We have created PIP-184 which helps users to perform cluster migration with Apache Pulsar. Cluster migration or Blue-Green cluster deployment is one of the proven solutions to migrate live traffic from one cluster to another. One of the examples is applications running on Kubernetes sometimes

Re: [VOTE] PIP-136: Sync Pulsar policies across multiple clouds

e > partitioned metadata > to the remote cluster's metadata store? > > Thanks, > Penghui > > On Wed, Mar 23, 2022 at 3:39 AM Rajan Dhabalia > wrote: > > > >> Do we need to provide the ability for users to decide to replicate the > > ACLs and replication cl

Re: Build Pulsar Server on Java 17- too strict ?

Hi, I do not agree to force client applications to use jdk-17 and that will not be good Pulsar as a project because that will force users to find another alternative of Pulsar for their messaging usecases. In large org where Pulsar is being used as a managed service and used by a large number of a

Re: Discuss about creating partitioned topic in V1

> Having said that, V1 admin API was in effect considered as "soft deprecated" since Pulsar 2.0. All documentation has been focused on v2 APIs. There are Orgs which have large scale Pulsar clusters and use V1 apis. There is no way we can deprecate V1 api and it has to be maintained until we find a

Re: [VOTE] PIP-136: Sync Pulsar policies across multiple clouds

/devops/geo-replication-in-apache-pulsar-part-2-patterns-and-practices.html > [2] > > https://github.com/apache/pulsar/blob/4dcb166e0bfcce7fc85fd8d59a25b881f6f9c6fa/pulsar-common/src/main/java/org/apache/pulsar/common/events/PulsarEvent.java#L36 > [3] > > https://github.com/apach

Re: [VOTE] PIP-136: Sync Pulsar policies across multiple clouds

to the remote > cluster? > > Currently, we already have a PulsarEvent struct to define the pulsar system > events, > Looks like we can use a unified event definition by PulsarEvent. > > Others look good to me. > > Regards, > Penghui > > > > On Sat, Mar 19

[VOTE] PIP-136: Sync Pulsar policies across multiple clouds

Hi, I would like to start VOTE on PIP-136: https://github.com/apache/pulsar/issues/13728 Thanks, Rajan On Tue, Feb 8, 2022 at 4:58 PM Rajan Dhabalia wrote: > > >> How do we designate the host broker? Is it manual? How does it work > when the host broker is removed from the cl

Re: [DISCUSS] PIP-136: Sync Pulsar policies across multiple clouds

will watch the metadata > > > changes and publish the changes to the event topic. > > > > How do we designate the host broker? Is it manual? How does it work > > when the host broker is removed from the cluster? > > > > If we collocate the active con

Re: [DISCUSS] PIP-136: Sync Pulsar policies across multiple clouds

o design it in an extensible fashion. > >> I think answer (2) addresses this concern as well. > Thanks, > Michael > > [0] https://lists.apache.org/thread/pj4n4wzm3do8nkc52l7g7obh0sktzm17 > [1] https://lists.apache.org/thread/h4cbvwjdomktsq2jo66x5qpvhdrqk871 > [2] https://lists.ap

[DISCUSS] PIP-136: Sync Pulsar policies across multiple clouds

Hi, I would like to start a discussion about PIP-136: Sync Pulsar policies across multiple clouds. PIP documentation: https://github.com/apache/pulsar/issues/13728 *Motivation* Apache Pulsar is a cloud-native, distributed messaging framework which natively provides geo-replication. Many organiza

[VOTE] PIP-100: Pulsar pluggable topic factory

I would like to start voting on the PIP. PIP: https://github.com/apache/pulsar/wiki/PIP-100:-Pulsar-pluggable-topic-factory PR: https://github.com/apache/pulsar/pull/12235 Thanks, Rajan On Tue, Sep 28, 2021 at 1:23 PM Rajan Dhabalia wrote: > Hi, > > I would like to propose a Plugga

PIP-100: Pulsar pluggable topic factory

Hi, I would like to propose a Pluggable topic factory in pulsar. Users select the topic type (persistent/non-persistent) based on the application use case and requirement of topic behavior. However, in some circumstances, users need some additional behavior on top of the existing implementation an

Re: [DISCUSS] Return the message ID of the first chunk when sending chunked messages

Hi, Though chunked messages are sequential for a specific producer, it's not guaranteed that they will be contiguous when the broker receives them and writes them to a ledger. So, it will be a little tricky to find out the first message-id of any chunked-message at any given time unless broker tag

Re: [DISCUSS] Contribute "Reactive Streams adapter for the Apache Pulsar Java Client" to the Apache Pulsar project

*My two cents about developing and contributing adapters/connectors for Apache Pulsar. Apache Pulsar was open-sourced with Spark and Storm adapters initially and then such adapter families evolved with multiple other adapters, connectors and IO connectors (sink/source) into apache pulsar git repo.

Re: Proposing a meetup organizing committee

*I think the Apache project should avoid vendor-specific branding, in that way users will have a clear understanding about what they can receive from the free Apache project. In support of that, the Apache project should be driven by the community/PMC who are the real contributors to the project an

Re: Treating lookup request timeout as equivalent to TooManyRequests

> so that timeouts on lookup type requests also closes the connection (and hopefully allows establishment of connection on a working broker). +1. Thanks, Rajan On Thu, Aug 12, 2021 at 11:30 AM Ivan Kelly wrote: > Hi folks, > > Rajan's opinion would be particularly useful here as they appears t

Re: Lack of retries on TooManyRequests

*The history behind introducing TooManyRequest error is to handle backpressure for zookeeper by throttling a large number of concurrent topics loading during broker cold restart. Therefore, pulsar has lookup throttling at both client and server-side that slows down lookup because lookup ultimately

  1   2   >