Re: Enhancing Pulsar CI with Netty leak detection and reporting

I agree with this direction. On Thu, May 8, 2025 at 1:12 AM Lari Hotari wrote: > > Hello all, > > Currently, Pulsar CI lacks the capability to systematically detect and > report Netty buffer leaks. This creates a significant blind spot in > our quality assurance process, as these leaks can slip t

[VOTE] PIP-390 Improve the reusability of Pulsar test code and best practice for unit and integration tests for Pulsar

Hi all, Please help review PIP-390 and cast your vote. Discussion: https://lists.apache.org/thread/fn3rk1x7v9291qh3g6vf4jxhvq6zc4mm PIP: https://github.com/apache/pulsar/pull/23565 Thanks, Heesung

Re: [DISCUSS] Add Common Integ Test Modules

direction, then I guess we can start using Junit-5 for new tests and for common tests in pulsar-inttest-client as well. Thanks, Heesung On Tue, Nov 5, 2024 at 2:35 PM Heesung Sohn wrote: > > Hi all, > > I raised a PIP PR here. > > https://github.com/apache/pulsar/pull/2356

Re: [DISCUSS] Add Common Integ Test Modules

Hi all, I raised a PIP PR here. https://github.com/apache/pulsar/pull/23565 Thanks, Heesung On Sat, Nov 2, 2024 at 6:31 PM Heesung Sohn wrote: > > Hi, > > Sure. Let me open a PIP for Pulsar Test Improvement. > > I think I can summarize this discussion and document it as

Re: [DISCUSS] Add Common Integ Test Modules

Hi, Sure. Let me open a PIP for Pulsar Test Improvement. I think I can summarize this discussion and document it as a PIP for future reference and as the agreed direction. Thanks, Heesung On Sat, Nov 2, 2024 at 2:13 AM Lari Hotari wrote: > > On Sat, 2 Nov 2024 at 02:32, Heesung Sohn

Re: [DISCUSS] Add Common Integ Test Modules

y to have any module > access anything it needs without thinking much about modularity. > > -Lari > > On 2024/11/01 03:05:33 Heesung Sohn wrote: > > Hi, > > > > > if we're not going to mock some behaviors of ZK or BK, it might not be > > > good. >

Re: [DISCUSS] Add Common Integ Test Modules

On Fri, Nov 1, 2024 at 9:55 AM Jie crossover wrote: > > > > LGTM, the current test code is relatively confusing, and this proposal can > > improve this situation. > > -- > > Best Regards！ > > crossoverJie > > > > > > Heesung Sohn 于2024年11月1日周五 0

[DISCUSS] Add Common Integ Test Modules

Hi all, I think we can refactor/split the `tests` module(the integration test module) to reuse the integration test code better. Currently, there are good test classes to reuse(such as topic message test cases, container setup, and other test util funcs), but this module is built under "test" sco

Re: [VOTE] PIP-382: Add a label named reason for topic_load_failed_total

+1 nonbinding Thanks, Heesung On Tue, Oct 29, 2024 at 6:57 PM guo jiwei wrote: > > +1 (binding) > > > Regards > Jiwei Guo (Tboy) > > > On Wed, Oct 16, 2024 at 12:50 PM Tao Jiuming wrote: > > > +1 nonbinding > > > > Yubiao Feng 于2024年10月16日 周三01:50写道： > > > > > Hi all > > > > > > I want to start

Re: [VOTE] PIP-380: Support setting up specific namespaces to skipping the load-shedding

+1(non-binding) Thanks, Heesung On Tue, Oct 29, 2024 at 11:52 PM Zike Yang wrote: > > +1 (binding) > > Thanks, > Zike Yang > > On Wed, Oct 30, 2024 at 2:25 PM 徐昀泽 wrote: > > > > +1 (binding) > > > > Thanks, > > Yunze > > > > > On Oct 30, 2024, at 12:50, Tao Jiuming wrote: > > > > > > +1 nonbin

Re: [DISCUSS] PIP-381: Handle large PositionInfo state

> we might be able to solve storing > 10M-100M unack messages but the question is if a broker really has that many unack messages then will broker run with such huge memory pressure and will it really serve large scale usecases for which Pulsar was built? I am sure, it might be useful for small use

Re: [DISCUSS] PIP-381: Handle large PositionInfo state

>> I'd hope to see these used for storing individual acks in Pulsar, but it >> seems to be too late to handle that when there are already other >> implementations that are sufficient for solving the problem. I think we can make PositionInfo ser/dser code and its interface generic, so that Pulsar

Re: [DISCUSS] PIP-381: Handle large PositionInfo state

-> do not send. On Wed, Sep 25, 2024 at 2:59 PM Heesung Sohn wrote: > > Hi team, > Can we use a space-efficient probabilistic data structure, such as > bloom filter to store the acked msg ids, if re-delivering acked msgs > are not so strict? > > > On Tue, Sep

Re: [DISCUSS] PIP-381: Handle large PositionInfo state

Hi team, Can we use a space-efficient probabilistic data structure, such as bloom filter to store the acked msg ids, if re-delivering acked msgs are not so strict? On Tue, Sep 24, 2024 at 5:28 PM Andrey Yegorov wrote: > > Penghui, > > Thank you for the thorough and detailed response. > > I have

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

Hi all, I realized that sampling outlier topic/subscription/consumer/producer metrics can be useful to operate Pulsar. I noticed that Pulsar's performance issues are mostly caused by outlier topics/subscription/consumers/producers. For example, disk usage can grow a lot when consumers are very s

Re: [DISCUSS] PIP-378 Add ServiceUnitStateTableView abstraction in ExtensibleLoadMangerImpl

/5d25ef8535bde62df7fff53b31198ea67ddf6ab5 The idea is to sync system topic and metadata store during migration operation (this syncer is run by the leader only under the dynamic config). Thanks, Heesung On Thu, Sep 12, 2024 at 4:06 PM Heesung Sohn wrote: > > Hi All, > > I drafted a pro

[VOTE] PIP-378 Add ServiceUnitStateTableView abstraction (ExtensibleLoadMangerImpl only)

Hi all, I would like to start a vote on PIP-378 to add ServiceUnitStateTableView abstraction and make it pluggable. Proposal link: https://github.com/apache/pulsar/pull/23300 Discussion link: https://lists.apache.org/thread/v7sod21r56hkt2cjxl9pp348r4jxo6o8 Thanks, Heesung

[DISCUSS] PIP-378 Add ServiceUnitStateTableView abstraction in ExtensibleLoadMangerImpl

Hi All, I drafted a proposal to add ServiceUnitStateTableView abstraction in ExtensibleLoadMangerImpl. This abstraction could help to event-source bundle ownership via other stores like ZK in addition to Pulsar system topics. https://github.com/apache/pulsar/pull/23300 I'm looking forward to hear

[DISCUSS] PIP-371 Support to specify message listeners in TableView constructor

Hi all, I shared a proposal to specify message listeners in TableView constructor. I'm looking forward to hearing from you. pip: https://github.com/apache/pulsar/pull/23167 Thank you, Heesung

Re: [DISCUSS] PIP-368: Support client properties for the lookup

to a different rack(rack B)? 2. Given that the latest load shedders(TransferShedder and AvgShedder) determine the destination brokers from the leader, how would the leader make sure to unload bundles(topics) to the brokers in the same rack? Thanks, Heesung On Fri, Jul 26, 2024 at 8:29 AM Heesung

Re: [DISCUSS] PIP-368: Support client properties for the lookup

+1 (not binding) Heesung On Thu, Jul 25, 2024 at 7:52 PM Jie crossover wrote: > > +1 > I left some comments in the PR. > -- > Best Regards！ > crossoverJie > > > Zike Yang 于2024年7月25日周四 11:15写道： > > > Hi, all, > > > > I proposed a new proposal to add client properties support for the > > lookup:

Re: [VOTE] PIP-364: Introduce a new load balance algorithm AvgShedder.

+1 (not binding) Heesung On Wed, Jun 26, 2024 at 12:36 AM Lari Hotari wrote: > > +1 (binding) > > -Lari > > On 2024/06/26 03:00:02 thetumbled wrote: > > Hi, Pulsar Community. > > I would like to start the voting thread for PIP-364: Introduce a new load > > balance algorithm AvgShedder. > > P

Re: [ANNOUNCE] Apache Pulsar 3.0.3 released

/pulsar-site/commit/7f5743008dd3a60dd91ee2dab9ca3c743fc3015e) to clarify the upload process, so this file size issue should be fixed in the next LTS release, 3.0.4. Regards, Heesung On Sun, Mar 10, 2024 at 1:06 PM Heesung Sohn wrote: > The Apache Pulsar team is proud to announce Apache Pul

[ANNOUNCE] Apache Pulsar 3.0.3 released

The Apache Pulsar team is proud to announce Apache Pulsar version 3.0.3. Pulsar is a highly scalable, low latency messaging platform running on commodity hardware. It provides simple pub-sub semantics over topics, guaranteed at-least-once delivery of messages, automatic cursor management for subsc

Re: [VOTE] Pulsar Release 3.0.3 Candidate 1

ari Hotari wrote: > > > > > +1 (binding) > > > > > > - Built from source > > > - Checked the signatures of the source and binary release artifacts > > > - Run standalone > > > - Checked producer and consumer > > > - Verified th

[VOTE] Pulsar Release 3.0.3 Candidate 1

ease download the source package, and follow the README to build and run the Pulsar standalone service. Regards, Heesung Sohn (heesung)

[DISCUSS] Apache Pulsar 3.0.3 Release

Dear the Apache Pulsar Dev community, I would like to propose releasing Apache Pulsar 3.0.3. It's over two months since the release of 3.0.2, and there are 95 new commits in branch-3.0: https://github.com/apache/pulsar/compare/v3.0.2...branch-3.0 Please let me know if you have any essential fixe

Re: [Discuss] Release Pulsar C++ Client 3.5.0

Hi, I raised this PR to support blue-green migration for cpp client. PR: https://github.com/apache/pulsar-client-cpp/pull/402 Thanks, Heseung On Wed, Feb 7, 2024 at 2:28 PM Heesung Sohn wrote: > Hi, > > Yes, I am currently working on C++ client update for the blue-green > migra

Re: [Discuss] Release Pulsar C++ Client 3.5.0

Hi, Yes, I am currently working on C++ client update for the blue-green migration support. I think I can raise a PR by early next week if everything goes smoothly. Thanks, Heesung On Wed, Feb 7, 2024 at 12:55 PM Matteo Merli wrote: > Heesung is finalizing the work for adding support for PIP-1

Re: [E] Re: [DISCUSS] PIP-188(BlueGreenClusterMigration) Remove redirectedClusterURI in Pulsar Client

blue-green migration logic. Regards, Heesung On Sun, Jan 28, 2024 at 3:23 PM Heesung Sohn wrote: > I see. Thank you for clarifying this requirement. > > Then, I am wondering if we need to refactor the current > `lookup/urlLookupMap` access code to avoid using `look

Re: [E] Re: [DISCUSS] PIP-188(BlueGreenClusterMigration) Remove redirectedClusterURI in Pulsar Client

sed on its state decided by the server. > > > Sent from my iPhone > > > On Jan 26, 2024, at 9:12 AM, Heesung Sohn wrote: > > > > To add more context, I am trying to add this blue-green migration logic > in > > the cpp and golang client, and before copy

Re: [DISCUSS] PIP-188(BlueGreenClusterMigration) Remove redirectedClusterURI in Pulsar Client

clients. Also, I updated the sample code for better synchronization and dedup. https://github.com/heesung-sn/pulsar/pull/59/ Thanks, Heesung On Thu, Jan 25, 2024 at 10:51 PM Heesung Sohn wrote: > I am sorry, but the current implementation of blue-green migration is for > all topics (produce

Re: [DISCUSS] PIP-188(BlueGreenClusterMigration) Remove redirectedClusterURI in Pulsar Client

lient because all of the topics (of consumers and producers) will eventually migrate to the new cluster? Having said this, having redirectedClusterURI in HandlerState appears to be futuristic. Heesung On Thu, Jan 25, 2024 at 10:36 PM Heesung Sohn wrote: > > This might not work because same P

Re: [DISCUSS] PIP-188(BlueGreenClusterMigration) Remove redirectedClusterURI in Pulsar Client

me topic) so, changing > service url will not work for blue-green migration redirection for all > topics served by that PulsarClient. > > Sent from my iPhone > > > On Jan 25, 2024, at 6:57 PM, Heesung Sohn wrote: > > > > Hi dev, > > > > I would like to d

[DISCUSS] PIP-188(BlueGreenClusterMigration) Remove redirectedClusterURI in Pulsar Client

Hi dev, I would like to discuss this change with the community. Motivation: By this BlueGreenClusterMigration PIP-188( https://github.com/apache/pulsar/pull/17962), `redirectedClusterURI` member var has been introduced in the pulsar client to hold the new cluster's endpoint. Accordingly, pulsar c

[Discussion] (Extensible Load Balancer) Expose loadBalancerInFlightServiceUnitStateWaitingTimeInMillis and loadBalancerServiceUnitStateMonitorIntervalInSeconds to ServiceConfiguration

Hello Pulsar dev, I propose that ExtensibleLoadBalancer expose the following two configurations in the ServiceConfiguration. loadBalancerInFlightServiceUnitStateWaitingTimeInMillis=3 loadBalancerServiceUnitStateMonitorIntervalInSeconds=60 PR: https://github.com/apache/pulsar/pull/21668/files

Re: [ANNOUNCE] Yubiao Feng as new PMC member in Apache Pulsar

Congrats! Heesung On Mon, Nov 13, 2023 at 6:57 AM Yubiao Feng wrote: > Thank you all > > Yubiao Feng > > > > On Mon, Nov 13, 2023 at 3:36 PM mattison chao > wrote: > > > Dear Community, > > > > We are thrilled to announce that Yubiao Feng > > https://github.com/poorbarcode has been invited and

Re: [Discuss] Disable `Rebase and Merge` on Pulsar Repo

e it and > enable it temporarily when necessary. Although it introduces a bit chore > commits to toggle the button, due to its low frequency, it should be fine. > > Best, > tison. > > > Heesung Sohn 于2023年11月6日周一 12:07写道： > > > Hi, > > > > This is the

Re: [Discuss] Disable `Rebase and Merge` on Pulsar Repo

> Thanks, > Rajan > > On Sun, Nov 5, 2023 at 7:36 PM Heesung Sohn wrote: > > > Hi, > > > > I recently merged my PR without squash by mistake. > > https://github.com/apache/pulsar/pull/21408 > > > > To prevent this human error, can we disable `r

[Discuss] Disable `Rebase and Merge` on Pulsar Repo

Hi, I recently merged my PR without squash by mistake. https://github.com/apache/pulsar/pull/21408 To prevent this human error, can we disable `rebase and merge` option, like below? [image: Screen Shot 2023-11-05 at 7.21.34 PM.png] How to disable Rebase and merge: https://docs.github.com/en/re

Re: [DISCUSS] Moving to Java 21 for development and Pulsar server component runtime

Hi, +1 for Java 21. I think Java 21 can open the door for Virtual Threads[1]. [1] https://openjdk.org/jeps/444 Thanks, Heesung On Wed, Oct 18, 2023 at 10:33 AM Lari Hotari wrote: > Yes, I agree that we don't need to rush in making Java 21 the default > runtime in docker images and the default

Re: [Vote] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

; > On Tue, Aug 8, 2023 at 2:55 AM Heesung Sohn > wrote: > > > Hi, > > > > I'd like to start a vote thread for the PIP-281. > > https://github.com/apache/pulsar/pull/20748 > > > > Discussion : > > https://lists.apache.org/thread/bdmx4qhn6hkoxm0xbtf67tq4kt5r8jmy > > > > Best, > > Heesung > > >

Re: [VOTE] PIP-301: Introduce LoadBalanceResources to unify the load-date CRUD

+1 (non-binding) On Tue, Sep 19, 2023 at 10:25 PM 丛搏 wrote: > +1 (binding) > > Thanks, > Bo >

Re: [Vote] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

t; wrote: > > > > +1 (binding) > > -- > > Matteo Merli > > > > > > > > On Mon, Aug 7, 2023 at 11:55 AM Heesung Sohn > > wrote: > > > > > Hi, > > > > > > I'd like to start a vote thread for the PIP-281. > > > h

Re: [Vote] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

you land regarding Michael’s ideas? > > Best, > Dave > > Sent from my iPhone > > > On Sep 5, 2023, at 11:56 AM, Heesung Sohn > > > wrote: > > > > Hi dev, > > > > It has been open for a month now. Could we have some votes here? >

Re: [Vote] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

Hi dev, It has been open for a month now. Could we have some votes here? Thanks, Heesung On Mon, Aug 7, 2023 at 11:55 AM Heesung Sohn wrote: > Hi, > > I'd like to start a vote thread for the PIP-281. > https://github.com/apache/pulsar/pull/20748 > > Discussion : >

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

se(chunkedMsgCtx.chunkedMsgBuffer); > > > > > > > } > > > > > > > chunkedMsgCtx.recycle(); > > > > > > > } > > > > > > > chunkedMessagesMap.remove(msgMetadata.getUuid()); > > > > > > >

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

0, Check the last chunkID to determine whether to > dedup. > If chunkID == 1, put chunkID into the map if absent. >IF chunkID = 0, dedup. > > BR, > xiangying > > On Sat, Aug 26, 2023 at 11:53 AM Heesung Sohn > wrote: > > > > However, wha

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

> [0] https://github.com/apache/pulsar/pull/21047 > > On Sat, Aug 26, 2023 at 10:58 AM Heesung Sohn > wrote: > > > > Actually, can we think about this case too? > > > > What happens if the cx sends the same chunked msg with the same seq id > when > >

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

hunks -1 = ctx.lastChunkId, the chunked msg already persisted. So, dedup 3. if m.currentChunkId <= ctx.lastChunkId , dedup. On Fri, Aug 25, 2023 at 7:56 PM Heesung Sohn wrote: > Actually, can we think about this case too? > > What happens if the cx sends the same chunked msg with the same

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

` should be treated as a new message. > UUID is solely used for the consumer side to identify different chunk > messages. > > BR > Xiangying > > [0] https://github.com/apache/pulsar/pull/21047 > > On Sat, Aug 26, 2023 at 9:34 AM Heesung Sohn > wrote: > > > > I

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

, the solution of optimizing the `uuid` is valuable to the new > implementation. > I will modify this in the PR[0]. Thank you very much for your reminder > and the provided UUID optimization solution. > > BR, > Xiangying > > [0] https://github.com/apache/pulsar/pull/20948 > &

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

ding the message can be found here: > > https://github.com/apache/pulsar/blob/e0c481e5f8d7fa5534d3327785928a234376789e/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java#L1892 > > BR, > Xiangying > > > > > > > > On Sat, Aug 26,

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

>> I think brokers can track the last chunkMaxMessageSize for each producer. > Using different chunkMaxMessageSize is just one of the aspects. In PIP-132 [0], we have included the message metadata size when checking maxMessageSize.The message metadata can be changed after splitting the chunks. We

Re: [DICUSS] Topic-Partition Auto Creation after Partitioned Topic Force-Removal

Hi, I see. Since the PIP has already been approved, I am closing this discussion. Thanks, Heesung On Wed, Aug 23, 2023 at 5:48 PM wrote: > Hi, Heesung Sohn > > After PIP https://github.com/apache/pulsar/issues/19239 is implemented, > the topic name includes `-partition-` keyw

[DICUSS] Topic-Partition Auto Creation after Partitioned Topic Force-Removal

Hi dev, I am proposing a behavior change in Pulsar, and I wonder if this is acceptable as a "bug fix." PR: https://github.com/apache/pulsar/pull/20554 Given that the topic-auto-creation is enabled, When a partitioned topic is deleted with force=true, the existing clients(consumers and producers)

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

> I think a simple better approach is to only check the deduplication for the last chunk of the large message. The consumer only gets the whole message after receiving the last chunk. We don't need to check the deduplication for all previous chunks. Also by doing this we only need bug fixes, we don

Re: [DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

old owner to disconnect clients, right? > Please correct me if I am wrong or missing something. > > Thanks, > Penghui > > > On Tue, Aug 22, 2023 at 4:53 AM Heesung Sohn > wrote: > > > Hi Penghui, > > > > I added one comment below. > > > >

Re: [DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

the decision to the community. > > > > I do not speak for the community (hopefully that's obvious), but I am > > a community member advocating for a slightly more complex solution > > that, in my opinion, will solve one part of the transfer problem. :) I > > don&#

Re: [DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

1/pulsar-common/src/main/proto/PulsarApi.proto#L242-L268 > [2] https://github.com/apache/pulsar/pull/20748#discussion_r1287927655 > > On Fri, Aug 4, 2023 at 3:41 PM Heesung Sohn > wrote: > > > > Hi, > > > > I updated the PIP to add more details about the recent

[Vote] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

Hi, I'd like to start a vote thread for the PIP-281. https://github.com/apache/pulsar/pull/20748 Discussion : https://lists.apache.org/thread/bdmx4qhn6hkoxm0xbtf67tq4kt5r8jmy Best, Heesung

Re: [DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

Hi, I updated the PIP to add more details about the recent discussion. Thanks, Heesung On Thu, Aug 3, 2023 at 6:46 PM Heesung Sohn wrote: > > > On Wed, Aug 2, 2023 at 8:02 PM PengHui Li wrote: > >> > One of the advantages of this proposal (add these optional dstBroke

Re: [DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

can also update the owner and then close the producer and consumers with > the new owner. > This PIP is based on PIP-192 because the proposed protocol change is based on the bundle states from PIP-192. I assume the community could raise other PIPs to utilize the close producer and consumer c

Re: [DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

eventually consistent, and the lookups should eventually converge. - if the ownership change is still in progress(`releasing`, `assigning`), this check will be deferred until the state becomes `owned` with a timeout. > Regards, > Penghui > > On Wed, Aug 2, 2023 at 1:18 AM Heesung Soh

Re: [DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

Hi, It has been over three weeks since this discussion started, and I replied to the questions in the PIP. I will send out a vote email tomorrow if looking good. Thanks, Heesung On Tue, Jul 25, 2023 at 1:12 PM Heesung Sohn wrote: > Hi, > > I replied to the comments from Kai and Ran i

Re: [DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

Hi, I replied to the comments from Kai and Ran in the PIP. Please review the PIP by any chance. Thank you, Heesung On Thu, Jul 6, 2023 at 4:32 PM Heesung Sohn wrote: > Hi dev, > > I proposed this PIP, https://github.com/apache/pulsar/pull/20748, to make > unloaded clients

Re: [DISCUSS] Cherry-pick PR-16059 to 2.10 to prevent infinite unloading

he/pulsar/pull/388 > > > > > > PengHui Li 于2023年7月1日周六 09:38写道： > > > > > > > > +1 for cherry-pick to branch-2.10 since users don't have a workaround > > > > for this issue, and the change is well-understand, low risk. > > > > &

Re: [DISCUSS] Unload Rate Limiting during Graceful Shutdown of Pulsar

we might end up having too many parameters 🤔 > > Thanks, > JooHyukKim(vince) > > On Fri, Jul 7, 2023 at 1:57 PM Heesung Sohn > wrote: > > > Hi, > > > > I agree with configuring the `shutdown.unloadNamespaceBundlesGracefully` > >

Re: [DISCUSS] Unload Rate Limiting during Graceful Shutdown of Pulsar

Hi, I agree with configuring the `shutdown.unloadNamespaceBundlesGracefully` behavior by a dynamic config. Also, I wonder if the better config is `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute` to support a slower rate. One bundle unloading per sec could be too fast if there are many top

[DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

Hi dev, I proposed this PIP, https://github.com/apache/pulsar/pull/20748, to make unloaded clients directly and gracefully connect to new owner brokers without lookups. Please take a look and let me know what you think. Regards, Heesung

[DISCUSS] Cherry-pick PR-16059 to 2.10 to prevent infinite unloading

Hi dev, I realized that `removeMostServicingBrokersForNamespace` func in the broker selection logic can cause infinite unloading. Suppose an overloaded broker unloaded a bundle and only has the minimum number of bundles(in that namespace) among brokers. In that case, the selection logic (`removeM

Re: Re:Re: [DISCUSS] PIP-255: Assign topic partitions to bundle by round robin

Hi, As pip-192(load balancer extension) has been added in pulsar-3.0, could you also clarify how this strategy will be compatible with the load balancer extension? >From my understanding, this partition assignment strategy can also be configurable in the load balancer extension. Can you confirm?

Re: Pulsar doesn't support cgroup v2 which became the default in Kubernetes v1.25+ (AKS v1.25, GKE v1.26, EKS v1.26)

Hi, +1 for the backport. If there are network bandwidth signals (if the below bandwidth metrics show), I think one could try to weigh more on bandwidth signals like the following. # bandwidth metrics pulsar_lb_bandwidth_in_usage Gauge The broker inbound bandwidth usage (in percent). pulsar_lb_ban

Re: [DISCUSS] PIP-257: Add Open ID Connect Support to Server Components

Hi, I am new to this security component, but how do we support token revoke? Thanks, Heesung On Mon, Mar 27, 2023 at 12:31 PM Michael Marshall wrote: > Here is the K8s integration: https://github.com/apache/pulsar/pull/19888. > > That PR makes it possible to configure a function running in > k

Re: Pulsar Flaky test report 2023-04-03 for PR builds in CI

o > flaky group would resolve that issue. > Once that it is proven to be stable, we can make a simple update to change > the test group. > I'll proceed with a PR for this soon. > > -Lari > > On 2023/04/04 16:39:21 Heesung Sohn wrote: > > Hi, > > > > T

Re: Pulsar Flaky test report 2023-04-03 for PR builds in CI

Hi, Thank you for the report. We are fixing the errors from ExtensibleLoadManagerImplTest. Regards, Heesung On Tue, Apr 4, 2023 at 3:01 AM Nicolò Boschi wrote: > Dear community, > Here's a report of the flaky tests in Pulsar CI during the observation > period of 2023-03-27 - 2023-04-03 > > >

Re: [DISCUSS] PIP-255: Assign topic partitions to bundle by round robin

Hi, I see. I have follow-up questions below. - This appears to be the "round-robin topic-to-bundle mapping" option in the `fundBundle` function. Is this the only place that needs an update? Can you list what change is required? - How do we enable this "round-robin topic-to-bundle mapping option"

Re: [DISCUSS] PIP-255: Assign topic partitions to bundle by round robin

Hi, Thank you for sharing this. In general, I think this can be another good option for Pulsar load assignment logic. However, I have some comments below. > The load managed by each Bundle is not even. Even if the number of partitions managed by each bundle is the same, there is no guarantee t

Re: [Discuss] Deprecate Index-based Publisher Stat Aggregation in Topics Partitioned-Stats

Hi dev, If there are no particular concerns, I will send out a vote email. Thanks, Heesung On Tue, Jan 3, 2023 at 10:28 AM Heesung Sohn wrote: > Hi Pulsar dev, > > I raised a PIP for the default config change(change the > aggregatePublisherStatsByProducerName config default

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

On Sun, Feb 5, 2023 at 2:26 AM Yan Zhao wrote: > On 2023/02/03 18:14:53 Heesung Sohn wrote: > > There are some cases to trigger it. > > 1. A cursor be removed. > > 2. Close the current ledger and create a new ledger. > > 3. Consumer ack the message, the slowest curso

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

2023 at 10:14 AM Heesung Sohn wrote: > > > There are some cases to trigger it. > 1. A cursor be removed. > 2. Close the current ledger and create a new ledger. > 3. Consumer ack the message, the slowest cursor move forward. > 4. User trigger truncateTopic by admin. > >

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

There are some cases to trigger it. 1. A cursor be removed. 2. Close the current ledger and create a new ledger. 3. Consumer ack the message, the slowest cursor move forward. 4. User trigger truncateTopic by admin. I see that this pip is for the internal ledger deletion cases(1-3 above), and it ap

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

peration state, but this will show some eventual consistency because of the delay between step 3 and step 5. Thanks, Heesung > > On Tue, Jan 31, 2023 at 9:06 PM Heesung Sohn > wrote: > > > On Tue, Jan 31, 2023 at 6:43 AM Yan Zhao wrote: > > > > > > - Have we

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

On Tue, Jan 31, 2023 at 6:43 AM Yan Zhao wrote: > > - Have we considered a metadata store to persist and dedup deletion > > requests instead of the system topic? Why is the system topic the better > > choice than a metadata store for this problem? > If we use the metadata store to store the middl

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

Hi, I assume the deletion APIs are async(when a user requests deletion, pulsar first returns success to the user if the request is persisted. Then, Pulsar asynchronously runs the deletion flow) - Have we considered a metadata store to persist and dedup deletion requests instead of the system top

Re: [DISCUSS] PIP-232: Introduce thread monitor to check if thread is blocked for long time.

Hi, This is a great idea. This will significantly help to debug production issues caused by long-waiting threads. I agree with Lari that we could start with the basic metrics. Like Lari shared in the reference, we could simply collect the current thread pool queue size like the following. ((Thr

Re: [Discuss] Deprecate Index-based Publisher Stat Aggregation in Topics Partitioned-Stats

On Fri, Dec 9, 2022 at 2:12 PM Heesung Sohn wrote: > Hi, > > I raised a PR to the master branch here, > https://github.com/apache/pulsar/pull/18807. > > PTAL. > > Thank you, > Heesung > > On Wed, Nov 30, 2022 at 4:06 AM Enrico Olivelli > wrote: > >> H

Re: [Discuss] Deprecate Index-based Publisher Stat Aggregation in Topics Partitioned-Stats

versions. > For a new major version we COULD break compatibility only as a last resort. > > If we have a way to improve the code and use less resources, I believe > that it is the best way. > > Enrico > > Il giorno mer 30 nov 2022 alle ore 11:56 Heesung Sohn > ha scritto: >

Re: [Discuss] Deprecate Index-based Publisher Stat Aggregation in Topics Partitioned-Stats

Hi, I raised this PR to my local fork to sync our direction. https://github.com/heesung-sn/pulsar/pull/15/files Please let me know if this direction looks good. Then, I will proceed to raise an official PR. Thanks, Heesung

Re: [Discuss] Deprecate Index-based Publisher Stat Aggregation in Topics Partitioned-Stats

d keep room for choice. > > > > > > > > In order to allow users (here Pulsar server-side admins in > particular) > > to > > > > choose the performance or backward-compatibility, how about > > introducing a > > > > "for

Re: [DISCUSS] PIP-215: Configurable Topic Compaction Strategy

Hi, The vote is open in this email thread. https://lists.apache.org/thread/6y8407pw4fv21n2n0cbrvsspg5tok0h7 Regards, Heesung On Fri, Nov 4, 2022 at 3:07 PM Heesung Sohn wrote: > Hi, > > > I would like to understand what the performance tradeoffs are for the > changes

[Discuss] AsyncAPI Pulsar Bindings

Dear Pulsar Community, It appears that there is strong demand for AsyncAPI Pulsar bindings. The discussion is already ongoing, and there is a PR open for this work. PR: https://github.com/asyncapi/bindings/pull/173 Regards, Heesung

Re: [Vote] PIP-192 New Pulsar Broker Load Balancer

Dear Pulsar Community, To track sub-PIPs and PRs, I put a project status in the PIP here. https://github.com/apache/pulsar/issues/16691#issuecomment-1318959420 Please help to review them to move this project forward. Thank you, Heesung On Mon, Oct 17, 2022 at 6:07 PM Heesung Sohn wrote

Re: [Discuss] Deprecate Index-based Publisher Stat Aggregation in Topics Partitioned-Stats

tatsByProducerName config but change the default to aggregatePublisherStatsByProducerName=true. Option 4: As a long-term fix, create separate Admin-APIs for publisher and subscription stats and drop their stats from `topics partitioned-stats` as it is expensive to aggregate them on the fly. (for

[Vote] PIP-220 TransferShedder (only for PIP-192 New Broker Load Balancer)

Dear Pulsar Community, Please review and vote on this PIP. PIP link: https://github.com/apache/pulsar/issues/18215 Thank you, -Heesung

[Discuss] Deprecate Index-based Publisher Stat Aggregation in Topics Partitioned-Stats

Dear Pulsar Community, We recently found a bug in `pulsar-admin topics partitioned-stats api` that could incur a memory burst, high GC time, or OOM. For this issue, I proposed a fix by deprecating the aggregatePublisherStatsByProducerName config and a

[Vote] PIP-215: Configurable TopicCompactionStrategy for StrategicTwoPhaseCompactor and TableView

Dear Pulsar Community, Please review and vote on this PIP. PIP link: https://github.com/apache/pulsar/issues/18099 Thank you, -Heesung

Re: [DISCUSS] PIP-215: Configurable Topic Compaction Strategy

Thanks, Heesung On Fri, Nov 4, 2022 at 11:11 AM Dave Fisher wrote: > > > > On Nov 4, 2022, at 10:28 AM, Heesung Sohn > > > wrote: > > > > Hi, > > > > I think `bool shouldKeepLeft(T prev, T cur)` is clearer. I updated the > PIP. > > &

Re: [DISCUSS] PIP-215: Configurable Topic Compaction Strategy

what the result will do. > > > Regarding redundant deserialization, the input type `T` is the type of > > message value, so the input values are already deserialized. > > Great, I should have realized that. That takes care of that concern. > > Thanks, > Michael >

Re: [DISCUSS] PIP-215: Configurable Topic Compaction Strategy

ta, and messageId copies. We could initially define `isValid()` only in CompactionStrategy, and add `isMergeEnabled() and merge()` later in the CompactionStrategy interface if requested. Regards, Heesung On Thu, Nov 3, 2022 at 9:50 AM Heesung Sohn wrote: > Oops! Michael, I apologize for the

  1   2   >