PIP-156: Build and Run Pulsar Server on Java 17

Dear Pulsar dev community, My name is Heesung Sohn, and I would like to open a discussion here about PIP-156: Build and Run Pulsar Server on Java 17 <https://github.com/apache/pulsar/issues/15207>. @Nicolò Boschi, I see your similar PR <https://github.com/apache/pulsar/pull/14355>

[VOTE] PIP-156: Build and Run Pulsar Server on Java 17

Dear Pulsar Community, Please review and vote on this PIP. PIP link : https://github.com/apache/pulsar/issues/15207 Thank you, -- <https://streamnative.io> Heesung Sohn Platform Engineer e: heesung.s...@streamnative.io streamnative.io <http://github.com/streamnativ

Re: JDK 17 runtime compatibility

Hi dev, I am definitely a latecomer to this issue, but I think we could also support java17 build as well as this java17 runtime support. I added this java17 build support and created a pip here, https://github.com/apache/pulsar/issues/15207. I think I can help with upgrading the build for java1

Re: PIP-156: Build and Run Pulsar Server on Java 17

. > > In this case, it would have been fair to provide feedback on the "JDK 17 > runtime compatibility" thread that Nicolo started in February or comment on > the PR https://github.com/apache/pulsar/pull/14355 which takes the move > to use Java 17 for CI and for the Docker ima

Re: PIP-156: Build and Run Pulsar Server on Java 17

, 2022 at 4:00 AM Jiuming Tao wrote: > +1 > Thanks, > Tao Jiuming > > > 2022年4月19日 上午2:25，Heesung Sohn > 写道： > > > > Dear Pulsar dev community, > > > > My name is Heesung Sohn, and I would like to open a discussion here > > about PIP-156: &g

[DISCUSS] Pulsar Broker Load Balancer Improvement Project

Dear Pulsar community, Our engineers from StreamNative would like to contribute to improving the broker load balancer. To collaborate with the community from the early phase, we opened a slack channel and summarized the improvement ideas in this document. pulsar slack channel : *#broker-load-bal

Re: Pulsar Flaky test report 2022-06-02 for PR builds in CI

Hi, Can I get this PR reviewed? This PR is to make some of the tests less flaky. https://github.com/apache/pulsar/pull/16011 Thanks, Heesung On Thu, Jun 2, 2022 at 5:06 AM Lari Hotari wrote: > Dear Pulsar community members, > > Here's a report of the flaky tests in Pulsar CI during the obs

Re: [DISCUSS] PIP-181: Provide new load balance placement strategy implementation for ModularLoadManagerStrategy

Hi 叶韵, The proposal looks good to me, and I left comments in the PR, https://github.com/apache/pulsar/pull/16281/. Regards, Heesung On Wed, Jun 29, 2022 at 5:39 AM 叶韵 wrote: > Hi Pulsar community: > I open a pip to discuss "Shadow Topic, an alternative way to support > readonly topic ownership.

[DISCUSS] PIP-192 New Pulsar Broker Load Balancer

Dear Pulsar dev community, We would like to open a discussion here about PIP-192: New Pulsar Broker Load Balancer . Regards, Heesung

Re: [DISCUSS] PIP-192 New Pulsar Broker Load Balancer

ng the existing logic. - This new load manager will be disabled in the first releases until proven stable. > Thanks, > Rajan > > > > On Tue, Jul 19, 2022 at 10:15 PM Heesung Sohn > wrote: > > > Dear Pulsar dev community, > > > > We would like t

Re: [DISCUSS] PIP-192 New Pulsar Broker Load Balancer

Hi Rajan, As discussed in the Pulsar Community Meeting today, I added the "Modification Summary" section that compares the current solutions and our proposals. Regards, Heesung On Wed, Jul 20, 2022 at 4:24 PM Matteo Merli wrote: > > On Wed, Jul 20, 2022 at 1:37 PM Rajan Dhabalia wrote: > > > A

[Vote] PIP-192 New Pulsar Broker Load Balancer

Dear Pulsar Community, Please review and vote on this PIP. PIP link: https://github.com/apache/pulsar/issues/16691 Thank you, -Heesung

Re: [VOTE] [PIP-169] Support split bundle by flow or qps

+1 (non-binding) Heesung On Tue, Aug 23, 2022 at 10:00 AM Anon Hxy wrote: > > +1 (non-binding) > > Thanks, > Xiaoyu Hou > > lordcheng10 <1572139...@qq.com.invalid> 于2022年8月23日周二 15:10写道： > > > Hi Pulsar Community, I would like to start a VOTE on "Support split bundle > > by flow or qps."(PIP-169

Re: [Vote] PIP-192 New Pulsar Broker Load Balancer

, Heesung On Tue, Aug 9, 2022 at 4:56 AM guo jiwei wrote: > +1 > > Regards > Jiwei Guo (Tboy) > > > On Tue, Aug 9, 2022 at 3:12 PM Qiang Huang > wrote: > > > +1 (non-binding) > > > > Kai Wang 于2022年8月5日周五 10:18写道： > > > > > +1 (non-binding) &

Re: [DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

old owner to disconnect clients, right? > Please correct me if I am wrong or missing something. > > Thanks, > Penghui > > > On Tue, Aug 22, 2023 at 4:53 AM Heesung Sohn > wrote: > > > Hi Penghui, > > > > I added one comment below. > > > >

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

> I think a simple better approach is to only check the deduplication for the last chunk of the large message. The consumer only gets the whole message after receiving the last chunk. We don't need to check the deduplication for all previous chunks. Also by doing this we only need bug fixes, we don

[DICUSS] Topic-Partition Auto Creation after Partitioned Topic Force-Removal

Hi dev, I am proposing a behavior change in Pulsar, and I wonder if this is acceptable as a "bug fix." PR: https://github.com/apache/pulsar/pull/20554 Given that the topic-auto-creation is enabled, When a partitioned topic is deleted with force=true, the existing clients(consumers and producers)

Re: [DICUSS] Topic-Partition Auto Creation after Partitioned Topic Force-Removal

Hi, I see. Since the PIP has already been approved, I am closing this discussion. Thanks, Heesung On Wed, Aug 23, 2023 at 5:48 PM wrote: > Hi, Heesung Sohn > > After PIP https://github.com/apache/pulsar/issues/19239 is implemented, > the topic name includes `-partition-` keyw

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

>> I think brokers can track the last chunkMaxMessageSize for each producer. > Using different chunkMaxMessageSize is just one of the aspects. In PIP-132 [0], we have included the message metadata size when checking maxMessageSize.The message metadata can be changed after splitting the chunks. We

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

ding the message can be found here: > > https://github.com/apache/pulsar/blob/e0c481e5f8d7fa5534d3327785928a234376789e/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java#L1892 > > BR, > Xiangying > > > > > > > > On Sat, Aug 26,

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

, the solution of optimizing the `uuid` is valuable to the new > implementation. > I will modify this in the PR[0]. Thank you very much for your reminder > and the provided UUID optimization solution. > > BR, > Xiangying > > [0] https://github.com/apache/pulsar/pull/20948 > &

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

` should be treated as a new message. > UUID is solely used for the consumer side to identify different chunk > messages. > > BR > Xiangying > > [0] https://github.com/apache/pulsar/pull/21047 > > On Sat, Aug 26, 2023 at 9:34 AM Heesung Sohn > wrote: > > > > I

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

hunks -1 = ctx.lastChunkId, the chunked msg already persisted. So, dedup 3. if m.currentChunkId <= ctx.lastChunkId , dedup. On Fri, Aug 25, 2023 at 7:56 PM Heesung Sohn wrote: > Actually, can we think about this case too? > > What happens if the cx sends the same chunked msg with the same

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

> [0] https://github.com/apache/pulsar/pull/21047 > > On Sat, Aug 26, 2023 at 10:58 AM Heesung Sohn > wrote: > > > > Actually, can we think about this case too? > > > > What happens if the cx sends the same chunked msg with the same seq id > when > >

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

0, Check the last chunkID to determine whether to > dedup. > If chunkID == 1, put chunkID into the map if absent. >IF chunkID = 0, dedup. > > BR, > xiangying > > On Sat, Aug 26, 2023 at 11:53 AM Heesung Sohn > wrote: > > > > However, wha

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

se(chunkedMsgCtx.chunkedMsgBuffer); > > > > > > > } > > > > > > > chunkedMsgCtx.recycle(); > > > > > > > } > > > > > > > chunkedMessagesMap.remove(msgMetadata.getUuid()); > > > > > > >

Re: [Vote] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

Hi dev, It has been open for a month now. Could we have some votes here? Thanks, Heesung On Mon, Aug 7, 2023 at 11:55 AM Heesung Sohn wrote: > Hi, > > I'd like to start a vote thread for the PIP-281. > https://github.com/apache/pulsar/pull/20748 > > Discussion : >

Re: [Vote] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

you land regarding Michael’s ideas? > > Best, > Dave > > Sent from my iPhone > > > On Sep 5, 2023, at 11:56 AM, Heesung Sohn > > > wrote: > > > > Hi dev, > > > > It has been open for a month now. Could we have some votes here? >

Re: [Vote] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

t; wrote: > > > > +1 (binding) > > -- > > Matteo Merli > > > > > > > > On Mon, Aug 7, 2023 at 11:55 AM Heesung Sohn > > wrote: > > > > > Hi, > > > > > > I'd like to start a vote thread for the PIP-281. > > > h

Re: [VOTE] PIP-301: Introduce LoadBalanceResources to unify the load-date CRUD

+1 (non-binding) On Tue, Sep 19, 2023 at 10:25 PM 丛搏 wrote: > +1 (binding) > > Thanks, > Bo >

Re: [Vote] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

; > On Tue, Aug 8, 2023 at 2:55 AM Heesung Sohn > wrote: > > > Hi, > > > > I'd like to start a vote thread for the PIP-281. > > https://github.com/apache/pulsar/pull/20748 > > > > Discussion : > > https://lists.apache.org/thread/bdmx4qhn6hkoxm0xbtf67tq4kt5r8jmy > > > > Best, > > Heesung > > >

Re: [DISCUSS] Moving to Java 21 for development and Pulsar server component runtime

Hi, +1 for Java 21. I think Java 21 can open the door for Virtual Threads[1]. [1] https://openjdk.org/jeps/444 Thanks, Heesung On Wed, Oct 18, 2023 at 10:33 AM Lari Hotari wrote: > Yes, I agree that we don't need to rush in making Java 21 the default > runtime in docker images and the default

[Discuss] Disable `Rebase and Merge` on Pulsar Repo

Hi, I recently merged my PR without squash by mistake. https://github.com/apache/pulsar/pull/21408 To prevent this human error, can we disable `rebase and merge` option, like below? [image: Screen Shot 2023-11-05 at 7.21.34 PM.png] How to disable Rebase and merge: https://docs.github.com/en/re

Re: [Discuss] Disable `Rebase and Merge` on Pulsar Repo

> Thanks, > Rajan > > On Sun, Nov 5, 2023 at 7:36 PM Heesung Sohn wrote: > > > Hi, > > > > I recently merged my PR without squash by mistake. > > https://github.com/apache/pulsar/pull/21408 > > > > To prevent this human error, can we disable `r

Re: [Discuss] Disable `Rebase and Merge` on Pulsar Repo

e it and > enable it temporarily when necessary. Although it introduces a bit chore > commits to toggle the button, due to its low frequency, it should be fine. > > Best, > tison. > > > Heesung Sohn 于2023年11月6日周一 12:07写道： > > > Hi, > > > > This is the

Re: [ANNOUNCE] Yubiao Feng as new PMC member in Apache Pulsar

Congrats! Heesung On Mon, Nov 13, 2023 at 6:57 AM Yubiao Feng wrote: > Thank you all > > Yubiao Feng > > > > On Mon, Nov 13, 2023 at 3:36 PM mattison chao > wrote: > > > Dear Community, > > > > We are thrilled to announce that Yubiao Feng > > https://github.com/poorbarcode has been invited and

[Discussion] (Extensible Load Balancer) Expose loadBalancerInFlightServiceUnitStateWaitingTimeInMillis and loadBalancerServiceUnitStateMonitorIntervalInSeconds to ServiceConfiguration

Hello Pulsar dev, I propose that ExtensibleLoadBalancer expose the following two configurations in the ServiceConfiguration. loadBalancerInFlightServiceUnitStateWaitingTimeInMillis=3 loadBalancerServiceUnitStateMonitorIntervalInSeconds=60 PR: https://github.com/apache/pulsar/pull/21668/files

[DISCUSS] PIP-188(BlueGreenClusterMigration) Remove redirectedClusterURI in Pulsar Client

Hi dev, I would like to discuss this change with the community. Motivation: By this BlueGreenClusterMigration PIP-188( https://github.com/apache/pulsar/pull/17962), `redirectedClusterURI` member var has been introduced in the pulsar client to hold the new cluster's endpoint. Accordingly, pulsar c

Re: [DISCUSS] PIP-188(BlueGreenClusterMigration) Remove redirectedClusterURI in Pulsar Client

me topic) so, changing > service url will not work for blue-green migration redirection for all > topics served by that PulsarClient. > > Sent from my iPhone > > > On Jan 25, 2024, at 6:57 PM, Heesung Sohn wrote: > > > > Hi dev, > > > > I would like to d

Re: [DISCUSS] PIP-188(BlueGreenClusterMigration) Remove redirectedClusterURI in Pulsar Client

lient because all of the topics (of consumers and producers) will eventually migrate to the new cluster? Having said this, having redirectedClusterURI in HandlerState appears to be futuristic. Heesung On Thu, Jan 25, 2024 at 10:36 PM Heesung Sohn wrote: > > This might not work because same P

Re: [DISCUSS] PIP-188(BlueGreenClusterMigration) Remove redirectedClusterURI in Pulsar Client

clients. Also, I updated the sample code for better synchronization and dedup. https://github.com/heesung-sn/pulsar/pull/59/ Thanks, Heesung On Thu, Jan 25, 2024 at 10:51 PM Heesung Sohn wrote: > I am sorry, but the current implementation of blue-green migration is for > all topics (produce

Re: [E] Re: [DISCUSS] PIP-188(BlueGreenClusterMigration) Remove redirectedClusterURI in Pulsar Client

sed on its state decided by the server. > > > Sent from my iPhone > > > On Jan 26, 2024, at 9:12 AM, Heesung Sohn wrote: > > > > To add more context, I am trying to add this blue-green migration logic > in > > the cpp and golang client, and before copy

Re: [E] Re: [DISCUSS] PIP-188(BlueGreenClusterMigration) Remove redirectedClusterURI in Pulsar Client

blue-green migration logic. Regards, Heesung On Sun, Jan 28, 2024 at 3:23 PM Heesung Sohn wrote: > I see. Thank you for clarifying this requirement. > > Then, I am wondering if we need to refactor the current > `lookup/urlLookupMap` access code to avoid using `look

Re: [Discuss] Release Pulsar C++ Client 3.5.0

Hi, Yes, I am currently working on C++ client update for the blue-green migration support. I think I can raise a PR by early next week if everything goes smoothly. Thanks, Heesung On Wed, Feb 7, 2024 at 12:55 PM Matteo Merli wrote: > Heesung is finalizing the work for adding support for PIP-1

Re: [Discuss] Release Pulsar C++ Client 3.5.0

Hi, I raised this PR to support blue-green migration for cpp client. PR: https://github.com/apache/pulsar-client-cpp/pull/402 Thanks, Heseung On Wed, Feb 7, 2024 at 2:28 PM Heesung Sohn wrote: > Hi, > > Yes, I am currently working on C++ client update for the blue-green > migra

[DISCUSS] Apache Pulsar 3.0.3 Release

Dear the Apache Pulsar Dev community, I would like to propose releasing Apache Pulsar 3.0.3. It's over two months since the release of 3.0.2, and there are 95 new commits in branch-3.0: https://github.com/apache/pulsar/compare/v3.0.2...branch-3.0 Please let me know if you have any essential fixe

[VOTE] Pulsar Release 3.0.3 Candidate 1

ease download the source package, and follow the README to build and run the Pulsar standalone service. Regards, Heesung Sohn (heesung)

Re: [VOTE] Pulsar Release 3.0.3 Candidate 1

ari Hotari wrote: > > > > > +1 (binding) > > > > > > - Built from source > > > - Checked the signatures of the source and binary release artifacts > > > - Run standalone > > > - Checked producer and consumer > > > - Verified th

[ANNOUNCE] Apache Pulsar 3.0.3 released

The Apache Pulsar team is proud to announce Apache Pulsar version 3.0.3. Pulsar is a highly scalable, low latency messaging platform running on commodity hardware. It provides simple pub-sub semantics over topics, guaranteed at-least-once delivery of messages, automatic cursor management for subsc

Re: [ANNOUNCE] Apache Pulsar 3.0.3 released

/pulsar-site/commit/7f5743008dd3a60dd91ee2dab9ca3c743fc3015e) to clarify the upload process, so this file size issue should be fixed in the next LTS release, 3.0.4. Regards, Heesung On Sun, Mar 10, 2024 at 1:06 PM Heesung Sohn wrote: > The Apache Pulsar team is proud to announce Apache Pul

Re: [VOTE] PIP-364: Introduce a new load balance algorithm AvgShedder.

+1 (not binding) Heesung On Wed, Jun 26, 2024 at 12:36 AM Lari Hotari wrote: > > +1 (binding) > > -Lari > > On 2024/06/26 03:00:02 thetumbled wrote: > > Hi, Pulsar Community. > > I would like to start the voting thread for PIP-364: Introduce a new load > > balance algorithm AvgShedder. > > P

Re: [DISCUSS] PIP-368: Support client properties for the lookup

+1 (not binding) Heesung On Thu, Jul 25, 2024 at 7:52 PM Jie crossover wrote: > > +1 > I left some comments in the PR. > -- > Best Regards！ > crossoverJie > > > Zike Yang 于2024年7月25日周四 11:15写道： > > > Hi, all, > > > > I proposed a new proposal to add client properties support for the > > lookup:

Re: [DISCUSS] PIP-368: Support client properties for the lookup

to a different rack(rack B)? 2. Given that the latest load shedders(TransferShedder and AvgShedder) determine the destination brokers from the leader, how would the leader make sure to unload bundles(topics) to the brokers in the same rack? Thanks, Heesung On Fri, Jul 26, 2024 at 8:29 AM Heesung

[DISCUSS] PIP-371 Support to specify message listeners in TableView constructor

Hi all, I shared a proposal to specify message listeners in TableView constructor. I'm looking forward to hearing from you. pip: https://github.com/apache/pulsar/pull/23167 Thank you, Heesung

[DISCUSS] PIP-378 Add ServiceUnitStateTableView abstraction in ExtensibleLoadMangerImpl

Hi All, I drafted a proposal to add ServiceUnitStateTableView abstraction in ExtensibleLoadMangerImpl. This abstraction could help to event-source bundle ownership via other stores like ZK in addition to Pulsar system topics. https://github.com/apache/pulsar/pull/23300 I'm looking forward to hear

[VOTE] PIP-378 Add ServiceUnitStateTableView abstraction (ExtensibleLoadMangerImpl only)

Hi all, I would like to start a vote on PIP-378 to add ServiceUnitStateTableView abstraction and make it pluggable. Proposal link: https://github.com/apache/pulsar/pull/23300 Discussion link: https://lists.apache.org/thread/v7sod21r56hkt2cjxl9pp348r4jxo6o8 Thanks, Heesung

Re: [DISCUSS] PIP-378 Add ServiceUnitStateTableView abstraction in ExtensibleLoadMangerImpl

/5d25ef8535bde62df7fff53b31198ea67ddf6ab5 The idea is to sync system topic and metadata store during migration operation (this syncer is run by the leader only under the dynamic config). Thanks, Heesung On Thu, Sep 12, 2024 at 4:06 PM Heesung Sohn wrote: > > Hi All, > > I drafted a pro

Re: [Vote] PIP-192 New Pulsar Broker Load Balancer

wrote: > +1 (binding) > > What's the status of this PIP ? > > Enrico > > Il giorno gio 25 ago 2022 alle ore 05:04 Hang Chen > ha scritto: > > > > +1 (binding) > > > > Awesome job! > > > > Thanks, > > Hang > > &g

[DISCUSS] PIP-215: Configurable Topic Compaction Strategy

Hi pulsar-dev community, I raised a pip to discuss : PIP-215: Configurable Topic Compaction Strategy PIP link: https://github.com/apache/pulsar/issues/18099 Regards, Heesung

Re: [DISCUSS] PIP-215: Configurable Topic Compaction Strategy

break the existing load balance logic. Users will be able to configure old/new broker load balancer.) Thank you for sharing your questions about PIP-192 here. But I think this PIP-215 is independent of PIP-192(though PIP-192 needs some of the features in PIP-215). Thanks, Heesung > Thanks, >

Re: [DISCUSS] PIP-215: Configurable Topic Compaction Strategy

new classes), so it should not break the existing load balance logic. Users will be able to configure old/new broker load balancer.) On Wed, Oct 19, 2022 at 10:17 AM Heesung Sohn wrote: > Hi, > > On Wed, Oct 19, 2022 at 2:06 AM 丛搏 wrote: > >> Hi, Heesung: >> I have some

Re: [DISCUSS] PIP-215: Configurable Topic Compaction Strategy

the owner for each bundle state channel partition.* *Channel Owner Discovery* *Then, in brokers’ TopicLookUp logic, we will add a special case to return the current leader(the elected BSC owner) for the BSC topics.* > > Maybe I missed something. > > Thanks, > Penghui > > On Thu, Oct 2

Re: [DISCUSS] PIP-217: LoadShedding Strategy Improment

Hi, Currently, the historical score is based on exponential smoothing. As you described, the current default value is conservative (by default 90%), meaning it weighs significantly more on the historical load than the current load in the formula. historicalPecentage is configurable, and one could

Re: [DISCUSS] PIP-215: Configurable Topic Compaction Strategy

igned by broker 1 5. bundle 3 splitting by broker 1 // in the process of split 6. bundle 3 assigned by broker 2 Regards, Heesung > Thanks, > Penghui > > On Sat, Oct 22, 2022 at 12:23 AM Heesung Sohn > wrote: > > > Hi Penghui, > > > > I put my answers inline

Re: [DISCUSS] PIP-215: Configurable Topic Compaction Strategy

topicPolicies get-compaction-strategy options Regards, Heesung On Mon, Oct 24, 2022 at 1:41 PM Heesung Sohn wrote: > Hi, please find my answers inline. > > On Sun, Oct 23, 2022 at 7:11 PM PengHui Li wrote: > >> Sorry, heesung, >> >> I think I used a confusing name "lea

[DISCUSS] PIP-220 TransferShedder (for PIP-192 New Broker Load Balancer)

Dear Pulsar Community, I raised a proposal to discuss a new shedder, TransferShedder that uses the bundle transfer protocol introduced in the PIP-192. PIP link : https://github.com/apache/pulsar/issues/18215 Regards, Heesung

Re: [DISCUSS] PIP-215: Configurable Topic Compaction Strategy

ot;isValid", "isMergeEnabled", and "merge" methods with a single > > method that lets the implementation handle one or all tasks. That > > would also remove the need to deserialize payloads multiple times too. > > > > I also feel like mentioning that

Re: [DISCUSS] PIP-215: Configurable Topic Compaction Strategy

ex to implement (leaderless conflict resolution and requires a single topic) 4. it is not a "one-way door" decision (we could add the pre-filter logic as well later) Regards, Heesung On Sat, Oct 29, 2022 at 1:03 PM Heesung Sohn wrote: > Hi Michael, > > For the pre-prefilte

Re: [DISCUSS] PIP-215: Configurable Topic Compaction Strategy

d add it later, but at that point, we > will have added this new API for compaction strategy. Are we confident > that pluggable compaction is independently an important addition to > Pulsar's > features, or would it make sense to make this API only exposed in the > broker? > >

Re: [DISCUSS] PIP-215: Configurable Topic Compaction Strategy

general. - Agreed with your point that we should merge CompactionStrategy APIs. I updated the interface proposal in the PIP. I replaced `"isValid", "isMergeEnabled", and "merge"` apis with "compact" api. Thanks, Heesung On Tue, Nov 1, 2022 at 11:26 AM Hee

Re: [DISCUSS] PIP-215: Configurable Topic Compaction Strategy

Oops! Michael, I apologize for the typo in your name. On Thu, Nov 3, 2022 at 9:47 AM Heesung Sohn wrote: > Hi Machel, > > Here are my additional comments regarding your earlier email. > > - I updated the PIP title to show that this will impact table view as well. > &

Re: [DISCUSS] PIP-215: Configurable Topic Compaction Strategy

ta, and messageId copies. We could initially define `isValid()` only in CompactionStrategy, and add `isMergeEnabled() and merge()` later in the CompactionStrategy interface if requested. Regards, Heesung On Thu, Nov 3, 2022 at 9:50 AM Heesung Sohn wrote: > Oops! Michael, I apologize for the

Re: [DISCUSS] PIP-215: Configurable Topic Compaction Strategy

what the result will do. > > > Regarding redundant deserialization, the input type `T` is the type of > > message value, so the input values are already deserialized. > > Great, I should have realized that. That takes care of that concern. > > Thanks, > Michael >

Re: [DISCUSS] PIP-215: Configurable Topic Compaction Strategy

Thanks, Heesung On Fri, Nov 4, 2022 at 11:11 AM Dave Fisher wrote: > > > > On Nov 4, 2022, at 10:28 AM, Heesung Sohn > > > wrote: > > > > Hi, > > > > I think `bool shouldKeepLeft(T prev, T cur)` is clearer. I updated the > PIP. > > &

[Vote] PIP-215: Configurable TopicCompactionStrategy for StrategicTwoPhaseCompactor and TableView

Dear Pulsar Community, Please review and vote on this PIP. PIP link: https://github.com/apache/pulsar/issues/18099 Thank you, -Heesung

[Discuss] Deprecate Index-based Publisher Stat Aggregation in Topics Partitioned-Stats

Dear Pulsar Community, We recently found a bug in `pulsar-admin topics partitioned-stats api` that could incur a memory burst, high GC time, or OOM. For this issue, I proposed a fix by deprecating the aggregatePublisherStatsByProducerName config and a

[Vote] PIP-220 TransferShedder (only for PIP-192 New Broker Load Balancer)

Dear Pulsar Community, Please review and vote on this PIP. PIP link: https://github.com/apache/pulsar/issues/18215 Thank you, -Heesung

Re: [Discuss] Deprecate Index-based Publisher Stat Aggregation in Topics Partitioned-Stats

tatsByProducerName config but change the default to aggregatePublisherStatsByProducerName=true. Option 4: As a long-term fix, create separate Admin-APIs for publisher and subscription stats and drop their stats from `topics partitioned-stats` as it is expensive to aggregate them on the fly. (for

Re: [Vote] PIP-192 New Pulsar Broker Load Balancer

Dear Pulsar Community, To track sub-PIPs and PRs, I put a project status in the PIP here. https://github.com/apache/pulsar/issues/16691#issuecomment-1318959420 Please help to review them to move this project forward. Thank you, Heesung On Mon, Oct 17, 2022 at 6:07 PM Heesung Sohn wrote

[Discuss] AsyncAPI Pulsar Bindings

Dear Pulsar Community, It appears that there is strong demand for AsyncAPI Pulsar bindings. The discussion is already ongoing, and there is a PR open for this work. PR: https://github.com/asyncapi/bindings/pull/173 Regards, Heesung

Re: [DISCUSS] PIP-215: Configurable Topic Compaction Strategy

Hi, The vote is open in this email thread. https://lists.apache.org/thread/6y8407pw4fv21n2n0cbrvsspg5tok0h7 Regards, Heesung On Fri, Nov 4, 2022 at 3:07 PM Heesung Sohn wrote: > Hi, > > > I would like to understand what the performance tradeoffs are for the > changes

Re: [Discuss] Deprecate Index-based Publisher Stat Aggregation in Topics Partitioned-Stats

d keep room for choice. > > > > > > > > In order to allow users (here Pulsar server-side admins in > particular) > > to > > > > choose the performance or backward-compatibility, how about > > introducing a > > > > "for

Re: [Discuss] Deprecate Index-based Publisher Stat Aggregation in Topics Partitioned-Stats

Hi, I raised this PR to my local fork to sync our direction. https://github.com/heesung-sn/pulsar/pull/15/files Please let me know if this direction looks good. Then, I will proceed to raise an official PR. Thanks, Heesung

Re: [Discuss] Deprecate Index-based Publisher Stat Aggregation in Topics Partitioned-Stats

versions. > For a new major version we COULD break compatibility only as a last resort. > > If we have a way to improve the code and use less resources, I believe > that it is the best way. > > Enrico > > Il giorno mer 30 nov 2022 alle ore 11:56 Heesung Sohn > ha scritto: >

Re: [Discuss] Deprecate Index-based Publisher Stat Aggregation in Topics Partitioned-Stats

On Fri, Dec 9, 2022 at 2:12 PM Heesung Sohn wrote: > Hi, > > I raised a PR to the master branch here, > https://github.com/apache/pulsar/pull/18807. > > PTAL. > > Thank you, > Heesung > > On Wed, Nov 30, 2022 at 4:06 AM Enrico Olivelli > wrote: > >> H

Re: [DISCUSS] PIP-232: Introduce thread monitor to check if thread is blocked for long time.

Hi, This is a great idea. This will significantly help to debug production issues caused by long-waiting threads. I agree with Lari that we could start with the basic metrics. Like Lari shared in the reference, we could simply collect the current thread pool queue size like the following. ((Thr

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

Hi, I assume the deletion APIs are async(when a user requests deletion, pulsar first returns success to the user if the request is persisted. Then, Pulsar asynchronously runs the deletion flow) - Have we considered a metadata store to persist and dedup deletion requests instead of the system top

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

On Tue, Jan 31, 2023 at 6:43 AM Yan Zhao wrote: > > - Have we considered a metadata store to persist and dedup deletion > > requests instead of the system topic? Why is the system topic the better > > choice than a metadata store for this problem? > If we use the metadata store to store the middl

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

peration state, but this will show some eventual consistency because of the delay between step 3 and step 5. Thanks, Heesung > > On Tue, Jan 31, 2023 at 9:06 PM Heesung Sohn > wrote: > > > On Tue, Jan 31, 2023 at 6:43 AM Yan Zhao wrote: > > > > > > - Have we

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

There are some cases to trigger it. 1. A cursor be removed. 2. Close the current ledger and create a new ledger. 3. Consumer ack the message, the slowest cursor move forward. 4. User trigger truncateTopic by admin. I see that this pip is for the internal ledger deletion cases(1-3 above), and it ap

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

2023 at 10:14 AM Heesung Sohn wrote: > > > There are some cases to trigger it. > 1. A cursor be removed. > 2. Close the current ledger and create a new ledger. > 3. Consumer ack the message, the slowest cursor move forward. > 4. User trigger truncateTopic by admin. > >

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

On Sun, Feb 5, 2023 at 2:26 AM Yan Zhao wrote: > On 2023/02/03 18:14:53 Heesung Sohn wrote: > > There are some cases to trigger it. > > 1. A cursor be removed. > > 2. Close the current ledger and create a new ledger. > > 3. Consumer ack the message, the slowest curso

Re: [Discuss] Deprecate Index-based Publisher Stat Aggregation in Topics Partitioned-Stats

Hi dev, If there are no particular concerns, I will send out a vote email. Thanks, Heesung On Tue, Jan 3, 2023 at 10:28 AM Heesung Sohn wrote: > Hi Pulsar dev, > > I raised a PIP for the default config change(change the > aggregatePublisherStatsByProducerName config default

Re: [DISCUSS] PIP-255: Assign topic partitions to bundle by round robin

Hi, Thank you for sharing this. In general, I think this can be another good option for Pulsar load assignment logic. However, I have some comments below. > The load managed by each Bundle is not even. Even if the number of partitions managed by each bundle is the same, there is no guarantee t

Re: [DISCUSS] PIP-255: Assign topic partitions to bundle by round robin

Hi, I see. I have follow-up questions below. - This appears to be the "round-robin topic-to-bundle mapping" option in the `fundBundle` function. Is this the only place that needs an update? Can you list what change is required? - How do we enable this "round-robin topic-to-bundle mapping option"

Re: Pulsar Flaky test report 2023-04-03 for PR builds in CI

Hi, Thank you for the report. We are fixing the errors from ExtensibleLoadManagerImplTest. Regards, Heesung On Tue, Apr 4, 2023 at 3:01 AM Nicolò Boschi wrote: > Dear community, > Here's a report of the flaky tests in Pulsar CI during the observation > period of 2023-03-27 - 2023-04-03 > > >

Re: Pulsar Flaky test report 2023-04-03 for PR builds in CI

o > flaky group would resolve that issue. > Once that it is proven to be stable, we can make a simple update to change > the test group. > I'll proceed with a PR for this soon. > > -Lari > > On 2023/04/04 16:39:21 Heesung Sohn wrote: > > Hi, > > > > T

Re: [DISCUSS] PIP-257: Add Open ID Connect Support to Server Components

Hi, I am new to this security component, but how do we support token revoke? Thanks, Heesung On Mon, Mar 27, 2023 at 12:31 PM Michael Marshall wrote: > Here is the K8s integration: https://github.com/apache/pulsar/pull/19888. > > That PR makes it possible to configure a function running in > k

Re: Pulsar doesn't support cgroup v2 which became the default in Kubernetes v1.25+ (AKS v1.25, GKE v1.26, EKS v1.26)

Hi, +1 for the backport. If there are network bandwidth signals (if the below bandwidth metrics show), I think one could try to weigh more on bandwidth signals like the following. # bandwidth metrics pulsar_lb_bandwidth_in_usage Gauge The broker inbound bandwidth usage (in percent). pulsar_lb_ban

Re: Re:Re: [DISCUSS] PIP-255: Assign topic partitions to bundle by round robin

Hi, As pip-192(load balancer extension) has been added in pulsar-3.0, could you also clarify how this strategy will be compatible with the load balancer extension? >From my understanding, this partition assignment strategy can also be configurable in the load balancer extension. Can you confirm?

  1   2   >