Re: [DISCUSS] PIP-229: Add a common interface to get fields of MessageIdData

Hi Bo, > But when `topicName` is not in the `PulsarApiMessageId`, they should use > `messageId instanceof TopicMessageId` to get the `topicName` The scenario only happens for `acknowledge` APIs because I don't want to add more overloads in PIP-224. The new `seek` overloads in PIP-224 just accept

Re: [VOTE] PIP-228: Refactor the information architecture of Pulsar client docs

+1 (non-binding) Thanks, Yunze On Tue, Dec 20, 2022 at 3:06 PM Zike Yang wrote: > > +1 (non-binding) > > Thanks, > Zike Yang > > On Tue, Dec 13, 2022 at 4:38 PM Jun Ma wrote: > > > > Hi all, > > > > I'm going to start the vote for PIP-228 [Refactor the information > > architecture of Pulsar cl

Re: [VOTE] PIP-228: Refactor the information architecture of Pulsar client docs

+1 (non-binding) Thanks, Zike Yang On Tue, Dec 13, 2022 at 4:38 PM Jun Ma wrote: > > Hi all, > > I'm going to start the vote for PIP-228 [Refactor the information > architecture of Pulsar client > docs](https://github.com/apache/pulsar/issues/18822). > > And this is the original thread for dis

Re: [DISCUSS] PIP-228: Refactor the information architecture of Pulsar client docs

Hi Jun, Thanks for your explanation. Looks good to me. BR, Zike Yang On Mon, Dec 19, 2022 at 2:35 PM Jun Ma wrote: > > Hi Zike, > > Thanks for your feedback! > > This inaccessible link can be removed because the proposed IA has been shown > in the following image. Thanks for pointing it out. >

Re: [DISCUSS] Change the default IO threads and listener threads of Java Client

+1 My concern is whether this change will affect some users who are creating many clients. I think we can wait for other users to confirm it. (If this will be affected, maybe we can give it a max_io_thread_num and then expand the size from 1 to max_io_thread_num when adding a new consumer or pr

[DISCUSS] Change the default IO threads and listener threads of Java Client

Hi all, I noticed the Java Client (I haven't checked other clients) uses 1 IO thread and 1 listener thread by default. It will require users to update the thread configuration if they have multiple cores and desired high throughput. Here is the example that we change to 16 IO threads in openmessa

Re: [DISCUSS] PIP-232: Introduce thread monitor to check if thread is blocked for long time.

With this proposal, I just remembered something. Apache SkyWalking provided a mechanism which called: Thread Profiling The core of the mechanism is start a schedul

Re: [DISCUSS] PIP-232: Introduce thread monitor to check if thread is blocked for long time.

+1 It’s a valuable feature that could help us to improve the stability of pulsar Thanks, Tao Jiuming > 2022年12月19日 20:17，adobewjl 写道： > > Hello pulsar community, > I've opened `PIP-232: Introduce thread monitor to check if thread is blocked > for long time.` to discuss. > For more details, ple

Re: [DISCUSS] Proposal required for Admin API/CLI and metrics changes

+1 The metrics, CLI, Admin API changes will directly face our users. I think a proposal will ask the whole community to consider the rationality of modifying it Not only the people in review know it Thanks, Tao Jiuming > 2022年12月7日 11:38，PengHui Li 写道： > > Hi all, > > I would like to start

Re: [DISCUSS] PIP-231: Add metric for topic load failed

Hi Asaf, > 1. When a topic fails to load, what happens exactly at that stage? Does > it switch over to another broker? Is there a retry on the current broker? There are 2 possibilities: 1. Create a new topic by CLI Tools, Admin API or Producer/Consumer auto create In this condition, if a topi

Re: [DISCUSS] Proposal required for Admin API/CLI and metrics changes

Hi Penghui: < All of the discussions here are about the Admin API / CLI changes < he metrics changes are about adding new metrics added, adding new labels, < removing labels, etc. It's not about the implementations. A bug fix usually < happens to the implementation, not the API definition or metri

Re: [DISCUSS] PIP-232: Introduce thread monitor to check if thread is blocked for long time.

Agree +1. I remember I mentioned this idea before https://lists.apache.org/thread/qwcxrr7o29q66dgs4lqkk1nvvvhbjl2s FYI: https://github.com/eclipse-vertx/vert.x/blob/d711e50174f84a06744c855dd29222beaa97bdeb/src/main/java/io/vertx/core/impl/btc/BlockedThreadChecker.java IMO, we can use one propos

Re: [DISCUSS] PIP-229: Add a common interface to get fields of MessageIdData

Hi Yunze : I have no reason to must add the `getTopicName()` into the `PulsarApiMessageId` (or named `MessageIdAdv `). I just thought of a scenario that users would use. For the consumer side, they don't know whether the topic is a partitioned topic or a non-partitioned topic. Maybe they don't car

Re: [PROPOSAL] Website precommit and move the source of docs to the site repo

Hi tison, Thanks for raising this up! Our community had a similar discussion previously and chose to "keep" the doc repo stay in the Pulsar main repo at that time. [1] lists the pros and cons of "keep" and "not keep" solutions. I'm +0 on this proposal because I think the total scores of these t

[DISCUSS] PIP-232: Introduce thread monitor to check if thread is blocked for long time.

Hello pulsar community, I've opened `PIP-232: Introduce thread monitor to check if thread is blocked for long time.` to discuss. For more details, please read the PIP at https://github.com/apache/pulsar/issues/18985 I'm looking forward to hearing what you think. Also the demo PR link at https://

Re: [DISCUSS] Release Pulsar 2.10.3

Hi Lari, I open a PR to cherry-pick this PR. Could you please take a look when you have time? Sincerely, Xiangying On Fri, Dec 16, 2022 at 4:21 PM Lari Hotari wrote: > There's at least one very important PR that wasn't marked for 2.10.3 . > For per

[DISCUSS] Introduce oshi library to sensory OS resources

Hi, All I would like to introduce a new library oshi[1] to help Apache Pulsar sensory OS resources. It can help us to get away from the complex file manipulation and cross-platform compatibility issues in some operating systems. code example:   https://github.com/apache/pulsar/pull/18984 Plea

[GitHub] [pulsar] liguangcheng added a comment to the discussion: geo replicated subscribers

GitHub user liguangcheng added a comment to the discussion: geo replicated subscribers @volfco & @sijie Now whether the replication subscription status is available, the version I tested is 2.8.0, and it does not work normally at present. One site produces 10 messages and consumes two. anothe

[GitHub] [pulsar] volfco added a comment to the discussion: geo replicated subscribers

GitHub user volfco added a comment to the discussion: geo replicated subscribers @merlimat Sure. Prometheus Metrics: ```# TYPE pulsar_replicated_subscriptions_pending_snapshots gauge pulsar_replicated_subscriptions_pending_snapshots{cluster="iad"} 0.0 # TYPE pulsar_replicated_subscriptions_snap

[GitHub] [pulsar] merlimat added a comment to the discussion: geo replicated subscribers

GitHub user merlimat added a comment to the discussion: geo replicated subscribers Can you check on the broker logs if it's enabling the replicated subscription on that topic? GitHub link: https://github.com/apache/pulsar/discussions/18983#discussioncomment-4447451 This is an automati

[GitHub] [pulsar] volfco added a comment to the discussion: geo replicated subscribers

GitHub user volfco added a comment to the discussion: geo replicated subscribers @merlimat These logs tell me that replication has been enabled: ``` 4:55:14.350 [bookkeeper-ml-workers-OrderedExecutor-3-0] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://develo

[GitHub] [pulsar] merlimat added a comment to the discussion: geo replicated subscribers

GitHub user merlimat added a comment to the discussion: geo replicated subscribers > This is not what I'm seeing. I can start the consumer in AUS, it'll start > reading. I close it at message 10 and wait a few seconds and start an IAD > consumer. Ok, missed that you mentioned you're waiting

[GitHub] [pulsar] merlimat added a comment to the discussion: geo replicated subscribers

GitHub user merlimat added a comment to the discussion: geo replicated subscribers Ok, that doesn't look correct then. Can you enable debug logs on `org.apache.pulsar.broker.service.persistent.{ReplicatedSubscriptionsController,ReplicatedSubscriptionSnapshotCache,ReplicatedSubscriptionsSnapsho

[GitHub] [pulsar] sijie added a comment to the discussion: geo replicated subscribers

GitHub user sijie added a comment to the discussion: geo replicated subscribers @volfco you might check out the replicated subscription introduced in 2.4.0. that can potentially solve you problem. GitHub link: https://github.com/apache/pulsar/discussions/18983#discussioncomment-4447446

[GitHub] [pulsar] volfco added a comment to the discussion: geo replicated subscribers

GitHub user volfco added a comment to the discussion: geo replicated subscribers @sijie I'll try that tomorrow and let you know if it meets my need. I didn't see that in my search of the docs GitHub link: https://github.com/apache/pulsar/discussions/18983#discussioncomment-4447447 This

[GitHub] [pulsar] merlimat added a comment to the discussion: geo replicated subscribers

GitHub user merlimat added a comment to the discussion: geo replicated subscribers Keep in mind that the subscription state in asynchronously replicated at configurable interval. Take a look at broker.conf: ``` # Frequency of snapshots for replicated subscriptions tracking. replicatedSubscri

[GitHub] [pulsar] aahmed-se added a comment to the discussion: geo replicated subscribers

GitHub user aahmed-se added a comment to the discussion: geo replicated subscribers There has been some discussion on geo replicated subscriptions @merlimat can provide more insights. GitHub link: https://github.com/apache/pulsar/discussions/18983#discussioncomment-4447444 This is an a

[GitHub] [pulsar] volfco added a comment to the discussion: geo replicated subscribers

GitHub user volfco added a comment to the discussion: geo replicated subscribers @sijie I've gotten a MVP cluster setup between two DCs that are ~40ms apart with replication between them. This is my code (I'm horrible at java): ```java import org.apache.pulsar.client.api.*; public class Main

[GitHub] [pulsar] volfco added a comment to the discussion: geo replicated subscribers

GitHub user volfco added a comment to the discussion: geo replicated subscribers The one thing I thought of in the background is that messages would have to be replicated with origin DC and origin ID or part of an Avro schema- as currently message IDs are unique per DC (from my understanding)

[GitHub] [pulsar] volfco created a discussion: geo replicated subscribers

GitHub user volfco created a discussion: geo replicated subscribers One of the big things missing in my book, is a way to have multiple subscribers across datacenters consume messages "exactly" once. DC A produces 5 messages. DC A has a consume subscription named TEST. DC B has a consumer wi

Re: [PROPOSAL] Website precommit and move the source of docs to the site repo

Thanks for your feedback! @Asaf > pre-commit I mean CI checks before merging a patch. Currently, we don't run checks for the content before merging them. This causes a series of syntax errors and broken links issues. If we hold docs under site2 folder in the main repo and then copied to the site

[VOTE] PIP-230: Throw exception when MessageIdImpl and BatchMessageIdImpl compare with each othe

Hi, pulsar community: I start a PIP about `Throw exception when MessageIdImpl and BatchMessageIdImpl compare with each other` discussion thread: https://lists.apache.org/thread/oylkh76z1wv8h754c1tvtnspwjojqwmx PIP: https://github.com/apache/pulsar/issues/18957 PR: https://github.com/apache/pulsa

Re: [DISCUSS] PIP-231: Add metric for topic load failed

I have several context-related questions: 1. When a topic fails to load, what happens exactly at that stage? Does it switch over to another broker? Is there a retry on the current broker? 2. Is there a way to provide a gauge showing how many topics are unloaded due to non-recoverable

Re: [DISCUSS] PIP-230: Modify MessageIdImpl and BatchMessageIdImpl compareTo(MessageId o) method

Hi, Xiangying: I have to change the PIP when `MessageIdImpl compareTo(BatchMessageIdImpl)` or `BatchMessageIdImpl compareTo(MessageIdImpl)` will throw an exception. Thanks, Bo Xiangying Meng 于2022年12月19日周一 08:13写道： > > Hi Bo > I am confused about this proposal. > Why do there exist MessageID an

Re: [DISCUSS] PIP-230: Modify MessageIdImpl and BatchMessageIdImpl compareTo(MessageId o) method

Hi Yubiao: > I'm a little curious why the message will lose, could you explain it? for example, user have two messageId batchMessageId = (ledgerId = 1, entryId = 1, batchIndex = 0, batchSize = 20) messageId = (ledgerId = 1, entryId = 1, batchSize = 20) if the user wants to ack the smallest messag

Re: [PROPOSAL] Website precommit and move the source of docs to the site repo

+1 I support moving them to the website repo. Thanks, Penghui On Mon, Dec 19, 2022 at 12:04 PM Yunze Xu wrote: > +1. The most significant point to me is that we can preview all the > content of the website without synchronizing contents from the > apache/pulsar repo. > > Thanks, > Yunze > > On

Re: [DISCUSS] PIP-231: Add metric for topic load failed

+1 Penghui On Mon, Dec 19, 2022 at 4:20 PM Jiuming Tao wrote: > Hello pulsar community, > > I've opened `PIP-231: Add metric for topic load failed` to discuss. > > Motivation: > Currently, we have topic_load_times > > metri

Re: [DISCUSS] Proposal required for Admin API/CLI and metrics changes

Hi Bo, > yes, need a proposal for Admin API/CLI and metrics changes. but It is > difficult for us to judge whether a PR is a bug fix or needs PIP or > doc/required through automatic detection. We can standardize the > committer review process and increase the description of the review > document,

[DISCUSS] PIP-231: Add metric for topic load failed

Hello pulsar community, I've opened `PIP-231: Add metric for topic load failed` to discuss. Motivation: Currently, we have topic_load_times metric to tracking how long a topic load succeed. But when loading a topic, there are