-385.md
Discussion Thread:
https://lists.apache.org/thread/9wddmj4o5mrdst427r40rr7phqb05y6s
Please review the PIP document and vote!
Regards
--
Girish Sharma
omments on the
> proposal with few questions and suggestions.
>
> Thank,
> Rajan
>
> On Tue, Oct 29, 2024 at 10:48 PM Girish Sharma
> wrote:
>
> > Hello everyone, gentle reminder.
> >
> > If there are no further comments, then please close the github comments
>
che.org/blog/2024/10/24/announcing-apache-pulsar-4-0/#enhanced-quality-of-service-qos-controls
>
> -Lari
>
> On 2024/10/10 11:23:30 Girish Sharma wrote:
> > I've updated the proposal with suggestions from Lari about utilization
> > based rate limit exceptions on clie
> > > (at
> > > > >> least 1 binding VOTE) and non-negative binding VOTE can move
> forward
> > > > after
> > > > >> waiting for max 1 month. This way , we can still have an approved
> > > review
> > > > >> from binding VOTE and max time can give contributors hope to get
> their
> > > > >> change available to get the benefit of Pulsar for their
> organization.
> > > > >> We should really improve the process as it is really painful for
> the
> > > > org or
> > > > >> contributors who have to wait such long for useful changes in
> Pulsar.
> > > > >>
> > > > >> Thanks,
> > > > >> Rajan
> > > > >>
> > > > >>
> > > > >> On Thu, Sep 5, 2024 at 8:08 AM Lari Hotari
> > > wrote:
> > > > >>
> > > > >>> +1 (binding)
> > > > >>>
> > > > >>> -Lari
> > > > >>>
> > > > >>> On 2024/09/04 04:38:01 Rajan Dhabalia wrote:
> > > > >>>> Hi,
> > > > >>>>
> > > > >>>> I would like to start a voting thread for PIP-326 to support the
> > > admin
> > > > >>> API
> > > > >>>> to read schema metadata and display in readable format.
> > > > >>>>
> > > > >>>> PIP design PR:
> > > > >>>> https://github.com/apache/pulsar/pull/22913
> > > > >>>>
> > > > >>>>
> > > > >>>> Thread:
> > > > >>>>
> https://lists.apache.org/thread/8s8m6k7oprmkn3jpblgxqkdh6d8z43x2
> > > > >>>>
> > > > >>>> Thanks,
> > > > >>>> Rajan
> > > > >>>>
> > > > >>>
> > > >
> > > >
> > >
> >
>
--
Girish Sharma
p will have to make some assumptions and oversubscribe the available
memory between those 100 clients which can lead to OOM if many partitions
are throttling.
Hope this helps and gives more context around how the PIP is useful.
Regards
On Sat, Oct 5, 2024 at 12:53 PM Girish Sharma
wrote:
>
; control. This addition will improve the ability to set and meet SLAs
> across
> > various Pulsar use cases, which is invaluable for many of our users.
> >
> > Thank you for driving this important improvement. It's contributions like
> > these that continue to enhance Pu
-protocol/pip/pip-385.md
Please share your thoughts on this proposal along with any concerns or
suggestions.
Regards
--
Girish Sharma
; Benefits:
> > > > 1. Improved message ordering guarantees
> > > > 2. Reduced unnecessary message blocking
> > > > 3. Better scalability and performance
> > > > 4. Enhanced observability
> > > >
> > > > This proposal would replace the existing "recently joined consumers"
> > > > mechanism, addressing its limitations while providing a more robust
> > > > solution.
> > > >
> > > > The full proposal can be found at:
> > > > https://github.com/apache/pulsar/pull/23309
> > > > The direct link to the rendered version of the markdown file is:
> > > > https://github.com/lhotari/pulsar/blob/lh-pip-379/pip/pip-379.md
> > > >
> > > > I welcome your feedback and discussion on this proposal. Please share
> > > > your thoughts, concerns, or suggestions.
> > > >
> > > > -Lari
> > > >
> > >
> >
>
--
Girish Sharma
Hi Tao,
We are planning to get a PIP out in this quarter.
Regards
On Mon, Jul 22, 2024 at 1:35 PM Tao Jiuming wrote:
> Hi,
>
> How's the proposal going?
>
> On 2024/01/19 06:28:14 Girish Sharma wrote:
> > Hello everyone,
> >
> > A a true cloud native platf
at's the ROI there?
So I would say that in your proposal, option 2 (current) is more accurate
(while not being the best) than option 1.
Regards
--
Girish Sharma
One suggestion, I think you can make do with storing just begin timestamp.
Any search utilising these values will work the same way with just one of
those timestamps compared to both begin and end.
Any particular reason you need both the timestamps?
Regards
On Fri, Mar 15, 2024, 9:39 AM 太上玄元道君
the
> > > > Pulsar base image with quite a bit of dependencies, from
> > `pulsar-client`
> > > > Python SDK, to gRPC which is quite a heavy package with many
> transitive
> > > > dependencies.
> > > >
> > > > Given that the vast majority would be using the `pulsar` base image
> to
> > > run
> > > > brokers and not python functions, it would make sense to split the
> > Python
> > > > support into a different image, like `pulsar-functions-python`, which
> > > > extends from the base image and adds all the needed Python
> > dependencies.
> > > >
> > > > This way it will be very easy for users to select the appropriate
> image
> > > and
> > > > we wouldn't be carrying a big amount of useless Python dependencies
> to
> > > > users who don't need them.
> > > >
> > > >
> > > > What are people's opinions with respect to this?
> > > >
> > > > Matteo
> > > >
> > > > --
> > > > Matteo Merli
> > > >
> > > >
> > >
> >
>
>
> --
> Best Regards,
> Neng
>
--
Girish Sharma
On Thu, Mar 7, 2024 at 11:38 AM Yunze Xu wrote:
> Regarding PIP-332 and PIP 310, similar to PIP-337, there is no
> discussion mail in the dev mail list. David left a comment [1] in
>
There is for 310 -
https://lists.apache.org/thread/13ncst2nc311vxok1s75thl2gtnk7w1t
Regards
--
Girish Sharma
On Fri, Feb 16, 2024 at 6:31 PM Girish Sharma
wrote:
> There have been a few discussions in the past on the slack channel and I
> recently also started a similar thread [0] regarding if we can skip certain
> releases while upgrading towards pulsar 3.0 and beyond. Starting this dev
>
;
> > What do you think about migrating CLI parse from jcommander to picocli?
> >
> > Thanks,
> > Zixuan
> >
> > [1] - https://github.com/cbeust/jcommander
> > [2] - https://picocli.info/
> > [3] - https://github.com/remkop/picocli/wiki/picocli-vs-JCommander
>
--
Girish Sharma
of the broker - from
dedupe, to transactions and beyond. I think almost all of the broker level
feature rely on the fact that a partition will always be owned by a single
topic at any given time. This will lead to an active partition for a single
partition across brokers..
--
Girish Sharma
ecome a committer and we
> > are pleased to announce that he has accepted.
> >
> > Welcome and Congratulations, Asaf Mesika!
> >
> > Please join us in congratulating and welcoming Asaf onboard!
> >
> > Best Regards,
> >
> > Lari Hotari
> > on behalf of the Pulsar PMC
> >
>
--
Girish Sharma
n order to align the
recommendation.
[0] https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1705392242948349
[1]
https://pulsar.apache.org/contribute/release-policy/#compatibility-between-releases
[2] https://pulsar.apache.org/blog/2024/02/12/announcing-apache-pulsar-3-2/
--
Girish Sharma
; > > > Hi Pulsar Community,
> > > > >
> > > > > Do we consider the 2.10 release line EOL? If not, is there a
> > > > > committer that would like to volunteer to release 2.10.6?
> > > > >
> > > > > We briefly discussed keeping 2.10 alive in June [0], and that was
> > > > > followed by a 2.10.5 release in July. Given that we already have
> > > > > 2.11, 3.0, 3.1, and now a discussion on 3.2, it seems
> > > > > unsustainable to keep
> > > > > 2.10 going much longer.
> > > > >
> > > > > Thanks,
> > > > > Michael
> > > > >
> > > > > [0]
> > > > > https://lists.apache.org/thread/w4jzk27qhtosgsz7l9bmhf1t7o9mxjhp
> > > > >
> > >
> >
>
--
Girish Sharma
mment for the same here -
https://github.com/apache/pulsar/pull/22039#discussion_r1487332735
On Tue, Feb 13, 2024 at 12:53 PM Lari Hotari wrote:
> On 2024/02/13 06:46:29 Girish Sharma wrote:
> > Personally, while this may be a much cleaner approach or may be not solve
> > the core
ified that there is a bug in the
current code. We are trying to do a bug fix here. The goal is not to deep
dive into a much bigger design problem as we want to hotfix this ASAP in
our system, but also want an alignment with the community so as to not
maintain this patch locally, internally, for every version we upgrade to.
> -Lari
>
> 1 - https://github.com/apache/pulsar/pull/12056
> 2 - https://github.com/apache/pulsar/pull/12072
> 3 - https://github.com/apache/pulsar/pull/12072#issuecomment-921663472
>
>
--
Girish Sharma
Reply inline, and also replied to the GH comment.
On Mon, Feb 12, 2024 at 9:37 PM Lari Hotari wrote:
> The confusing detail is that in PIP-61 [1], the alternative that has been
> implemented in the Pulsar code base has been marked as the rejected
> alternative ("Return all advertised listeners(r
ka via re-mapping). For that
purpose, I've kept it out of scope. But what I would ensure is that adding
this new feature of partitions scale down is not increasing the complexity
or difficulty of providing seamless partition count change for ordered
consumption in future.
--
Girish Sharma
Bumping this up! Hoping this can be discussed so that I get rule out that
this approach has any fatal flaws.
Regards
On Fri, Jan 19, 2024 at 11:58 AM Girish Sharma
wrote:
> Hello everyone,
>
> A a true cloud native platform, which supports scale up and scale down, I
> feel like the
/retention)
Goal is that there is no data movement involved and no impact on existing
partitions during this scale down.
Looking forward to the discussion.
[0]
https://docs.google.com/document/d/1sbGQSwDihQftIRsxAXg5Zm4uxKQ0kRk9HadKYRFTswI/edit?usp=sharing
Regards
--
Girish Sharma
refactoring rate limiter codebase which is
already covered by Lari in PIP-322.
I will be creating a new parent PIP soon about the high level requirements.
Thank you everyone who participated in the thread and the discussion on
23rd dev meeting.
Regards
On Thu, Nov 23, 2023 at 8:26 PM Girish Sharma
ead:
> https://lists.apache.org/thread/p559tsphr7kvbh2qqw8vsow0ylytonnz
>
>
>
>
>
>
>
>
>
> Ruihong
--
Girish Sharma
+1 (non-binding)
On Mon, Dec 11, 2023 at 3:49 PM PengHui Li wrote:
> +1 (binding)
>
> Just one minor comment on the proposal PR
>
> Regards,
> Penghui
>
--
Girish Sharma
n,
then instead of following a long exercise of moving that topic to a new
namespace, you can use this feature.
[0] - https://github.com/apache/pulsar/pull/12136
>
> On Thu, Dec 7, 2023 at 2:28 AM Girish Sharma
> wrote:
>
> > Hello, replies inline.
> >
> > On Wed, D
d, Dec 6, 2023 at 12:49 PM Joe F wrote:
>
> > Girish,
> >
> > Thank you for making my point much better than I did ..
> >
> > -Joe
> >
> > On Tue, Dec 5, 2023 at 1:45 AM Girish Sharma
> > wrote:
> >
> > > Hello Xiangying,
> >
are the replication-clusters.
> > >
> > > >Allowed clusters defined at tenant level
> > > >will restrict tenants to create namespaces in regions/clusters where
> > they
> > > >are not allowed.
> > > >As Rajan also mentioned, allowed-cl
8
>
> I'm looking forward to hearing from you.
>
> BR
> Xiangying
>
--
Girish Sharma
> Hi Girish,
> >
> > replies inline and after that there are some updates about my
> > preparation for the community meeting on Thursday. (there's
> > https://github.com/lhotari/async-tokenbucket with a PoC for a
> > low-level high performance token bucket imp
rt :) . At most if things don't go as desired,
then we would end up doing this and adding plugging logic in a pulsar fork.
>From a PoC perspective, we may try it out soon.
> > Would love to hear a plan from your point of view and what you envision.
>
> Thanks, I hope I was able to express most of it in these long emails.
> I'm having a break next week and after that I was thinking of
> summarizing this discussion from my viewpoint and then meet in the
> Pulsar community meeting on November 23rd to discuss the summary and
> conclusions and the path forward. Perhaps you could also prepare in a
>
Sounds like a plan!
> similar way where you summarize your viewpoints and we discuss this on
> Nov 23rd in the Pulsar community meeting together with everyone who is
> interested to participate. If we have completed the preparation before
> the meeting, we could possibly already exchange our summaries
> asynchronously before the meeting. Girish, Would this work for you?
>
> Yes, we can exchange it before 23rd. I can come back on my final
requirements and plan by end of next week.
Regards
--
Girish Sharma
of refactoring and changes in the
> based
> > design as we have been discussing here.
>
> I appreciate your efforts on initiating PIP-310 and getting the ball
> rolling. Since you are operating Pulsar at scale, your contributions
> and feedback are very valuable in improving Pulsar's capacity manage.
> I happen to have a different view of how a custom rate limiter
> implemented with the possible pluggable interface could help with
> overall capacity management in Pulsar.
> We need to go beyond PIP-310 in solving multi-tenant capacity
> management/SOP/SLA challenges with Pulsar. The resource groups work
> started with PIP-81 is a good start point, but there's a need to
> improve and revisit the design to be able to meet the competition, the
> closed source Confluent Kora.
>
As mentioned in the comment above, resource groups shouldn't be discussed
here.. they are out of scope of the discussion involving partition level
rate limiting and I did not intend to bring them into the discussion.
I would like to see a comment on my proposal about the state of pulsar rate
limiter. I believe that's true "meeting in the middle".
>
> Thanks for providing such detailed and useful feedback! I think that
> this has already been a valuable interaction.
>
Thank you for painstakingly replying to these long emails.. This probably
is the longest thread in pulsar ML in recent times :)
>
> The improvements happen one step at a time. We can make things happen
> when we work together. I'm looking forward to that!
>
Would love to hear a plan from your point of view and what you envision.
>
> -Lari
>
--
Girish Sharma
Hello Lari, replies inline
On Thu, Nov 9, 2023 at 6:50 AM Lari Hotari wrote:
> Hi Girish,
>
> replies inline.
>
> On Thu, 9 Nov 2023 at 00:29, Girish Sharma
> wrote:
> > While dual-rate dual token bucket looks promising, there is still some
> > challenge with resp
ady- all be it
refilling only every 1 second, rather than distributing the tokens across
that second.
> algorithm is. That could be a starting point until we start covering
> the advanced cases which require a dual token bucket algorithm.
> If someone has the bandwidth, it would be fine to start experimenting
> in this area with code to learn more.
>
>
I think this is a big assumption. Since this is a critical use case in my
organisation, I will have to contribute everything here myself. Now I do
understand that reviews can take time in the OSS world, but this can't be
left at "simple token based approach and then letting anyone pick and
explore to extend/enhance it to dual token approach". This probably is one
of the main reasons why pluggability is important here :)
--
Girish Sharma
extra time in discussing and closing the improved rate limiter design. Of
course, I will keep the interface definition in mind while proposing the
improved rate limiter and vice versa.
What are your thoughts here?
>
>
> -Lari
>
> On Wed, 8 Nov 2023 at 10:46, Girish Sharma
> wrot
t different types of
> implementation.
>
> Thanks,
> Rajan
>
> On Tue, Nov 7, 2023 at 10:02 AM Lari Hotari wrote:
>
> > Hi Girish,
> >
> > Replies inline.
> >
> > On Tue, 7 Nov 2023 at 15:26, Girish Sharma
> > wrote:
> > >
> >
n't visibile and it's also very
> hard to observe.
> In a case where 2 producers with very different rate limiting options
> share a connection, it definetely is a problem in the current
>
Yes actually, I was talking to the team and we did observe a case where
there was a client app with 2 producer objects writing to two different
topics. When one of their topics was breaching quota, the other one was
also observing rate limiting even though it was under quota. So eventually,
this surely needs to be solved. One short term solution is to at least not
share connections across producer objects. But I still feel it's out of
scope of this discussion here.
What do you think about the quick solution of not sharing connections
across producer objects? I can raise a github issue explaining the
situation.
--
Girish Sharma
o more details of the intended behavior
> and define what bursting really means and how we believe that solves a
> problem with "hot source" producers.
> Makes sense?
>
>
I am happy if the discussion concludes eitherways - a pluggable
implementation or a 99% use-case capturing configurable rate limiter. But
from what I've seen, the participation in OSS threads can be very random
and thus, I am afraid it might take a while before more folks pitch in
their inputs and a clear direction of discussion is formed.
--
Girish Sharma
rive, they can easily be handled by
moving the different producers to different topic and the consumers
subscribing to more than one topic in case they need data from all of those
topics.
Regards
>
> -Lari
>
> la 4. marrask. 2023 klo 17.24 Lari Hotari kirjoitti:
>
> > Replies i
iting aspect.
>
> Are there other community members with input on the design and
> implementation of an improved rate limiter?
> I’m eager to continue this conversation and work together towards a robust
> solution.
>
Again, I would love this to land in pieces so that TAT for actual usage is
much faster. What do you suggest from that perspective?
>
> -Lari
>
--
Girish Sharma
improvement area. Together we could find a good solution that works for
> multiple use cases and addresses existing challenges in producer flow
> control and rate limiting.
>
> -Lari
>
> On 2023/11/03 11:16:37 Girish Sharma wrote:
> > Hello Lari,
> > Thanks for bringin
t; Brokers have to reply to clients after receiving and decode the message,
> but the broker also has the back-pressure mechanism. Broker cannot read
> messages because the channel is `disableAutoRead`.
>
> So the rate-limiters have to adapt to back-pressure.
>
> Thanks,
> Tao
ld improve.
> I think it might be time to do so since there's a requirement to improve
> rate limiting. I guess that's the main motivation also for PIP-310.
>
> -Lari
>
> 1 - https://github.com/apache/pulsar/pull/21399/files
> 2 - https://lists.apache.org/thread/
that we can't
use precise rate limiter in the resource group level.
Thus, in order to support custom rate limiters, I've created the PIP-310
This is the discussion thread. Please go through the PIP and provide your
inputs.
Link - https://github.com/apache/pulsar/pull/21399
Regards
ny PRs, actively engaging on the mailing list, triaging GitHub
> > Issues, and helping out with the website.
> >
> > On behalf of the Pulsar PMC, welcome and congratulations to tison!
> >
> > Best,
> > Michael
> >
> --
> Nicolò Boschi
>
--
Girish Sharma
so that the load balancing can be better, utilizing lower
priority consumers as well.
I was also thinking that from a user perspective, one would think that a
lower priority consumer is meant for backup in case one active consumer
goes down - which also doesn't work like that.
Regards
> Re
priority. Example, if ten consumers (consumer priority 1 through 10)
are connected to 10 partitions, all 10 partitions would only send data to
just one of the consumers at any given time.
Regards
> Regards,
> Penghui
>
> On Mon, Jul 3, 2023 at 2:49 PM Girish Sharma
> wrote:
>
Bumping this up. Would really like to discuss this in the community.
Regards
On Wed, Jun 28, 2023 at 11:49 PM Girish Sharma
wrote:
> Hi everyone, I am trying to understand the failover subscription logic a
> bit more in detail. Specifically, the doc
> <https://pulsar.apache.or
sly in above example
Regards
--
Girish Sharma
e new functionality for those who want to use regex
> for deletion.
>
> I hope this clears up any confusion and addresses your concerns.
> Please let me know if you have any further questions or suggestions.
>
> Best regards,
> Xiangying Meng
>
> On Mon, Apr 24, 2023 at 6:23
at 3:24 PM Yubiao Feng
wrote:
> Hi Girish Sharma
>
> > What additional advantage would one get by using that approach
> > rather than simply using a one liner script to just call delete
> > topic for each of those topics if the list of topics is known.
>
> If users
gement
> > for
> > subscribers, and cross-datacenter replication.
> >
> > For Pulsar release details and downloads, visit:
> >
> > https://pulsar.apache.org/download
> >
> > Release Notes are at:
> > https://pulsar.apache.org/release-notes
>
gt; > working with Apache Pulsar.
> >
> > I'd like to discuss the feasibility of this suggestion and gather
> feedback
> > from the community.
> > If everyone agrees, I can work on implementing this feature and submit a
> > pull request for review.
> >
> > Looking forward to hearing your thoughts on this.
> >
> > Best regards,
> > Xiangying
> >
>
--
Girish Sharma
and namespace
> structures
> > in place
> > while cleaning up their contents.
> > I hope this clarifies my intention, and I would like to hear your
> thoughts
> > on this proposal.
> >
> > Best regards,
> > Xiangying
> >
> > On Sat, Apr 15, 2023 at
; I'd like to discuss the feasibility of this suggestion and gather feedback
> from the community.
> If everyone agrees, I can work on implementing this feature and submit a
> pull request for review.
>
> Looking forward to hearing your thoughts on this.
>
> Best regards,
> Xiangying
>
--
Girish Sharma
On Fri, Mar 31, 2023 at 7:09 PM Enrico Olivelli wrote:
> I agree that we should finally have PIPs committed to git somewhere.
>
> When a PIP is approved it can be committed, but we must run the VOTE
> and wait for 3 binding +1,
> this is hard to do with a PR.
> It already happened a few times tha
re.
>
> Before you start, you search Pull Requests with label PIP in GitHub (`is:pr
> "PIP-" in:title`)
> Take the biggest number and add 1.
> It is super rare to have two people create PR at the same time.
>
> *Show me all approved PIPs:*
> Search merged PRs labeled PIP.
> Look at "pip" folder
>
> *Show me rejected PIPs:*
> Search closed PRs with "PIP-" in title, or labeled PIP.
>
>
> This is very similar to how Apache BK works.
>
> WDYT?
>
--
Girish Sharma
6:23 PM Asaf Mesika wrote:
> On Sun, Feb 26, 2023 at 2:49 PM Girish Sharma
> wrote:
>
> > Good proposal Asaf.
> > I've also wondered why the PIP creation and discussion process is so
> > separated. The PIP discussion and voting starts off as a GitHub issue,
o achieve your high level
> design
> > * It should include the following if applicable:
> > * REST API Changes
> > * Protocol Changes
> >
> > # Monitoring
> > * Describe exactly what you will add to Pulsar allowing you to
> > monitor/observe this proposal?
> > * If those are metrics, provide the names, description, labels and
> units
> > * Explain what constitutes abnormal that I should pay attention to
> >
> > # Backward Compatibility
> > * Describe exact instructions if someone needs to revert from a version
> > containing it to a previous version
> >
> > # Alternatives
> > * Describe alternative design decisions and why you have not opted for
> them
> >
> > # General notes
> > * Any general notes you wish to make
> >
> > # Links (Updated afterwards)
> > * Mailing List discussion thread:
> > * Mailing List voting thread:
> >
> > ==
> > Would love to hear what you think about it, before opening a PR about
> this.
> >
>
--
Girish Sharma
tra work to
> maintaining alarm rule.
>
> Yike Xiao
>
> ____
> 发件人: Girish Sharma
> 发送时间: 2022年12月31日 0:07
> 收件人: dev@pulsar.apache.org
> 主题: Re: [DISCUSS] PIP-235: Add metric for subscription back size
>
> How about adding a new label sub
k size · Issue #19112 ·
> apache/pulsar<https://github.com/apache/pulsar/issues/19112>
> Motivation Now we have pulsar_storage_backlog_size for topic backlog size,
> user can create an alarm rule like pulsar_storage_backlog_size >
> THRESHOLD, typically this alarm is going to notify cor...
> github.com
>
>
>
> Yike Xiao
>
--
Girish Sharma
discussions emails from GitBox do not thread properly as
the subject of the email contains unique text like "XYZ commented on thread
"
2. There is already a way to subscribe to github discussions via GitHub
UI so these emails are duplicated.
Regards
--
Girish Sharma
65 matches
Mail list logo
