Hi,
As pip-192(load balancer extension) has been added in pulsar-3.0,
could you also clarify how this strategy will be compatible with the load
balancer extension?
>From my understanding, this partition assignment strategy can also be
configurable in the load balancer extension. Can you confirm?
This proposal is easier to understand than before. Overall LGTM. But I
think these `onBundleXXX` methods could be default so that we can
implement it with a simple lambda.
Thanks,
Yunze
On Wed, Apr 19, 2023 at 10:22 AM Lin Lin wrote:
>
> We make this configuration item to be a dynamic configura
We make this configuration item to be a dynamic configuration.
We change change it on broker level.
If we can change it on namespace level, even load of bundle in some namespace
is balanced, it is still difficult to make broker balance
On 2023/04/16 16:07:45 lifepuzzlefun wrote:
> I think this
I think this feature is very helpful on heavy traffic topic which have
continuous stable load on each partition.
Is there a way we can set some kind of namespace policy to set the plugin
PartitionAssigner. Hope this can be set on namespace level,
if this can be achieved, it is more adoptable t
As I mentioned in the implementation of PIP, we will plug-in the partition
assignment strategy.
However, in the same cluster, it is impossible for some Brokers to use
consistent hashing and some Brokers to use round robin.
On 2023/04/11 07:37:19 Xiangying Meng wrote:
> Hi Linlin,
> > This is a
Hi Linlin,
> This is an incompatible modification, so the entire cluster needs to be
upgraded, not just a part of the nodes
Appreciate your contribution to the new feature in PIP-255.
I have a question regarding the load-balancing aspect of this feature.
You mentioned that this is an incompatibl
> Do we expect that the bundles should have the same loads?
Yes.If the bundle load is similar, it will be easier to achieve a balanced
state during subsequent load balancing.
If the bundle loads are different or there are hot bundles, it will trigger
bundle split or unload.
Unload the hotspot bu
Hi Lin,
> The load managed by each Bundle is not even. Even if the number of
partitions managed
by each bundle is the same, there is no guarantee that the sum of the
loads of these partitions
will be the same.
Do we expect that the bundles should have the same loads? The bundle is the
base
> This appears to be the "round-robin topic-to-bundle mapping" option in
> the `fundBundle` function. Is this the only place that needs an update? Can
> you list what change is required?
In this PIP, we only discuss topic-to-bundle mapping
Change is required:
1)
When lookup, partitions is assign
The namespace level bundle-unload can be performed in
NamespaceService#splitAndOwnBundleOnceAndRetry
A new judgment will be added here.
After splitting the bundle, it should determine whether to unload at the
namespace level.
On 2023/03/19 09:53:07 lifepuzzlefun wrote:
> I'm interest on the imp
Hi, I see. I have follow-up questions below.
- This appears to be the "round-robin topic-to-bundle mapping" option in
the `fundBundle` function. Is this the only place that needs an update? Can
you list what change is required?
- How do we enable this "round-robin topic-to-bundle mapping option"
Thanks for joining this discussion
> 1. where is the partition to bundle mapping stored？
We don't need to store the mapping relationship, it can be calculated
dynamically. The first is the starting bundle, partition-0 is calculated
directly through consistent hashing. Subsequent partitions a
I'm interest on the implementation details.
1. where is the partition to bundle mapping stored？when upgrade origin logic to
the new round robin logic. how the current code distinguish partition assigned
by origin logic and the new created topic assign by round robin logic.
2. can you explain
Hi, pulsar community. I'm insterest in this topic and want to join the
discussion : - )
I test to create one partitioned topic with 1024 partitions in a namespace with
256 bundle.
the distribution for this topic is post below.
key is the total assigned partition number to the bundle.
value is
Thanks for your review.
> Could you clarify the limitation of the current logic?
The current logic cannot guarantee that the traffic of each bundle is the same,
and must be balanced through split.
However, the load of the topic is not always the same, and the traffic of the
business changes wi
Hi,
Thank you for sharing this.
In general, I think this can be another good option for Pulsar load
assignment logic.
However, I have some comments below.
> The load managed by each Bundle is not even.
Even if the number of partitions managed by each bundle is the same,
there is no guarantee t
Hi all,
I created a proposal to
assign topic partitions to bundles by round robin:
https://github.com/apache/pulsar/issues/19806
It is already running in our production environment,
and it has a good performance.
Thanks!
17 matches
Mail list logo
