Hi.
The proposed approach to improve rebalancing efficiency looks a little bit
naive to me.
A mature solution to a problem should include a computation of a "cost" for
each rebalancing type.
The cost formula should take into consideration various aspects, like
number of pages in the memory cache
>
> I think we can modify the heuristic so
> 1) Exclude partitions by threshold (IGNITE_PDS_WAL_REBALANCE_THRESHOLD -
> reduce it to 500)
> 2) Select only that partition for historical rebalance where difference
> between counters less that partition size.
Agreed, let's go this way.
On Thu, Jul
I completely forget about another promise to favor of using historical
rebalance where it is possible. When cluster decided to use a full balance,
demander nodes should clear not empty partitions.
This can to consume a long time, in some cases that may be compared with a
time of rebalance.
It also
Ivan,
I agree with a combined approach: threshold for small partitions and count
of update for partition that outgrew it.
This helps to avoid partitions that update not frequently.
Reading of a big WAL piece (more than 100Gb) it can happen, when a client
configured it intentionally.
There are no
Hi Vladislav,
Thanks for raising this topic.
Currently present IGNITE_PDS_WAL_REBALANCE_THRESHOLD (default is 500_000)
is controversial. Assuming that the default number of partitions is 1024,
cache should contain a really huge amount of data in order to make WAL
delta rebalancing possible. In fac
Hi guys,
I want to implement a more honest heuristic for historical rebalance.
Before, a cluster makes a choice between the historical rebalance or not it
only from a partition size. This threshold more known by a name of property
IGNITE_PDS_WAL_REBALANCE_THRESHOLD.
It might prevent a historical r
