[
https://issues.apache.org/jira/browse/KUDU-3731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18074313#comment-18074313
]
Gabriella Lotz commented on KUDU-3731:
--------------------------------------
h2. Auto-Rebalancer Leader Rebalancing Scaling Test
Cluster: 3 tservers, 1 master. Active flags during test:
auto_leader_rebalancing_interval_seconds=60,
leader_rebalancing_max_moves_per_round=10.
Test table: loadgen_auto_c1d157c5d87c4a54b713b00bb453d683: 90 range partitions,
1 hash bucket, RF=3, 90 tablets total.
h3. What I did
Created a dedicated 90-tablet RF=3 test table using kudu perf loadgen with
--table_num_range_partitions=90 --table_num_hash_partitions=1
--table_num_replicas=3 --keep_auto_table.
Creation was initially rejected with "tablet replicas over maximum permitted at
creation time (180)" because 90 tablets × RF=3 = 270 replicas exceeded
max_create_tablets_per_ts (60) × 3 tservers = 180. Fixed by setting
max_create_tablets_per_ts=100 at runtime before retrying.
With the table live, I started a watch -n 60 loop on ksck and monitored master
log lines matching leader rebalance finish and All tables' leader rebalancing
finished to track per-table per-round transfer counts. I then stopped
tserver-2, let leaders re-elect, brought it back, and recorded the convergence.
h3. Findings
h4. 1. The convergence formula holds.
Both initial equalization (post-creation, leaders skewed onto one tserver by
Raft) and post-bounce recovery converged in exactly 3 rounds = 3 minutes,
matching the prediction:
{code:java}
rounds = ceil(total_excess / leader_rebalancing_max_moves_per_round)
wall_time = rounds × auto_leader_rebalancing_interval_seconds{code}
Observed transfer sequence: 10 → 10 → 8 (initial), then 11 → 10 → 7
(post-bounce). Both hit zero on the 4th round check.
Projection: The formula scales linearly: 300-tablet table predicts 10 rounds,
900-tablet predicts 30 rounds.
h4. 2. auto_leader_rebalancing_interval_seconds defaults to 3600s.
At the default, the same 3-round recovery for a 90-tablet cluster would take 3
hours instead of 3 minutes. A 300-tablet cluster would take 10 hours. This flag
must be set explicitly at startup (e.g.
--auto_leader_rebalancing_interval_seconds=60). It is runtime-tagged so it can
be changed without restart, but not setting it is a silent production hazard.
h4. 3. RF=1 tables are invisible to the auto leader rebalancer.
The leader rebalancer has an early-return guard for RF=1 tables. ksck's "Tablet
Leaders" count includes RF=1 leaders, which can make the cluster look more
imbalanced than what the rebalancer actually acts on. This matters when
interpreting ksck output on clusters with mixed RF tables.
h4. 4. leader_rebalancing_max_moves_per_round is a soft cap, not a hard ceiling.
One round produced 11 transfers against a cap of 10. The outer loop over
overloaded source tservers only logs when the limit is hit but does not break,
so after source-A fills to 10, the outer loop continues to source-B, which can
add a few more tasks. In practice the overshoot is 1–2 RPCs and harmless, but
the flag should be understood as approximate.
h4. 5. Replica rebalancer correctly halted throughout.
Every round while ts2 was down logged Not found: tserver <uuid> not available
for placement and skipped. No spurious replica moves were attempted. Recovery
was immediate once ts2 re-registered.
> Long-running smoke test
> -----------------------
>
> Key: KUDU-3731
> URL: https://issues.apache.org/jira/browse/KUDU-3731
> Project: Kudu
> Issue Type: Sub-task
> Reporter: Gabriella Lotz
> Assignee: Gabriella Lotz
> Priority: Major
>
> h4. Step 0: Start a cluster with 1 master and 3 tablet servers.
> h4. Step 1: Set the following flags.
> --auto_rebalancing_enabled=true
> --auto_rebalancing_interval_seconds=60
> --auto_leader_rebalancing_enabled=true
> h4. Step 2: Create the following tables.
> {code:java}
> kudu table create <master> '{
> "table_name": "user_events",
> "schema": {
> "columns": [
> {"column_name": "user_id", "column_type": "STRING", "is_nullable":
> false},
> {"column_name": "event_id", "column_type": "INT64", "is_nullable":
> false},
> {"column_name": "data", "column_type": "STRING", "is_nullable":
> true}
> ],
> "key_column_names": ["user_id", "event_id"]
> },
> "partition": {
> "hash_partitions": [{"columns": ["user_id", "event_id"], "num_buckets":
> 8}]
> },
> "num_replicas": 3
> }'
> kudu table create <master> '{
> "table_name": "time_series_table",
> "schema": {
> "columns": [
> {"column_name": "ts", "column_type": "UNIXTIME_MICROS",
> "is_nullable": false},
> {"column_name": "sensor_id", "column_type": "STRING",
> "is_nullable": false},
> {"column_name": "value", "column_type": "DOUBLE",
> "is_nullable": true}
> ],
> "key_column_names": ["ts", "sensor_id"]
> },
> "partition": {
> "range_partition": {
> "columns": ["ts"],
> "range_bounds": [
> {"upper_bound": {"bound_type": "exclusive", "bound_values":
> ["1704067200000000"]}},
> {
> "lower_bound": {"bound_type": "inclusive", "bound_values":
> ["1704067200000000"]},
> "upper_bound": {"bound_type": "exclusive", "bound_values":
> ["1735689600000000"]}
> },
> {"lower_bound": {"bound_type": "inclusive", "bound_values":
> ["1735689600000000"]}}
> ]
> }
> },
> "num_replicas": 3
> }' {code}
> h4. Step 3: Start loadgen in tmux
> tmux new -s smoke
> Inside tmux:
> while true; do
> kudu perf loadgen <master> \
> --table_name=user_events \
> --num_threads=4 \
> --num_rows_per_thread=500000 \
> --flush_per_n_rows=1000 \
> --run_cleanup # cleanup so that disk doesn't fill up
> sleep 10
> done
> h4. Step 4: Monitor findings for 2 weeks. (start 2026.03.13.)
> # Is the cluster healthy?
> kudu cluster ksck <master>
> # Check whether auto-rebalancer is running.
> grep -i rebalanc /var/log/kudu/kudu-master.INFO | tail -20
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
