[
https://issues.apache.org/jira/browse/KUDU-3731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18068959#comment-18068959
]
Gabriella Lotz commented on KUDU-3731:
--------------------------------------
h3. Scenario 1: Single Tablet Server Down and Back Up
Setup
Cluster: 1 master, 3 tablet servers
Table: user_events, RF=3, 8 tablets (hash-partitioned)
h3. What was tested
TServer 3 was taken offline. After ~10 minutes it was brought back up. Cluster
behavior was observed throughout via ksck and master logs.
h3. Observations
h4. While TServer was down
All 8 tablets transitioned to UNDER_REPLICATED immediately. The two remaining
TServers maintained quorum (RF=3, majority=2) so writes continued. The
auto_rebalancer correctly skipped the unavailable server every iteration:
{code:java}
W auto_rebalancer.cc:222] Could not retrieve cluster info:
Not found: tserver 9ae997b69e694e3a8c508628d98a7ee7 not available for
placement{code}
After 300 seconds (follower_unavailable_considered_failed_sec), the catalog
manager began attempting to replace the missing replica via ADD_PEER:NON_VOTER.
This attempt failed continuously and produced misleading log spam:
{code:java}
W catalog_manager.cc:5161] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ...:
Not found: could not select location for extra replica:
not enough tablet servers to satisfy replica placement policy:
the total number of registered tablet servers (3) does not allow for adding
an extra replica; consider bringing up more to have at least 4 tablet
servers{code}
Root cause of the failure: The ADD_PEER:NON_VOTER recovery mechanism works by
temporarily adding a 4th copy of the replica on a spare server. With RF=3
across exactly 3 tservers, every server already hosts a copy of every tablet,
there is no spare server. This makes the entire ADD_PEER recovery path
permanently unavailable in this configuration. The warning retried every ~60
seconds for the entire duration the TServer was down.
h4. After TServer came back up
The TServer re-registered with the master and its replicas rejoined the Raft
consensus groups via normal follower catch-up (WAL replay). All 8 tablets
transitioned back to HEALTHY without manual intervention. The auto leader
rebalancer then redistributed tablet leaders:
{code:java}
I auto_leader_rebalancer.cc:359] table: user_events,
leader rebalance finish, leader transfer count: 0{code}
Final leader distribution: 3 / 3 / 2 across the servers - acceptably balanced.
Total recovery time: approximately 10 minutes from TServer restart to full
cluster health.
> Long-running smoke test
> -----------------------
>
> Key: KUDU-3731
> URL: https://issues.apache.org/jira/browse/KUDU-3731
> Project: Kudu
> Issue Type: Sub-task
> Reporter: Gabriella Lotz
> Assignee: Gabriella Lotz
> Priority: Major
>
> h4. Step 0: Start a cluster with 1 master and 3 tablet servers.
> h4. Step 1: Set the following flags.
> --auto_rebalancing_enabled=true
> --auto_rebalancing_interval_seconds=60
> --auto_leader_rebalancing_enabled=true
> h4. Step 2: Create the following tables.
> {code:java}
> kudu table create <master> '{
> "table_name": "user_events",
> "schema": {
> "columns": [
> {"column_name": "user_id", "column_type": "STRING", "is_nullable":
> false},
> {"column_name": "event_id", "column_type": "INT64", "is_nullable":
> false},
> {"column_name": "data", "column_type": "STRING", "is_nullable":
> true}
> ],
> "key_column_names": ["user_id", "event_id"]
> },
> "partition": {
> "hash_partitions": [{"columns": ["user_id", "event_id"], "num_buckets":
> 8}]
> },
> "num_replicas": 3
> }'
> kudu table create <master> '{
> "table_name": "time_series_table",
> "schema": {
> "columns": [
> {"column_name": "ts", "column_type": "UNIXTIME_MICROS",
> "is_nullable": false},
> {"column_name": "sensor_id", "column_type": "STRING",
> "is_nullable": false},
> {"column_name": "value", "column_type": "DOUBLE",
> "is_nullable": true}
> ],
> "key_column_names": ["ts", "sensor_id"]
> },
> "partition": {
> "range_partition": {
> "columns": ["ts"],
> "range_bounds": [
> {"upper_bound": {"bound_type": "exclusive", "bound_values":
> ["1704067200000000"]}},
> {
> "lower_bound": {"bound_type": "inclusive", "bound_values":
> ["1704067200000000"]},
> "upper_bound": {"bound_type": "exclusive", "bound_values":
> ["1735689600000000"]}
> },
> {"lower_bound": {"bound_type": "inclusive", "bound_values":
> ["1735689600000000"]}}
> ]
> }
> },
> "num_replicas": 3
> }' {code}
> h4. Step 3: Start loadgen in tmux
> tmux new -s smoke
> Inside tmux:
> while true; do
> kudu perf loadgen <master> \
> --table_name=user_events \
> --num_threads=4 \
> --num_rows_per_thread=500000 \
> --flush_per_n_rows=1000 \
> --run_cleanup # cleanup so that disk doesn't fill up
> sleep 10
> done
> h4. Step 4: Monitor findings for 2 weeks. (start 2026.03.13.)
> # Is the cluster healthy?
> kudu cluster ksck <master>
> # Check whether auto-rebalancer is running.
> grep -i rebalanc /var/log/kudu/kudu-master.INFO | tail -20
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
