Chovan Zoltan created KUDU-3818:
-----------------------------------
Summary: Master can't restart after interrupted sys catalog
creation
Key: KUDU-3818
URL: https://issues.apache.org/jira/browse/KUDU-3818
Project: Kudu
Issue Type: Improvement
Reporter: Chovan Zoltan
Assignee: Chovan Zoltan
Attachments: test-output.txt
When a new multi-master cluster starts, each master creates the system catalog
tablet in SysCatalogTable::CreateNew(). The steps run in this order:
1. Write the tablet metadata to disk.
2. Resolve the other masters' UUIDs over RPC. This blocks until those masters
are up, for up to --raft_get_node_instance_timeout_ms (30s by default).
3. Write the consensus metadata to disk.
If the master is stopped or crashes between steps 1 and 3, or if step 2 times
out, the tablet metadata is on disk but the consensus metadata isn't. Each
later start then takes the "load existing" path and fails for good:
{noformat}
E20261027 13:19:53.309684 31360 master.cc:417] Unable to init master catalog
manager: Not found: Unable to initialize catalog manager: Failed to initialize
sys tables async: Unable to load consensus metadata for tablet
00000000000000000000000000000000: ...
/wal/consensus-meta/00000000000000000000000000000000: No such file or directory
(error 2)
{noformat}
The master can't recover without manual action (wiping its data directories),
even though it never joined the Raft config.
This showed up as a flaky
{{TestSecurity.testRenewAndReacquireKeberosCredentials}} in a TSAN pre-commit
run. The test restarts all masters about a second after the mini cluster
reports it has started. The mini cluster only waits for each master's RPC
server to come up, not for the catalog to be created. Under TSAN, two of the
three masters were still resolving peer UUIDs when they were killed. After the
restart both failed with the error above, no leader could be elected, and
{{listTabletServers()}} timed out after 30s ("Master config ... has no
leader"). The same error also appears earlier in the same log during
{{testConnectToNonLeaderMasters}}. That test passed only because just one
master was affected, so the other two still had a majority.
The full test output from the failed run is attached. The relevant section
starts at 13:19:40, the masters are restarted at 13:19:51, and the test fails
at 13:20:31.
Proposed fix: build the Raft config (including resolving the peer UUIDs) before
writing anything to disk. Also let {{SysCatalogTable::Load()}} finish an
interrupted creation, but only when no Raft state (consensus metadata or WAL)
exists for the sys catalog tablet, i.e. the master has never been part of the
Raft config.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)