Chovan Zoltan created KUDU-3818:
-----------------------------------

             Summary: Master can't restart after interrupted sys catalog 
creation 
                 Key: KUDU-3818
                 URL: https://issues.apache.org/jira/browse/KUDU-3818
             Project: Kudu
          Issue Type: Improvement
            Reporter: Chovan Zoltan
            Assignee: Chovan Zoltan
         Attachments: test-output.txt

When a new multi-master cluster starts, each master creates the system catalog 
tablet in SysCatalogTable::CreateNew(). The steps run in this order:
1. Write the tablet metadata to disk.
2. Resolve the other masters' UUIDs over RPC. This blocks until those masters 
are up, for up to --raft_get_node_instance_timeout_ms (30s by default).
3. Write the consensus metadata to disk.

If the master is stopped or crashes between steps 1 and 3, or if step 2 times 
out, the tablet metadata is on disk but the consensus metadata isn't. Each 
later start then takes the "load existing" path and fails for good:

{noformat}
E20261027 13:19:53.309684 31360 master.cc:417] Unable to init master catalog 
manager: Not found: Unable to initialize catalog manager: Failed to initialize 
sys tables async: Unable to load consensus metadata for tablet 
00000000000000000000000000000000: ... 
/wal/consensus-meta/00000000000000000000000000000000: No such file or directory 
(error 2)
{noformat}

The master can't recover without manual action (wiping its data directories), 
even though it never joined the Raft config.

This showed up as a flaky 
{{TestSecurity.testRenewAndReacquireKeberosCredentials}} in a TSAN pre-commit 
run. The test restarts all masters about a second after the mini cluster 
reports it has started. The mini cluster only waits for each master's RPC 
server to come up, not for the catalog to be created. Under TSAN, two of the 
three masters were still resolving peer UUIDs when they were killed. After the 
restart both failed with the error above, no leader could be elected, and 
{{listTabletServers()}} timed out after 30s ("Master config ... has no 
leader"). The same error also appears earlier in the same log during 
{{testConnectToNonLeaderMasters}}. That test passed only because just one 
master was affected, so the other two still had a majority.

The full test output from the failed run is attached. The relevant section 
starts at 13:19:40, the masters are restarted at 13:19:51, and the test fails 
at 13:20:31.

Proposed fix: build the Raft config (including resolving the peer UUIDs) before 
writing anything to disk. Also let {{SysCatalogTable::Load()}} finish an 
interrupted creation, but only when no Raft state (consensus metadata or WAL) 
exists for the sys catalog tablet, i.e. the master has never been part of the 
Raft config.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to