[
https://issues.apache.org/jira/browse/HDDS-12174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18129424#comment-18129424
]
Siyao Meng commented on HDDS-12174:
-----------------------------------
Hit this again in the {{upgrade}} acceptance test HA cluster (scm2 running
{{ozone scm --bootstrap}}), about 3 of 9 cluster starts on a loaded host.
Cause: on a bootstrap SCM, {{StorageContainerManager#start}} starts Ratis and
then calls {{persistSCMCertificates}}, which reads and writes the local cert
tables through {{SCMCertStore}} (neither call goes through Ratis). Meanwhile
the StateMachineUpdater thread runs {{SCMHAManagerImpl#installCheckpoint}},
which closes the SCM DB in {{stopServices}}, swaps the directory and reopens it
in {{startServices}}. A cert table access inside that window fails with "Rocks
Database is closed", {{start()}} throws and SCM exits. A new SCM installs a
snapshot right after joining the ring, so the window is easy to hit.
Proposed fix: hold a lock across the DB swap in {{installCheckpoint}}, exposed
as {{SCMHAManager#getDBReplaceLock}}, and take it in {{persistSCMCertificates}}
around the local cert table access (the {{listCACertificate}} call stays
outside the lock). Unit test in {{TestSCMHAManagerImpl}}.
[~pifta], are you still working on this? If not, I can take it.
> SCM start failed as checkpoint replaced DB
> ------------------------------------------
>
> Key: HDDS-12174
> URL: https://issues.apache.org/jira/browse/HDDS-12174
> Project: Apache Ozone
> Issue Type: Bug
> Components: SCM
> Affects Versions: 1.4.1
> Reporter: Attila Doroszlai
> Assignee: István Fajth
> Priority: Major
> Attachments: acceptance-upgrade.zip
>
>
> {code}
> 2025-01-31 05:49:46,878 [main] INFO ha.SCMRatisServerImpl: starting ratis
> server 0.0.0.0:9894
> ...
> 2025-01-31 05:49:47,543 [grpc-default-executor-0] INFO
> impl.SnapshotInstallationHandler:
> 8e991fe1-76b7-45a9-974d-cbd80cf68516@group-8FC1CA2C16E2: receive
> installSnapshot:
> f02ef16f-92c4-4b45-9414-1dade74e823c->8e991fe1-76b7-45a9-974d-cbd80cf68516#0-t2,notify:(t:1,
> i:0)
> ...
> 2025-01-31 05:49:49,714
> [8e991fe1-76b7-45a9-974d-cbd80cf68516@group-8FC1CA2C16E2-StateMachineUpdater]
> INFO ha.SCMHAManagerImpl: Replaced DB with checkpoint, term: 2, index: 58
> ...
> 2025-01-31 05:49:49,951 [main] ERROR server.StorageContainerManagerStarter:
> SCM start failed with exception
> org.apache.hadoop.hdds.scm.exceptions.SCMException: java.io.IOException:
> Rocks Database is closed
> ...
> at
> org.apache.hadoop.hdds.scm.server.StorageContainerManager.persistSCMCertificates(StorageContainerManager.java:1598)
> at
> org.apache.hadoop.hdds.scm.server.StorageContainerManager.start(StorageContainerManager.java:1543)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
