Siyao Meng created HDDS-17139:
---------------------------------
Summary: Allow scaling OM and SCM in HA docker-compose
environments beyond 3 nodes
Key: HDDS-17139
URL: https://issues.apache.org/jira/browse/HDDS-17139
Project: Apache Ozone
Issue Type: Sub-task
Components: OM HA, test, SCM HA
Reporter: Siyao Meng
Datanodes in the compose environments can be scaled with {{docker compose up
--scale datanode=N}} (used by {{run.sh}} and {{testlib.sh}}), because they are
interchangeable. OM and SCM cannot: {{ozone-ha}} and {{ozonesecure-ha}} define
exactly 3 of each, so trying a 5, 7 or 9 node ring (HDDS-17122) means copying
services and config by hand, as {{om-bootstrap.yaml}} /
{{scm-decommission.yaml}} do for a 4th node.
h3. What ties OM/SCM to 3 today
# *One service per node with a fixed identity:* {{om1..om3}} / {{scm1..scm3}}
each have their own service and {{hostname}}; om1 and scm1 publish fixed host
ports ({{9874:9874}}, {{9876:9876}}), so replicas of one service could not
share them.
# *Hard-coded HA config* in {{docker-config}}:
{{ozone.om.nodes.omservice=om1,om2,om3}}, {{ozone.om.address.omservice.omN}},
{{ozone.scm.nodes.scmservice=scm1,scm2,scm3}},
{{ozone.scm.address.scmservice.scmN}}, {{ozone.scm.primordial.node.id=scm1}}.
# *Per-node startup roles:* scm1 sets {{ENSURE_SCM_INITIALIZED}}; scm2/scm3 set
{{ENSURE_SCM_BOOTSTRAPPED}} with a {{WAITFOR}} chain (scm2 waits for scm1, scm3
for scm2, OMs for scm3). In {{dockerlibexec/entrypoint.sh}} the two are an
{{if/elif}}, so identical replicas cannot use SCM auto-bootstrap, although
SCM-HA.md documents that {{scm --init}} plus {{scm --bootstrap}} on every node
is safe when {{ozone.scm.primordial.node.id}} is set.
# *Tests use node names:* ~30 compose test scripts and robot files reference
om1..3 / scm1..3 (e.g. {{smoketest/omha/testOMHA.robot}},
{{smoketest/scmha/primordial-scm.robot}},
{{ozonesecure-ha/test-leadership.sh}}).
Kerberos is not a blocker: {{entrypoint.sh}} fetches keytabs per {{hostname
-f}} from the KDC issuer, and the OM/SCM principals ({{om/om@}}, {{scm/scm@}})
are not per host.
h3. Proposal
*A (recommended): generate the N-node definition.* Add a helper (e.g. in
{{testlib.sh}}, used by {{run.sh}}) that, given {{OZONE_OM_COUNT}} /
{{OZONE_SCM_COUNT}} (default 3), writes a compose override with services
{{om1..omN}} / {{scm1..scmM}} (from one YAML anchor each) and the matching
{{OZONE-SITE.XML_ozone.om.*}} / {{ozone.scm.*}} entries. Node names stay
{{omN}} / {{scmN}}, so existing tests keep working unchanged; only om1/scm1
publish host ports. Usage: {{OZONE_OM_COUNT=5 OZONE_SCM_COUNT=5 ./run.sh -d}},
and the same for {{ozonesecure-ha}}.
*B: true {{--scale om=N --scale scm=M}}.* Replicas get container names
{{<project>-om-<i>}}, resolvable on the compose network, but they share one
environment and do not know their index. This needs the node list generated
from a count anyway, SCM auto-bootstrap in {{entrypoint.sh}} (run {{--init}}
then {{--bootstrap}} when both are requested, instead of {{elif}}), and tests
that discover node names instead of using om1/scm1. More moving parts for
little gain over A.
h3. Done when
* {{ozone-ha}} and {{ozonesecure-ha}} start with 3 (default, unchanged), 5, 7
and 9 OMs/SCMs; {{ozone admin om roles}} / {{ozone admin scm roles}} show one
leader and N-1 followers; the smoketests pass.
* HDDS-17132 (5-node acceptance tests) can then be built on top of this instead
of a copied compose file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
