[
https://issues.apache.org/jira/browse/HDDS-17140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18128662#comment-18128662
]
Stephen O'Donnell commented on HDDS-17140:
------------------------------------------
This issue is more of an Ozone client problem than S3G specific, as there may
be other custom applications built using the Ozone client that would have
similar behavior. Ideally we should fix the client to periodically check if
the OM version has changed and if so update its view of the "apparent version"
to use. One unfortunate problem is that the client may poll all the time when
it does not need to. Eg if we tell it to poll once every 1 or 10 minutes, then
most of the time a cluster is not in an upgrade flow and its a waste of
resources. However, does a simple heartbeat every 10 minutes from every client
matter? I guess it could if there are 1000's of clients, but even then its a
fairly low resource use compared to other operations.
On the other hand there may be some way the client can tell if OM is
pre-finalized and it should keep checking, and therefore not poll when it does
not need to. In the general case, an application could be using a client that
is newer than the cluster version, so we cannot just compare the client version
and the OM version to decide whether to poll.
The solution for S3G alone is easier as it will have a copy of the
OzoneManagerVersion enum, so it can check if the apparent version returned by
OM is older than the software version, indicating a pre-finalized state and
have it tell the client to refresh.
You could also argue that a long lived custom application cannot use any new
features without a code change itself, and therefore needs to be restarted
after the cluster has been upgraded to start using the features, so maybe
fixing S3G is enough for now?
We should let [~erose] add his thoughts before jumping into the code change.
> S3 Gateway does not pick up the new OM version after ZDU finalization
> ----------------------------------------------------------------------
>
> Key: HDDS-17140
> URL: https://issues.apache.org/jira/browse/HDDS-17140
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Chung-En Lee
> Assignee: Andy Huang
> Priority: Major
>
> The ZDU design doc ("Order of Operations During the Upgrade", step 5) says to
> rolling-restart S3 Gateway and other client processes on the new software
> before the cluster is finalized. At that point OM is still pre-finalized, so
> it reports the old version to clients.
> S3 Gateway learns the OM version only once, when it handles its first request
> after starting, and keeps it for the life of the process. In a ZDU rolling
> restart S3 Gateway keeps serving traffic, so it captures the old version
> before finalization. After the admin finalizes the cluster, OM reports the
> new version, but S3 Gateway keeps the old one. Any new feature that depends
> on the finalized OM version stays disabled in S3 Gateway until it is
> restarted again.
> Proposed fix (options):
> 1. Change the documented upgrade order so S3 Gateway is restarted after
> finalization.
> 2. Keep the current order, and document that S3 Gateway must be restarted
> again after finalization.
> 3. Make S3 Gateway aware OM version changes (e.g. after finalization) and
> re-initialize its OzoneClient automatically, so it picks up the latest OM
> version without a manual restart.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
