[
https://issues.apache.org/jira/browse/HDDS-16383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18112260#comment-18112260
]
Luis Pigueiras commented on HDDS-16383:
---------------------------------------
Thanks for digging into the actual mechanism, the {{fillInStackTrace}} override
sounds reasonable. Happy to see a patch :)
> Ozone client logs full stack trace at INFO on every OM leader failover
> (RetryInvocationHandler)
> -----------------------------------------------------------------------------------------------
>
> Key: HDDS-16383
> URL: https://issues.apache.org/jira/browse/HDDS-16383
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Client
> Affects Versions: 2.2.0
> Reporter: Luis Pigueiras
> Priority: Minor
>
> h3. Summary
> When accessing Ozone via hdfs commands (like {{hdfs dfs -ls ofs://...}},
> every freshly-created client starts by contacting a random OM rather than a
> cached/known leader. With N configured OMs in HA, this means there is only a
> 1-in-N chance of hitting the actual leader on the first try, the other
> (N-1)/N of the time, the client gets redirected via an
> {{OMNotLeaderException}}, which is logged at INFO with a full exception and
> stack trace.
> As far as I understand, this logging retrial mechanism is normal with a OM HA
> setup, so having it as a full INFO-level stack trace overstates its severity
> and adds noise to normal client commands.
> h3. Steps to reproduce
> # Repeat any {{hdfs dfs}} / {{ofs://}} command against a cluster:
> {code}
> hdfs dfs -ls ofs://<cluster>/<existingpath>/
> {code}
> # Eventually you will get some message like
> {code}
> INFO retry.RetryInvocationHandler: org.apache...ServiceException:
> org.apache.hadoop.ipc_.RemoteException(...OMNotLeaderException):
> OM:<om-host> is not the leader. Suggested leader is OM:<om-host2>
> at org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisServer...
> ... (full stack trace)
> {code}
> h3. Expected behavior
> This should probably not be logged as a full INFO-level stack trace? Maybe
> DEBUG level, or a terse single-line INFO message without the stack trace,
> would be more appropriate, this is an expected, transparently-handled event,
> not an error condition. Right now we have modified the log4j.properties to
> add this to reduce the noise from client commands.
> {code}
> log4j.logger.org.apache.hadoop.io_.retry.RetryInvocationHandler=WARN
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
