adoroszlai opened a new pull request, #7354: URL: https://github.com/apache/ozone/pull/7354
## What changes were proposed in this pull request? If client sends unknown message to OM (e.g. new client talking to old OM), OM rejects it with error: `Message missing required fields`. Client then retries it according to policy. Since the message is sent unchanged, retries will not improve the situation. Repro: ``` $ cd hadoop-ozone/dist/target/ozone-2.0.0-SNAPSHOT/compose/xcompat $ OZONE_VERSION=1.4.0 COMPOSE_FILE=old-cluster.yaml:clients.yaml docker-compose up -d $ COMPOSE_FILE=old-cluster.yaml:clients.yaml docker-compose exec new_client bash bash-4.4$ kinit -kt /etc/security/keytabs/testuser.keytab testuser/[email protected] bash-4.4$ OZONE_LOGLEVEL=DEBUG ozone repair quota status --service-host=om ... 2024-10-24 18:09:33,779 [main] INFO retry.RetryInvocationHandler: com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(com.google.protobuf.InvalidProtocolBufferException): Message missing required fields: cmdType at com.google.protobuf.UninitializedMessageException.asInvalidProtocolBufferException(UninitializedMessageException.java:81) at com.google.protobuf.AbstractParser.checkMessageInitialized(AbstractParser.java:71) at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:89) at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:95) at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49) at org.apache.hadoop.ipc.RpcWritable$ProtobufWrapperLegacy.readFrom(RpcWritable.java:170) at org.apache.hadoop.ipc.RpcWritable$Buffer.getValue(RpcWritable.java:232) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:476) at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:595) at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:573) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1227) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1094) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1017) at java.base/java.security.AccessController.doPrivileged(AccessController.java:712) at java.base/javax.security.auth.Subject.doAs(Subject.java:439) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3048) , while invoking $Proxy21.submitRequest over nodeId=null,nodeAddress=om:9862 after 1 failover attempts. Trying to failover after sleeping for 4000ms. Current retry count: 1. ... ``` This causes command that sends unsupported request to "hang". https://issues.apache.org/jira/browse/HDDS-11608 ## How was this patch tested? Used same repro steps to verify that command fails after first request: ``` bash-4.4$ OZONE_LOGLEVEL=DEBUG ozone repair quota status --service-host=om ... 2024-10-24 18:15:57,344 [main] DEBUG ha.OMFailoverProxyProviderBase: RetryProxy: OM om1: RemoteException: Message missing required fields: cmdType at com.google.protobuf.UninitializedMessageException.asInvalidProtocolBufferException(UninitializedMessageException.java:81) at com.google.protobuf.AbstractParser.checkMessageInitialized(AbstractParser.java:71) at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:89) at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:95) at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49) at org.apache.hadoop.ipc.RpcWritable$ProtobufWrapperLegacy.readFrom(RpcWritable.java:170) at org.apache.hadoop.ipc.RpcWritable$Buffer.getValue(RpcWritable.java:232) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:476) at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:595) at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:573) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1227) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1094) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1017) at java.base/java.security.AccessController.doPrivileged(AccessController.java:712) at java.base/javax.security.auth.Subject.doAs(Subject.java:439) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3048) Message missing required fields: cmdType bash-4.4$ ``` CI: https://github.com/adoroszlai/ozone/actions/runs/11503306578 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
