Hi All, I would like to propose merging the S3g Grpc Persistent Connection Feature branch, HDDS-4440-s3-performance, into the Ozone master. This feature branch brings servicing s3 transactions through the S3 gateway, s3g, over Grpc to the Ozone Manager, om. In a production ready configuration with an ozone cluster configured with security enabled and encryption on the wire, the feature shows at least a 2x performance improvement in transaction processing rates.
The implementation processes S3 gateway requests through a gRPC OmTransport, a design that allows a persistent OzoneClient connection and aws signature authentication on a per request basis. The implementation is a result of the HDDS-4440 design proposal for a "persistent OM connection for S3 gateway". A supporting Jira patch, HDDS-5881, improved upon ozone 1.2.0 s3g by introducing a persistent Hadoop RPC, hRPC, connection between the s3g and om. This feature branch, s3g over Grpc, supersedes the patch by further improving the performance by over 2x with encryption on the wire (TLS). The feature branch proposed to be merged represents HDDS-4440 - consisting of 14 subtasks (1-15) and 2 supporting Jiras HDDS-5881 & HDDS-5630. All Jiras are represented in the branch to be merged with no outstanding work. I'd like to start a formal VOTE to merge HDDS-4440-s3-performance branch to master. The merge checklist for the feature branch is - S3g Grpc Persistent Connections Feature Merge (HDDS-4440) : https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=199540957 The feature design doc is found in: https://ci-hadoop.apache.org/view/Hadoop%20Ozone/job/ozone-doc-master/lastSuccessfulBuild/artifact/hadoop-hdds/docs/public/design/s3-performance.html The ASF slack channel for the feature is: #ozone-s3g-grpc, https://the-asf.slack.com/archives/C020VL6Q7T5 The feature Jira containing supporting tasks, sub-tasks and design docs is: HDDS-4440, https://issues.apache.org/jira/browse/HDDS-4440 The mailing list ANNOUNCE for this feature branch is: https://lists.apache.org/thread/7l4fdplrxp90n42186lkl89w5rmhm45b Thanks to all contributors for your input into the design, implementing, testing and review of this feature. Thanks to Marton for design, design discussions and for getting the feature off the ground from conception through to realization. Thanks to many others that were involved through the whole process, Alex, Michel, Angela, Jasen, Arpit, Sid, George, Aravindan, Ritesh, Attila, Bharat. I am starting with my +1. *The vote will run for 7 days, ending on Apr 12 at 8:00 PT.* Thanks! Best Regards, Neil -- NJ
