Hi, We recently created a proof-of-concept, POC, implementation for processing S3 gateway, s3g, requests with the transport of OmRequests over gRPC. This follows HDDS-4440 https://ozone.apache.org/docs/1.1.0/design/s3-performance.html.
Current commit implementation can be found in: https://github.com/neils-dev/ozone/tree/HDDS-4440-s3-performance. Slack channel #ozone-s3g-grpc . The implementation processes S3 gateway requests through a gRPC OmTransport, a design that allows a constant OzoneClient connection and aws signature authentication on a per request basis. This differs from the current implementation relying on HadoopRPC that creates an OzoneClient connection for each request. The need to create a 'client' connection for each request results in overhead impacting request processing times. Another limitation to the existing HadoopRPC based OmTransport implementation is that errors in request processing are shown as ambiguous Http 500 server responses to S3 gateway callers. With the redesigned s3g, proper error responses describing the error can be relayed to the caller. Preliminary performance comparisons between the existing s3g HadoopRPC OmTransport and the POC gRPC OmTransport show at least a 2x improvement in request processing rates for simple file operations (create buckets). See slack channel #ozone-s3g-gprc for more info and inquiries. Branch HDDS-4440-s3-performance is also created. Please let us know, if you have any comments/questions... Best - Neil & Jasen G-Research -- NJ
