jojochuang commented on code in PR #582: URL: https://github.com/apache/ozone-site/pull/582#discussion_r4351478110
########## docs/04-user-guide/01-client-interfaces/03-s3/01-s3-api.md: ########## @@ -74,30 +94,58 @@ The Ozone S3 Gateway implements a substantial subset of the Amazon S3 REST API. | ✅ [GetObjectTagging](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectTagging.html) | Retrieves the tag set of an object. | Returns the tag set as defined by the S3 API. | | ✅ [DeleteObjectTagging](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjectTagging.html) | Removes the tag set from an object. | Clears all tags associated with the object. | +### Object Attributes Operations + +| **API Name** | **Feature** | **Note** | +| ------------ | ----------- | -------- | +| ✅ [GetObjectAttributes](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectAttributes.html) | Returns selected metadata about an object without the object body. | Served as `GET /{bucket}/{key}?attributes`. Use the `x-amz-object-attributes` header to request `ETag`, `ObjectSize`, `StorageClass`, `ObjectParts`, and `Checksum`. For multipart objects, `x-amz-max-parts` and `x-amz-part-number-marker` paginate part metadata. **Non-compliant behavior:** `Checksum` is accepted in the request but not returned, because Ozone does not persist `x-amz-checksum-*` on upload. The `versionId` query parameter and SSE-C headers are ignored. On OBS and LEGACY buckets, `ObjectParts` omits per-part `Part` elements unless an additional checksum was stored at upload time; FSO buckets always return part details. | + ### Multipart Upload Operations | **API Name** | **Feature** | **Note** | | ------------ | ----------- | -------- | | ✅ [CreateMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html) | Starts a multipart upload session. | Returns an UploadId for subsequent parts. | | ✅ [UploadPart](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html) | Uploads an individual part for a multipart upload. | Part numbering and ETag computation follow AWS semantics. | +| ✅ [UploadPartCopy](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.html) | Uploads a part by copying data from an existing object. | Uses the `x-amz-copy-source` header, and supports `x-amz-copy-source-range` to copy a byte range of the source object. | | ✅ [ListMultipartUploads](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListMultipartUploads.html) | Lists in-progress multipart uploads in a bucket. | Up to 1000 multipart uploads are returned in a batch by default. | | ✅ [ListParts](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListParts.html) | Lists uploaded parts for an in-progress multipart upload. | Supports pagination if many parts exist. | -| ✅ [CompleteMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html) | Assembles all uploaded parts into a final object. | Returns an ETag that follows AWS multipart rules. | -| ✅ [AbortMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html) | Aborts an in-progress multipart upload. | Frees up storage used by uploaded parts. Incomplete multipart uploads older than 7 days are automatically aborted by Ozone. | +| ✅ [CompleteMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html) | Assembles all uploaded parts into a final object. | Returns an ETag that follows AWS multipart rules. See [Conditional requests](#conditional-requests). | +| ✅ [AbortMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html) | Aborts an in-progress multipart upload. | Frees up storage used by uploaded parts. The OM automatically aborts incomplete multipart uploads that are older than `ozone.om.open.mpu.expire.threshold` (default 30 days). To abort them earlier for a bucket, use a lifecycle rule with `AbortIncompleteMultipartUpload`. | ### Additional Operations | **API Name** | **Feature** | **Note** | | -------------------------- | ------------- | ---------- | -| ✅ Generate Presigned URL | Generates a temporary URL for accessing an object. | Uses AWS Signature V4. **Non-compliant behavior:** The generated URL may include a fixed default region rather than dynamically reflecting the bucket’s location. Ozone now supports generating presigned URLs for all major S3 operations, including `GetObject`, `PutObject`, `DeleteObject`, `HeadObject`, `HeadBucket`, `MultipartUpload`. | +| ✅ Generate Presigned URL | Generates a temporary URL for accessing an object. | Uses AWS Signature V4. **Non-compliant behavior:** The generated URL may include a fixed default region rather than dynamically reflecting the bucket’s location. The S3 Gateway accepts presigned URLs (AWS Signature V4 query authentication) for all supported operations, such as `GetObject`, `PutObject`, `DeleteObject`, `HeadObject`, `HeadBucket`, and the multipart upload operations. | + +### Conditional Requests + +The S3 Gateway partially supports [conditional requests](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-requests.html): + +- **GetObject and HeadObject:** Support `If-Match`, `If-None-Match`, `If-Modified-Since`, and `If-Unmodified-Since`. A failed `If-Match` or `If-Unmodified-Since` returns `412 Precondition Failed`. A matching `If-None-Match` or a failed `If-Modified-Since` returns `304 Not Modified`. +- **CopyObject:** Supports `x-amz-copy-source-if-match`, `x-amz-copy-source-if-none-match`, `x-amz-copy-source-if-modified-since`, and `x-amz-copy-source-if-unmodified-since` on the source object. A failed condition returns `412 Precondition Failed`. +- **PutObject, CopyObject, and CompleteMultipartUpload:** Support `If-Match` with an ETag, to write only if the existing object matches, and `If-None-Match: *`, to write only if the object does not exist. `If-None-Match` with a specific ETag is not supported and returns `InvalidRequest`. `If-Match` and `If-None-Match` cannot be used together. Review Comment: If-None-Match with an Etag is supported by https://github.com/apache/ozone/pull/11366 (just added) ########## docs/04-user-guide/01-client-interfaces/03-s3/01-s3-api.md: ########## @@ -148,6 +196,30 @@ ozone s3 revokesecret -u om/[email protected] -y S3 secret revoked. ``` +## Health check + +The S3 Gateway serves two unauthenticated health endpoints on its web admin server (default port `19878`), separate from the S3 listener (`9878`). Because they are served on the admin port, they do not collide with S3 bucket names and do not require AWS SigV4 signing, so a load balancer can poll them directly. + +### Liveness + +```bash +curl http://localhost:19878/health/live +OK +``` + +Returns `200 OK` while the gateway process is up and its web admin server can serve requests. It does not check OM reachability. + +### Readiness + +```bash +curl http://localhost:19878/health/ready +READY +``` + +Returns `200 READY` only when the gateway can reach OM, and `503 NOT READY` during startup or while OM is unreachable. A background probe refreshes OM reachability off the request path, so the endpoint always responds immediately and cannot be blocked by a slow OM. The probe cadence and its timeout are tunable with `ozone.s3g.health-check.probe.interval` and `ozone.s3g.health-check.probe.timeout`. + +When S3 Gateway is horizontally scaled, point the load balancer's HTTP health check at the readiness endpoint so it only routes S3 traffic to gateways that can serve it; see the bundled `compose/common/s3-haproxy.cfg` for a worked HAProxy example. Both endpoints can be turned off with `ozone.s3g.health-check.enabled=false`, which removes them entirely. Note that the bundled Kubernetes manifests and HAProxy configs probe `/health/live` and `/health/ready` unconditionally, so if you disable the endpoints you must also remove those probes from your deployment; otherwise Kubernetes restarts the pods on failed liveness checks and HAProxy marks every backend down. Review Comment: the bundled HA proxy config file actually accepts missing health check points hadoop-ozone/dist/src/main/compose/common/s3-haproxy.cfg but in any case, the bundle HA proxy config is simple, not meant for production deployments, I suggest don't mention the HA Proxy config. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
