[
https://issues.apache.org/jira/browse/HDDS-16531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chung-En Lee updated HDDS-16531:
--------------------------------
Description:
h5. Motivation
In current Ozone deployments, all S3 traffic (both metadata and payload)
traverses the S3 Gateway (S3G). S3G acts as an active reverse-proxy,
translating HTTP streams into internal Ozone client RPC/gRPC/Ratis calls.
This architecture has major drawbacks for write-intensive workloads:
# {*}Network Double-Hop{*}: Client -> S3G -> DataNodes, making S3G instances a
massive network and CPU bottleneck.
# {*}Unnecessary Proxy Overhead{*}: S3G must buffer and manage streaming
chunks, increasing latency and memory pressure.
# {*}Deviation from Modern Object Storage Pattern{*}: Standard S3 Presigned
PUT is intended to offload data transmission directly to storage nodes after
obtaining authorization from the control plane.
h5. Proposed Architecture: Decoupled S3 Data Path
We propose decoupling S3G into a pure *Control Plane* (Metadata & Signing) and
leveraging *DataNodes (or co-located Data Workers)* as the {*}Data Plane{*}:
{code:java}
1. Request Presigned URL
Client -------------------> S3G / OM (Control Plane)
<------------------- Returns Presigned URL (pointing to target DN)
2. Direct Payload Upload (HTTP PUT)
Client -----------------------------------------------------> Leader DataNode
(Data Plane)
│
├──> Replicate
via Ratis Pipeline
│ (Follower
DNs)
└──> Ack /
Commit {code}
h5. Goal of this JIRA
Start an architectural discussion and gather feedback on:
* Feasibility of embedding an HTTP receiver into DataNode.
* Security model for validating presigned parameters on storage nodes.
* Preparation of a formal design doc for community review.
was:
h5. Motivation
In current Ozone deployments, all S3 traffic (both metadata and payload)
traverses the S3 Gateway (S3G). S3G acts as an active reverse-proxy,
translating HTTP streams into internal Ozone client RPC/gRPC/Ratis calls.
This architecture has major drawbacks for write-intensive workloads:
# {*}Network Double-Hop{*}: Client -> S3G -> DataNodes, making S3G instances a
massive network and CPU bottleneck.
# {*}Unnecessary Proxy Overhead{*}: S3G must buffer and manage streaming
chunks, increasing latency and memory pressure.
# {*}Deviation from Modern Object Storage Pattern{*}: Standard S3 Presigned
PUT is intended to offload data transmission directly to storage nodes after
obtaining authorization from the control plane.
h5. Proposed Architecture: Decoupled S3 Data Path
We propose decoupling S3G into a pure *Control Plane* (Metadata & Signing) and
leveraging *DataNodes (or co-located Data Workers)* as the {*}Data Plane{*}:
{code:java}
1. Request Presigned URL
Client -------------------> S3G / OM (Control Plane)
<------------------- Returns Presigned URL (pointing to target DN)
2. Direct Payload Upload (HTTP PUT)
Client -----------------------------------------------------> Leader DataNode
(Data Plane)
│
├──> Replicate
via Ratis Pipeline
│ (Follower
DNs)
└──> Ack /
Commit {code}
h5. Key Technical Considerations
# {*}DataNode HTTP Ingress{*}:
** Introduce a lightweight HTTP/Netty ingress service on DataNodes (disabled
by default or on a dedicated port).
** It accepts HTTP PUT requests carrying pre-authenticated authorization
context.
# {*}Authentication & Token Encapsulation{*}:
** How S3G encodes Ozone credentials: S3G validates AWS SigV4 / credentials
during URL generation, requests block/container allocation from OM, and embeds
an Ozone Block Token & pipeline metadata into the Presigned URL.
** DN validates the embedded token before accepting the payload.
# {*}Ratis Bridge on Leader DataNode{*}:
** The Leader DN acts as the stream head, converting incoming HTTP payload
directly into Ratis log entries / zero-copy chunk writes across the allocated
pipeline.
# {*}Lifecycle & Multipart Upload{*}:
** For single PUT: Automatic OM commit upon stream completion, or explicit
finalization.
** For Multipart Upload: Individual parts uploaded directly to DNs;
{{CompleteMultipartUpload}} processed by S3G / OM as a metadata-only
transaction.
h5. Goal of this JIRA
Start an architectural discussion and gather feedback on:
* Feasibility of embedding an HTTP receiver into DataNode vs. a standalone
edge sidecar.
* Security model for validating presigned parameters on storage nodes.
* Preparation of a formal design doc for community review.
> Support direct S3 upload via Presigned PUT to DataNodes
> -------------------------------------------------------
>
> Key: HDDS-16531
> URL: https://issues.apache.org/jira/browse/HDDS-16531
> Project: Apache Ozone
> Issue Type: Improvement
> Reporter: Chung-En Lee
> Priority: Major
>
> h5. Motivation
> In current Ozone deployments, all S3 traffic (both metadata and payload)
> traverses the S3 Gateway (S3G). S3G acts as an active reverse-proxy,
> translating HTTP streams into internal Ozone client RPC/gRPC/Ratis calls.
> This architecture has major drawbacks for write-intensive workloads:
> # {*}Network Double-Hop{*}: Client -> S3G -> DataNodes, making S3G instances
> a massive network and CPU bottleneck.
> # {*}Unnecessary Proxy Overhead{*}: S3G must buffer and manage streaming
> chunks, increasing latency and memory pressure.
> # {*}Deviation from Modern Object Storage Pattern{*}: Standard S3 Presigned
> PUT is intended to offload data transmission directly to storage nodes after
> obtaining authorization from the control plane.
> h5. Proposed Architecture: Decoupled S3 Data Path
> We propose decoupling S3G into a pure *Control Plane* (Metadata & Signing)
> and leveraging *DataNodes (or co-located Data Workers)* as the {*}Data
> Plane{*}:
>
> {code:java}
> 1. Request Presigned URL
> Client -------------------> S3G / OM (Control Plane)
> <------------------- Returns Presigned URL (pointing to target DN)
>
> 2. Direct Payload Upload (HTTP PUT)
> Client -----------------------------------------------------> Leader DataNode
> (Data Plane)
> │
> ├──>
> Replicate via Ratis Pipeline
> │
> (Follower DNs)
> └──> Ack /
> Commit {code}
> h5. Goal of this JIRA
> Start an architectural discussion and gather feedback on:
> * Feasibility of embedding an HTTP receiver into DataNode.
> * Security model for validating presigned parameters on storage nodes.
> * Preparation of a formal design doc for community review.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
