[
https://issues.apache.org/jira/browse/HDDS-12542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Andika updated HDDS-12542:
-------------------------------
Description:
Currently we use SignedChunkInputStream for parsing chunk payload with chunk
signatures. However, no chunk signatures verification is done.
We can support this feature. We might need to dynamically change the WriteChunk
size based on the S3 chunk size specified (i.e. 1 S3 chunk = 1 WriteChunk) to
fit more to S3 SDK behavior and prevent multiple round-trips per S3 chunk
payload (Edit: might not be good since a single Ozone chunk is significantly
larger than S3 chunk). Additionally, we also need to handle the trailer (if
any).
I expect that the solution is not straightforward since there are differences
in architecture between Ozone and AWS S3, so some design document might be
needed. Few things to note:
* We might need to keep track of the previous chunk signature since the
subsequent chunk signature is derived from the previous chunk signature
* Support trailer SignedChunksInputStream: Perhaps adding a simple boolean
flag and check for the trailer afterwards
* Checksum verification location: S3G or DN?
Resources
[https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html]
[https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming-trailers.html]
!screenshot-1.png|width=690,height=350!
was:
Currently we use SignedChunkInputStream for parsing chunk payload with chunk
signatures. However, no chunk signatures verification is done.
We can support this feature. We might need to dynamically change the WriteChunk
size based on the S3 chunk size specified (i.e. 1 S3 chunk = 1 WriteChunk) to
fit more to S3 SDK behavior and prevent multiple round-trips per S3 chunk
payload. Additionally, we also need to handle the trailer (if any).
I expect that the solution is not straightforward since there are differences
in architecture between Ozone and AWS S3, so some design document might be
needed. Few things to note:
* We might need to keep track of the previous chunk signature since the
subsequent chunk signature is derived from the previous chunk signature
* Support trailer SignedChunksInputStream: Perhaps adding a simple boolean
flag and check for the trailer afterwards
* Checksum verification location: S3G or DN?
Resources
[https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html]
[https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming-trailers.html]
!screenshot-1.png|width=690,height=350!
> Support S3 signed chunk payload verification
> --------------------------------------------
>
> Key: HDDS-12542
> URL: https://issues.apache.org/jira/browse/HDDS-12542
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Ivan Andika
> Priority: Major
> Attachments: screenshot-1.png
>
>
> Currently we use SignedChunkInputStream for parsing chunk payload with chunk
> signatures. However, no chunk signatures verification is done.
> We can support this feature. We might need to dynamically change the
> WriteChunk size based on the S3 chunk size specified (i.e. 1 S3 chunk = 1
> WriteChunk) to fit more to S3 SDK behavior and prevent multiple round-trips
> per S3 chunk payload (Edit: might not be good since a single Ozone chunk is
> significantly larger than S3 chunk). Additionally, we also need to handle the
> trailer (if any).
> I expect that the solution is not straightforward since there are differences
> in architecture between Ozone and AWS S3, so some design document might be
> needed. Few things to note:
> * We might need to keep track of the previous chunk signature since the
> subsequent chunk signature is derived from the previous chunk signature
> * Support trailer SignedChunksInputStream: Perhaps adding a simple boolean
> flag and check for the trailer afterwards
> * Checksum verification location: S3G or DN?
> Resources
> [https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html]
> [https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming-trailers.html]
> !screenshot-1.png|width=690,height=350!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
