jayakasadev opened a new pull request, #3227:
URL: https://github.com/apache/iggy/pull/3227
Phase 1b of the S3-as-primary-storage milestone. Adds the compio-native S3
backend behind cargo features = ["object-storage"] (default off). When the
feature is on and [system.storage] kind = "object", bootstrap returns an
S3Storage built on rusty-s3 (SigV4 + request shaping) and cyper (compio
HTTP client, rustls TLS) — no tokio, no opendal.
What lands:
- rusty-s3 + url added as optional deps under the `object-storage` feature.
cyper and rustls were already in the server crate.
- S3Storage with all ObjectStorage methods (put/put_if_absent/put_multipart/
get_range/head/list_prefix/delete) plus an S3Multipart handle.
- BufferedMultipartWriter that coalesces sub-5-MiB iggy flushes into S3-legal
parts; the small-segment path aborts the multipart and falls back to a
single PUT under threshold (the Phase 0 spike confirmed S3 rejects parts
< 5 MiB with EntityTooSmall otherwise).
- main.rs installs rustls::crypto::ring::default_provider() at boot when the
feature is on (idempotent — `install_default` returns Err if already set).
- bootstrap::resolve_object_storage now returns S3Storage for kind = "object"
+ feature on; without the feature, logs a warning and returns fs.
- 5 new unit tests (4 on BufferedMultipartWriter, 1 IGGY_TEST_MINIO-gated
S3 wire round-trip).
- core/server-ng/config.toml gets the matching [system.storage] block so its
embedded-toml validation test still passes after the SystemConfig field
add.
Two findings from the Phase 0 spike that this commit bakes in:
- AWS ETags arrive wrapped in quotes (`"abc"`); rusty-s3's
CompleteMultipartUpload re-wraps them, so callers must `.trim_matches('"')`
before passing the etag back. S3Multipart::upload_part does this.
- Multipart minimum part size is 5 MiB except the final part. The
BufferedMultipartWriter `part_size` argument is debug-asserted >= that.
Stacks on Phase 1a. ClickUp: 86ahbmwn9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
