Hi Muhammet, Thanks for the proposal — the motivation resonates strongly. I've been working on PR #3884 [1] to fix a shaded-Jackson leak from fluss-fs-s3 into uber-jars: the Hadoop/AWS SDK v1 dependency chain pulls in jackson-databind whose Multi-Release JAR entries are not relocated by the shade plugin, causing NoSuchMethodError in downstream apps. That fix is a band-aid within the current Hadoop-wrapper architecture; a native v2 implementation would eliminate the root cause.
One question on the write path, specifically around S3 request rate limits: The 16 MB default part size results in roughly 3.7x more PUT requests per GB compared to S3A's 64 MB blocks (66 vs 18 requests per 1 GB segment). The proposal makes a good case that this is acceptable from a cost perspective, but I'm curious about the rate-limiting angle. S3's per-prefix PUT limit is around 3,500 req/s. Fluss's remote layout (/<db>/<table>/<bucket>/<segment-uuid>/<offset>.log) naturally distributes writes across prefixes via segment UUIDs, which helps. However, in a high-ingestion scenario where multiple buckets of a single table are tiering simultaneously, writes concentrate under the same /<db>/<table>/ prefix tree. At 3.7x the request volume, the path to hitting 503 Slow Down responses is correspondingly shorter. Have you considered whether the 16 MB default might need deployment-specific guidance for high-ingestion clusters, or is the expectation that the bounded concurrency (8 concurrent uploads) is sufficient to stay within S3 rate limits in practice? [1] https://github.com/apache/fluss/pull/3884 Best regards, Zhe Wang Muhammet Orazov via dev <[email protected]> 于2026年8月4日周二 19:02写道： > Hi community, > > I would like to start a discussion on FIP-XXX: Native S3 FileSystem (FIP > number to be added later), which proposes `fluss-fs-s3-native`: an S3 > filesystem plugin built directly on the AWS SDK for Java v2, with no > Hadoop dependency. > > Today every S3 read and write in Fluss (remote log segment tiering, KV > snapshot upload and download, client-side remote reads) goes through > `fluss-fs-s3`, a wrapper around Hadoop's `S3AFileSystem`. > > Four problems follow from that: > > 1. **The AWS SDK for Java v1 reached end-of-support on December 31, > 2025.** > > Fluss pins version 1.12.319 in `fluss-fs-s3`, so it receives no further > updates, including security fixes. The Hadoop-sanctioned upgrade path is > `hadoop-aws` 3.4.x, which reaches SDK v2 only through the monolithic > `aws-sdk-java-bundle`. This coupling is also why `fluss-filesystems` is > still on Hadoop 3.3.4 while the rest of Fluss builds against 3.4.0. > > 2. **Dependency weight.** > > The shaded plugin jar is about 30 MB and its NOTICE lists 47 bundled > artifacts. The pom carries roughly 30 hand-maintained exclusions, and we > ship a patched copy of an SDK-internal class to work around a JDK bug in > the bundled SDK's XML parsing. > > 3. **Release-cycle coupling.** > > Adopting new S3 capabilities waits on a Hadoop release and then a Fluss > dependency upgrade, rather than a direct SDK bump. > > Flink faced the same situation and resolved it in FLIP-555, now accepted > and merged. Its benchmark measured roughly 2x sustained throughput > (about 200 MB/s against 90 MB/s) compared to the SDK v1 based connector. > > Scope and compatibility: new S3 FS only, no SPI changes, and no data or > layout change. > > The full proposal is here: > > https://docs.google.com/document/d/1E-CjiYB7IXhdWOq_nOhAtRTb5nZNaJLHIpi19NW-ySI/edit?usp=sharing > > Feedback is welcome! These areas where I expect your attention and > discussions: Write path and S3 native defaults. > > Thanks and best, > Muhammet >
