Hi community,
I would like to start a discussion on FIP-XXX: Native S3 FileSystem (FIP
number to be added later), which proposes `fluss-fs-s3-native`: an S3
filesystem plugin built directly on the AWS SDK for Java v2, with no
Hadoop dependency.
Today every S3 read and write in Fluss (remote log segment tiering, KV
snapshot upload and download, client-side remote reads) goes through
`fluss-fs-s3`, a wrapper around Hadoop's `S3AFileSystem`.
Four problems follow from that:
1. **The AWS SDK for Java v1 reached end-of-support on December 31,
2025.**
Fluss pins version 1.12.319 in `fluss-fs-s3`, so it receives no further
updates, including security fixes. The Hadoop-sanctioned upgrade path is
`hadoop-aws` 3.4.x, which reaches SDK v2 only through the monolithic
`aws-sdk-java-bundle`. This coupling is also why `fluss-filesystems` is
still on Hadoop 3.3.4 while the rest of Fluss builds against 3.4.0.
2. **Dependency weight.**
The shaded plugin jar is about 30 MB and its NOTICE lists 47 bundled
artifacts. The pom carries roughly 30 hand-maintained exclusions, and we
ship a patched copy of an SDK-internal class to work around a JDK bug in
the bundled SDK's XML parsing.
3. **Release-cycle coupling.**
Adopting new S3 capabilities waits on a Hadoop release and then a Fluss
dependency upgrade, rather than a direct SDK bump.
Flink faced the same situation and resolved it in FLIP-555, now accepted
and merged. Its benchmark measured roughly 2x sustained throughput
(about 200 MB/s against 90 MB/s) compared to the SDK v1 based connector.
Scope and compatibility: new S3 FS only, no SPI changes, and no data or
layout change.
The full proposal is here:
https://docs.google.com/document/d/1E-CjiYB7IXhdWOq_nOhAtRTb5nZNaJLHIpi19NW-ySI/edit?usp=sharing
Feedback is welcome! These areas where I expect your attention and
discussions: Write path and S3 native defaults.
Thanks and best,
Muhammet
