Hello Zhe Wang,
For a multipart writer every part is exactly one PUT-class request.
Therefore, the request rate a deployment can generate is its upload
bandwidth divided by the part size, plus about five fixed requests per
segment: `CreateMultipartUpload`, `CompleteMultipartUpload`, and the
three index files (`.index`, `.timeindex`, `.writer_snapshot`), which
are single `PutObject` calls on both plugins.
The bandwidth needed to reach the limit is 3,500 x part size, which
gives the sustained upload rate needed to hit 3,500 PUT/s inside one
partitioned prefix:
Part size Sustained upload into Tablet
servers at a
one partitioned prefix saturated 10
Gbps
--------------------------------- ----------------------
-------------------
5 MB (flink-s3-fs-native default) ~17 GiB/s ~15
16 MB (proposed) ~55 GiB/s ~47
64 MB (S3A today) ~219 GiB/s ~188
With 16 MB default, hitting 3,500 PUT/s inside one prefix takes roughly
47 tablet servers, each pushing a full 10 Gbps of tiering traffic into
that same prefix, at the same time, continuously.
The 16 MB default does lower the upload rate one prefix can sustain
before hitting 3,500 PUT/s by 3.7x compared to S3A: from ~219 GiB/s down
to ~55 GiB/s. Still, even the lower ~55 GiB/s rate is far above what a
real deployment can generate.
That table is also another argument for not carrying Flink's 5 MB
default across, which the FIP argues on heap grounds today. At 5 MB the
number drops to 15 servers, and a 15-server cluster is a real
deployment.
I hope this answers your first question.
To answer your second question about the option:
`s3.upload.max.concurrent.uploads` bounds heap usage, not request rate.
It limits the number of in-flight part buffers per stream, so the memory
a server can spend on uploads is a fixed amount instead of scaling with
the host CPU. That does cap the request rate as a side effect, but it is
not why a running cluster stays safe: the reason clusters sit far below
3,500 PUT/s is the bandwidth arithmetic above, not this knob.
~~
But your questions helped me identify a gap in the FIP related to Fluss
auto-partitioned tables.
The remote paths Fluss creates are
`log/{db}/{table}-{tableId}/{partitionName}-p{partitionId}/{bucket}/{segment-uuid}/`,
so an auto-partitioned table creates a fresh prefix subtree per
partition. On a daily partitioned table that is a cold prefix every day.
The AWS S3 performance guide
([optimizing-performance](https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html))
is explicit about this:
The scaling, in the case of both read and write operations, happens
gradually and is not instantaneous, and actual performance will vary
based on your specific workload characteristics, usage patterns, and
system configuration. While Amazon S3 is scaling to your new higher
request rate, you may see some 503 (Slow Down) errors.
You're right that smaller parts make 503 (Slow Down) responses more
likely. The exposure is each time a new partition starts writing to a
brand-new prefix, before S3 has raised that prefix's capacity, and on a
daily auto-partitioned table that happens every day.
**Retry Budget**
I think the real gap your question uncovered is the retry budget, and it
has nothing to do with the part size.
fluss-fs-s3 today FIP as drafted
(hadoop-aws 3.3.4)
----------------- ---------------------------------
---------------------
Client attempts fs.s3a.attempts.maximum = 10 4
(s3.retry.max-num-
retries = 3, plus
the
initial attempt)
Throttle-specific fs.s3a.retry.throttle.limit = 20 none, shares the
retries at fs.s3a.retry.throttle.interval budget above
= 500 ms
The difference matters more than the attempt counts suggest. With
`s3.retry.throttle.base-delay` at 1 s doubling per attempt, three
retries wait about 1 + 2 + 4 = 7 seconds in total before the client
gives up on a throttled prefix. S3A's 20 throttle-specific retries keep
the client trying for minutes. Seven seconds is not enough to cover the
scale-up window on a cold prefix, so the FIP as drafted does not address
it.
I am going to verify these numbers against the code and update the FIP
draft accordingly, likely raising the retry default to match what
`fluss-fs-s3` ships today.
Please let me know if you have any other questions. I am going to keep
you posted
Best,
Muhammet
