First, I am +1 on the design proposed in this PIP.
> Could you please explain that why the design will leads to unnecessary load
> on the broker?
> IMO, I don't think it will bring huge expenses.
I did not mean the throttling design proposed by this PIP was
expensive. I meant that running the of
Hi Michael,
> The main trade off with throttling is that we might
> leave performance on the table if the cluster is not heavily utilized.
It is indeed possible. When a broker don’t publish/consume many messages and
offloaders are in the throttling,
the performance of the broker cannot be relea
> Yes, the PIP’s key point is protect the broker, to prevent offloading takes
> too much broker resources.
Throttling also protects the bookkeeper. Reads that are used to
offload data are the lowest priority reads since they are not serving
an actual client. Since we don't have a way to tell book
Hi Michael,
> This PIP is similar to autorecovery throttling. I think the feature
> makes sense for the same reasons that throttling autorecovery makes
> sense.
Yes, the PIP’s key point is protect the broker, to prevent offloading takes too
much broker resources.
> Tangentially, can we decoup
This PIP is similar to autorecovery throttling. I think the feature
makes sense for the same reasons that throttling autorecovery makes
sense.
Tangentially, can we decouple writes to tiered storage from the broker
hosting the topic being offloaded? An independent service could write
to tiered stor
> One alternative would be to throttle offload in the write path instead of
> adding additional logic to the read path in managed ledgers.
This is really a feasible method.
But we need to make changes in FileSystem and BlobStore offloaders, event
custom offloaders. I think this is not universal.
One alternative would be to throttle offload in the write path instead of
adding additional logic to the read path in managed ledgers.
One simple way to do this is to to limit how many threads can write offloaded
ledgers. This is the same way that reading of offloaded ledgers are already
“throt
