Hi Michael, > This PIP is similar to autorecovery throttling. I think the feature > makes sense for the same reasons that throttling autorecovery makes > sense.
Yes, the PIP’s key point is protect the broker, to prevent offloading takes too much broker resources. > Tangentially, can we decouple writes to tiered storage from the broker > hosting the topic being offloaded? Do you mean that add a new broker type? And this type broker only for Offload processing? I think that the introduction of a new broker type is relatively heavyweight in order to implement offload throttling. We can do it in a simpler way Thanks, Tao Jiuming > 2022年11月8日 上午8:08，Michael Marshall <mmarsh...@apache.org> 写道： > > This PIP is similar to autorecovery throttling. I think the feature > makes sense for the same reasons that throttling autorecovery makes > sense. > > Tangentially, can we decouple writes to tiered storage from the broker > hosting the topic being offloaded? An independent service could write > to tiered storage without impacting the broker and could easily scale > as with the work. The primary complication for the service would be > figuring out which ledgers to offload. Perhaps the managed ledger > could "offer" ledgers up that need to be offloaded, and the new > service would only need to consume those events. > > Although, a new service would complicate the pulsar deployment. > > Thanks, > Michael > > On Mon, Nov 7, 2022 at 10:30 AM Jiuming Tao > <jm...@streamnative.io.invalid> wrote: >> >>> One alternative would be to throttle offload in the write path instead of >>> adding additional logic to the read path in managed ledgers. >> >> This is really a feasible method. >> But we need to make changes in FileSystem and BlobStore offloaders, event >> custom offloaders. I think this is not universal. >> >>> One simple way to do this is to to limit how many threads can write >>> offloaded ledgers. This is the same way that reading of offloaded ledgers >>> are already “throttled” by that thread count defaulting to 2. >> >> Yes, the offloader thread count is defaulting to 2, but, it does not >> effectively limit traffic. If the reading rate of BK is very fast, it also >> leads to high CPU/Memory/Network usage >> >> Thanks, >> Tao Jiuming >> >>> 2022年11月2日 上午1:43，Dave Fisher <w...@apache.org> 写道： >>> >>> One alternative would be to throttle offload in the write path instead of >>> adding additional logic to the read path in managed ledgers. >>> >>> One simple way to do this is to to limit how many threads can write >>> offloaded ledgers. This is the same way that reading of offloaded ledgers >>> are already “throttled” by that thread count defaulting to 2. >>> >>> Regards, >>> Dave >>> >>> Sent from my iPhone >>> >>>> On Nov 1, 2022, at 10:27 AM, Jiuming Tao <jm...@streamnative.io.invalid> >>>> wrote: >>>> >>>> Hi pulsar community, >>>> >>>> I opened a PIP to discuss: PIP-211: Introduce offload throttling >>>> >>>> PIP link: https://github.com/apache/pulsar/issues/18004 >>>> <https://github.com/apache/pulsar/issues/18004> >>>> >>>> Thanks, >>>> Tao Jiuming >>> >>
