>I see, but when a broker loads a bundle with many topics, you trigger immediately offloading for them, even if they don't have much traffic. This is kind of a "bad surprise" when you update a cluster to the newer version and you don't expect load to spike unexpectedly. My concerns is only about the upgrade path for users coming from older versions
I see. That makes sense. Let me add a flag to control this feature. Thanks, Hang Enrico Olivelli <eolive...@gmail.com> 于2024年5月6日周一 15:21写道： > > Il giorno lun 6 mag 2024 alle ore 04:15 Hang Chen <chenh...@apache.org> ha > scritto: > > > Hi Enrico, > > I agree with Jiuming, we do not need a flag to control this > > behavior. We will call `maybeOffloadInBackground` method in the > > managedledger init stage. This method only checks the offload policies > > to decide if we need to trigger offload. If the offload conditions are > > met, use the scheduled executor to trigger offload. The whole check > > process has no blocking calls and won't impact the managedledger init. > > > > https://github.com/apache/pulsar/blob/eee3694f00e269eef0f75d791521d0d35d8ff411/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L2475-L2489 > > > > > > I see, but when a broker loads a bundle with many topics, you trigger > immediately offloading for them, even if they don't have much traffic. > This is kind of a "bad surprise" when you update a cluster to the newer > version and you don't expect load to spike unexpectedly. > My concerns is only about the upgrade path for users coming from older > versions > > Maybe it is not a big deal ? > > Enrico > > > > > > Thanks, > > Hang > > > > 太上玄元道君 <dao...@apache.org> 于2024年5月6日周一 00:31写道： > > > > > > hi Enrico, > > > > > > I still think it doesn't matter > > > > > > 1. Offload a topic or not, configured by > > > > > `managedLedgerOffloadAutoTriggerSizeThresholdBytes`/`managedLedgerOffloadThresholdInSeconds`, > > > even though we may trigger a bunch of stuff, it's expected behavior > > > > > > 2. After triggered to offload a topic, the offload task will be added to > > an > > > Executor, currently, we have 2 threads to execute offload with sync mode, > > > so, even though there are thousands of topics, only 2 tasks can be > > executed > > > at the same time, other tasks wait. > > > > > > 3. If you are worrying about system overwhelmed, there is a PR to > > introduce > > > offload throttling(https://github.com/apache/pulsar/pull/22385), > > > it is for the purpose of protect the broker, it has been pending in > > review > > > for a loooong time, I'm very appreciate it that if you can help review > > > > > > 4. There are toooo many flags in pulsar(~500), I think we should be > > > cautious when introducing new FLAGs > > > > > > Thanks, > > > Tao Jiuming > > > > > > Enrico Olivelli <eolive...@gmail.com>于2024年5月5日 周日13:44写道： > > > > > > > daojun, > > > > > > > > Il giorno sab 4 mag 2024 alle ore 22:06 太上玄元道君 <dao...@apache.org> ha > > > > scritto: > > > > > > > > > it doesn't matter, we don't need to wait until offload finished when > > > > > loading topic, since offloading may take a long time. > > > > > it just trigger offloading, and we don't need to handle the result. > > > > > > > > > > > > > We don't have to wait for the results, but we still trigger a bunch of > > > > stuff and if the broker has thousand of topics, there may be an herd > > effect > > > > as this is a behavior change, I think it is safest to have a flag to > > > > disable this new behavior, > > > > otherwise upgrading an existing big cluster may lead to unexpected > > results, > > > > like overwhelmed systems > > > > > > > > > > > > Enrico > > > > > > > > > > > > > > > > > > > > > > Enrico Olivelli <eolive...@gmail.com>于2024年5月5日 周日02:41写道： > > > > > > > > > > > Overall I think it is a good idea. > > > > > > > > > > > > What about adding a flag to enable this behavior? > > > > > > > > > > > > In case there are problems with the offloader the topic won't be > > able > > > > to > > > > > > load. > > > > > > > > > > > > Enrico > > > > > > > > > > > > Il Sab 4 Mag 2024, 17:12 Hang Chen <chenh...@apache.org> ha > > scritto: > > > > > > > > > > > > > Hi guys, > > > > > > > I pushed a new proposal to improve the tiered storage > > offload > > > > > > > efficiency. Please help take a look, thanks. > > > > > > > > > > > > > > PIP: https://github.com/apache/pulsar/pull/22650 > > > > > > > > > > > > > > Thanks, > > > > > > > Hang > > > > > > > > > > > > > > > > > > > > > > > >
