Hi, Pulsar has 2 configurations for the backlog eviction: > backlogQuotaDefaultLimitBytes and backlogQuotaDefaultLimitSecond, if > topic backlog reaches the threshold of any item, backlog eviction will be > triggered.
This seems like default values, not the actual values. Can you please provide an explanation in the PIP and link to read more: 1. Where do you define the backlog quota exactly? What is the granularity (subscription?) 2. Is the backlog quota on by default? If so, what are the default values? *Notes* 1. When the backlog quota limit is defined in Bytes, and you wish to know how close a subscription is to its bytes limit, you need to calculate the backlog size in bytes. From my understanding, there is an accurate calculation (which is costly in terms of I/O) and there is an estimate of it. I presume you would want to use the estimated one, is that correct? The backlog quota itself, uses the accurate or the estimated when it starts evicting entries (i.e. marking them as acknowledged)? 2. For the backlog limit specifying in time units, there is no estimate, as it must be calculated all the time (earliest unacknowledged message distance from now). How do you plan to calculate the current age of the earliest message without bearing that I/O cost on each metric calculation? 3. In the Goal section, you specify that your goal is to add a "proximity" metric. a) You must define that - what is proximity metric exactly? What are its units? How are you planning to calculate it? b) Proximity is not a good term IMO. I personally have never seen this term used in software systems, unless it's in the aviation/space industry. Once you explain (a) I hope I can help provide alternative names. 4. Maybe we should provide the used quota percentage for both limits, instead of one per both, since it's easier to act upon the alert when you need which one triggered it. 5. I didn't understand the "slowest_subscription" label used when describing the metric label. Can you please provide an explanation? 6. I suggest writing a "High Level Design" section, and add everything you need to know for this proposal, so I don't need to read the implementation details below (code). Thanks, Asaf On Wed, Feb 22, 2023 at 4:52 PM 太上玄元道君 <dao...@apache.org> wrote: > Hi all, > > I've started a PIP to discuss: PIP-248 Add backlog eviction metric > > ### Motivation: > > Pulsar has 2 configurations for the backlog eviction: > `backlogQuotaDefaultLimitBytes` and `backlogQuotaDefaultLimitSecond`, if > topic backlog reaches the threshold of any item, backlog eviction will be > triggered. > > Before backlog eviction happens, we don't have a metric to monitor how long > that it can reaches the threshold. > > We can provide a progress bar metric to tell users some topics is about to > trigger backlog eviction. And users can subscribe the alert to schedule > consumers. > > For more details, please read the PIP at > https://github.com/apache/pulsar/issues/19601 > > Thanks, > Tao Jiuming >
