Hi Aasf, I've updated the PIP, PTAL Thanks, Tao Jiuming
Asaf Mesika <asaf.mes...@gmail.com> 于2023年3月5日周日 21:00写道： > On Thu, Mar 2, 2023 at 12:57 PM 太上玄元道君 <dao...@apache.org> wrote: > > > > I think you should fix this explanation: > > > > Thanks! I would like to copy the context you provide to the PIP > motivation, > > your description is more detailed, so developers don't have to go through > > the code. > > > > Sure > > > > > > > Today the quota is checked periodically, right? So that's how the > > operator > > > knows the cost in terms of I/O is limited. > > > Now you are adding one additional I/O per collection, every 1 min by > > > default. That's a lot perhaps. How long is the check interval today? > > > > Actually, I don't want to introduce additional costs, I thought we > > could cache its result, so that it won't introduce additional costs. > > It may be that I did not make it clear in the PIP and caused this > > misunderstanding, sorry. > > > > Ok, just to verify: You plan to modify the code that runs periodically the > backlog quota check, so the result will be cached there? This way when you > pull that information from that code every 1min to expose it as a metric it > will have 0 I/O cost? > > > > > > > > The user today can calculate quota used for size based limit, since > there > > > are two metrics that are exposed today on a topic level: " > > > pulsar_storage_backlog_quota_limit" and "pulsar_storage_backlog_size". > > You > > > can just divide the two to get a percentage. > > > For the time-based limit, the only metric exposed today is quota > itself , > > " > > > pulsar_storage_backlog_quota_limit_time". > > > > I only noticed `pulsar_storage_backlog_size` but missed > > `pulsar_storage_backlog_quota_limit` and > > `pulsar_storage_backlog_quota_limit_time`. Many thanks for your reminder. > > > > > > So, in this condition, we already have the following topic-level metrics: > > `pulsar_storage_backlog_size`: The total backlog size of the topics of > this > > topic owned by this broker (in bytes). > > `pulsar_storage_backlog_quota_limit`: The total amount of the data in > this > > topic that limits the backlog quota (bytes). > > `pulsar_storage_backlog_quota_limit_time`: The backlog quota limit in > > time(seconds). (This metric does not exists in the doc, need to improve) > > > > > > We just need to add a new metric named > > `pulsar_storage_earliest_msg_publish_time_in_backlog` in the topic-level > > that indicates the publish time of the earliest message in the backlog. > > So users could get `pulsar_backlog_size_quota_used_percentage` by divide > > `pulsar_storage_backlog_size ` and > > `pulsar_storage_backlog_quota_limit`(`pulsar_storage_backlog_size` / > > `pulsar_storage_backlog_quota_limit`), > > and could get `pulsar_backlog_time_quota_used_percentage` by divide `now > - > > pulsar_storage_earliest_msg_publish_time_in_backlog` and > > `pulsar_storage_backlog_quota_limit_time` (`now - > > pulsar_storage_earliest_msg_publish_time_in_backlog` / > > `pulsar_storage_backlog_quota_limit_time`). > > > > I think there is a problem with the name > `pulsar_storage_earliest_msg_publish_time_in_backlog` in the topic-level: > * First, I prefer exposing the age rather than the publish time. > * Second, it's a bit hard to figure out the meaning of the earliest msg in > the backlog. > > Maybe `pulsar_storage_backlog_age_seconds`? In the explanation you can > write: "The age (time passed since it was published) of the earliest > unacknowledged message based on the topic's > existing subscriptions" ? > > > > > > > The backlog quota time checker runs periodically, so we can cache its > > result, so it won't lead to much costs. > > > > Pulsar also exposed subscription-level `backlogSize` and > > `earliestMsgPublishTimeInBacklog` in Pulsar-Admin > > < > > > https://github.com/apache/pulsar/blob/master/pulsar-client-admin-api/src/main/java/org/apache/pulsar/client/admin/Topics.java#L1139 > > > > > if > > `subscriptionBacklogSize` and `getEarliestTimeInBacklog` are true. > > We can also expose `backlogQuotaLimiteSize` and `backlogQuotaLimitTime` > of > > the topic to PulsarAdmin. > > > > What is the relationship you see between Pulsar exposing > subscriptionBacklogSize and earliestMsgPublishTimeInBacklog in > subscription, to exposing the backlog quota limits in pulsar admin? > > Limits can be exposed to Pulsar Admin, since it has 0 cost associated with > it. > I think it's a good idea to do that. > The quota usage can also be exposed to pulsar admin, since we pull that > data from the backlog quota checker cache, so it has 0 cost as well. > > As we said in previous email we can also expose > `backlogQuotaTimeOldestBacklogAgeSubscriptionName` > > > > > > After users receive the backlog alert from metrics alerting systems, they > > can get the topic name, then, they can request Topics#getStats > > < > > > https://github.com/apache/pulsar/blob/master/pulsar-client-admin-api/src/main/java/org/apache/pulsar/client/admin/Topics.java#L1139 > > > > > to > > get which subscriptions are in the huge backlog. > > > > > I agree users can use PulsarAdmin getStats for topic , with > getEarliestTimeInBacklog=true to find the oldest subscription responsible > for exceeding quota, but we can give them that information with 0 cost > since we already have that subscription name cached (we spent the I/O to > find out who that subscription is, let's just cache it and provide it). > > > > > > Thanks, > > Tao Jiuming > > > > Asaf Mesika <asaf.mes...@gmail.com> 于2023年3月1日周三 23:42写道： > > > > > > > > > > Pulsar has 2 configurations for the backlog eviction > > > > < > > > > > > https://pulsar.apache.org/docs/2.11.x/cookbooks-retention-expiry/#backlog-quotas > > > > > > > > : backlogQuotaDefaultLimitBytes and backlogQuotaDefaultLimitSecond. > > > > By default, backlog eviction is disabled, and also, there is a field > > > named > > > > backlogQuotaMap in TopicPolicies > > > > < > > > > > > https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/java/org/apache/pulsar/common/policies/data/HierarchyTopicPolicies.java#L45 > > > > > > > > /NamespaceSpacePolicies > > > > < > > > > > > https://github.com/apache/pulsar/blob/master/pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/policies/data/Policies.java#L41 > > > > > > assists > > > > in controlling Topic/Namespace level backlog quota. > > > > > > > > If topic backlog reaches the threshold of any item, backlog eviction > > will > > > > be triggered, Pulsar will move subscription's cursor to skip > > > unacknowledged > > > > messages. > > > > > > > > Before backlog eviction happens, we don't have a metric to monitor > how > > > > long that it can reaches the threshold. > > > > > > > > > > I think you should fix this explanation: > > > > > > In Pulsar, a subscription maintains a state of message acknowledged. A > > > subscription backlog is the set of messages which are unacknowledged. > > > A subscription backlog size is the sum of size of unacknowledged > messages > > > (in bytes). > > > A topic can have many subscriptions. > > > A topic backlog is defined as the backlog size of the subscription > which > > > has the oldest unacknowledged message. Since acknowledged messages can > be > > > interleaved with unacknowledged messages, calculating the exact size of > > > that subscription can be expensive as it requires I/O operations to > read > > > from the messages from the ledgers. > > > For that reason, the topic backlog is actually defined to be the > > estimated > > > backlog size of that subscription. It does so by summarizing the size > of > > > all the ledgers, starting from the current active one, up to the ledger > > > which contains the oldest unacknowledged message (There is actually a > > > faster way to calculate it, but this is the definition of the > > estimation). > > > > > > A topic backlog age is the age of the oldest unacknowledged message (in > > any > > > subscription). If that message was written 30 minutes ago, its age is > 30 > > > minutes. > > > > > > Pulsar has a feature called backlog quota (place link). It allows the > > user > > > to define a quota - in effect, a limit - which limits the topic > backlog. > > > There are two types of quotas: > > > * Size based: The limit is for the topic backlog size (as we defined > > > above). > > > * Time based: The limit is for the topic's backlog age (as we defined > > > above). > > > > > > Once a topic backlog exceeds either one of those limits, an action is > > taken > > > upon messages written to the topic: > > > * The producer write is placed on hold for a certain amount of time > > before > > > failing. > > > * The producer write is failed > > > * The subscriptions oldest unacknowledged messages will be acknowledged > > in > > > order until both the topic backlog size or age will fall inside the > limit > > > (quota). The process is called backlog eviction (happens every > interval) > > > > > > The quotas can be defined as a default value for any topic, by using > the > > > following broker configuration keys: backlogQuotaDefaultLimitBytes , > > > backlogQuotaDefaultLimitSecond. It can also be specified directly for > all > > > topics in a given namespace using the namespace policy, or a specific > > topic > > > using a topic policy. > > > > > > The user today can calculate quota used for size based limit, since > there > > > are two metrics that are exposed today on a topic level: " > > > pulsar_storage_backlog_quota_limit" and "pulsar_storage_backlog_size". > > You > > > can just divide the two to get a percentage. > > > For the time-based limit, the only metric exposed today is quota itself > > , " > > > pulsar_storage_backlog_quota_limit_time". > > > > > > ------------ > > > > > > I would create two metrics: > > > > > > `pulsar_backlog_size_quota_used_percentage` > > > `pulsar_backlog_time_quota_used_percentage` > > > > > > You would like to know what triggered the alert, hence two. > > > It's not the quota percentage, it's the quota used percentage. > > > > > > ---------- > > > > > > It checks if the backlog size exceeds the threshold( > > > > backlogQuotaDefaultLimitBytes), and it gets the current backlog size > by > > > > calculating LedgerInfo > > > > < > > > > > > https://github.com/apache/pulsar/blob/master/managed-ledger/src/main/proto/MLDataFormats.proto#L54 > > > >, > > > > it will not lead to I/O. > > > > > > This is not correct. > > > It checks against the topic / namespace policy, and if it doesn't > exist, > > it > > > falls back on the default configuration key mentioned above. > > > > > > It checks if the backlog time exceeds the threshold( > > > > backlogQuotaDefaultLimitSecond). If preciseTimeBasedBacklogQuotaCheck > > is > > > > set to be true, it will read an entry from Bookkeeper, but the > default > > > > value is false, which means it gets the backlog time by calculating > > > > LedgerInfo > > > > < > > > > > > https://github.com/apache/pulsar/blob/master/managed-ledger/src/main/proto/MLDataFormats.proto#L54 > > > >. > > > > So in general, we don't need to worry about it will lead to I/O. > > > > > > > > > I'm afraid of that. > > > Today the quota is checked periodically, right? So that's how the > > operator > > > knows the cost in terms of I/O is limited. > > > Now you are adding one additional I/O per collection, every 1 min by > > > default. That's a lot perhaps. How long is the check interval today? > > > > > > Perhaps in the backlog quota check, you can persist the check result, > and > > > use it? Persist the age that is. > > > > > > > > > ------ > > > > > > Regarding "slowest_subscription" > > > I think the cost is too high, because the subscriptions will keep > > > alternating, which can generate so many unique time series. Since > > > Prometheus flush only every 2 hours, or any there TSDB, it will cost > you > > > too much. > > > > > > I suggest exposing the name via the topic stats. This way they can > issue > > a > > > REST call to grab that subscription name only when the alert fires. > > > > > > Thanks, > > > > > > Asaf > > > > > > > > > > > > > > > > > > On Tue, Feb 28, 2023 at 9:29 AM 太上玄元道君 <dao...@apache.org> wrote: > > > > > > > Hi Asaf, > > > > I've updated the PIP, PTAL > > > > > > > > Thank, > > > > Tao Jiuming > > > > > > > > Asaf Mesika <asaf.mes...@gmail.com> 于2023年2月26日周日 23:03写道： > > > > > > > > > Hi, > > > > > > > > > > Pulsar has 2 configurations for the backlog eviction: > > > > > > backlogQuotaDefaultLimitBytes and backlogQuotaDefaultLimitSecond, > > if > > > > > > topic backlog reaches the threshold of any item, backlog eviction > > > will > > > > be > > > > > > triggered. > > > > > > > > > > This seems like default values, not the actual values. Can you > please > > > > > provide an explanation in the PIP and link to read more: > > > > > 1. Where do you define the backlog quota exactly? What is the > > > granularity > > > > > (subscription?) > > > > > 2. Is the backlog quota on by default? If so, what are the default > > > > values? > > > > > > > > > > > > > > > > > > > > *Notes* > > > > > 1. When the backlog quota limit is defined in Bytes, and you wish > to > > > know > > > > > how close a subscription is to its bytes limit, you need to > calculate > > > the > > > > > backlog size in bytes. From my understanding, there is an accurate > > > > > calculation (which is costly in terms of I/O) and there is an > > estimate > > > of > > > > > it. I presume you would want to use the estimated one, is that > > correct? > > > > > The backlog quota itself, uses the accurate or the estimated when > it > > > > starts > > > > > evicting entries (i.e. marking them as acknowledged)? > > > > > > > > > > 2. For the backlog limit specifying in time units, there is no > > > estimate, > > > > as > > > > > it must be calculated all the time (earliest unacknowledged message > > > > > distance from now). How do you plan to calculate the current age of > > the > > > > > earliest message without bearing that I/O cost on each metric > > > > calculation? > > > > > > > > > > 3. In the Goal section, you specify that your goal is to add a > > > > "proximity" > > > > > metric. > > > > > a) You must define that - what is proximity metric exactly? What > are > > > its > > > > > units? How are you planning to calculate it? > > > > > b) Proximity is not a good term IMO. I personally have never seen > > this > > > > term > > > > > used in software systems, unless it's in the aviation/space > industry. > > > > Once > > > > > you explain (a) I hope I can help provide alternative names. > > > > > > > > > > 4. Maybe we should provide the used quota percentage for both > limits, > > > > > instead of one per both, since it's easier to act upon the alert > when > > > you > > > > > need which one triggered it. > > > > > > > > > > 5. I didn't understand the "slowest_subscription" label used when > > > > > describing the metric label. Can you please provide an explanation? > > > > > > > > > > 6. I suggest writing a "High Level Design" section, and add > > everything > > > > you > > > > > need to know for this proposal, so I don't need to read the > > > > > implementation details below (code). > > > > > > > > > > Thanks, > > > > > > > > > > Asaf > > > > > > > > > > > > > > > On Wed, Feb 22, 2023 at 4:52 PM 太上玄元道君 <dao...@apache.org> wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > I've started a PIP to discuss: PIP-248 Add backlog eviction > metric > > > > > > > > > > > > ### Motivation: > > > > > > > > > > > > Pulsar has 2 configurations for the backlog eviction: > > > > > > `backlogQuotaDefaultLimitBytes` and > > `backlogQuotaDefaultLimitSecond`, > > > > if > > > > > > topic backlog reaches the threshold of any item, backlog eviction > > > will > > > > be > > > > > > triggered. > > > > > > > > > > > > Before backlog eviction happens, we don't have a metric to > monitor > > > how > > > > > long > > > > > > that it can reaches the threshold. > > > > > > > > > > > > We can provide a progress bar metric to tell users some topics is > > > about > > > > > to > > > > > > trigger backlog eviction. And users can subscribe the alert to > > > schedule > > > > > > consumers. > > > > > > > > > > > > For more details, please read the PIP at > > > > > > https://github.com/apache/pulsar/issues/19601 > > > > > > > > > > > > Thanks, > > > > > > Tao Jiuming > > > > > > > > > > > > > > > > > > > > >
