the limitation seems make no sense, after
http://github.com/apache/pulsar/pull/22472 merged, I'll try to fix it.
Thanks,
Tao Jiuming
Yike Xiao <km...@live.com>于2024年4月10日 周三16:58写道：
> Here's a hint as to why this restriction was introduced.
> After reviewing the current implementation of the retention policy, I
> found that the logic for determining whether a ledger exceeds the retention
> size and should be deleted is based on the following logic (pseudocode):
>
> ```
> long totalSizeToDelete = 0;
> for (ledger in ledgersBeforeSlowestReaderLedgerId) {
> totalSizeToDelete += ledger.size;
> if (managedLedgerTotalSize - totalSizeToDelete >= retentionSize) {
> // add this ledger to ledgersToDelete list
> }
> }
> ```
>
> The actual code can be seen at: [1]
>
> The issue lies in the use of `managedLedgerTotalSize - totalSizeToDelete
> >= retentionSize`, where `managedLedgerTotalSize` is used instead of
> `SUM(ledgerSize(ledgersBeforeSlowestReaderLedgerId))`. This causes ledgers
> that have not been acknowledged to also be included in the retention size
> comparison.
>
> Here is an example to illustrate the problem:
> Suppose the current ManagedLedger has 5 ledgers, [1, 2, 3, 4, 5], each
> ledger's size is 1MB, and the retentionSize is set to 2MB. The current
> slowestReaderLedgerId is 3. Based on the current implementation, ledgers 1
> and 2 are expected to be retained but will be deleted.
> The retention policy will only actually take effect when the retentionSize
> exceeds the backlog quota.
>
> [1]:
> https://github.com/apache/pulsar/blob/master/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L2695-L2696
>
> Regards,
> Yike
> ________________________________
> From: Yike Xiao <km...@live.com>
> Sent: Wednesday, April 3, 2024 15:36
> To: dev@pulsar.apache.org <dev@pulsar.apache.org>
> Subject: [DISCUSS] Remove the limitation between backlog quota and
> retention policy
>
> Hi all,
>
> In Pulsar, backlogs consist of unacknowledged messages. Once a message is
> acknowledged by all subscriptions, it enters the retention stage, as
> clearly illustrated in the official documentation [1]. The backlog quota
> limits the size and/or time threshold of the backlog, while the retention
> policy dictates the duration or volume of messages (acknowledged by all
> subscriptions) to retain.
>
> However, I encountered an undocumented restriction that mandates the
> backlog quota to be greater than the retention policy. For example, a
> namespace with a backlog quota 20GB in size and unlimited in time cannot
> have a 2GB in size and 7 days in time retention policy. This detail is
> implemented in the code [3].
>
> This restriction poses challenges in scenarios where there's a need for a
> larger backlog quota to accommodate extensive usage while desiring a
> shorter retention period for traceback purposes. Such configurations seem
> unattainable under the current constraint.
>
> Given that this limitation has been part of Pulsar since its initial
> open-source release, I am curious about the rationale behind it.
> Understanding the original intent or the technical considerations that led
> to this restriction would be immensely helpful.
>
> Therefore, I propose reevaluating and potentially removing this
> restriction to allow for more flexible configurations of backlog quota and
> retention policy.
>
> I would greatly appreciate your thoughts on this matter.
>
> Thank you for your time and consideration. I look forward to your response
> and am eager to contribute further to this discussion.
>
> [1]:
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpulsar.apache.org%2Fdocs%2Fnext%2Fcookbooks-retention-expiry%2F%23retention-policies&data=05%7C02%7C%7Ccb059c912d5a4f98327308dc53b0ded0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638477266343295533%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=wRjCYXkwGq4TAhYet8FdxkpnaehpCay7ST1AZi2jKAo%3D&reserved=0
> <
> https://pulsar.apache.org/docs/next/cookbooks-retention-expiry/#retention-policies
> >
> [3]:
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fblob%2F7315aeb6258b7adc9d874268d50acb95ffc0cf2b%2Fpulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fadmin%2FAdminResource.java%23L374-L393&data=05%7C02%7C%7Ccb059c912d5a4f98327308dc53b0ded0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638477266343313621%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=JmKZUCH%2F9dD07pGIknGpOnTfGUNmaDAn3bM7f%2FirgKg%3D&reserved=0
> <
> https://github.com/apache/pulsar/blob/7315aeb6258b7adc9d874268d50acb95ffc0cf2b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/AdminResource.java#L374-L393
> >
>
> Regards,
> Yike
>
