[DISCUSSION] PIP-130: Apply redelivery backoff policy for ack timeout
https://github.com/apache/pulsar/issues/13528 Pasted below for quoting convenience. - PIP 130: Apply redelivery backoff policy for ack timeout ## Motivation PIP 106 https://github.com/apache/pulsar/wiki/PIP-106%3A-Negative-acknowledgment-backoff introduced negative acknowledgment message redelivery backoff which allows users to achieve more flexible message redelivery delay time control. But the redelivery backoff policy only apply to the negative acknowledgment API, for users who use ack timeout to trigger the message redelivery, not the negative acknowledgment API, they can't use the new features introduced by PIP 106. So the proposal is to apply the message redelivery policy for the ack timeout mechanism. Users can specify an ack timeout redelivery backoff, for example, apply an exponential backoff with 10 seconds ack timeout: ```java client.newConsumer() .ackTimeout(10, TimeUnit.SECOND) .ackTimeoutRedeliveryBackoff( ExponentialRedeliveryBackoff.builder() .minDelayMs(1000) .maxDelayMs(6).build()); .subscribe(); ``` The message redelivery behavior should be: | Redelivery count | Redelivery delay | | | | | 1 | 10 + 1 seconds | | 2 | 10 + 2 seconds | | 3 | 10 + 4 seconds | | 4 | 10 + 8 seconds | | 5 | 10 + 16 seconds | | 6 | 10 + 32 seconds | | 7 | 10 + 60 seconds | | 8 | 10 + 60 seconds | ## Goal Add an API to the Java Client to provide the ability to specify the ack timeout message redelivery backoff and the message redelivery behavior should abide by the redelivery backoff policy. ## API Changes 1. Change `NegativeAckRedeliveryBackoff` to `RedeliveryBackoff`, so that we can use the MessageRedeliveryBackoff for both negative acknowledgment API and ack timeout message redelivery. 2. Change `NegativeAckRedeliveryExponentialBackoff` to `ExponentialRedeliveryBackoff`, and add `multiplier` for `RedeliveryExponentialBackoff` with default value 2. ExponentialRedeliveryBackoff.builder() .minDelayMs(1000) .maxDelayMs(6) .multiplier(5) .build() 3. Add `ackTimeoutRedeliveryBackoff` method for the `ConsumerBuilder`: ```java client.newConsumer() .ackTimeout(10, TimeUnit.SECOND) .ackTimeoutRedeliveryBackoff( ExponentialRedeliveryBackoff.builder() .minDelayMs(1000) .maxDelayMs(6).build()); .subscribe(); ``` ## Compatibility and migration plan Since the negative acknowledgment message, redelivery backoff has not been released yet, so we can modify the API directly. ## Tests plan - Verify the introduced `multiplier` of ExponentialRedeliveryBackoff - Verify the ack timeout message redelivery work as expected Regards, Penghui
Re: [DISCUSSION] Produce chunk messages failed when topic level maxMessageSize is set
+1,
We can only skip the topic level messages size check for the chunk message.
Regards,
Penghui
On Mon, Dec 20, 2021 at 3:37 PM Haiting Jiang
wrote:
> Hi Pulsar Community,
>
> I discovered a bug that chunk messages producing fails if topic level
> maxMessageSize is set [1]. The root cause of this issue is because chunk
> message is using broker level maxMessageSize as chunk size. And topic level
> maxMessageSize is always <= broker level maxMessageSize. So once it is set,
> the on-going chunk message producing fails.
>
> ## Proposed changes
> I would like to fix this by just skipping topic level maxMessageSize check
> in
> org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize.Topic
> level maxMessageSize is introduced in [2], for the purpose of "easier to
> plan resource quotas for client allocation". And IMO this change will not
> bring further complex into this.
>
> ## Alternative
> Add a client side topic level maxMessageSize and keep it synced with
> broker.
>
> Required changes:
> - [client] Add a new field
> org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to store this
> client side topic level maxMessageSize.
> - [PulsarApi.proto] Add a new field maxMessageSize in the
> CommandProducerSuccess for the initial value of ProducerBase#maxMessageSize
> - [PulsarApi.proto] Add a new Command like
> CommandUpdateClientPolicy{producerId, maxMessageSize} to update
> ProducerBase#maxMessageSize when topic level maxMessageSize is updated.
> Further more, some other data consistency issues need be handled very
> carefully when maxMessageSize is updated.
> This alternative is complex but can also solve other topic level
> maxMessageSize issue [3] when batching is enabled (non-batching case is
> solved with PR [4]).
>
> Any suggestions or other use cases of topic level maxMessageSize will be
> appreciated.
>
> Thanks,
> Haiting Jiang
>
> [1] https://github.com/apache/pulsar/issues/13360
> [2] https://github.com/apache/pulsar/pull/8732
> [3] https://github.com/apache/pulsar/issues/12958
> [4] https://github.com/apache/pulsar/pull/13147
>
Re: [DISCUSSION] Produce chunk messages failed when topic level maxMessageSize is set
+1
We'd better skip the topic level maxMessageSize check for chunk messages.
Best,
Hang
PengHui Li 于2021年12月27日周一 22:07写道：
>
> +1,
>
> We can only skip the topic level messages size check for the chunk message.
>
> Regards,
> Penghui
>
> On Mon, Dec 20, 2021 at 3:37 PM Haiting Jiang
> wrote:
>
> > Hi Pulsar Community,
> >
> > I discovered a bug that chunk messages producing fails if topic level
> > maxMessageSize is set [1]. The root cause of this issue is because chunk
> > message is using broker level maxMessageSize as chunk size. And topic level
> > maxMessageSize is always <= broker level maxMessageSize. So once it is set,
> > the on-going chunk message producing fails.
> >
> > ## Proposed changes
> > I would like to fix this by just skipping topic level maxMessageSize check
> > in
> > org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize.Topic
> > level maxMessageSize is introduced in [2], for the purpose of "easier to
> > plan resource quotas for client allocation". And IMO this change will not
> > bring further complex into this.
> >
> > ## Alternative
> > Add a client side topic level maxMessageSize and keep it synced with
> > broker.
> >
> > Required changes:
> > - [client] Add a new field
> > org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to store this
> > client side topic level maxMessageSize.
> > - [PulsarApi.proto] Add a new field maxMessageSize in the
> > CommandProducerSuccess for the initial value of ProducerBase#maxMessageSize
> > - [PulsarApi.proto] Add a new Command like
> > CommandUpdateClientPolicy{producerId, maxMessageSize} to update
> > ProducerBase#maxMessageSize when topic level maxMessageSize is updated.
> > Further more, some other data consistency issues need be handled very
> > carefully when maxMessageSize is updated.
> > This alternative is complex but can also solve other topic level
> > maxMessageSize issue [3] when batching is enabled (non-batching case is
> > solved with PR [4]).
> >
> > Any suggestions or other use cases of topic level maxMessageSize will be
> > appreciated.
> >
> > Thanks,
> > Haiting Jiang
> >
> > [1] https://github.com/apache/pulsar/issues/13360
> > [2] https://github.com/apache/pulsar/pull/8732
> > [3] https://github.com/apache/pulsar/issues/12958
> > [4] https://github.com/apache/pulsar/pull/13147
> >
[ANNOUNCE] Apache Pulsar 2.7.4 released
The Apache Pulsar team is proud to announce Apache Pulsar version 2.7.4. Pulsar is a highly scalable, low latency messaging platform running on commodity hardware. It provides simple pub-sub semantics over topics, guaranteed at-least-once delivery of messages, automatic cursor management for subscribers, and cross-datacenter replication. For Pulsar release details and downloads, visit: https://pulsar.apache.org/download Release Notes are at: http://pulsar.apache.org/release-notes We would like to thank the contributors that made the release possible. Regards Jiwei Guo (Tboy)
Re: [DISCUSSION] Produce chunk messages failed when topic level maxMessageSize is set
+1
Skip the topic level setting is better
Hang Chen 於 2021年12月27日 週一 22:09 寫道：
> +1
>
> We'd better skip the topic level maxMessageSize check for chunk messages.
>
> Best,
> Hang
>
> PengHui Li 于2021年12月27日周一 22:07写道：
> >
> > +1,
> >
> > We can only skip the topic level messages size check for the chunk
> message.
> >
> > Regards,
> > Penghui
> >
> > On Mon, Dec 20, 2021 at 3:37 PM Haiting Jiang
> > wrote:
> >
> > > Hi Pulsar Community,
> > >
> > > I discovered a bug that chunk messages producing fails if topic level
> > > maxMessageSize is set [1]. The root cause of this issue is because
> chunk
> > > message is using broker level maxMessageSize as chunk size. And topic
> level
> > > maxMessageSize is always <= broker level maxMessageSize. So once it is
> set,
> > > the on-going chunk message producing fails.
> > >
> > > ## Proposed changes
> > > I would like to fix this by just skipping topic level maxMessageSize
> check
> > > in
> > >
> org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize.Topic
> > > level maxMessageSize is introduced in [2], for the purpose of "easier
> to
> > > plan resource quotas for client allocation". And IMO this change will
> not
> > > bring further complex into this.
> > >
> > > ## Alternative
> > > Add a client side topic level maxMessageSize and keep it synced with
> > > broker.
> > >
> > > Required changes:
> > > - [client] Add a new field
> > > org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to store this
> > > client side topic level maxMessageSize.
> > > - [PulsarApi.proto] Add a new field maxMessageSize in the
> > > CommandProducerSuccess for the initial value of
> ProducerBase#maxMessageSize
> > > - [PulsarApi.proto] Add a new Command like
> > > CommandUpdateClientPolicy{producerId, maxMessageSize} to update
> > > ProducerBase#maxMessageSize when topic level maxMessageSize is updated.
> > > Further more, some other data consistency issues need be handled very
> > > carefully when maxMessageSize is updated.
> > > This alternative is complex but can also solve other topic level
> > > maxMessageSize issue [3] when batching is enabled (non-batching case is
> > > solved with PR [4]).
> > >
> > > Any suggestions or other use cases of topic level maxMessageSize will
> be
> > > appreciated.
> > >
> > > Thanks,
> > > Haiting Jiang
> > >
> > > [1] https://github.com/apache/pulsar/issues/13360
> > > [2] https://github.com/apache/pulsar/pull/8732
> > > [3] https://github.com/apache/pulsar/issues/12958
> > > [4] https://github.com/apache/pulsar/pull/13147
> > >
>
Re: [DISCUSSION] Produce chunk messages failed when topic level maxMessageSize is set
+1
Regards
Jiwei Guo (Tboy)
On Tue, Dec 28, 2021 at 5:44 AM 陳智弘 wrote:
>
> +1
>
> Skip the topic level setting is better
>
> Hang Chen 於 2021年12月27日 週一 22:09 寫道：
>
> > +1
> >
> > We'd better skip the topic level maxMessageSize check for chunk messages.
> >
> > Best,
> > Hang
> >
> > PengHui Li 于2021年12月27日周一 22:07写道：
> > >
> > > +1,
> > >
> > > We can only skip the topic level messages size check for the chunk
> > message.
> > >
> > > Regards,
> > > Penghui
> > >
> > > On Mon, Dec 20, 2021 at 3:37 PM Haiting Jiang
> > > wrote:
> > >
> > > > Hi Pulsar Community,
> > > >
> > > > I discovered a bug that chunk messages producing fails if topic level
> > > > maxMessageSize is set [1]. The root cause of this issue is because
> > chunk
> > > > message is using broker level maxMessageSize as chunk size. And topic
> > level
> > > > maxMessageSize is always <= broker level maxMessageSize. So once it is
> > set,
> > > > the on-going chunk message producing fails.
> > > >
> > > > ## Proposed changes
> > > > I would like to fix this by just skipping topic level maxMessageSize
> > check
> > > > in
> > > >
> > org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize.Topic
> > > > level maxMessageSize is introduced in [2], for the purpose of "easier
> > to
> > > > plan resource quotas for client allocation". And IMO this change will
> > not
> > > > bring further complex into this.
> > > >
> > > > ## Alternative
> > > > Add a client side topic level maxMessageSize and keep it synced with
> > > > broker.
> > > >
> > > > Required changes:
> > > > - [client] Add a new field
> > > > org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to store this
> > > > client side topic level maxMessageSize.
> > > > - [PulsarApi.proto] Add a new field maxMessageSize in the
> > > > CommandProducerSuccess for the initial value of
> > ProducerBase#maxMessageSize
> > > > - [PulsarApi.proto] Add a new Command like
> > > > CommandUpdateClientPolicy{producerId, maxMessageSize} to update
> > > > ProducerBase#maxMessageSize when topic level maxMessageSize is updated.
> > > > Further more, some other data consistency issues need be handled very
> > > > carefully when maxMessageSize is updated.
> > > > This alternative is complex but can also solve other topic level
> > > > maxMessageSize issue [3] when batching is enabled (non-batching case is
> > > > solved with PR [4]).
> > > >
> > > > Any suggestions or other use cases of topic level maxMessageSize will
> > be
> > > > appreciated.
> > > >
> > > > Thanks,
> > > > Haiting Jiang
> > > >
> > > > [1] https://github.com/apache/pulsar/issues/13360
> > > > [2] https://github.com/apache/pulsar/pull/8732
> > > > [3] https://github.com/apache/pulsar/issues/12958
> > > > [4] https://github.com/apache/pulsar/pull/13147
> > > >
> >
Re: [DISCUSS] Release Pulsar 2.7.4
Hi Shivji, The 2.7.4 has been already released and includes 2.17.0. Regards Jiwei Guo (Tboy) On Tue, Dec 21, 2021 at 6:47 PM Shivji Kumar Jha wrote: > > Hi Pulsar Team, > > 2.16.0 is prone to DDoS attacks [1]. Is it possible to move to 2.17.0 in > pulsar 2.7.4 ? > > [1] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45105 > > Regards, > Shivji Kumar Jha > http://www.shivjijha.com/ > +91 8884075512 > > > On Tue, 21 Dec 2021 at 02:46, Michael Marshall wrote: > > > Thank you Penghui, Technoboy, and BewareMyPower for picking up where I > > left off to get the 2.7.4 release ready to go. You cherry-picked many > > commits, > > resolved many conflicts, and fixed a bunch of tests! > > > > Thanks, > > Michael > > > > > > On Thu, Dec 16, 2021 at 7:37 PM guo jiwei wrote: > > > > > > Hi, > > > After we have fixed some issue like ZookeeperCache NPE, listing namespace > > > exception, and skip some flaky tests (verified locally), now the CI have > > > passed. > > > Skipped flaky tests are tracked here: > > > https://github.com/apache/pulsar/issues/13299 > > > Now we decide to vote for releasing 2.7.4. > > > > > > Regards > > > Jiwei Guo (Tboy) > > > > > > > > > On Tue, Dec 14, 2021 at 11:58 AM PengHui Li wrote: > > > > > > > Thanks for the update, I will move it 2.7.5 > > > > > > > > Thanks, > > > > Penghui > > > > > > > > On Tue, Dec 14, 2021 at 9:47 AM Matteo Merli > > > > wrote: > > > > > > > > > Let's take https://github.com/apache/pulsar/pull/12484 out of the > > > > > picture since it's failing the tests. > > > > > > > > > > > > > > > -- > > > > > Matteo Merli > > > > > > > > > > > > > > > On Sun, Dec 12, 2021 at 11:06 PM PengHui Li > > wrote: > > > > > > > > > > > > Yes, > > > > > > > > > > > > https://github.com/apache/pulsar/pull/13215 has cherry-picked, so > > we > > > > can > > > > > > close it. > > > > > > https://github.com/apache/pulsar/pull/12484 blocked by the test. > > > > > > > > > > > > Penghui > > > > > > > > > > > > On Mon, Dec 13, 2021 at 2:35 PM Dave Fisher < wave4d...@comcast.net > > > > > > > > wrote: > > > > > > > > > > > > > I see 2 PRs still open at > > > > > > > > > > > > > > > > > > https://github.com/apache/pulsar/pulls?q=is%3Aopen+is%3Apr+label%3Arelease%2F2.7.4 > > > > > > > > > > > > > > Sent from my iPhone > > > > > > > > > > > > > > > On Dec 12, 2021, at 8:22 PM, guo jiwei > > > > wrote: > > > > > > > > > > > > > > > > I have pushed out some fixes in > > > > > > > https://github.com/apache/pulsar/pull/13243 > > > > > > > > After the tests get passed, I will send out the RC-1 VOTE for > > 2.7.4 > > > > > > > > > > > > > > > > Regards > > > > > > > > Jiwei Guo (Tboy) > > > > > > > > > > > > > > > > > > > > > > > >> On Sun, Dec 12, 2021 at 3:11 PM PengHui Li < > > peng...@apache.org> > > > > > wrote: > > > > > > > >> > > > > > > > >> Just put an update here. We have done the PR cherry-picking > > > > > > > >> > > > > > > > >> https://github.com/apache/pulsar/commits/branch-2.7 > > > > > > > >> > > > > > > > >> And most of the integration tests are fixed due to the docker > > > > image > > > > > > > issue > > > > > > > >> or the testcontainer issue, now some integration tests get > > passed, > > > > > but > > > > > > > some > > > > > > > >> are not. > > > > > > > >> And there are some failed tests, maybe a flaky test, we need > > to > > > > > ensure > > > > > > > it's > > > > > > > >> not a regression. > > > > > > > >> > > > > > > > >> We are continuing on the test part. > > > > > > > >> > > > > > > > >> Penghui > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >>> On Sat, Dec 11, 2021 at 5:36 PM PengHui Li < > > peng...@apache.org> > > > > > wrote: > > > > > > > >>> > > > > > > > >>> Hi Michael, > > > > > > > >>> > > > > > > > >>> +1, > > > > > > > >>> > > > > > > > >>> Thanks for the great work. > > > > > > > >>> We will continue on the PR cherry-picking and the release > > process > > > > > to > > > > > > > make > > > > > > > >>> sure the urgent release can be done ASAP. > > > > > > > >>> > > > > > > > >>> Penghui > > > > > > > >>> > > > > > > > >>> On Sat, Dec 11, 2021 at 3:42 PM Michael Marshall < > > > > > mmarsh...@apache.org > > > > > > > > > > > > > > > >>> wrote: > > > > > > > >>> > > > > > > > Given the log4j CVE, we should work to release 2.7.4. > > > > > > > > > > > > > > I started preparing the release today by cherry-picking > > merged > > > > PRs > > > > > > > that have the `release/2.7.4` label but have not yet been > > > > > > > cherry-picked to `branch-2.7` [0]. There are still 37 PRs > > that > > > > > have > > > > > > > not been cherry picked. I think it will take too long to > > cherry > > > > > pick > > > > > > > all of these commits, as many have conflicts, and we should > > > > > prioritize > > > > > > > releasing 2.7.4. The main commits that we should get > > > > cherry-picked > > > > > > > before creating the git tag are any labeled with > > > > > `component/security`. > > > > >
Re: [DISCUSSION] Produce chunk messages failed when topic level maxMessageSize is set
+ 1 for skipping the topic level message size check for the chunk message
in the broker.
On Tue, Dec 28, 2021 at 9:14 AM guo jiwei wrote:
> +1
>
>
> Regards
> Jiwei Guo (Tboy)
>
> On Tue, Dec 28, 2021 at 5:44 AM 陳智弘 wrote:
> >
> > +1
> >
> > Skip the topic level setting is better
> >
> > Hang Chen 於 2021年12月27日 週一 22:09 寫道：
> >
> > > +1
> > >
> > > We'd better skip the topic level maxMessageSize check for chunk
> messages.
> > >
> > > Best,
> > > Hang
> > >
> > > PengHui Li 于2021年12月27日周一 22:07写道：
> > > >
> > > > +1,
> > > >
> > > > We can only skip the topic level messages size check for the chunk
> > > message.
> > > >
> > > > Regards,
> > > > Penghui
> > > >
> > > > On Mon, Dec 20, 2021 at 3:37 PM Haiting Jiang <
> jianghait...@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Pulsar Community,
> > > > >
> > > > > I discovered a bug that chunk messages producing fails if topic
> level
> > > > > maxMessageSize is set [1]. The root cause of this issue is because
> > > chunk
> > > > > message is using broker level maxMessageSize as chunk size. And
> topic
> > > level
> > > > > maxMessageSize is always <= broker level maxMessageSize. So once
> it is
> > > set,
> > > > > the on-going chunk message producing fails.
> > > > >
> > > > > ## Proposed changes
> > > > > I would like to fix this by just skipping topic level
> maxMessageSize
> > > check
> > > > > in
> > > > >
> > >
> org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize.Topic
> > > > > level maxMessageSize is introduced in [2], for the purpose of
> "easier
> > > to
> > > > > plan resource quotas for client allocation". And IMO this change
> will
> > > not
> > > > > bring further complex into this.
> > > > >
> > > > > ## Alternative
> > > > > Add a client side topic level maxMessageSize and keep it synced
> with
> > > > > broker.
> > > > >
> > > > > Required changes:
> > > > > - [client] Add a new field
> > > > > org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to store
> this
> > > > > client side topic level maxMessageSize.
> > > > > - [PulsarApi.proto] Add a new field maxMessageSize in the
> > > > > CommandProducerSuccess for the initial value of
> > > ProducerBase#maxMessageSize
> > > > > - [PulsarApi.proto] Add a new Command like
> > > > > CommandUpdateClientPolicy{producerId, maxMessageSize} to update
> > > > > ProducerBase#maxMessageSize when topic level maxMessageSize is
> updated.
> > > > > Further more, some other data consistency issues need be handled
> very
> > > > > carefully when maxMessageSize is updated.
> > > > > This alternative is complex but can also solve other topic level
> > > > > maxMessageSize issue [3] when batching is enabled (non-batching
> case is
> > > > > solved with PR [4]).
> > > > >
> > > > > Any suggestions or other use cases of topic level maxMessageSize
> will
> > > be
> > > > > appreciated.
> > > > >
> > > > > Thanks,
> > > > > Haiting Jiang
> > > > >
> > > > > [1] https://github.com/apache/pulsar/issues/13360
> > > > > [2] https://github.com/apache/pulsar/pull/8732
> > > > > [3] https://github.com/apache/pulsar/issues/12958
> > > > > [4] https://github.com/apache/pulsar/pull/13147
> > > > >
> > >
>
--
Zike Yang
Re: [DISCUSSION] Produce chunk messages failed when topic level maxMessageSize is set
It looks loke you already prepared a awesome PIP!
Please copy your proposal on a PIP.
I am +1
Enrico
Il Mar 28 Dic 2021, 04:27 Zike Yang ha
scritto:
> + 1 for skipping the topic level message size check for the chunk message
> in the broker.
>
> On Tue, Dec 28, 2021 at 9:14 AM guo jiwei wrote:
>
> > +1
> >
> >
> > Regards
> > Jiwei Guo (Tboy)
> >
> > On Tue, Dec 28, 2021 at 5:44 AM 陳智弘 wrote:
> > >
> > > +1
> > >
> > > Skip the topic level setting is better
> > >
> > > Hang Chen 於 2021年12月27日 週一 22:09 寫道：
> > >
> > > > +1
> > > >
> > > > We'd better skip the topic level maxMessageSize check for chunk
> > messages.
> > > >
> > > > Best,
> > > > Hang
> > > >
> > > > PengHui Li 于2021年12月27日周一 22:07写道：
> > > > >
> > > > > +1,
> > > > >
> > > > > We can only skip the topic level messages size check for the chunk
> > > > message.
> > > > >
> > > > > Regards,
> > > > > Penghui
> > > > >
> > > > > On Mon, Dec 20, 2021 at 3:37 PM Haiting Jiang <
> > jianghait...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi Pulsar Community,
> > > > > >
> > > > > > I discovered a bug that chunk messages producing fails if topic
> > level
> > > > > > maxMessageSize is set [1]. The root cause of this issue is
> because
> > > > chunk
> > > > > > message is using broker level maxMessageSize as chunk size. And
> > topic
> > > > level
> > > > > > maxMessageSize is always <= broker level maxMessageSize. So once
> > it is
> > > > set,
> > > > > > the on-going chunk message producing fails.
> > > > > >
> > > > > > ## Proposed changes
> > > > > > I would like to fix this by just skipping topic level
> > maxMessageSize
> > > > check
> > > > > > in
> > > > > >
> > > >
> >
> org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize.Topic
> > > > > > level maxMessageSize is introduced in [2], for the purpose of
> > "easier
> > > > to
> > > > > > plan resource quotas for client allocation". And IMO this change
> > will
> > > > not
> > > > > > bring further complex into this.
> > > > > >
> > > > > > ## Alternative
> > > > > > Add a client side topic level maxMessageSize and keep it synced
> > with
> > > > > > broker.
> > > > > >
> > > > > > Required changes:
> > > > > > - [client] Add a new field
> > > > > > org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to
> store
> > this
> > > > > > client side topic level maxMessageSize.
> > > > > > - [PulsarApi.proto] Add a new field maxMessageSize in the
> > > > > > CommandProducerSuccess for the initial value of
> > > > ProducerBase#maxMessageSize
> > > > > > - [PulsarApi.proto] Add a new Command like
> > > > > > CommandUpdateClientPolicy{producerId, maxMessageSize} to update
> > > > > > ProducerBase#maxMessageSize when topic level maxMessageSize is
> > updated.
> > > > > > Further more, some other data consistency issues need be handled
> > very
> > > > > > carefully when maxMessageSize is updated.
> > > > > > This alternative is complex but can also solve other topic level
> > > > > > maxMessageSize issue [3] when batching is enabled (non-batching
> > case is
> > > > > > solved with PR [4]).
> > > > > >
> > > > > > Any suggestions or other use cases of topic level maxMessageSize
> > will
> > > > be
> > > > > > appreciated.
> > > > > >
> > > > > > Thanks,
> > > > > > Haiting Jiang
> > > > > >
> > > > > > [1] https://github.com/apache/pulsar/issues/13360
> > > > > > [2] https://github.com/apache/pulsar/pull/8732
> > > > > > [3] https://github.com/apache/pulsar/issues/12958
> > > > > > [4] https://github.com/apache/pulsar/pull/13147
> > > > > >
> > > >
> >
>
>
> --
> Zike Yang
>
Re: [DISCUSSION] PIP-130: Apply redelivery backoff policy for ack timeout
Penghui I am overall +1 to this proposal but I am afraid about compatibility. Amd I won't to be sure that we are not breaking anything. Pulsar client and Consumer are configurable using a map of key value pairs. So we must take care of not changing the behaviour. What do you mean with 'redelivery backorder has not been released yet'? Enrico Il Lun 27 Dic 2021, 14:25 PengHui Li ha scritto: > https://github.com/apache/pulsar/issues/13528 > > Pasted below for quoting convenience. > > - > > PIP 130: Apply redelivery backoff policy for ack timeout > > ## Motivation > > PIP 106 > > https://github.com/apache/pulsar/wiki/PIP-106%3A-Negative-acknowledgment-backoff > introduced negative acknowledgment message redelivery backoff which allows > users to achieve > more flexible message redelivery delay time control. But the redelivery > backoff policy only > apply to the negative acknowledgment API, for users who use ack timeout to > trigger the message > redelivery, not the negative acknowledgment API, they can't use the new > features introduced by > PIP 106. > > So the proposal is to apply the message redelivery policy for the ack > timeout mechanism. > Users can specify an ack timeout redelivery backoff, for example, apply an > exponential backoff > with 10 seconds ack timeout: > > ```java > client.newConsumer() > .ackTimeout(10, TimeUnit.SECOND) > .ackTimeoutRedeliveryBackoff( > ExponentialRedeliveryBackoff.builder() > .minDelayMs(1000) > .maxDelayMs(6).build()); > .subscribe(); > ``` > > The message redelivery behavior should be: > > | Redelivery count | Redelivery delay | > | | | > | 1 | 10 + 1 seconds | > | 2 | 10 + 2 seconds | > | 3 | 10 + 4 seconds | > | 4 | 10 + 8 seconds | > | 5 | 10 + 16 seconds | > | 6 | 10 + 32 seconds | > | 7 | 10 + 60 seconds | > | 8 | 10 + 60 seconds | > > ## Goal > > Add an API to the Java Client to provide the ability to specify the ack > timeout message redelivery > backoff and the message redelivery behavior should abide by the redelivery > backoff policy. > > > ## API Changes > > 1. Change `NegativeAckRedeliveryBackoff` to `RedeliveryBackoff`, so that we > can use the > MessageRedeliveryBackoff for both negative acknowledgment API and ack > timeout message redelivery. > > 2. Change `NegativeAckRedeliveryExponentialBackoff` to > `ExponentialRedeliveryBackoff`, and add `multiplier` > for `RedeliveryExponentialBackoff` with default value 2. > > ExponentialRedeliveryBackoff.builder() > .minDelayMs(1000) > .maxDelayMs(6) > .multiplier(5) > .build() > > 3. Add `ackTimeoutRedeliveryBackoff` method for the `ConsumerBuilder`: > > ```java > client.newConsumer() > .ackTimeout(10, TimeUnit.SECOND) > .ackTimeoutRedeliveryBackoff( > ExponentialRedeliveryBackoff.builder() > .minDelayMs(1000) > .maxDelayMs(6).build()); > .subscribe(); > ``` > > ## Compatibility and migration plan > > Since the negative acknowledgment message, redelivery backoff has not been > released yet, > so we can modify the API directly. > > ## Tests plan > > - Verify the introduced `multiplier` of ExponentialRedeliveryBackoff > - Verify the ack timeout message redelivery work as expected > > Regards, > Penghui >
