Hi Nitin,
Michael's point about the decoupling of producer and consumer makes sense.
Your use case is more like a queue model, not a pub-sub model. So the
decoupling is not the issue for you.
But Pulsar is mainly a pub-sub system by nature. So we must take it
into more consideration.
As in your case, If we only provide setting max-retry-time in
`Consumer#reconsumeLater`, and forbid Producer to set this value (as
in Producer API).
Will it solve the problem?
Best,
Haiting
On Tue, Jan 17, 2023 at 3:43 PM Asaf Mesika <asaf.mes...@gmail.com> wrote:
>
> From my knowledge, when a client does negative ack, after a certain amount
> of time, it sends the following command to the broker containing the
> negatively acked messages ids:
>
> message CommandRedeliverUnacknowledgedMessages {
> required uint64 consumer_id = 1;
> repeated MessageIdData message_ids = 2;
> optional uint64 consumer_epoch = 3;
> }
>
>
> On Tue, Jan 17, 2023 at 5:24 AM Nitin Goyal <nitin.goyal....@gmail.com>
> wrote:
>
> > Hello all,
> >
> > I have a few questions.
> >
> > 1. When a message is NACKed does it still send the commands to the broker?
> > 2. Do we have any plan to replace the ReconsumeLater function with NACK
> > Backoff interface in future?
> >
> >
> > On Tue, Jan 10, 2023 at 7:17 PM Nitin Goyal <nitin.goyal....@gmail.com>
> > wrote:
> >
> > > Hello Team,
> > >
> > > Thanks for pointing these things out.
> > >
> > > First thing is to consider that this addon feature is for the message
> > > which needs to go to DLQ after a retryLater reaches the maximum limit of
> > > consumer level retries. It allows an additional maximum limit if a
> > message
> > > has one at the time of producing.
> > >
> > > There are two types of retry in current architecture which consumers can
> > > go with.
> > >
> > > 1. Process later with an NACK backoff policy: In this case if someone
> > > wants to retry and wants to set a maximum retry at message processor
> > level.
> > > They can get the number of times the message has been retried so far
> > > by calling the `reconsumeTimes` function. they will match the number with
> > > their custom retry. Now there are two things that will happen. Either
> > they
> > > will call ACK that particular message or they have to call the
> > > `reconsumeLater` function based on system requirement. If they call ACK
> > > then the message is marked as delete in the ledger. And other cases would
> > > be explained in point 2.
> > >
> > > 2. By calling reconsumeLater: In some real cases if someone is calling
> > > reconsumeLater with a delay like they want to process a message again but
> > > after a particular time. so the new message gets created in the consumer
> > > RETRY topic which was again processed by the consumer (they may call the
> > > reconsumeLater again if the system requires based on use cases.). Now if
> > > the message reaches its retry limit which was configured at consumer
> > level
> > > they will be sent to the DLQ message.
> > >
> > > So my proposal will not make any changes to 1st scenario which is to use
> > > NACK backoff for retrying the message and process. Consumers can use the
> > > same mechanism as they are doing now.
> > >
> > > It adds an additional feature to pulsar in which if a producer knows how
> > > many times an event should be processed a max apart from the consumer max
> > > retry and send to DLQ if reaches its limit. Then the above change will
> > > check if the limit is reached for reconsume later then it will send to
> > DLQ
> > > even if the consumer has higher reconsme later retry limit.
> > >
> > > if producers do not pass the max retry limit it will obey the consumer
> > > retry limit.
> > >
> > > Also in this PIP we are not removing the consumer own retry limit it
> > would
> > > be there. but get overridden by message limit if passed by the producer.
> > if
> > > a consumer wants to re-override that limit ( temporarily change in the
> > > retry limit) which also can be achieved by sending a new max retry as
> > > custom property to the `reconsumeLater` function.
> > >
> > > So let me rephrase the subject for this pip is to `*MAX RECONSUME TIMES
> > > per Message*`.
> > >
> > > Michael's point about adding relation between producer and consumer?
> > >
> > > Yeah, It added a relation b/w them but it would be loose coupling.
> > Because
> > > producers can send the max retry limit in a message which can be
> > overridden
> > > by the consumer if needed. so consumers do not have to obey the limit if
> > > needed.
> > >
> > > Why I prefer message properties over `MessageMetadata` is because if a
> > > consumer wants to override the max limit (temporarily or permanent) then
> > it
> > > would be difficult to update if required.
> > >
> > > I hope the above content answers most of the doubts about this PIP.
> > >
> > > On Tue, Jan 10, 2023 at 5:59 PM r...@apache.org <ranxiaolong...@gmail.com
> > >
> > > wrote:
> > >
> > >> Thanks for submitting this PIP, Nitin Goyal.
> > >>
> > >> Seeing that you have submitted a related pull request in the Go SDK
> > >> community, I am sorry that I made the request changes first.
> > >>
> > >> For the retry processing of a single message, the methods currently
> > >> provided are:
> > >>
> > >> -ReconsumeLater
> > >> -Nack
> > >>
> > >> In actual usage scenarios, it may be a more elegant way to retry if we
> > >> verify Nack multiple times. The current implementation of ReconsumeLater
> > >> relies on delayed messages. This is not an elegant way, and there is no
> > >> way
> > >> to record the number of retries.
> > >>
> > >> In order to support backoff retries, we introduced the NackBackoff
> > >> strategy, which itself is an interface and exposed to users. Based on
> > this
> > >> interface, we can do more customized things. If the NackBackoff
> > interface
> > >> can't meet our current needs, I prefer that we implement more parameters
> > >> in
> > >> the NackBackoff strategy instead of continuing to add new functional
> > logic
> > >> in ReconsumeLater.
> > >>
> > >> --
> > >> Thanks
> > >> Xiaolong Ran
> > >>
> > >> Enrico Olivelli <eolive...@gmail.com> 于2023年1月10日周二 16:23写道：
> > >>
> > >> > I think that Michael's point is very important.
> > >> >
> > >> > Producers and Consumers are decoupled and this PIP would introduce
> > >> > a new concept.
> > >> > Also the same Message can be consumed using multiple subscriptions
> > >> > (typically Applications)
> > >> > and all the applications will process the message in a different way
> > >> > (by definition).
> > >> >
> > >> > Isn't it possible to implement what you want with an Interceptor or
> > >> > with some custom handling for the DLQ ?
> > >> > The DLQ (currently) in Pulsar is mostly a client-side feature.
> > >> > The producer can tag the messages with a message property and the
> > >> > client can decide what to do
> > >> > and how many times to retry the message or give up.
> > >> >
> > >> > Enrico
> > >> >
> > >> > Il giorno mar 10 gen 2023 alle ore 01:00 Michael Marshall
> > >> > <mmarsh...@apache.org> ha scritto:
> > >> > >
> > >> > > Thanks for submitting this PIP, Nitin Goyal.
> > >> > >
> > >> > > At the heart of this PIP is an assumption about the relationship
> > >> > > between producers and consumers. The PIP assumes a producer knows
> > how
> > >> > > many times a consumer should attempt to consume a message before
> > >> > > giving up and sending it to the DLQ. Does anyone have strong
> > opinions
> > >> > > on the boundaries between producers and consumers in relation to
> > this
> > >> > > PIP?
> > >> > >
> > >> > > This PIP expands the relationship between producer and consumer by
> > >> > > letting the producer tell the consumer's pulsar client when to send
> > a
> > >> > > message to the DLQ, and as such, we should be very intentional about
> > >> > > accepting this PIP.
> > >> > >
> > >> > > Because a user can easily implement this feature on their own and
> > >> > > because it tightly couples producers and consumers, I think we
> > should
> > >> > > not move forward with this PIP. I am open to discussion, though.
> > >> > >
> > >> > > > for instance in case of major system failure you don't want to
> > lose
> > >> > > > all your messages.
> > >> > >
> > >> > > For what it's worth, the retry letter topic feature, which this PIP
> > >> > > relies on, sends all messages to the DLQ, so this feature does not
> > >> > > introduce conditions for message loss.
> > >> > >
> > >> > > As an aside, if we move forward with this feature, we need to make
> > >> > > sure that the protocol documentation is updated and we should
> > consider
> > >> > > putting this field in the `MessageMetadata` protobuf object, not in
> > a
> > >> > > properties map.
> > >> > >
> > >> > > Thanks,
> > >> > > Michael
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Mon, Jan 9, 2023 at 9:47 AM Nitin Goyal <
> > nitin.goyal....@gmail.com
> > >> >
> > >> > wrote:
> > >> > > >
> > >> > > > Hello Enrico,
> > >> > > >
> > >> > > > For your concern about temporarily increasing of retry. It can be
> > >> > achieved
> > >> > > > using overriding msg property while calling reconsuming later with
> > >> > custom
> > >> > > > properties..
> > >> > > >
> > >> > > > About msg immutable as per current design if consumer call
> > >> > reconsumeLater
> > >> > > > function it creates a new msg in the system adding few properties
> > >> to it
> > >> > > > like how many times it get consumed. That also allow other custom
> > >> > > > properties to be added in newly generated msg.. so if msg needs
> > >> > temporary
> > >> > > > high retry or change in retry count on msg they can override it
> > >> using
> > >> > > > custom properties…
> > >> > > >
> > >> > > > Thanks
> > >> > > > Nitin Goyal
> > >> > > >
> > >> > > >
> > >> > > > On Mon, 9 Jan 2023 at 1:11 PM, Enrico Olivelli <
> > eolive...@gmail.com
> > >> >
> > >> > wrote:
> > >> > > >
> > >> > > > > I don't think that this is a good idea.
> > >> > > > >
> > >> > > > > Because IIUC we want to add a property per message that sets the
> > >> > > > > maximum time of retries.
> > >> > > > >
> > >> > > > > Unfortunately in a real system sometimes you have to change the
> > >> > number
> > >> > > > > of retries temporarily,
> > >> > > > > for instance in case of major system failure you don't want to
> > >> lose
> > >> > > > > all your messages.
> > >> > > > >
> > >> > > > > If we allow you to set a property on a message then you won't be
> > >> able
> > >> > > > > to change it because the message is immutable.
> > >> > > > >
> > >> > > > > TTL (time to live) is a similar concept but it is related to the
> > >> > > > > concept of "physical time", if you have message that represents
> > >> > > > > a task to be executed within a given deadline then it makes
> > sense
> > >> to
> > >> > > > > state it in the message metadata.
> > >> > > > >
> > >> > > > > But the "number of retries" depends on how you deal with the
> > >> retries:
> > >> > > > > - how much time do you wait ?
> > >> > > > > - how often do you have a temporary failure to retry ?
> > >> > > > >
> > >> > > > > It would make more sense to have a QOS (quality of service)
> > >> attribute
> > >> > > > > on the message, like "important/"non-important"/"foo"/"bar"
> > >> > > > > and have a way for the brokers and the clients to handle that. I
> > >> am
> > >> > > > > pretty sure that with interceptors you can already do something.
> > >> > > > >
> > >> > > > >
> > >> > > > > I am against hard coding the behaviour described in the PIP
> > (and I
> > >> > > > > voted -1 in the VOTE thread)
> > >> > > > >
> > >> > > > > Enrico
> > >> > > > >
> > >> > > > > Il giorno ven 6 gen 2023 alle ore 09:11 Zike Yang <
> > >> z...@apache.org>
> > >> > ha
> > >> > > > > scritto:
> > >> > > > > >
> > >> > > > > > Hi,
> > >> > > > > >
> > >> > > > > > This looks good to me.
> > >> > > > > > +1
> > >> > > > > >
> > >> > > > > > I was thinking if we could add a new API for `reconsumeLater`
> > to
> > >> > let
> > >> > > > > > users set the max retry times easily. But I saw that there are
> > >> too
> > >> > > > > > many reconsumeLater API and this will make the consumer more
> > >> > complex.
> > >> > > > > >
> > >> > > > > > Thanks,
> > >> > > > > > Zike Yang
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On Thu, Jan 5, 2023 at 3:58 PM Zixuan Liu <node...@gmail.com>
> > >> > wrote:
> > >> > > > > > >
> > >> > > > > > > +1
> > >> > > > > > >
> > >> > > > > > > Thanks,
> > >> > > > > > > Zixuan
> > >> > > > > > >
> > >> > > > > > > Nitin Goyal <nitin.goyal....@gmail.com> 于2023年1月5日周四
> > 13:50写道：
> > >> > > > > > >
> > >> > > > > > > > Hi all,
> > >> > > > > > > >
> > >> > > > > > > > I made a PIP to discuss:
> > >> > > > > https://github.com/apache/pulsar/issues/19136
> > >> > > > > > > >
> > >> > > > > > > > Thanks,
> > >> > > > > > > > Nitin Goyal
> > >> > > > > > > >
> > >> > > > >
> > >> > > > --
> > >> > > > Regards
> > >> > > > Nitin Goyal
> > >> >
> > >>
> > >
> > >
> > > --
> > > Regards
> > > Nitin Goyal
> > >
> >
> >
> > --
> > Regards
> > Nitin Goyal
> >
