Thanks for submitting this PIP, Nitin Goyal. Seeing that you have submitted a related pull request in the Go SDK community, I am sorry that I made the request changes first.
For the retry processing of a single message, the methods currently provided are: -ReconsumeLater -Nack In actual usage scenarios, it may be a more elegant way to retry if we verify Nack multiple times. The current implementation of ReconsumeLater relies on delayed messages. This is not an elegant way, and there is no way to record the number of retries. In order to support backoff retries, we introduced the NackBackoff strategy, which itself is an interface and exposed to users. Based on this interface, we can do more customized things. If the NackBackoff interface can't meet our current needs, I prefer that we implement more parameters in the NackBackoff strategy instead of continuing to add new functional logic in ReconsumeLater. -- Thanks Xiaolong Ran Enrico Olivelli <eolive...@gmail.com> 于2023年1月10日周二 16:23写道： > I think that Michael's point is very important. > > Producers and Consumers are decoupled and this PIP would introduce > a new concept. > Also the same Message can be consumed using multiple subscriptions > (typically Applications) > and all the applications will process the message in a different way > (by definition). > > Isn't it possible to implement what you want with an Interceptor or > with some custom handling for the DLQ ? > The DLQ (currently) in Pulsar is mostly a client-side feature. > The producer can tag the messages with a message property and the > client can decide what to do > and how many times to retry the message or give up. > > Enrico > > Il giorno mar 10 gen 2023 alle ore 01:00 Michael Marshall > <mmarsh...@apache.org> ha scritto: > > > > Thanks for submitting this PIP, Nitin Goyal. > > > > At the heart of this PIP is an assumption about the relationship > > between producers and consumers. The PIP assumes a producer knows how > > many times a consumer should attempt to consume a message before > > giving up and sending it to the DLQ. Does anyone have strong opinions > > on the boundaries between producers and consumers in relation to this > > PIP? > > > > This PIP expands the relationship between producer and consumer by > > letting the producer tell the consumer's pulsar client when to send a > > message to the DLQ, and as such, we should be very intentional about > > accepting this PIP. > > > > Because a user can easily implement this feature on their own and > > because it tightly couples producers and consumers, I think we should > > not move forward with this PIP. I am open to discussion, though. > > > > > for instance in case of major system failure you don't want to lose > > > all your messages. > > > > For what it's worth, the retry letter topic feature, which this PIP > > relies on, sends all messages to the DLQ, so this feature does not > > introduce conditions for message loss. > > > > As an aside, if we move forward with this feature, we need to make > > sure that the protocol documentation is updated and we should consider > > putting this field in the `MessageMetadata` protobuf object, not in a > > properties map. > > > > Thanks, > > Michael > > > > > > > > On Mon, Jan 9, 2023 at 9:47 AM Nitin Goyal <nitin.goyal....@gmail.com> > wrote: > > > > > > Hello Enrico, > > > > > > For your concern about temporarily increasing of retry. It can be > achieved > > > using overriding msg property while calling reconsuming later with > custom > > > properties.. > > > > > > About msg immutable as per current design if consumer call > reconsumeLater > > > function it creates a new msg in the system adding few properties to it > > > like how many times it get consumed. That also allow other custom > > > properties to be added in newly generated msg.. so if msg needs > temporary > > > high retry or change in retry count on msg they can override it using > > > custom properties… > > > > > > Thanks > > > Nitin Goyal > > > > > > > > > On Mon, 9 Jan 2023 at 1:11 PM, Enrico Olivelli <eolive...@gmail.com> > wrote: > > > > > > > I don't think that this is a good idea. > > > > > > > > Because IIUC we want to add a property per message that sets the > > > > maximum time of retries. > > > > > > > > Unfortunately in a real system sometimes you have to change the > number > > > > of retries temporarily, > > > > for instance in case of major system failure you don't want to lose > > > > all your messages. > > > > > > > > If we allow you to set a property on a message then you won't be able > > > > to change it because the message is immutable. > > > > > > > > TTL (time to live) is a similar concept but it is related to the > > > > concept of "physical time", if you have message that represents > > > > a task to be executed within a given deadline then it makes sense to > > > > state it in the message metadata. > > > > > > > > But the "number of retries" depends on how you deal with the retries: > > > > - how much time do you wait ? > > > > - how often do you have a temporary failure to retry ? > > > > > > > > It would make more sense to have a QOS (quality of service) attribute > > > > on the message, like "important/"non-important"/"foo"/"bar" > > > > and have a way for the brokers and the clients to handle that. I am > > > > pretty sure that with interceptors you can already do something. > > > > > > > > > > > > I am against hard coding the behaviour described in the PIP (and I > > > > voted -1 in the VOTE thread) > > > > > > > > Enrico > > > > > > > > Il giorno ven 6 gen 2023 alle ore 09:11 Zike Yang <z...@apache.org> > ha > > > > scritto: > > > > > > > > > > Hi, > > > > > > > > > > This looks good to me. > > > > > +1 > > > > > > > > > > I was thinking if we could add a new API for `reconsumeLater` to > let > > > > > users set the max retry times easily. But I saw that there are too > > > > > many reconsumeLater API and this will make the consumer more > complex. > > > > > > > > > > Thanks, > > > > > Zike Yang > > > > > > > > > > > > > > > On Thu, Jan 5, 2023 at 3:58 PM Zixuan Liu <node...@gmail.com> > wrote: > > > > > > > > > > > > +1 > > > > > > > > > > > > Thanks, > > > > > > Zixuan > > > > > > > > > > > > Nitin Goyal <nitin.goyal....@gmail.com> 于2023年1月5日周四 13:50写道： > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > I made a PIP to discuss: > > > > https://github.com/apache/pulsar/issues/19136 > > > > > > > > > > > > > > Thanks, > > > > > > > Nitin Goyal > > > > > > > > > > > > > > -- > > > Regards > > > Nitin Goyal >
