I think that Michael's point is very important. Producers and Consumers are decoupled and this PIP would introduce a new concept. Also the same Message can be consumed using multiple subscriptions (typically Applications) and all the applications will process the message in a different way (by definition).
Isn't it possible to implement what you want with an Interceptor or with some custom handling for the DLQ ? The DLQ (currently) in Pulsar is mostly a client-side feature. The producer can tag the messages with a message property and the client can decide what to do and how many times to retry the message or give up. Enrico Il giorno mar 10 gen 2023 alle ore 01:00 Michael Marshall <mmarsh...@apache.org> ha scritto: > > Thanks for submitting this PIP, Nitin Goyal. > > At the heart of this PIP is an assumption about the relationship > between producers and consumers. The PIP assumes a producer knows how > many times a consumer should attempt to consume a message before > giving up and sending it to the DLQ. Does anyone have strong opinions > on the boundaries between producers and consumers in relation to this > PIP? > > This PIP expands the relationship between producer and consumer by > letting the producer tell the consumer's pulsar client when to send a > message to the DLQ, and as such, we should be very intentional about > accepting this PIP. > > Because a user can easily implement this feature on their own and > because it tightly couples producers and consumers, I think we should > not move forward with this PIP. I am open to discussion, though. > > > for instance in case of major system failure you don't want to lose > > all your messages. > > For what it's worth, the retry letter topic feature, which this PIP > relies on, sends all messages to the DLQ, so this feature does not > introduce conditions for message loss. > > As an aside, if we move forward with this feature, we need to make > sure that the protocol documentation is updated and we should consider > putting this field in the `MessageMetadata` protobuf object, not in a > properties map. > > Thanks, > Michael > > > > On Mon, Jan 9, 2023 at 9:47 AM Nitin Goyal <nitin.goyal....@gmail.com> wrote: > > > > Hello Enrico, > > > > For your concern about temporarily increasing of retry. It can be achieved > > using overriding msg property while calling reconsuming later with custom > > properties.. > > > > About msg immutable as per current design if consumer call reconsumeLater > > function it creates a new msg in the system adding few properties to it > > like how many times it get consumed. That also allow other custom > > properties to be added in newly generated msg.. so if msg needs temporary > > high retry or change in retry count on msg they can override it using > > custom properties… > > > > Thanks > > Nitin Goyal > > > > > > On Mon, 9 Jan 2023 at 1:11 PM, Enrico Olivelli <eolive...@gmail.com> wrote: > > > > > I don't think that this is a good idea. > > > > > > Because IIUC we want to add a property per message that sets the > > > maximum time of retries. > > > > > > Unfortunately in a real system sometimes you have to change the number > > > of retries temporarily, > > > for instance in case of major system failure you don't want to lose > > > all your messages. > > > > > > If we allow you to set a property on a message then you won't be able > > > to change it because the message is immutable. > > > > > > TTL (time to live) is a similar concept but it is related to the > > > concept of "physical time", if you have message that represents > > > a task to be executed within a given deadline then it makes sense to > > > state it in the message metadata. > > > > > > But the "number of retries" depends on how you deal with the retries: > > > - how much time do you wait ? > > > - how often do you have a temporary failure to retry ? > > > > > > It would make more sense to have a QOS (quality of service) attribute > > > on the message, like "important/"non-important"/"foo"/"bar" > > > and have a way for the brokers and the clients to handle that. I am > > > pretty sure that with interceptors you can already do something. > > > > > > > > > I am against hard coding the behaviour described in the PIP (and I > > > voted -1 in the VOTE thread) > > > > > > Enrico > > > > > > Il giorno ven 6 gen 2023 alle ore 09:11 Zike Yang <z...@apache.org> ha > > > scritto: > > > > > > > > Hi, > > > > > > > > This looks good to me. > > > > +1 > > > > > > > > I was thinking if we could add a new API for `reconsumeLater` to let > > > > users set the max retry times easily. But I saw that there are too > > > > many reconsumeLater API and this will make the consumer more complex. > > > > > > > > Thanks, > > > > Zike Yang > > > > > > > > > > > > On Thu, Jan 5, 2023 at 3:58 PM Zixuan Liu <node...@gmail.com> wrote: > > > > > > > > > > +1 > > > > > > > > > > Thanks, > > > > > Zixuan > > > > > > > > > > Nitin Goyal <nitin.goyal....@gmail.com> 于2023年1月5日周四 13:50写道： > > > > > > > > > > > Hi all, > > > > > > > > > > > > I made a PIP to discuss: > > > https://github.com/apache/pulsar/issues/19136 > > > > > > > > > > > > Thanks, > > > > > > Nitin Goyal > > > > > > > > > > > -- > > Regards > > Nitin Goyal
