I am trying to draw a conclusion on this email thread. > Maybe some way to plug to the broker some logic without interfering with its core? > In our business fixed delay at consumer level regardless of any producer > configuration is a big win due to easy implementation and usage.
Based on Ezequiel's last comment, if we are able to find a way to plug a new "fixed delay" dispatcher without touching other dispatcher logic, is that a good approach for the community to proceed on this direction? - Sijie On Wed, Feb 20, 2019 at 8:26 AM 李鹏辉gmail <codelipeng...@gmail.com> wrote: > Sorry for hear that DLQ causes GC. > > Agree with discussed before, Dispatcher is a performance sensitive piece > of code. > If we make changes on the dispatcher, we must pay attention to memory > overhead and blocking. > > I prefer fixed delayed message solution(aka delayed time level). User > can define multi topics with deferent delay.Topic is still a FIFO model. > > Improve user experience by packaging client API, topics can be created > automatically, User can customize the delay level. > > In our scene, This can already meet most of the needs. Currently depends > on DLQ feature. We know from the user where the experience is not very > good. > User need to maintain the message expired. > > So, If we can avoid complexity of use and do not impose a performance > burden > on message dispatching. I prefer implement it on broker side(broker do not > need to sorting messages by time, just need to check the tail message > can be dispatch, i don’t think this will cause dispatching performance > problem). > > For more complicated delayed messages(e.g. arbitrary delayed delivery). > I don’t think pulsar need to support such complicated scene(after we > discussed before). > In our scene, we have more complicated message requirement(e.g. delay > message can be > paused, stoped, and re-run. e.g. cron messages). > > However these case is not very widely used. > > - Penghui > > > > 在 2019年2月20日，06:37，Sebastián Schepens > <sebastian.schep...@mercadolibre.com.INVALID> 写道： > > > > Hi, > > I am really not into any details of the proposed implementation, but was > > just wondering, has anyone had a look at how Uber implemented this in > > Cherami? Cherami seems very similar to Pulsar, its storage system also > > seems very similar to bookkeeper. They seem to implement delayed queues > by > > storing the time as part of the key in rocksdb and using sorted > iterators, > > could this be done in Pulsar as well? > > > > Cheers, > > Sebastian > > > > On Tue, Feb 19, 2019 at 6:02 PM Dave Fisher <dave2w...@comcast.net> > wrote: > > > >> Hi - > >> > >> Well, it does, but can this be implemented without building a > delayQueue? > >> It seems to me that a delayQueue both breaks resiliency if the broker > goes > >> down and would certainly add overhead. Perhaps my idea to discard > responses > >> that are too new and then retrieve once they are out of the delayed > >> timeframe would be simpler? > >> > >> Again I am somewhat naive to the details. I’m not sure that the path > >> through the code is kept to an absolute minimum when you have a Consumer > >> with a nonzero delay? > >> > >> Regards, > >> Dave > >> > >>> On Feb 19, 2019, at 12:39 PM, Ezequiel Lovelle < > >> ezequiellove...@gmail.com> wrote: > >>> > >>> Hi Dave! > >>> > >>>> I wonder if clients can add an optional argument to the broker call > when > >>> pulling events. The argument would be the amount of delay. Any messages > >>> younger than the delay are not returned by the broker. > >>> > >>> This is exactly what https://github.com/apache/pulsar/pull/3155 does > :). > >>> We still need to decide if we want to add this feature at client side > or > >>> broker side, the pull request does it on the broker. > >>> > >>> -- > >>> *Ezequiel Lovelle* > >>> > >>> > >>> On Tue, 19 Feb 2019 at 17:06, Dave Fisher <dave2w...@comcast.net> > wrote: > >>> > >>>> Hi - > >>>> > >>>> My thoughts here may be completely useless but I wonder if clients can > >> add > >>>> an optional argument to the broker call when pulling events. The > >> argument > >>>> would be the amount of delay. Any messages younger than the delay are > >> not > >>>> returned by the broker. > >>>> > >>>> Regards, > >>>> Dave > >>>> > >>>>> On Feb 19, 2019, at 11:47 AM, Ezequiel Lovelle < > >>>> ezequiellove...@gmail.com> wrote: > >>>>> > >>>>>> The recent changes made to support DLQ caused major problems with > >>>> garbage > >>>>> collection > >>>>> > >>>>> If garbage collection is a big concern maybe we could add some config > >>>>> parameter on the broker to disable the usage of this feature and > return > >>>>> BrokerMetadataException in this situation, giving the power to the > >>>>> administrator whether to offer this feature or not. > >>>>> > >>>>>> is it acceptable to do it at broker side? > >>>>> > >>>>> I think this is the big question that needs to be answered. > >>>>> > >>>>>> can we just have a separated dispatcher for fixed delayed > >> subscription? > >>>>> > >>>>> I will try to do a completely new approach, simpler, and more > isolated > >>>>> from broker logic. Maybe some way to plug to the broker some logic > >>>> without > >>>>> interfering with its core? > >>>>> > >>>>> In our business fixed delay at consumer level regardless of any > >> producer > >>>>> configuration is a big win due to easy implementation and usage. > >>>>> > >>>>> -- > >>>>> *Ezequiel Lovelle* > >>>>> > >>>>> > >>>>> On Wed, 13 Feb 2019 at 23:25, Sijie Guo <guosi...@gmail.com> wrote: > >>>>> > >>>>>> Agreed that dispatcher is a performance sensitive piece of code. > Feel > >>>> bad > >>>>>> to hear that DLQ causes GC. Are there any issues tracking those > items > >>>> you > >>>>>> guys identified with DLQ changes? > >>>>>> > >>>>>>> How is this different from a subscription running behind? > >>>>>> > >>>>>> As far as I understand form the discussion at #3155, I don't think > >>>> there is > >>>>>> a fundamental difference from a backlogged subscriber. > >>>>>> The discussion point will mainly be - if a delayed subscription can > be > >>>>>> implemented with a simpler approach at broker side without changing > >>>> other > >>>>>> dispatcher logic, > >>>>>> is it acceptable to do it at broker side? So we don't have to > >>>> reimplement > >>>>>> the same mechanism at different language clients. I think that's the > >>>> same > >>>>>> tradeoff we were discussing for generic delayed messages. > >>>>>> > >>>>>> My thought would be - can we just have a separated dispatcher for > >> fixed > >>>>>> delayed subscription? The logic can be ISOLATED from other normal > >>>>>> dispatchers. if users don't enable delayed subscription, they will > not > >>>>>> exercise that dispatcher. This can be a good direction to explore > for > >>>>>> future changes that are related to dispatchers. > >>>>>> > >>>>>> - Sijie > >>>>>> > >>>>>> > >>>>>> On Thu, Feb 14, 2019 at 8:43 AM Joe F <joefranc...@gmail.com> > wrote: > >>>>>> > >>>>>>> Delayed subscription is simpler, and probably worth doing in the > >> broker > >>>>>> IF > >>>>>>> done right. > >>>>>>> > >>>>>>> How is this different from a subscription running behind? Why does > >>>>>>> supporting that require this complex a change in the dispatcher, > when > >>>> we > >>>>>>> already support backlogged subscribers? > >>>>>>> > >>>>>>> I am extremely wary of changes in the dispatcher. The recent > changes > >>>> made > >>>>>>> to support DLQ caused major problems with garbage collection, > broker > >>>>>>> failure and service interruptions for us. Even though we ARE NOT > >> using > >>>>>> the > >>>>>>> DLQ feature. Not a pleasant experience. > >>>>>>> > >>>>>>> This is a very performance sensitive piece of code, and it should > be > >>>>>>> treated as such. > >>>>>>> > >>>>>>> Joe > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Wed, Feb 13, 2019 at 3:58 PM Sijie Guo <guosi...@gmail.com> > >> wrote: > >>>>>>> > >>>>>>>> Hi all, > >>>>>>>> > >>>>>>>> I am going to wrap up the discussion regarding delayed delivery > use > >>>>>>> cases. > >>>>>>>> > >>>>>>>> For arbitrary delayed delivery, there are a few +1s to doing > PIP-26 > >> in > >>>>>>>> functions. I am assuming that we will go down this path, unless > >> there > >>>>>> are > >>>>>>>> other proposals. > >>>>>>>> > >>>>>>>> However there is a use case Lovelle pointed out about "Fixed > Delayed > >>>>>>>> Message". More specifically it is > >>>>>>>> https://github.com/apache/pulsar/pull/3155 > >>>>>>>> (The caption in #3155 is a bit misleading). IMO it is a "delayed > >>>>>>>> subscription", basically all messages in the subscription is > delayed > >>>> to > >>>>>>>> dispatch in a given time interval. The consensus of this feature > is > >>>> not > >>>>>>> yet > >>>>>>>> achieved. Basically, there will be two approaches for this: > >>>>>>>> > >>>>>>>> a) DONT treat "fixed delayed message" as a different case. Just > use > >>>> the > >>>>>>>> same approach as in PIP-26. > >>>>>>>> b) treat "fixed delayed message" as a different case, e.g. we can > >>>>>> better > >>>>>>>> call it "delayed subscription" or whatever can distinguish it from > >>>>>>> general > >>>>>>>> arbitrary delayed delivery. Use the approach proposed/discussed in > >>>>>> #3155. > >>>>>>>> > >>>>>>>> I would like the community to discuss this and also come to an > >>>>>> agreement. > >>>>>>>> So Lovelle can move forward with the approach agreed by the > >> community. > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Sijie > >>>>>>>> > >>>>>>>> On Tue, Jan 29, 2019 at 6:30 AM Ezequiel Lovelle < > >>>>>>>> ezequiellove...@gmail.com> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> "I agree, but that is *not what #3155 tries to achieve." > >>>>>>>>> > >>>>>>>>> This typo made this phrase nonsense, sorry! > >>>>>>>>> > >>>>>>>>> On Mon, 28 Jan 2019, 16:44 Ezequiel Lovelle < > >>>>>> ezequiellove...@gmail.com > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>>> What exactly is the delayed delivery use case? > >>>>>>>>>> > >>>>>>>>>> This is helpful on systems relaying on pulsar for persistent > >>>>>>> guarantees > >>>>>>>>>> and using it for synchronization or some sort of checks, but on > >>>>>> such > >>>>>>>>>> systems is common to have some overhead committing data on > >>>>>> persistent > >>>>>>>>>> storage maybe due to buffered mechanism or distributing the data > >>>>>>> across > >>>>>>>>>> the network before being available. > >>>>>>>>>> > >>>>>>>>>> Surely would be more use cases I don't came across right now. > >>>>>>>>>> > >>>>>>>>>>> Random insertion and deletion is not what FIFO queues like > Pulsar > >>>>>>> are > >>>>>>>>>> designed for. > >>>>>>>>>> > >>>>>>>>>> I agree, but that is now what #3155 tries to achieve. #3155 is > >>>>>> just a > >>>>>>>>>> fixed delay for all message in a consumer, that's the reason > that > >>>>>> the > >>>>>>>>>> implementation of #3155 is quite trivial. > >>>>>>>>>> > >>>>>>>>>> +1 from me for doing PIP-26 in functions. > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> *Ezequiel Lovelle* > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Sat, 26 Jan 2019 at 09:57, Yuva raj <uvar...@gmail.com> > wrote: > >>>>>>>>>> > >>>>>>>>>>> Considering the way pulsar is built +1 for doing PIP-26 in > >>>>>>> functions. > >>>>>>>> I > >>>>>>>>> am > >>>>>>>>>>> more of thinking in a way like publish it pulsar we will make > it > >>>>>>>>> available > >>>>>>>>>>> in a different queuing system if you need priority and delay > >>>>>>> messages > >>>>>>>>>>> support. Pulsar functions would go enough for this kind of use > >>>>>>> cases. > >>>>>>>>>>> > >>>>>>>>>>> On Fri, 25 Jan 2019 at 22:29, Ivan Kelly <iv...@apache.org> > >>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>>> Correct. PIP-26 can be implemented in Functions. I believe > the > >>>>>>>> last > >>>>>>>>>>>>> discussion in PIP-26 thread kind of agree on functions > >>>>>> approach. > >>>>>>>>>>>>> If the community is okay with PIP-26 in functions, I think > >>>>>> that > >>>>>>> is > >>>>>>>>>>>> probably > >>>>>>>>>>>>> a good approach to start. > >>>>>>>>>>>> > >>>>>>>>>>>> +1 for doing it in functions. > >>>>>>>>>>>> > >>>>>>>>>>>> -Ivan > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> *Thanks* > >>>>>>>>>>> > >>>>>>>>>>> *Yuvaraj L* > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>> > >>>> > >> > >> > >
