If consumer control the delayed message specific execution time we must trust clock of consumer, this can cause delayed message process ahead of time, some applications cannot tolerate this condition.
> My concern of this category of approaches is "bandwidth" usage. It is > basically trading bandwidth for complexity. @Sijie Guo <si...@apache.org> Agree with you, such an trading can cause the broker's out going network to be more serious. Come back to the user scene, we have a lot of delayed message requirement: 1. Fixed timeout, e.g.(with 10s, 30s, 10min delayed), this is the largest proportion in throughput of delayed message . A subscription with a fixed delayed time can approach to this scene. 2. Arbitrary timeout, e.g.(with 10min, 60min, 62min, 78min…. delayed), deal with this scenario from the client is difficult(delayed time to much distributed), PIP-26 can approach to this scene. Through the previous discussions around the delayed message delivery proposals, we should try to avoid adding complexity in the broker dispatching code. Impletement Fixed delayed subscrption can be simple as https://github.com/apache/pulsar/pull/3155, but can’t approach to the arbitrary timeout scene. I have a idea, We can turn arbitrary timeout feature into an optional component like function worker, sql worker, websocket proxy, by this way, we can keep broker dispatching simpler and the component can separate deployment. Sijie Guo <guosi...@gmail.com> 于2019年1月17日周四 下午5:43写道： > On Thu, Jan 17, 2019 at 2:58 PM Matteo Merli <mme...@apache.org> wrote: > > > After a long delay (no pun intended), I finally got through the > > previous discussions around the delayed message delivery proposals. > > I'm referring to PIP-26 > > https://github.com/apache/pulsar/wiki/PIP-26%3A-Delayed-Message-Delivery > > and the Pull Request at #3155 > > https://github.com/apache/pulsar/pull/3155 > > > > To summarize these proposals (correct me if I'm getting any point wrong): > > > > * PIP-26 > > - Producer sets arbitrary timeout on each message > > - Broker keeps a hash-wheel timer (backed by ledger) to keep track > > of messages for which the dispatch has to be deferred > > > > * PR #3155 > > - Consumer specify a fixed time delay to consume messages > > - Broker will defer delivery by that time > > > > As I stated previously, we should try to avoid adding complexity in > > the broker dispatching code, unless there's a clear benefit compared > > to do the same operation in client library. > > > > After discussing with Ivan, I wanted to share this alternative approach. > > > > Key points: > > * Application set arbitrary timeout on each message > > * Broker is unchanged > > * Consumer (in client library) will make these messages visible to > > application after delay has expired > > > > Implementation notes: > > * Producer change is trivial. We just need to add new field in > > message metadata (similar as described in PIP-26) > > * On consumer side, the following will happen: > > - Messages get added to receiverQueue > > - When application calls receive, we might get from the queue a > > message with delay. > > - This message is not passed to application. Rather insert the > > message ID into a priority queue (or equivalent structure), ordered by > > target time. > > - At this point messages are not added to the ack-timeout tracker > > - Periodically, we check the head of the priority queue to see if > > there's anything ready > > - If so, we request the broker to "redeliver" these messages, > > using the same mechanism as ack-timeout: > > CommandRedeliverUnacknowledgedMessages > > with a list of message ids) > > > > If I understand this correctly, this is an enhanced variation of > "ack-timeout"-ish approach that supports arbitrary delays. > > My concern of this category of approaches is "bandwidth" usage. It is > basically trading bandwidth for complexity. > > A *deferred* message will be delivered "twice" from brokers (if there is no > cache or cache miss). If a topic's traffic is M bytes/second, > there are N subscriptions. The traffic will effectively be 2 * M * N, which > can potentially be a red flag to the users who > rely on this feature. > > But I am also fine with approach if most of people don't in favor of > changing the dispatch logic at broker side. > > > > > This approach will ensure: > > * We can support arbitrary delays > > * No changes and no overhead in broker - No need to configure > > policies for delay activation > > * Works well with existing flow control mechanism: messages are > > dequeued so that we can process messages with smaller delays > > * Amount of memory required in client side is limited. > > - We just keep message ids (we could consider caching few > > messages as well, as an optimization) > > - Broker has a limit of unacked messages pushed to a consumer > > (default 50K). I don't expect this being a particular problem. > > If there a lot of messages with big differences in the delay > > value, the worst case would be that the applied delay to be higher > > for some of the messages. > > > > Any thoughts on this? > > > > -- > > Matteo Merli > > <mme...@apache.org> > > >
