> On Jun 16, 2020, at 12:39 PM, Wietse Venema <wie...@porcupine.org> wrote:
>
> - Adding layer of sender-based round-robin selection. Not sure if
> that would explode at large scale.
When the input mix approximates steady-state, FIFO is essentially optimal,
with each type of message getting its average share of the output in a fair
and timely manner. If/when we stray from FIFO, we need to be carefuly not
be simply underserving some fraction of the expected steady-state load
distribution.
This would mean being able to somehow detect a "burst" of traffic and
characterise to distinguish the messages that are contributing to the
burst from other messages. That's a tricky problem. The sender address
for bulk traffic is liable to have VERP tags, the client IP of interest
may be a few hops back from the edge system that encounters delays in
delivering an input burst to remote organizations (ADMDs in the language
of RFC5598).
> - Adding some artificial 'cost' value that is computed while delivery
> requests are added to per-destination queues. Cost could depend on
> the number of delivery requests per sender email address, the message
> size, and so on. Then, the scheduler could choose what-to-deliver
> based on artificial cost in addition to the things that it already
> considers now.
Ideally we'd have an algorithm that could group related messages into
a set of logical "bulk" sources, and apply the current bulk message
preëmption algorithm not only to multi-recipient mail, but also to
multi-message bulk sources. The hard part is the classification,
especially in a single-threaded queue manager that needs to do this
in O(1 millisecond).
Perhaps the best proxy for related messages is origin domain (ignoring
localpart) + approximate message size. Related messages are likely to
carry similar content of approximately the same size to all recipients.
But completely ignoring the localpart may be too coarse. It is not
obvious to me how to extract common elements from a per-recipient-salted
localpart...
A traditional multi-recipient message automatically qualifies as a single
source (same message size for all recipients and same origin domain).
We could attempt to group "related" messages in a fuzzy manner as above,
and then apply the existing preëmption algorithm. Still not sure how to
do a good job of the aggregation.
--
--
Viktor.
