On 06Apr2021 23:12, Kurt Hackenberg <k...@panix.com> wrote: >On Wed, Apr 07, 2021 at 09:43:36AM +1000, Cameron Simpson wrote: >>My new tool streams the fetches: it issues RETRs for every message up >>front at maximum network speed - fully buffered and with no waits. A >>parallel worker thread collects the messages as they come in at full >>speed (the upstream server likely also gets to fully buffer); it issues >>DELEtes as each message is saved, also fully buffered. > >Slick. Clearly the right way to handle that high latency.
Just tried it on the satellite link with an overnight load of messages, normally a 10 minute exercise with getmail (give or take). 411 messages, 8.5 seconds. >Have you ever tried the program fdm? It fetches, filters, and >delivers mail, like getmail and procmail combined. I haven't tried >it, but it looks interesting. I have not. My mail filer is decoupled from my fetcher: it monitors spool Maildirs (which also means I can refile a message just by saving it to a spool Maildir). And it has its own syntax to my liking; other tools inherently will not :-) And looking at the conf file, it seems that (like procmail, which I abandoned years ago) it matches using regexps. These are appalling for email addresses. When testing addresses, my filer does a correct address parse and compares the inner component (eg "c...@cskk.id.au" from "Cameron Simpson <c...@cskk.id.au>"), and can do set membership tests on that eg "is this address in this group?". >This paragraph in its manual sounds like it might stream fetching like >your program: > >"fdm tries to queue a number of mails simultaneously, so that older >can be delivered while waiting for the server to provide the next. The >maximum length of the queue for each account is set by the >'queue-high' option (the default is two) and the maximum mail size >accepted by the 'maximum-size' option (the default is 32 MB). In >addition, the 'rewrite' action requires an additional temporary >mail. Although fdm will fail rather than dropping mail if the disk >becomes full, users should bear in mind the possibility and set the >size of the temporary directory and the fdm options according to their >needs." I've been reading the manual. I think it does not. I think it actually allows the filing/saving to proceeed while requesting/fetching the next message. So a simple form of parallelism, but not one which reduces the fetch latency between requests. >fdm is at github: ><https://github.com/nicm/fdm/> > >The paragraph quoted above is at about line 300 in the manual, which is here: ><https://github.com/nicm/fdm/blob/master/MANUAL> Thanks. An interesting read. Cheers, Cameron Simpson <c...@cskk.id.au>
