Wietse Venema put forth on 11/23/2010 6:57 PM: > Victor Duchovni: >> On Tue, Nov 23, 2010 at 07:35:49PM -0500, Wietse Venema wrote: >> >>> Victor Duchovni: >>>> On Tue, Nov 23, 2010 at 05:20:11PM -0600, Stan Hoeppner wrote: >>>> >>>>> Would you please give us the run down on why these map types (and maybe >>>>> others) shouldn't be used with proxymap due to performance reasons? You >>>>> mentioned something about this long ago but I can't seem to locate that >>>>> email in my archives. IIRC you didn't go into much technical detail as >>>>> to why the performance would be lower using proxymap. >>>> >>>> There is no point in using IPC to ask a server to read a file, when the >>>> client process can read it directly. Especially with CDB, since there >>>> is no per-client page pool adding some per-client memory overhead. >>> >>> With Stan's huge CIDR maps on a small machine, proxymap helps >>> to avoid running out of memory. >> >> CIDR maps are indeed a different kettle of fish, they are not >> "read-a-file via IPC", rather they are "do a memory lookup via IPC", and >> if the memory footprint is sufficiently high (lots Berkeley DB tables or >> huge CIDR tables), then indeed one may want to use proxymap. >> >> With "cdb" (the recommended read-only indexed table type for Postfix) direct >> access is best. > > proxymap is OK for moderate traffic to expensive or otherwise > inaccessible resources. Opening a CDB map is not expensive, an > SQL database handle or memory-based map (regexp etc.) can be > expensive, and /etc/passwd can be inaccessible due to lazy opens. > > proxymap is not the best solution if you need minimal latency at > all cost. > > Wietse
Thank you gentlemen. I relatively recently added the dnswl.org CIDR whitelist table, which contains over 97K+ entries. I use a perl script to consolidate the CIDRs, cutting it down to around 67K entries. Timing postmap -q it takes on average .5 seconds to complete a lookup in the consolidated file regardless of where the entry lies within the file. Timing queries on the original file shows .75 seconds per lookup. This is after caching. Testing cold is kind of irrelevant. I'm guessing the latency is actually higher when smtpd queries proxymap than when timing postmap -q. Is this the case? An extra ~.5-1.0 seconds isn't a big deal regarding this MX. However, each new table or dns lookup I add to my anti-spam checks adds up, and I don't want to reach a point where I'm taking 10 seconds per connection to determine if I'm going to reject it. So I try to continually optimize, especially after adding something like the dnswl.org CIDR map. -- Stan
