One missing word makes all the difference.
Wietse
Wietse Venema:
>> This week I implemented a memcache client for Postfix in the hope
>> that it would be useful to share postscreen(8) or verify(8) caches
>> among multiple MTAs.
...
> Considering that memcache patches are floating around with several
> problems, and the amount of time that I put into this, I have made
> the Postfix memcache client part of the 2.9 release. This version
> will at least reliably report problems with memcache servers.
Stan Hoeppner:
> Could a (heavily) modified proxymap/proxywrite daemon serve this multi
> host cache sharing duty? Or would that require more work than writing
> something from scratch?
Short answer: I think we need both to share the postscreen database
among multiple MTAs with adequate performance. The reason is that
proxymap and memcache solve very different problems.
Below is the long answer, based on an analysis of the different
problems that proxymap and memcache try to solve.
First a few bits of insight about memcache, then a quick look at
proxymap, and finally, how the two can be combined to get the best
of both worlds.
Quick look at memcache
----------------------
Each memcache server is effectively a world-writable network ramdisk
(with obvious security implications). If used by itself, a memcache
server would be suitable for storing low-value, easy-to-recreate,
data such as the postscreen or verify cache for a cluster of Postfix
MTAs. All read/writes are from/to memory so they can be very fast.
The advantage of memcache is that it already exists. The protocol
is so simple that I ripped out libmemcache last night and wrote
my own memcache client in a few hours (and that was mostly
pasting little blocks of existing Postfix code together).
Quick look at proxymap
----------------------
The Postfix proxymap server shares connections to (usually persistent)
databases. In principle, this could be used to share a postscreen or
verify cache between different hosts, after adding some new code:
- proxymap client/server support to forward first/next method calls.
These are needed for postscreen / verify cache cleanup support.
Only one proxymap client should invoke these methods.
- proxymap server support for network access control (by default,
only local client access should be allowed). As before, a proxymap
reader or writer process will open only databases that are listed
in main.cf:proxy_read_maps (or proxy_write_maps).
Limitations:
- Only one proxymap writer process per Postfix Berkeley DB database.
The database can quickly become a performance bottleneck.
Combining memcache and proxymap
-------------------------------
It seems that memcache is best for (surprise) doing what it was
designed for: a cache layer on top of a persistent database, where
the cache is maintained by the clients of the persistent database.
This means doing something like this:
/etc/postfix/main.cf:
postscreen_cache_map = memcache:/etc/postfix/postscreen-cache
/etc/postfix/postscreen-cache:
# Where is the memcache server?
hosts = host1:port1
# Where is the persistent database?
persistence = proxymap:inet:host2:port2:persistent-database
# Future: sharding (key hashing) to spread the load across
# multiple memory caches and persistent databases.
The idea is that the memcache client delegates persistent database
access to the proxymap server. Whenever the memcache client looks
up or modifies the persistent database, it updates the memcache
server to keep the cache up-to-date (clearly, cache consistency
won't be perfect, so information should be cached for only a limited
time).
Adding persistence to the Postfix memcache client requires adding
1) interfaces for delete and first/next methods, 2) support to
access a persistent database and 3) support to update the memcache
server whenever the Postfix memcache client looks up or modifies
the persistent database.
I am aware of memcachedb, which combines memcached with Berkeley DB.
The advantage of the approach above is that it would combine memcache
with any persistent database, for example, mysql, once Postfix write
support is completed. That said, Postfix will never allow the use
of memcache for security-sensitive data.
Wietse
