Hello,
I've written what I thought would be about the simplest possible after
queue content filter (appended), and it's behaving in ways I don't
expect.
The goal of the filter is to remove "Sender:" headers. FWIW, these
are added by the Gnu Mailman mailing list processor and can cause
upset in sensitive users of the MS Outlook MUA. See:
http://wiki.list.org/display/DOC/From+field+displayed+by+Microsoft+Outlook
I'm running Debian etch (4.0) with Mailman 1:2.1.9-7 and Postfix
2.3.8-2+etch1. It's on an old slow box with not a lot of RAM.
The awk is mawk.
The filter is a simple awk script run by postfix's spawn, exactly as
recommended in FILTER_README for post-queue content filters. It sends
stdin to stdout, except when it encounters a Sender: header which it
ignores. The awk output is piped to nc for delivery to postfix's
smtpd, and nc also sends smtpd's output to stdout so spawn can send it
back to the postfix smtp process feeding the filter.
Nc terminates only when the smtpd process closes the network socket,
not when it receives eof on stdin. This allows the last bit of nc's
stdin to be flushed through whatever buffering happens on the network
side.
What should happen, and what does happen a lot, is that the smtpd side
eventually receives a QUIT, responds with a 221, and closes the
connection. (At least I assume that's what's happening.) However,
sometimes, ps shows that the awk process has finished, is gone and
does not show, but the nc process is still running, and waiting for
smtpd to close it's side of the network socket. I can't determine the
conditions that will cause nc to be left running. In all cases the
mail is successfully delivered.
Offhand I can't think of a "regular" smtp protocol exchange that shuts
down the mail sending side of the network socket and leaves the mail
receiving side wanting to talk with the sending side, although this is
what seems to be happening. I'm guessing that the postfix smtp side
sometimes shuts down early, after sending a SMTP QUIT, leaving the
smtpd side still trying to send the 221 response. I can't think of
why it would do that, given that postfix is essentially talking to
itself.
The tests were all done by sending mail to the Mailman list address,
so I suppose it's possible that the condition occurs only when Mailman
bounces mail to the moderator or does something else that I just
wasn't noticing.
My solution is to use -q with nc, and tell it to quit 2 seconds (the
$DELAY value in the script) after receiving eof on stdin. This gives
the network time to flush and keeps the number of running processes
down so that the total number of concurrent filters configured in
master.cf is not reached. (Otherwise nc will wait for smtpd to reach
it's inactivity timeout and close the connection, a much longer
interval.)
I've a couple of questions. First, what's going on? Why doesn't
smtpd always finish it's SMTP session and gracefully close the network
socket? Is this somehow indicative of a problem in postfix? Second,
and perhaps most important, is 2 seconds enough? Too much? What's
the right value and why?
I'm also interested in hearing any comments on the script, (or why
what it does is a terrible idea, etc.) (I already know removing
Sender: headers violates the RFC, but it seems a relatively harmless
violation that at worst would confuse the end-user.)
I'd like to see a canonical example of a very simple after-queue
filter script included in the postfix distribution and/or
documentation. Perhaps my script could be a starting point.
Thank you.
---------------------<snip>------------------------
#!/bin/sh
# /usr/local/sbin/mailman-filter
#
# Copyright (C) 2008 Karl O. Pinc <[EMAIL PROTECTED]>
#
# This program is free software: you can redistribute it and/or
modify
# it under the terms of the GNU General Public License as published
by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# Get rid of Sender: headers for mailman email lists
#
# Karl O. Pinc <[EMAIL PROTECTED]>
#
# September 8, 2008
#
# Syntax: mailman-filter host port
#
# Arguments:
# host The host to send output
# port The port to send to
#
# Remarks:
# Outlook mungs it's displayed From info to show that mail was
# sent via the Sender: header address. This "fixes" the problem.
# See:
#
http://wiki.list.org/display/DOC/From+field+displayed+by+Microsoft+Outlook
#
# Invoked by a postfix header check that sends only those emails
# to us that need the Sender header removed, so we can indiscriminately
# remove the header herein.
#
# Bugs:
# Rate limited by the nc disconnect delay and the number of concurrent
# filter processes configured in master.cf, at least during those times
# when the smtp server does not disconnect on it's own.
#
# Uses \x characters, so works with mawk and gawk but probably won't
# work with a strictly posix awk.
#
# Requires:
# Netcat, awk.
# For postfix configuration see:
http://www.postfix.org/FILTER_README.html
#
# An mfilter user:
# useradd -c 'User to run mailman postfix filter' \
# -s /bin/false -d /tmp mfilter
#
# A regexp line (or lines) in /etc/postfix/header-checks:
# /^Sender: [EMAIL PROTECTED]/ FILTER
mfilter:127.0.0.1:10025
#
# A line in /etc/postfix/main.cf to turn on header checks:
# header_checks = regexp:/etc/postfix/header_checks
#
# Lines in /etc/postfix/master.cf:
#
## Mailman content filtering to get rid of Sender: headers for outlook
## (Note configuration is for 1 concurrent process, increase for
production.)
## (Entire transport is mfilter:localhost:10025)
##
## The transport that invokes the filter
#mfilter unix - - n - 1 smtp
# -o smtp_send_xforward_command=yes
# -o disable_mime_output_conversion=yes
# -o smtp_generic_maps=
## Run the content filter
#127.0.0.1:10025 inet n n n - 1 spawn
# user=mfilter argv=/usr/local/sbin/mailman-filter 127.0.0.1
10026
## Reinject mail back into postfix
#127.0.0.1:10026 inet n - n - 1 smtpd
# -o content_filter=
# -o
receive_override_options=no_unknown_recipient_checks,no_header_body_checks,no_milters
# -o smtpd_helo_restrictions=
# -o smtpd_client_restrictions=
# -o smtpd_sender_restrictions=
# -o smtpd_recipient_restrictions=permit_mynetworks,reject
# -o mynetworks=127.0.0.0/8
# -o smtpd_authorized_xforward_hosts=127.0.0.0/8
# How long (seconds) to delay after eof from the smtp sender before
# disconnecting the smtp server. This allows the traffic from the
# sender to the smtp server to be flushed. It's not clear why this is
# needed because the smtp server should disconnect the network socket.
# It is not always needed. I suspect a race condition where the
# client is disconnecting after the QUIT before the server finishes
# the 221 reply and the server buffer is not flushed so the server
# connection hangs around.
DELAY=2
host=$1
port=$2
awk -W interactive \
'BEGIN {headers = 1;
data = 0;};
/^DATA\r$/ {data = 1;};
/^\r$/ {if (data == 1) {
headers = 0;
}
};
{if (! (data == 1 && headers == 1 && $1 == "Sender:")) {
print;
if ($0 == ".\r") {
# End of data, smtp loops and allows another message
headers = 1;
data = 0;
}
}
}; ' \
| nc -q $DELAY $host $port
Karl <[EMAIL PROTECTED]>
Free Software: "You don't pay back, you pay forward."
-- Robert A. Heinlein
