Sometimes loosing mails when using Procmail as content filter
Hi, I'm using Procmail to filter incoming messages with Spamassassin and ClamAV. Today I recognized that mails are getting lost when a single mail contains multiple recipients that are hosted on this server. I've no clue what I'm doing wrong. Filtering the mails if only one of the recipients is hosted on this server works fine. I'm using Postfix version 2.6.6. It looks like the mail is lost directly during the first Procmail rule. I already tried to define a lock for the first rule, but it does not help. The first mail (out of multiple recipients in that single mail) arrives correctly at its recipient, all other are getting lost. Can someone tell me what I'm doing wrong? Thanks, Michael The corresponding log lines are: maillog = Jun 11 08:52:30 mx0 postfix/smtpd[31969]: connect from testrelay.tu-graz.ac.at[129.27.2.201] Jun 11 08:52:30 mx0 postfix/smtpd[31969]: 8581482404B8: client=testrelay.tu-graz.ac.at[129.27.2.201] Jun 11 08:52:30 mx0 postfix/cleanup[31977]: 8581482404B8: message-id=<5397fcaa.8060...@stud.tugraz.at> Jun 11 08:52:30 mx0 postfix/qmgr[31967]: 8581482404B8: from=, size=2241, nrcpt=4 (queue active) Jun 11 08:52:30 mx0 postfix/smtpd[31969]: disconnect from testrelay.tu-graz.ac.at[129.27.2.201] Jun 11 08:52:35 mx0 postfix/pickup[31968]: D3AC582437EE: uid=5000 from= Jun 11 08:52:35 mx0 postfix/pipe[31978]: 8581482404B8: to=, relay=procmail, delay=5.4, delays=0.13/0/0/5.3, dsn=2.0.0, status=sent (delivered via procmail service) Jun 11 08:52:35 mx0 postfix/pipe[31978]: 8581482404B8: to=, relay=procmail, delay=5.4, delays=0.13/0/0/5.3, dsn=2.0.0, status=sent (delivered via procmail service) Jun 11 08:52:35 mx0 postfix/pipe[31978]: 8581482404B8: to=, relay=procmail, delay=5.4, delays=0.13/0/0/5.3, dsn=2.0.0, status=sent (delivered via procmail service) Jun 11 08:52:35 mx0 postfix/pipe[31978]: 8581482404B8: to=, relay=procmail, delay=5.4, delays=0.13/0/0/5.3, dsn=2.0.0, status=sent (delivered via procmail service) Jun 11 08:52:35 mx0 postfix/qmgr[31967]: 8581482404B8: removed Jun 11 08:52:35 mx0 postfix/cleanup[31977]: D3AC582437EE: message-id=<5397fcaa.8060...@stud.tugraz.at> Jun 11 08:52:35 mx0 postfix/qmgr[31967]: D3AC582437EE: from=, size=2659, nrcpt=1 (queue active) Jun 11 08:52:35 mx0 dovecot: lmtp(32093): Connect from local Jun 11 08:52:35 mx0 dovecot: lmtp(32093, veron...@filestore.su): HyvbNLP8l1NdfQAAtCQUIQ: sieve: msgid=<5397fcaa.8060...@stud.tugraz.at>: stored mail into mailbox 'INBOX' Jun 11 08:52:35 mx0 postfix/lmtp[32003]: D3AC582437EE: to=, relay=mx0.filestore.su[private/dovecot-lmtp], delay=0.13, delays=0.02/0/0.01/0.1, dsn=2.0.0, status=sent (250 2.0.0 HyvbNLP8l1NdfQAAtCQUIQ Saved) Jun 11 08:52:35 mx0 dovecot: lmtp(32093): Disconnect from local: Successful quit Jun 11 08:52:35 mx0 postfix/qmgr[31967]: D3AC582437EE: removed = procmail.log = procmail: [32072] Wed Jun 11 08:52:30 2014 procmail: Locking "/var/mail/vhosts/backup-mail/veron...@filestore.su/.lock" procmail: Assigning "LASTFOLDER=/var/mail/vhosts/backup-mail/veron...@filestore.su/new/1402469550.32072_1.mx0.ovz" procmail: Unlocking "/var/mail/vhosts/backup-mail/veron...@filestore.su/.lock" procmail: Executing "/usr/local/bin/clamassassin" procmail: Match on "^X-Virus-Status: No" procmail: Executing "/usr/bin/spamassassin" procmail: [32072] Wed Jun 11 08:52:35 2014 procmail: Executing "/usr/sbin/sendmail,-f,michael.neur...@stud.tugraz.at,veron...@filestore.su" procmail: [32072] Wed Jun 11 08:52:35 2014 procmail: Assigning "LASTFOLDER=/usr/sbin/sendmail -f michael.neur...@stud.tugraz.at veron...@filestore.su" Subject: Re: test Folder: /usr/sbin/sendmail -f michael.neur...@stud.tugraz.at vero2532 = My procmail script is: = SHELL=/bin/sh SENDMAILFLAGS="-f $E_SENDER $E_RECIPIENT" LOGFILE="/var/log/procmail" VERBOSE=on # First make a backup of the mail :0c: /var/mail/vhosts/backup-mail/$E_RECIPIENT/ # Scan for viruses :0fw | /usr/local/bin/clamassassin # If no virus has been found, scan for Spam :0fw * ^X-Virus-Status: No | /usr/bin/spamassassin # Last action: inject back to Postfix :0w | /usr/sbin/sendmail $SENDMAILFLAGS = It is connected to Postfix as follows: = smtp inet n - n - - smtpd -o content_filter=procmail:dummy -o receive_override_options=no_address_mappings -o smtpd_recipient_restrictions=reject_unauth_destination = = procmail unix -
Re: CIDR Whitelist ?
El 10.06.2014 20:47, Viktor Dukhovni escribió: There is no single right answer. For many users a single linear list of conditions is simplest. For some users, where one wants a whitelist for one set of test to not short-circuit other tests, multiple lists are better. We should not be dogmatic about either approach. What I used to do some time ago was this: - Define three different pools of smtpd processes listening on 127.0.0.2, 127.0.0.3... etc. One of the pools would be devoted to residential IP address blocks, another one to "general service" and a third one to a whitelist of known-good servers for which there was reliable contact information. - Using the FreeBSD firewall, I created so called divert rules. For the residential group and the whitelist group I defined a couple of preffix/address tables, so that connections coming from those address spaces would be diverted to the relevant pool of smtpd processes. There were several advantages to this: - Limiting the consumption of resources by likely bots coming from residential addresses. - Fine tuning the different policies for different origins. For the well known servers I could bypass some checks. For example, at some point I had something like 16 smtpd processes for residential, 32 for well known servers, 200 for "general". Before adopting this solution, even using 1000 smtpd processes was hopeless, residential based bots quickly flooded all of them denying the reception of legitimate mail. Borja.
Re: Sometimes losing mails when using Procmail as content filter
On Wed, 11 Jun 2014 09:37:44 +0200 Michael Neurohr wrote: > I'm using Procmail to filter incoming messages with Spamassassin and > ClamAV. Today I recognized that mails are getting lost when a single > mail contains multiple recipients that are hosted on this server. > I've no clue what I'm doing wrong. Filtering the mails if only one of > the recipients is hosted on this server works fine. You post a lot of information but I don't see /var/log/procmail. Perhaps it has a clue. I also use procmail to run spamassassin but I do it from individual user .procmailrc files (generated from their preferences) and I wonder if that has something to do with it. What about SA and CAV system configs? Could there be something in there that uses a common file to check for dups? I have this in my .procmailrc files: :0 Wh: msgid.lock | formail -D 65536 $HOME/.msgid.cache If something similar is happening with those programs but with a common database (e.g. /var/db/msgid.cache) that might have the effect that you are seeing. Just spitballing here. The procmail log might shed more light. -- D'Arcy J.M. Cain System Administrator, Vex.Net http://www.Vex.Net/ IM:da...@vex.net VoIP: sip:da...@vex.net
Re: Sometimes loosing mails when using Procmail as content filter
Michael Neurohr: > Hi, > I'm using Procmail to filter incoming messages with Spamassassin and ClamAV. > Today I recognized that mails are getting lost when a single mail > contains multiple recipients that are hosted on this server. I've no > clue what I'm doing wrong. Filtering the mails if only one of the > recipients is hosted on this server works fine. Configure Postfix to make single-recipient deliveries. Set "procmail_destination_recipient_limit = 1" which is required just like it is required for local(8) deliveries. Wietse
Re: Sometimes loosing mails when using Procmail as content filter
On 11.06.2014 14:17, Wietse Venema wrote: > Configure Postfix to make single-recipient deliveries. > > Set "procmail_destination_recipient_limit = 1" which is required > just like it is required for local(8) deliveries. Yes, that did it. Many thanks! Is that setting needed for all filters that are used like unix pipes? Michael
Re: Sometimes losing mails when using Procmail as content filter
On 11.06.2014 13:02, D'Arcy J.M. Cain wrote: > You post a lot of information but I don't see /var/log/procmail. > Perhaps it has a clue. I wrote a wrong info before the procmail log. All I'm seeing in /var/log/procmail is: = procmail: [32072] Wed Jun 11 08:52:30 2014 procmail: Locking "/var/mail/vhosts/backup-mail/veron...@filestore.su/.lock" procmail: Assigning "LASTFOLDER=/var/mail/vhosts/backup-mail/veron...@filestore.su/new/1402469550.32072_1.mx0.ovz" procmail: Unlocking "/var/mail/vhosts/backup-mail/veron...@filestore.su/.lock" procmail: Executing "/usr/local/bin/clamassassin" procmail: Match on "^X-Virus-Status: No" procmail: Executing "/usr/bin/spamassassin" procmail: [32072] Wed Jun 11 08:52:35 2014 procmail: Executing "/usr/sbin/sendmail,-f,michael.neur...@stud.tugraz.at,veron...@filestore.su" procmail: [32072] Wed Jun 11 08:52:35 2014 procmail: Assigning "LASTFOLDER=/usr/sbin/sendmail -f michael.neur...@stud.tugraz.at veron...@filestore.su" Subject: Re: test Folder: /usr/sbin/sendmail -f michael.neur...@stud.tugraz.at vero2532 = The processing of the other users do not produce any log entries. That fact is really wired... But as Wietse wrote, the problem was the multiple recipient deliveries. The problem is solved now :-) Michael
Re: Sometimes loosing mails when using Procmail as content filter
On 6/11/2014 9:10 AM, Michael Neurohr wrote: > On 11.06.2014 14:17, Wietse Venema wrote: >> Configure Postfix to make single-recipient deliveries. >> >> Set "procmail_destination_recipient_limit = 1" which is required >> just like it is required for local(8) deliveries. > > Yes, that did it. Many thanks! > Is that setting needed for all filters that are used like unix pipes? > > Michael > That setting is needed for any transport or filter that expects only one recipient at a time. Very generally, this means final delivery, but there are exceptions. -- Noel Jones
Re: the after-queue simple filter -- I found my problem.
On Tue, Jun 10, 2014 at 08:02:55PM -0500, Noel Jones wrote: > On 6/10/2014 5:45 PM, Jay G. Scott wrote: > > > > After trying various combinations of things in > > main.cf and master.cf, I find that, using the script > > below, if the mail reaches the filter script, the > > resulting mail message bounces (too many hops) > > but the filter has processed the input. > > > > If I turn content_filter off in main.cf the > > mail gets delivered unfiltered, no bounces, no hops. > > > > main.cf: > > > > content_filter = filter > > > > got it filtered, but resulted in the hops bounce. > > > > What am I missing? > > > > j. > > > Near the bottom of > http://www.postfix.org/FILTER_README.html#simple_filter > the document describes how to turn on filtering for mail arriving > via SMTP, but leave it turned off as the main.cf default so mail > won't loop. I had two smtp lines in the master.cf file. Splutter. The change to the pickup line (per Dukhovni) also helped me. It's working for me now. Thanks. j. > > Of course this means you would need to submit mail via SMTP for > testing; mail submitted via the sendmail(1) interface is not > filtered. This is necessary since the "simple" filter resubmits the > mail to postfix via the sendmail command. > > > > -- Noel Jones -- Jay Scott 512-835-3553g...@arlut.utexas.edu Head of Sun Support, Sr. System Administrator Applied Research Labs, Computer Science Div. S224 University of Texas at Austin
Re: the after-queue simple filter -- I found my problem.
On Wed, Jun 11, 2014 at 11:37:46AM -0500, Jay G. Scott wrote: > I had two smtp lines in the master.cf file. Splutter. You're supposed to. One is for the Postfix SMTP server (inbound mail): smtp inet n - n - - smtpd the other is for the Postfix SMTP client (outbound mail): smtp unix - - n - - smtp > The change to the pickup line (per Dukhovni) also helped me. > > It's working for me now. Or not, if you've commented out or deleted one of the two standard SMTP services. -- Viktor.
Re: How to block offering SASL auth to clients based on RBL
Stan Hoeppner schrieb: > On 6/10/2014 3:39 PM, Wietse Venema wrote: >> Kai Krakow: >>> BTW: In this context, what's the best approach to put mailboxes on a >>> separate machine? Let the LDA drop mails into NFS mounts, or let postfix >>> transport the mails via transport_map into a machine which hosts the LDA >>> (dovecot in our case)? >> >> I recommend Dovecot via LMTP, but NFS would work, too, assuming one >> file per message. I can't say which approach would handle the most >> load. > > Dovecot's LMTP and LDA both perform index updates during delivery to the > mailbox. They also enable Sieve. Dovecot's speedy performance is due > in large part to its indexes. If you use the Postfix LDA to drop mail > directly into maildir files, Dovecot will need to stat the new files to > update its indexes, before responding to a LIST command. On a busy > server this can be expensive, and responsiveness at the MUA may be > sluggish. > > Thus I concur with Wietse. Use LMTP for performance, and to enable > Sieve scripting. Okay, thanks to both of you. That are the pointers I need. I'm currently in the mood of creating a new mail server architecture based on the impressions from the last weeks: * mailin server: does MX and outbound mail * mailout server: handle user submissions only * transport mails to local domains via dovecot LMTP / to mbox server * transport mails to remote domains by passing them to mailin server * bulkmail server: handle user and webserver bulk submissions * handles mails generated by webservers (e.g. webforms) * handles newsletters from worker processes * other bulk purposes * maybe handle outbound bulk mails * can transport to local domains directly * mbox server: handle pop3 and imap requests from users * accepts no external traffic, just from mailout / bulkmail * just a receiver for local domains * maybe handle dovecot outgoing mails (thou we didn't support anyway) With this setup I can place different policies and rate limits for inbound/outbound. The mail servers mailin and mailout are named by view of the user altough the first handles external inbound/outbound, and the second handles submissions. Access to the bulk server could be limited to user accounts flagged as such. Any ideas/suggestions? Do you see problems? I'm not sure yet if I deploy this to different VM instances or just put multiple postfix instances on the same machine... I'd probably prefer the first. There's already a central and separated user db not outlined in this setup and accessed via mysql. -- Replies to list only preferred.
Re: the after-queue simple filter -- I found my problem.
On Wed, Jun 11, 2014 at 04:42:42PM +, Viktor Dukhovni wrote: > On Wed, Jun 11, 2014 at 11:37:46AM -0500, Jay G. Scott wrote: > > > I had two smtp lines in the master.cf file. Splutter. > > You're supposed to. One is for the Postfix SMTP server > (inbound mail): > > smtp inet n - n - - smtpd > > the other is for the Postfix SMTP client (outbound mail): > > smtp unix - - n - - smtp > > > The change to the pickup line (per Dukhovni) also helped me. > > > > It's working for me now. > > Or not, if you've commented out or deleted one of the two standard > SMTP services. Fixed that, and it's working. Thanks. j. > > -- > Viktor. -- Jay Scott 512-835-3553g...@arlut.utexas.edu Head of Sun Support, Sr. System Administrator Applied Research Labs, Computer Science Div. S224 University of Texas at Austin
Re: How to block offering SASL auth to clients based on RBL
On Sat, Jun 7, 2014 at 3:33 PM, Kai Krakow wrote: > How is one supposed to automatically block such hijacked accounts within > postfix? A simple heuristic could be detecting unusual high mail volume for > that account, probably by detecting the always repeating or similar > subjects. What I do against this is; install CSF/LFD, the open/free suite of ConfigServer scripts. It has a wonderful option where you can prevent the blacklisted IP to even access postfix at all. Blocklists are controlled by modifying /etc/csf/csf.blocklists (I would recommend against using spamhaus or UCEprotect though, too many weird decisions there, and prone to false positives). So why not do that? In addition you get an awful lot of good security for your server. Regards, Julius Thijssen
How to deal with erroneous Return-Paths?
Hi, the German Free Mailer GMX sends its newsletters with a Return-Path like "#16155...@gmx.net". When I pass the message to Procmail with = procmail unix - n n - 10 pipe flags=Rq user=vmail null_sender= argv=/usr/bin/procmail -m E_SENDER=$sender E_RECIPIENT=$recipient /etc/procmailrc = the "$sender" stays empty because of the leading "#". I also recognized that sometimes the sending mail server does not provide a Return-Path at all. These cases result in Sendmail not being able to reinject the mail after Procmail has processed the message. How can I handle such cases? You can find my procmailrc at [1]. The corresponding log lines are: /var/log/maillog = Jun 12 07:39:15 mx0 postfix/smtpd[13440]: connect from mout.gmx.net[212.227.15.19] Jun 12 07:39:15 mx0 postfix/smtpd[13440]: B689B8240481: client=mout.gmx.net[212.227.15.19] Jun 12 07:39:15 mx0 postfix/cleanup[13447]: B689B8240481: message-id=<08f52951b1ef360622acf42743d2d...@nlsender13.ui-portal.com> Jun 12 07:39:15 mx0 postfix/qmgr[26217]: B689B8240481: from=<#16155...@gmx.net>, size=13755, nrcpt=1 (queue active) Jun 12 07:39:15 mx0 postfix/smtpd[13440]: disconnect from mout.gmx.net[212.227.15.19] Jun 12 07:39:24 mx0 postfix/sendmail[13474]: fatal: Recipient addresses must be specified on the command line or via the -t option Jun 12 07:39:24 mx0 postfix/postdrop[13475]: warning: stdin: unexpected EOF in data, record type 78 length 62 Jun 12 07:39:24 mx0 postfix/postdrop[13475]: fatal: uid=5000: malformed input Jun 12 07:39:24 mx0 postfix/pipe[13448]: B689B8240481: to=, relay=procmail, delay=9.1, delays=0.08/0.01/0/9, dsn=5.2.0, status=bounced (can't create user output file) Jun 12 07:39:24 mx0 postfix/cleanup[13447]: CC9D58243806: message-id=<20140612053924.cc9d58243...@mx0.filestore.su> Jun 12 07:39:24 mx0 postfix/qmgr[26217]: CC9D58243806: from=<>, size=15587, nrcpt=1 (queue active) Jun 12 07:39:24 mx0 postfix/bounce[13476]: B689B8240481: sender non-delivery notification: CC9D58243806 Jun 12 07:39:24 mx0 postfix/qmgr[26217]: B689B8240481: removed Jun 12 07:39:25 mx0 postfix/smtp[13482]: CC9D58243806: to=<#16155...@gmx.net>, relay=mx00.emig.gmx.net[213.165.67.114]:25, delay=0.44, delays=0.01/0.01/0.17/0.24, dsn=5.0.0, status=bounced (host mx00.emig.gmx.net[213.165.67.114] said: 550 Requested action not taken: mailbox unavailable (in reply to RCPT TO command)) Jun 12 07:39:25 mx0 postfix/qmgr[26217]: CC9D58243806: removed = /var/log/procmail = procmail: [13449] Thu Jun 12 07:39:15 2014 procmail: Locking "/var/mail/vhosts/backup-mail/mich...@filestore.su/.lock" procmail: Assigning "LASTFOLDER=/var/mail/vhosts/backup-mail/mich...@filestore.su/new/1402551555.13449_1.mx0.ovz" procmail: Unlocking "/var/mail/vhosts/backup-mail/mich...@filestore.su/.lock" procmail: Executing "/usr/local/bin/clamassassin" procmail: Match on "^X-Virus-Status: No" procmail: Executing "/usr/bin/spamassassin" procmail: [13449] Thu Jun 12 07:39:24 2014 procmail: Executing "/usr/sbin/sendmail,-f,mich...@filestore.su" sendmail: fatal: Recipient addresses must be specified on the command line or via the -t option procmail: [13449] Thu Jun 12 07:39:24 2014 procmail: Program failure (75) of "/usr/sbin/sendmail" procmail: Assigning "LASTFOLDER=/usr/sbin/sendmail -f mich...@filestore.su" Subject: Exklusiver Shopping-Zugang: Mode bis zu 80% reduziert - mit Gutschein Folder: **Bounced** 13945 postdrop: warning: stdin: unexpected EOF in data, record type 78 length 62 postdrop: fatal: uid=5000: malformed input = Cheers, Michael [1] http://marc.info/?l=postfix-users&m=140247232114778&w=2
[no subject]
Hi, I am not getting DSN from receiver to sender. My set up is as follows. I have two domains, say a.abc.com and b.abc.com and one relay server. When mail goes from one domain to other domain, mail goes through the relay server. I have tried to send mail form a.abc.com to b.abc.com through command prompt(I have enabled DSN by giving RET=HDRS near mail from command etc.). Mail has been successfully forwarded to the relay server and DSN is getting delivered to sender inbox. when Relay server forwards the mail to actual destination, Sender is not getting any notification. Even if relay server is failing to send mail to destination sender is not able to get that notification. How sender will get DSN if mail is going through the relay server? I searched a lot but didn’t get anything. Please help me. Thanking you in advance. --- Hyndavi Every 3000 Sheets of paper costs us a tree.. Save trees... Conserve Trees. Don't print this email or any Files unless you really need to Confidentiality Notice The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain confidential or privileged information. If you are not the intended recipient, please notify the sender at Bharat Electronics or supp...@bel.co.in immediately and destroy all copies of this message and any attachments.
Postfix-DSN support
Hi, I am not getting DSN from receiver to sender. My set up is as follows. I have two domains, say a.abc.com and b.abc.com and one relay server. When mail goes from one domain to other domain, mail goes through the relay server. I have tried to send mail form a.abc.com to b.abc.com through command prompt(I have enabled DSN by giving RET=HDRS near mail from command etc.). Mail has been successfully forwarded to the relay server and DSN is getting delivered to sender inbox. when Relay server forwards the mail to actual destination, Sender is not getting any notification. Even if relay server is failing to send mail to destination sender is not able to get that notification. How sender will get DSN if mail is going through the relay server? I searched a lot but didn’t get anything. Please help me. Thanking you in advance. --- Hyndavi Every 3000 Sheets of paper costs us a tree.. Save trees... Conserve Trees. Don't print this email or any Files unless you really need to Confidentiality Notice The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain confidential or privileged information. If you are not the intended recipient, please notify the sender at Bharat Electronics or supp...@bel.co.in immediately and destroy all copies of this message and any attachments.
