processing time metrics for rejected connections
Considering that spam accounts for the bulk of all client connections to an MX these days, it might be beneficial if we had log data showing total time per session, not just for queued mail, so an OP can see how long it's taking to reject at the smtpd stage, as well as time elapsed when rejecting messages at cleanup with header/body checks, or with a pre-queue content filter, etc. If a server accepts 10k smtpd connections a day and rejects 9k via smtpd_*_restrictions, 400 via a pre-queue content filter, and 100 via header checks, etc, it would seem that the amount of processing time required for the remaining 500 legitimate emails is less relevant in the absence of time metrics for the 95% of the mail rejected as spam. If we're using delays=a/b/c/d for troubleshooting that's fine. But if we're expecting to be tuning a server for performance based on log metric data we need time data for our rejected messages as well. The main reason I ask is that my filters are constantly becoming more complex and the hardware remains the same. I'd like to be able to see how much additional load I'm adding to my system with the constant addition of new filters. Any chance this is on the agenda? Or does this type of log data already exist, and I'm simply too blind to find it? Thanks. -- Stan
Re: Postfix with AD and Exchange
On Sat, 2010-09-04 at 12:11 +0530, Ashwin Muni wrote: > I am using exchange and want to migrate to postfix, The issue is > users should be authenticated from Active Directory and other thing is > i have got 1000 users from which 150 users will remain on exchange and > the rest mailboxes will be created on postfix (Linux Box). the 750 > users are actually not very important. > > >If user has mailbox in AD deliver > in Ad >|| > =(4)== >|| > || > [][ ]===(3)=<=[ >] [ v ] > (1)--> Port 25 ][ Postfix] Authentication from AD [ Active > Directory Box] ===[ Exchange] > [][Linux Box ]==>===(2)=== > [ ] > [] >|| Else Deliver on Postfix Machine > (5) >|| > [ v] > [ Storage on Linux Box ] > [ ] > > 1. Mail Entering Postfix machien using Port 25 > 2. Authentication from AD if user exists and mailbox location > 3 Results from AD > 4. If user on Mailbox on exchange then deliver to echange > 5. Else Deliver to postfix > Why do you want to query AD in real-time. Micro$$oft AD implements ldap very poorly and you will waste time on your postfix server waiting for AD replies. Instead do this * Ldap Query the Ad server in the cron and get list of all valid users with mailboxes. * create a transport file to deliver only these users to the Exchange Rest to your local delivery .. I assume you use lmtp ---/etc/postfix/transport exch_us...@domain.tld smtp:[exchange.server.ip] exch_us...@domain.tld smtp:[exchange.server.ip] exch_us...@domain.tld smtp:[exchange.server.ip] ... .. domain.tld lmtp:/path * Just make sure the unknown users are rejected at the smtpd level before the mail enters inside. Thanks Ram > Any help appreciated. > > -- > Ashwin Muni > http://www.linuxmaza.com > Linux Tutorials and Howtos
Re: Postfix with AD and Exchange
Thanks Ram But all my 1000 users are in AD and only few of them need to have mailboxes on exchange, how shall i bifurcate 250 users in exchange and rest 750 users in postfix. Again the idea of fetching valid users is great from AD will script it. Ashwin On Sat, Sep 4, 2010 at 1:38 PM, Ram wrote: > > > On Sat, 2010-09-04 at 12:11 +0530, Ashwin Muni wrote: > > I am using exchange and want to migrate to postfix, The issue is > > users should be authenticated from Active Directory and other thing is > > i have got 1000 users from which 150 users will remain on exchange and > > the rest mailboxes will be created on postfix (Linux Box). the 750 > > users are actually not very important. > > > > > >If user has mailbox in AD deliver > > in Ad > >|| > > =(4)== > >|| > > || > > [][ ]===(3)=<=[ >] [ v ] > > (1)--> Port 25 ][ Postfix] Authentication from AD [ Active > > Directory Box] ===[ Exchange] > > [][Linux Box ]==>===(2)=== > > [ ] > > [] > >|| Else Deliver on Postfix Machine > > (5) > >|| > > [ v] > > [ Storage on Linux Box ] > > [ ] > > > > 1. Mail Entering Postfix machien using Port 25 > > 2. Authentication from AD if user exists and mailbox location > > 3 Results from AD > > 4. If user on Mailbox on exchange then deliver to echange > > 5. Else Deliver to postfix > > > > Why do you want to query AD in real-time. Micro$$oft AD implements ldap > very poorly and you will waste time on your postfix server waiting for > AD replies. > > Instead do this > * Ldap Query the Ad server in the cron and get list of all valid users > with mailboxes. > * create a transport file to deliver only these users to the Exchange > Rest to your local delivery .. I assume you use lmtp > > ---/etc/postfix/transport > exch_us...@domain.tld smtp:[exchange.server.ip] > exch_us...@domain.tld smtp:[exchange.server.ip] > exch_us...@domain.tld smtp:[exchange.server.ip] > ... > .. > domain.tld lmtp:/path > > > > > * Just make sure the unknown users are rejected at the smtpd level > before the mail enters inside. > > > > Thanks > Ram > > > > > > > > > > > > > Any help appreciated. > > > > -- > > Ashwin Muni > > http://www.linuxmaza.com > > Linux Tutorials and Howtos > > > -- Ashwin Muni http://www.linuxmaza.com Linux Tutorials and Howtos
postscreen bug ?
Hi, i have recently set up the current postfix version 2.8 from the trunk, so far everything works fine. First some feedback to Wietse about postscreen in production environment: I have configured postscreen and its doing a great job in production environment, combined with grey listing there is rarely a spam coming through. The new pregreet check works really exceptionally well, also in combination with a DNSBL list. there is however one small thing which iam runing in, postfix keeps complaining close database /var/lib/postfix/ps_cache.db: No such file or directory i checked the file is there and is being used from postscreen (where else shall it store its temporary data ?) its also accessible (file permissions) and i have no functional problems. Is this a bug or am i doing something wrong ? If anyone has an idea please tell me, and start using the postscreen, its worth the minute configuring. Regards, Frank
Re: Customized transport with multiple recipients
On 09/02/2010 03:26 PM, Zhou, Yan wrote: Hi there, If Postfix server gets a mail message with multiple TO: address (i.e., multiple recipients), does Postfix send one message to each address? These decisions are not made when postfix receives mail. The message in the incoming queue includes all recipients that postfix _accepted_. When the qmgr(8) picks up the message, several settings are consulted, and by default postfix will send one copy of the message to each destination domain/nexthop. So if the message contains multiple recipients that can be delivered over one transport, they will be combined in one message. If so, at what stage does this happen? If you mean the exact process, that would be the qmgr(8). Take a look at how postfix operates: http://www.postfix.org/OVERVIEW.html The postfix log seems to indicate that. For my Postfix, I customized my transport in postfix like this, transport_maps = hash:/etc/postfix/transport where the transport file reads like this (connector is a script that calls a program to consume emails). connector: I observed that the transport only get one single message with multiple TO: address in it. So, this means my transport should do the work of sending to multiple destinations? There is no reason it "should", no. You can force postfix to split them up into single-recipient messages with the $transport_destination_recipient_limit = 1 parameter (where $transport is the name of your transport) - this will limit each message to that transport to a single recipient. This will often be necessary for delivery to programs (as opposed to mail servers) that can only handle one recipient at a time. If I do not use any customized transport, rather, use the default Maildir/ format, I am sure that Postfix will place one message in each destination folder. "The default transport" and "the default Maildir format" are not related. What happens to local mail is decided by the local(8) mail delivery agent (MDA) included with postfix. How physical mailboxes are addressed is configured there, and nowhere else. The qmgr(8) uses the term "destination" in two distinct ways, depending on the setting of $transport_destination_recipient_limit. In general (for any transports that do not have that setting defined), a destination refers to a mail server, not a message recipient. In the above case, however, the default for local_destination_recipient_limit is 1, so "destination" does mean "single recipient". Your transport is not local(8), and has no relation to it. If you don't explicitly set it, $yourtransport_destination_recipient_limit will be 50. My assumption, by using a customized transport, I am taking away what Postfix delivery agent does, which is copy one message to each destination? No. By default, local_destination_recipient_limit = 1, so the qmgr(8) already breaks them up into separate messages. For a custom transport, however, the built-in default is 50. The transport is not involved in either case - it's the qmgr(8) that decides this. In conclusion, yes, you can use a transport that only accepts a single recipient - if you configure it. J.
Re: postscreen bug ?
fdo...@network-steps.com: > postfix keeps complaining > > close database /var/lib/postfix/ps_cache.db: No such file or directory That is a Berkeley DB mis-feature. Newer Postfix snapshots ignore that error. Wietse
Re: processing time metrics for rejected connections
On Sat, Sep 04, 2010 at 03:02:00AM -0500, Stan Hoeppner wrote: > If we're using delays=a/b/c/d for troubleshooting that's fine. But if > we're expecting to be tuning a server for performance based on log > metric data we need time data for our rejected messages as well. The purpose of the a/b/c/d logging is to allow you to isolate the origin of message processing latency for mail that enters your queue. If you are running a reasonably recent Postfix release, and "stress-dependent" behaviour is never or rarely triggered (this is logged), then your input is fast enough. You cna monitor the average number of established connections to your server, and compare that to the maximum process limit for the "smtp inet" (i.e. smtpd) service. So long as the connection load stays below the maximum configured load, you are fine. If not, look into "postscreen" in the 2.8 snapshot. > The main reason I ask is that my filters are constantly becoming more > complex and the hardware remains the same. I'd like to be able to see > how much additional load I'm adding to my system with the constant > addition of new filters. What do you mean by "filters"? Connection concurrency is proportional to the connection lifetime and the connection rate. You can measure the impact of increased connection lifetime by observing the average number of concurrent connections to your server. Keep an eye on the CPU load and the disk I/O utilization. > Any chance this is on the agenda? Or does this type of log data already > exist, and I'm simply too blind to find it? Fine-grained logging of time spent in the various input stages has not proved necessary, so far. -- Viktor.
Re: postscreen bug ?
* fdo...@network-steps.com : > close database /var/lib/postfix/ps_cache.db: No such file or directory I'm also seeing this, but only very sporadically: Aug 20 08:49:23 mail-ausfall postfix/postscreen[15615]: close database /var/lib/postfix/ps_cache.db: No such file or directory Aug 20 16:05:43 mail-ausfall postfix/postscreen[28112]: close database /var/lib/postfix/ps_cache.db: No such file or directory Aug 20 20:54:47 mail-ausfall postfix/postscreen[5827]: close database /var/lib/postfix/ps_cache.db: No such file or directory Aug 22 00:00:02 mail-ausfall postfix/postscreen[17856]: close database /var/lib/postfix/ps_cache.db: No such file or directory Aug 23 08:36:09 mail-ausfall postfix/postscreen[4931]: close database /var/lib/postfix/ps_cache.db: No such file or directory Aug 24 07:37:55 mail-ausfall postfix/postscreen[21289]: close database /var/lib/postfix/ps_cache.db: No such file or directory Aug 25 08:37:49 mail-ausfall postfix/postscreen[2421]: close database /var/lib/postfix/ps_cache.db: No such file or directory Aug 25 16:58:53 mail-ausfall postfix/postscreen[30306]: close database /var/lib/postfix/ps_cache.db: No such file or directory Aug 26 06:59:48 mail-ausfall postfix/postscreen[21834]: close database /var/lib/postfix/ps_cache.db: No such file or directory Aug 26 09:09:57 mail-ausfall postfix/postscreen[30604]: close database /var/lib/postfix/ps_cache.db: No such file or directory Aug 26 15:37:12 mail-ausfall postfix/postscreen[2631]: close database /var/lib/postfix/ps_cache.db: No such file or directory Aug 27 11:09:09 mail-ausfall postfix/postscreen[20718]: close database /var/lib/postfix/ps_cache.db: No such file or directory Aug 27 16:29:04 mail-ausfall postfix/postscreen[22487]: close database /var/lib/postfix/ps_cache.db: No such file or directory Aug 28 15:11:20 mail-ausfall postfix/postscreen[2247]: close database /var/lib/postfix/ps_cache.db: No such file or directory Aug 28 15:17:49 mail-ausfall postfix/postscreen[726]: close database /var/lib/postfix/ps_cache.db: No such file or directory Aug 29 00:00:03 mail-ausfall postfix/postscreen[6613]: close database /var/lib/postfix/ps_cache.db: No such file or directory Aug 30 08:21:55 mail-ausfall postfix/postscreen[15896]: close database /var/lib/postfix/ps_cache.db: No such file or directory Aug 31 08:12:58 mail-ausfall postfix/postscreen[3321]: close database /var/lib/postfix/ps_cache.db: No such file or directory Aug 31 08:43:35 mail-ausfall postfix/postscreen[22243]: close database /var/lib/postfix/ps_cache.db: No such file or directory Aug 31 09:19:09 mail-ausfall postfix/postscreen[32409]: close database /var/lib/postfix/ps_cache.db: No such file or directory -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebra...@charite.de | http://www.charite.de
Re: postscreen bug ?
* Wietse Venema : > That is a Berkeley DB mis-feature. > Newer Postfix snapshots ignore that error. I'm still seeing it with postfix-2.8-20100830: Sep 1 05:14:38 mail postfix/postscreen[17745]: close database /var/lib/postfix/ps_cache.db: No such file or directory Sep 1 09:49:00 mail postfix/postscreen[25684]: close database /var/lib/postfix/ps_cache.db: No such file or directory Sep 2 08:26:28 mail postfix/postscreen[2276]: close database /var/lib/postfix/ps_cache.db: No such file or directory Sep 3 11:46:48 mail postfix/postscreen[19027]: close database /var/lib/postfix/ps_cache.db: No such file or directory Sep 3 15:01:25 mail postfix/postscreen[14512]: close database /var/lib/postfix/ps_cache.db: No such file or directory Sep 4 11:46:25 mail postfix/postscreen[29647]: close database /var/lib/postfix/ps_cache.db: No such file or directory Sep 4 14:50:02 mail postfix/postscreen[25263]: close database /var/lib/postfix/ps_cache.db: No such file or directory Sep 4 14:50:21 mail postfix/postscreen[27086]: close database /var/lib/postfix/ps_cache.db: No such file or directory -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebra...@charite.de | http://www.charite.de
Re: postscreen bug ?
Ralf Hildebrandt: [ Charset UTF-8 unsupported, converting... ] > * Wietse Venema : > > > That is a Berkeley DB mis-feature. > > Newer Postfix snapshots ignore that error. > > I'm still seeing it with postfix-2.8-20100830: > > Sep 1 05:14:38 mail postfix/postscreen[17745]: close database > /var/lib/postfix/ps_cache.db: No such file or directory Perhaps you notice that this is NOT LOGGED AS A WARNING. Wietse
Re: postscreen bug ?
On Sat, Sep 04, 2010 at 03:28:28PM +0200, Ralf Hildebrandt wrote:
> * Wietse Venema :
>
> > That is a Berkeley DB mis-feature.
> > Newer Postfix snapshots ignore that error.
>
> I'm still seeing it with postfix-2.8-20100830:
>
> Sep 1 05:14:38 mail postfix/postscreen[17745]: close database
> /var/lib/postfix/ps_cache.db: No such file or directory
As expected. Notice that this message is informational, not a warning
or an error:
/*
* With some Berkeley DB implementations, close fails with a bogus ENOENT
* error, while it reports no errors with put+sync, no errors with
* del+sync, and no errors with the sync operation just before this
* comment. This happens in programs that never fork and that never share
* the database with other processes. The bogus close error has been
* reported for programs that use the first/next iterator. Instead of
* making Postfix look bad because it reports errors that other programs
* ignore, I'm going to report the bogus error as a non-error.
*/
if (DICT_DB_CLOSE(dict_db->db) < 0)
msg_info("close database %s: %m", dict_db->dict.name);
Not reporting the anomaly at all may mask real problems in the future.
--
Viktor.
Re: postscreen bug ?
* Wietse Venema : > > Sep 1 05:14:38 mail postfix/postscreen[17745]: close database > > /var/lib/postfix/ps_cache.db: No such file or directory > > Perhaps you notice that this is NOT LOGGED AS A WARNING. Oh, well - Minor detail :) -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebra...@charite.de | http://www.charite.de
Re: postscreen bug ?
> As expected. Notice that this message is informational, not a warning
> or an error:
>
> /*
> * With some Berkeley DB implementations, close fails with a bogus ENOENT
> * error, while it reports no errors with put+sync, no errors with
> * del+sync, and no errors with the sync operation just before this
> * comment. This happens in programs that never fork and that never share
> * the database with other processes. The bogus close error has been
> * reported for programs that use the first/next iterator. Instead of
> * making Postfix look bad because it reports errors that other programs
> * ignore, I'm going to report the bogus error as a non-error.
> */
> if (DICT_DB_CLOSE(dict_db->db) < 0)
> msg_info("close database %s: %m", dict_db->dict.name);
>
> Not reporting the anomaly at all may mask real problems in the future.
Hm, so isn't that a bug in BerkeleyDB then - and should be reported?
--
Ralf Hildebrandt
Geschäftsbereich IT | Abteilung Netzwerk
Charité - Universitätsmedizin Berlin
Campus Benjamin Franklin
Hindenburgdamm 30 | D-12203 Berlin
Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
ralf.hildebra...@charite.de | http://www.charite.de
Re: postscreen bug ?
On Sat, Sep 04, 2010 at 04:45:08PM +0200, Ralf Hildebrandt wrote:
> > if (DICT_DB_CLOSE(dict_db->db) < 0)
> > msg_info("close database %s: %m", dict_db->dict.name);
> >
> > Not reporting the anomaly at all may mask real problems in the future.
>
> Hm, so isn't that a bug in BerkeleyDB then - and should be reported?
Yes, it looks like a Berkeley DB bug. The benefit of reporting it may
not be worth the effort, but if someone wants to be a good citizen,
by all means... This issue may already be known.
--
Viktor.
Re: processing time metrics for rejected connections
Victor Duchovni put forth on 9/4/2010 7:33 AM: > What do you mean by "filters"? Spam filters in the form of table lookups and dnsbl queries. I'm currently processing 12,581 CIDRs 1,568 regular expressions (PCRE) 5 dnsbl lookups per each inbound connection (assuming no hits). Obviously a hit in a table early on in the processing order is going to cut the time down. I'm just wondering how much time is being consumed by processing all of these and the lookup delays to the dnsbl servers. Is it so small it's irrelevant, or is it large enough it could become an issue as I add more tables/entries and/or dnsbls, and inbound mail volume increases? Say I add another 20k CIDR or 1,500 more PCRE entries? FYI this is an old dual CPU 500 MHz 32bit x86 machine. Now you may understand my concern about running large numbers of PCREs and CIDRs per connection. Even with these old CPUs, processing these tables may be a piece of cake, I simply don't know. That's why I'm asking. It may be that all of this is a total non issue, and if so, I'd like to know that. Thanks. -- Stan
Re: Postfix with AD and Exchange
"Ashwin Muni" wrote in message news:aanlktimqqpcmm6f89ioamsjtavtmsn3p+gxs0db0i...@mail.gmail.com... I am using exchange and want to migrate to postfix, The issue is users should be authenticated from Active Directory and other thing is i have got 1000 users from which 150 users will remain on exchange and the rest mailboxes will be created on postfix (Linux Box). the 750 users are actually not very important. If user has mailbox in AD deliver in Ad ||=(4)== || || [][ ]===(3)=<=[ ] [ v ] (1)--> Port 25 ][ Postfix] Authentication from AD [ Active Directory Box] ===[ Exchange] [][Linux Box ]==>===(2)=== [ ] [] || Else Deliver on Postfix Machine (5) || [ v] [ Storage on Linux Box ] [ ] 1. Mail Entering Postfix machien using Port 25 2. Authentication from AD if user exists and mailbox location 3 Results from AD 4. If user on Mailbox on exchange then deliver to echange 5. Else Deliver to postfix Any help appreciated. -- Ashwin Muni http://www.linuxmaza.com Linux Tutorials and Howtos You have the option of sending all mail to Postfix. You could then use an application to pop the mail from Postfix to Exchange for the 150 users. I have seen Windows admins pop mail from a hosted mail service to Exchange on the LAN. Would this work for you? Would you be running Postfix on a LAN?
Re: Postfix with AD and Exchange
On 2010-09-04 at 19:19:34 -0500, Jamrock wrote: > 1. Mail Entering Postfix machien using Port 25 > 2. Authentication from AD if user exists and mailbox location > 3 Results from AD > 4. If user on Mailbox on exchange then deliver to echange > 5. Else Deliver to postfix That's almost exactly how we handle it. > > Any help appreciated. > > You have the option of sending all mail to Postfix. You could then use an > application to pop the mail from Postfix to Exchange for the 150 users. > > I have seen Windows admins pop mail from a hosted mail service to Exchange on > the LAN. We use an Ubuntu box along with the likewise-open package to join the Ubuntu machine to the domain. Because the server is 'joined' to the domain, it treats all the Windows uses as 'local' users on the Linux box--so by default, all mail will be delivered to the Linux box. We override the exchange users specifically in the transport_maps file: u...@example.tldsmtp:[exc.han.ge.ip] -A
Re: Postfix with AD and Exchange
On 2010-09-04 at 21:35:44 -0700, Aaron C. de Bruyn wrote: > We use an Ubuntu box along with the likewise-open package to join the Ubuntu > machine to the domain. Bad form to reply to my own message--but I forgot to include: We do spamassassin filtering on the Linux box too which has benefits for Exchange users. If you are using Exchange 2003, you can create: C:\Program Files\Exchsrvr\bin\MSCFV2\MSExchange.UceContentFilter.xml using notepad With the content: http://schemas.microsoft.com/2005/CustomWeight";> Save the file as *UNICODE* and then make sure your IMF settings are correct in Exchange as well as making sure Spamassassin prefixes the message subject with '[SPAM]' and all your spammy messages will be moved by Exchange to your Junk folder. -A
