postfix dimensioning
Hi All, first post here, nice to meet you :) I've been using postfix and dovecot for years but always with few users. Now i must build a server for 1500 users, they will use various email software (thunderbird, outlook, ..) the webmail (i'm not sure if squirrelmail or roundcube) and blackberry devices (with the BIS service). There will be around 1000 domains (virtual_domains) and postfix will read its users and domains from mysql. I already have the hardware, it is a server with 4 physical cpu (Intel Xeon E5504 @ 2.00GHz) and 24 gb of ram, do you think it is enough ? Thanks for supporting, regards and have a nice day, Mike
Re: postfix dimensioning
On Thu, 01 Nov 2012 10:22:13 +0100 Robert Schetterer wrote: > Am 01.11.2012 09:14, schrieb Ram: > > > > On 11/01/2012 11:49 AM, [email protected] wrote: > >> Hi All, first post here, nice to meet you :) > >> > >> I've been using postfix and dovecot for years but always with few users. > >> > >> Now i must build a server for 1500 users, > >> they will use various email software (thunderbird, outlook, ..) > >> the webmail (i'm not sure if squirrelmail or roundcube) > >> and blackberry devices (with the BIS service). > >> > >> There will be around 1000 domains (virtual_domains) and postfix will > >> read its users and domains from mysql. > >> > >> I already have the hardware, it is a server with 4 physical cpu (Intel > >> Xeon E5504 @ 2.00GHz) and 24 gb of ram, > >> do you think it is enough ? > >> > > Usually postfix would not be the bottleneck for any mail server. You > > need to size for dovecot in your case. > > And sizing would be impossible unless you know how many mails / hr or > > concurrent imap / pop connections you are going to get. > > Please size your dovecot server accordingly. The disk i/o will also be > > very important when you size dovecot. > > > > Thank > > Ram > > PS: 1500 users with 1000 domains means hardly 2 users per domain .. > > Are these right numbers ? > > > > Hi, what you need is some idea of tec about backup, failure safety, > loadbalancing , cluster storage etc specially on the imap/pop side, > postfix side mostly is the more easy part in such setups > and not known for needing heavy hardware or producing bottlenecks > in short words, you have to think about the whole setup you wanna goal > first, > not special software like postfix, perhaps you should first look on > dovecot site and list for examples and hints > > Best Regards > MfG Robert Schetterer > > -- > [*] sys4 AG > > http://sys4.de, +49 (89) 30 90 46 64 > Franziskanerstraße 15, 81669 München > > Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 > Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer > Aufsichtsratsvorsitzender: Joerg Heidrich Hi Ram and Robert thank you very much for your pointers, I'll ask on the dovecot mailing list and do some homework :) > PS: 1500 users with 1000 domains means hardly 2 users per domain .. > Are these right numbers ? yes strange but true, almost each domain has just 1 user. Regards and have a nice day, Mike
postfix and cacti (snmp ?)
Hi All, I've put online a postfix+dovecot+mysql+roundcube server that will grow till serving around 1500 accounts. I've installed the usual cacti graphs, plus the beautiful percona suite for mysql's cacti plus some graphs for iops and iostat. I was lurking around for the best solution to graph postfix usage, the most detailed possible, in order to prevent and foresee problems. I'm finding sparse results, I'm not sure which one is the most current / complete. Do you have any suggestions ? Thank you very much for supporting, regards and have a nice day, Mike
custom filter and mailer-daemon
Hi postfix-arena ppl!
I've developed a simple PHP script that is supposed to capture returning mail
delivery errors ("Undelivered Mail Returned to Sender")
and mark on a database that the email address is bad. (100% legit traffic I
swear, I work for an ISP and hate spam / abuses as much as you do :|)
So at the end of my master.cf I've added the following:
myUndRetCustomFilter unix - n n - - pipe
flags=F user=vmail:vmail argv=/usr/local/bin/myUndRetCustomFilter.php
${sender} ${size} ${recipient}
and created a file /etc/postfix/myUndRetCustomFilter
with inside the definition:
[email protected] FILTER myUndRetCustomFilter:dummy
and then created the database with the command:
postmap /etc/postfix/myUndRetCustomFilter
it works ok when testing manually, from the command line, with the command:
/usr/local/bin/myUndRetCustomFilter.php <
Maildir/cur/1384946361.M523250P7714.linuxmailserver\,S\=3871\,W\=3978\:2\,
[email protected]
(where Maildir/cur/13849... is the email file with the "Undelivered Mail
Returned to Sender" error and attached the original email)
It also works ok when sending the same email directly to
[email protected] with an email client.
But with real traffic I can see in the syslog that the filter doesn't get
triggered.
My gut suggests me that emails generated by postfix itself:
[email protected] (Mail Delivery System)
bypass the master.cf filter rule.
If that's the case, do you know a way to enforce the processing of the custom
filter also for mailer-daemon ?
Other workarounds or alltogether solutions are as much greatly welcome :)
Thank you very much for supporting,
regards and have a nice day.
Mike
Re: custom filter and mailer-daemon
On Mon, 5 May 2014 20:19:50 -0400 (EDT) [email protected] (Wietse Venema) wrote: > [email protected]: > > But with real traffic I can see in the syslog that the filter > > doesn't get triggered. > > > > My gut suggests me that emails generated by postfix itself: > > [email protected] (Mail Delivery System) > > bypass the master.cf filter rule. If that's the case, do you know > > a way to enforce the processing of the custom filter also for > > mailer-daemon ? > > If you use smtpd_xxx_restrictions, then of course they won't have > any effect at all on bounces that Postfix sends itself. > > Instead of FILTER, use transport_maps. > > http://www.postfix.org/postconf.5.html#transport_maps > http://www.postfix.org/transport.5.html > > Wietse Hello Wietse (*so much respect toward you*), I've read the documentation, switched to transport_maps changing a bit the filter syntax and it seems to work. Thank you very much! It's really beautiful to 'plug' scripts into the engine logic, so flexible and powerful! Wish you a nice evening, Mike
transport_maps, custom script and 'too many hops'
Hi All!
I've created a simple PHP script that recognizes mail delivery errors
("Undelivered Mail Returned to Sender")
and does a number of things on a database (100% legit traffic I swear, I work
for an ISP and hate spam / abuses as much as you do :|)
I've added it to transport_maps
so it processed mails bounced by postfix itself
with the string:
[email protected] mySimplePhpScript:dummy
it works great but there is a problem:
when trying to send a normal (not an error) direct email to
[email protected]
it loops around 10 times and then dies with the error:
status=bounced (too many hops)
and the email is not delivered to [email protected]
The PHP script finishes with this:
$sendmail = '/usr/sbin/sendmail -G -i ' . implode(' ', $argv);
$handle = popen($sendmail, 'w');
fwrite($handle, $content);
$sendmail_return_value = pclose($handle);
so it uses 'sendmail' to try to delivery the email, but it loops.
I've read the transport_maps documentation but I have no idea how to prevent
this,
if I delete that part of code, the email looks delivered ok:
status=sent (delivered via mySimplePhpScript service)
but it doesn't show up in the inbox.
Thank you very much for supporting, any hint would be greatly appreciated!
Best Regards,
Mike
custom script and loops problem - looking for professional support
Hi I wrote this script: http://pastebin.com/BZGNDcdn which is configured in postfix in transport_maps like this: [email protected] myCustomScript:dummy and is supposed to catch emails that return errors ("Undelivered Mail Returned to Sender") when [email protected] sends emails. (btw I'm sending 100% legit traffic, nothing spammy here) It works very well but if I try to send an email to [email protected] it loops 10 times and then gives up (554 5.4.0 Error: too many hops). I'm seeking community or professional support to fix this issue. Thanks for your attention, regards and have a nice day, Mike
improving logging
Hi is it possible to log what postfix does to mysql ? For example each email sent or received, with sender and receiver, date, status, file length. And even better, is it possible to integrate that with BL / SA / AV in order to have on the same database row also the blacklist check, spam score and antivirus info ? Thanks for your attention, regards and have a nice day, Mike
Re: improving logging
On Mon, 26 May 2014 16:17:14 +0200 Ansgar Wiechers wrote: > On 2014-05-26 [email protected] wrote: > > Hi is it possible to log what postfix does to mysql ? > > For example each email sent or received, with sender and receiver, > > date, status, file length. > > And even better, is it possible to integrate that with BL / SA / AV > > in order to have on the same database row also the blacklist check, > > spam score and antivirus info ? > > Postfix logs to syslog and syslog (rsyslog at least) can be configured > to write to MySQL instead of files. See [1] for details. > > [1] http://www.rsyslog.com/doc/rsyslog_mysql.html > > Regards > Ansgar Wiechers > -- > "Abstractions save us time working, but they don't save us time learning." > --Joel Spolsky Hi yes sure but that would result in having 20 database rows for each email, since that is what happens in the syslog, each component (postfix, spamassassin, amavis, policyd, ...) writes few lines in the syslog for each email sent or received. Also, if aggregating, that information must be correlated. Well I guess the answer is no, perhaps qmail does it better ? Thanks for supporting, regards and have a nice day, Mike
Re: improving logging
On Mon, 26 May 2014 16:43:45 +0200 "[email protected]" wrote: > > Am 26.05.2014 16:31, schrieb [email protected]: > > Hi yes sure but that would result in having 20 database rows for each > > email, since that is what happens in the syslog, > > each component (postfix, spamassassin, amavis, policyd, ...) writes few > > lines in the syslog for each email sent or received. > > > > Also, if aggregating, that information must be correlated. > > Well I guess the answer is no, perhaps qmail does it better? > > *you* need to correlate that > > that's why the queue-id exists in the logs and if there > are serveral servers talking to each other you > get as last line in your own log even the queue-id > of the destination > > this *can not* be in one line because that is just how > email works - a message is accepted, queued, forwarded > to filters and back, tried several times if the detsination > is not available yet > > so there is per definition no process knowing the > whole flow of a mail from A to Z and given how > email works the first queue line may be written > now and the final line (sent or bounced) 5 days > later > ___ > > [root@srv-rhsoft:~]$ cat maillog | grep 3gcMzr47KVzBr0x > May 26 04:43:04 srv-rhsoft postfix/smtpd[19441]: 3gcMzr47KVzBr0x: > client=* > May 26 04:43:04 srv-rhsoft postfix/cleanup[19446]: 3gcMzr47KVzBr0x: > message-id=<3gcMzq5pg3z1LHc@> > May 26 04:43:04 srv-rhsoft postfix/qmgr[28478]: 3gcMzr47KVzBr0x: > from=**, size=3265, nrcpt=1 (queue active) > May 26 04:43:04 srv-rhsoft postfix/lmtp[19447]: 3gcMzr47KVzBr0x: to=, > relay=127.0.0.1[127.0.0.1]:24, > delay=0.44, delays=0.16/0.04/0/0.25, dsn=2.0.0, status=sent (215 Recipient > *** OK) > May 26 04:43:04 srv-rhsoft postfix/qmgr[28478]: 3gcMzr47KVzBr0x: removed Thanks for the info, appreciated. Just wondering, is there any 'packaged' / 'commercial' version of the email stack on linux, suitable for an ISP with around 3K email accounts ? Best Regards, Mike
Re: custom script and loops problem - looking for professional support
On Mon, 26 May 2014 17:33:52 -0500 Noel Jones wrote: > On 5/26/2014 6:19 AM, [email protected] wrote: > > Hi I wrote this script: http://pastebin.com/BZGNDcdn > > which is configured in postfix in transport_maps like this: > > [email protected] myCustomScript:dummy > > and is supposed to catch emails that return errors ("Undelivered Mail > > Returned to Sender") when [email protected] sends emails. > > (btw I'm sending 100% legit traffic, nothing spammy here) > > > > It works very well > > but if I try to send an email to [email protected] > > it loops 10 times and then gives up (554 5.4.0 Error: too many hops). > > A transport must deliver mail outside of postfix, such as to a > remote system, a mailbox, or another postfix instance. The > transport must not loop mail back into the same postfix instance. > > Perhaps a simple content_filter will fit your needs better than a > transport. Or maybe not. > http://www.postfix.org/FILTER_README.html#simple_filter > > > -- Noel Jones > > > > > > I'm seeking community or professional support to fix this issue. > > > > > > Thanks for your attention, > > regards and have a nice day, > > Mike > > > Hi Noel, thanks for your suggestion but unfortunately I've already been there. If you configure the script as a FILTER, instead of a transport_map, you'll not be able to catch bounced emails as suggested here: http://marc.info/?l=postfix-users&m=139933564703452&w=2 Thanks to the helpful community, regards and have a nice day, Mike
Re: custom script and loops problem - looking for professional support
On Tue, 27 May 2014 06:41:22 -0400 (EDT) [email protected] (Wietse Venema) wrote: > Noel Jones: > > Perhaps a simple content_filter will fit your needs better than a > > transport. Or maybe not. > > http://www.postfix.org/FILTER_README.html#simple_filter > > [email protected]: > > If you configure the script as a FILTER, instead of a transport_map, you'll > > not be able to catch bounced emails > > as suggested here: http://marc.info/?l=postfix-users&m=139933564703452&w=2 > > The FILTER has already handled the email BEFORE it bounced. > Why do you need to see it twice? > > Wietse Hi Wietse, thanks for your attention. When using the script as a FILTER, it isn't able to catch emails generated by postfix itself, as described here: http://marc.info/?l=postfix-users&m=139932881701528&w=2 instead, while using it as a transport_map, it works very well but works only with emails generated by postfix itself: when trying to send an email to it, it loops 10 times and then gives up with the 'too many hops' error. It is the last part of the script: http://pastebin.com/BZGNDcdn this part: $sendmail = '/usr/sbin/sendmail -G -i ' . implode(' ', $argv); $handle = popen($sendmail, 'w'); fwrite($handle, $content); $sendmail_return_value = pclose($handle); the script passes the email to the receiver (itself in this given problem case), but it loops and then it gives up. Thanks, regards and have a nice day, Mike
Re: custom script and loops problem - looking for professional support
On Tue, 27 May 2014 13:02:20 + Viktor Dukhovni wrote: > On Tue, May 27, 2014 at 11:36:46AM +0200, [email protected] wrote: > > > Hi Noel, thanks for your suggestion but unfortunately I've already been > > there. > > And yet Noel is right. Transport overrides that direct mail to a > script are NOT compatible with re-injection of the message back > into the same Postfix queue by the script. This obviously leads > to a loop, how could it not? > > Therefore, you must either use a FILTER (which works with re-injection > provided that re-injection does activate the filter again, see > FILTER_README), or else you MUST NOT re-inject the message back > into the same queue. > > Since you've not explained what goals this particular script is > intended to address, further help is difficult. So far, we're > tackling this at the wrong layer, discussing your solution, rather > than your problem. > > -- > Viktor. Hi I've explained my goal here: http://marc.info/?l=postfix-users&m=139932881701528&w=2 then here: http://marc.info/?l=postfix-users&m=140015047902115&w=2 and here: http://marc.info/?l=postfix-users&m=140110327807685&w=2 also suppliying code. Anyway I've 'fixed' the problem by using another email to catch those errors and adding a 'reply-to' header. Thanks and regards, Mike
allow an email account to receive emails from a blacklisted IP
Hi I have the same configuration and situation as the following: http://serverfault.com/questions/132750/postfix-whitelist-before-recipient-restrictions In short: " So i need a method of whitelisting ANY email that comes to domain.com, however i need it to take place before any of the recipient restrictions, they want no RBL or postgrey blocking at all. " but the accepted answer has two contradictions: first he suggest to use "check_sender_access" but in the example he uses "check_client_access", then he uses "smtpd_recipient_restrictions" but in the comments someone points out why not to use "smtpd_sender_restrictions" .. Can you please clarify which is the best method ? Thanks for your attention, regards and have a nice day, Mike
Re: allow an email account to receive emails from a blacklisted IP
On Tue, 3 Jun 2014 08:55:46 -0400 (EDT) [email protected] (Wietse Venema) wrote: > [email protected]: > > So i need a method of whitelisting ANY email that comes to domain.com, > > however i need it to take place before any of the recipient > > restrictions, they want no RBL or postgrey blocking at all. > > Birta Levente: > > If you want to whitelist recipient domain you need to check recipient > > access: > > > > http://www.postfix.org/postconf.5.html#smtpd_recipient_restrictions > > > > smtpd_recipient_restrictions = > > reject_unauth_destination, > > check_recipient_access somemapfile, > > reject_rbl_client b.barracudacentral.org, > > reject_rbl_client cbl.abuseat.org, > > reject_rbl_client bl.mailspike.net, > > check_policy_service unix:postgrey/socket > > > > > > somemapfile: > > domain.comOK > > This is correct, by placing the whitelist after reject_unauth_destination. > Otherwise you would have an exploitable open relay. > > Wietse Hi Wietse and all the community, thanks for enlighting on this. I applied the changes and I'm waiting for some spam to come in :) I'll report on this / confirm if everything goes well. Thanks for supporting, regards and have a nice day, Mike
postfix and exchange
Hi, I'm currently the following two parameters: reject_authenticated_sender_login_mismatch, reject_unauthenticated_sender_login_mismatch, under 'smtpd_recipient_restrictions' to enforce sasl username+password authentication. As you may know, M$ Exchange, when using an external SMTP, can't send username+password of each user but only 1 username+password for all the domain. In short, postfix must allow spoofing the username of the domain. Perhaps I'm missing something about all this, if not, the question is: is there a way in postfix to disable the two parameters above just for that domain (thus allowing spoofing of the username, only for that domain) ? Thank you.
Re: postfix and exchange
On Wed, 03 Dec 2014 13:18:45 +0100 "[email protected]" wrote: > > Am 03.12.2014 um 13:06 schrieb [email protected]: > > Hi, I'm currently the following two parameters: > > > > reject_authenticated_sender_login_mismatch, > > reject_unauthenticated_sender_login_mismatch, > > > > under 'smtpd_recipient_restrictions' > > to enforce sasl username+password authentication. > > > > As you may know, M$ Exchange, when using an external SMTP, can't send > > username+password of each user > > but only 1 username+password for all the domain. > > In short, postfix must allow spoofing the username of the domain. > > > > Perhaps I'm missing something about all this, > > if not, the question is: is there a way in postfix to disable the two > > parameters above just for that domain > > (thus allowing spoofing of the username, only for that domain)? > > put the exchange host in "mynetworks" and just add "permit_mynetworks" > *before* "reject_authenticated_sender_login_mismatch" > > you don't want to do this domain based with access lists because that > can be spoofed easily Hi, my whole 'smtpd_recipient_restrictions' is: smtpd_recipient_restrictions = permit_mynetworks, check_policy_service inet:127.0.0.1:10031, check_sender_access hash:/etc/postfix/sender_access, check_recipient_access hash:/etc/postfix/check_recipient_access_skip_blacklists, reject_authenticated_sender_login_mismatch, reject_unauthenticated_sender_login_mismatch, permit_sasl_authenticated, reject_non_fqdn_hostname, reject_non_fqdn_sender, reject_non_fqdn_recipient, reject_unauth_destination, reject_unauth_pipelining, reject_invalid_hostname, check_client_access hash:/etc/postfix/rbl_override reject_rbl_client zen.spamhaus.org, reject_rbl_client cbl.abuseat.org, reject_rbl_client bl.spamcop.net, this is a server with many domains and users (and filters, also custom antispam filters). Wouldn't your suggested modification disable all the rest of the processing logic ? I mean .. would reject_rbl_client zen.spamhaus.org still be considered ? If that's the case, wouldn't be easy to spoof the domain and abuse the server ? Thank you very much for supporting.
Re: postfix and exchange
On Wed, 03 Dec 2014 14:11:44 +0100 "[email protected]" wrote: > > Am 03.12.2014 um 13:40 schrieb [email protected]: > > On Wed, 03 Dec 2014 13:18:45 +0100 > > "[email protected]" wrote: > >> put the exchange host in "mynetworks" and just add "permit_mynetworks" > >> *before* "reject_authenticated_sender_login_mismatch" > > > > Hi, my whole 'smtpd_recipient_restrictions' is: > > > > smtpd_recipient_restrictions = > > permit_mynetworks, > > check_policy_service inet:127.0.0.1:10031, > > check_sender_access hash:/etc/postfix/sender_access, > > check_recipient_access > > hash:/etc/postfix/check_recipient_access_skip_blacklists, > > reject_authenticated_sender_login_mismatch, > > reject_unauthenticated_sender_login_mismatch, > > permit_sasl_authenticated, > > reject_non_fqdn_hostname, > > reject_non_fqdn_sender, > > reject_non_fqdn_recipient, > > reject_unauth_destination, > > reject_unauth_pipelining, > > reject_invalid_hostname, > > check_client_access hash:/etc/postfix/rbl_override > > reject_rbl_client zen.spamhaus.org, > > reject_rbl_client cbl.abuseat.org, > > reject_rbl_client bl.spamcop.net, > > > > this is a server with many domains and users (and filters, also custom > > antispam filters). > > > > Wouldn't your suggested modification disable all the rest of the processing > > logic ? > > I mean .. would reject_rbl_client zen.spamhaus.org still be considered ? > > If that's the case, wouldn't be easy to spoof the domain and abuse the > > server? > > if the sending host is controlled by you it makes no sense to do RBL > checks for one of your own machines and so it should be safe add the > host to "mynetworks" Hi, the Exchange server is not controlled by me, nor are the clients. I agree that skipping RBLs checks is ok but what about the other rules, will they be skipped ? I need policyd to track sending, together with amavis and spamd. Thanks.
Re: using a non fully qualified host name as relayhost
On Wed, 3 Dec 2014 16:55:47 + Viktor Dukhovni wrote: > On Wed, Dec 03, 2014 at 04:36:12PM +, James Bailey wrote: > > > is it possible to use a non fully qualified host name as relayhost? > > Not by default. And it is generally not a good idea to change this. > > However, if you replace the relayhost setting with: > > # Default empty value > # relayhost = > default_transport = nondns:[relayhost] > > and never use the "nondns" transport in transport_maps with > any other nexthop address, you can configure: > > master.cf: > nondns unix - - n - - smtp > -o smtp_host_lookup=native > # With Postfix >= 2.11 uncomment this, comment-out next: > # -o smtp_dns_support_level=disabled > -o disable_dns_lookups=yes > > > My client has a number of different relay hosts in DCs globally these are > > marked for example us1, us2, in1, au2. So a relay-host. I would simple like > > to be able roll out a default single main.cf globally with relayhost = > > my-relay rather than my-relay.$dc.example.com and have to change it for each > > DC. > > It is simplest to automate the construction of a site-specific > main.cf with a Makefile. > > -- > Viktor. Perfect, thanks to all for the suggestions and for supporting! :)
