Oddly formatted date (malfomed even)

[Chris G]

I have a mail sent across my LAN (from postfix to postfix) whose headers
start as follows:-

From r...@zbmc.eu  Tue Nov 23 08: 2:13 2010
Return-Path: 
X-Original-To: root
Delivered-To: r...@zbmc.eu
Received: by mws.zbmc.eu (Postfix, from userid 0)
id 1282856013B; Tue, 23 Nov 2010 08:42:13 + (GMT)
From: Cron Daemon 
To: r...@zbmc.eu
Subject: Cron  test -x /usr/sbin/anacron || ( cd / &&
run-parts --report /etc/cron.daily
)
Content-Type: text/plain; charset=ANSI_X3.4-1968
X-Cron-Env: 
X-Cron-Env: 

X-Cron-Env: 
X-Cron-Env: 
Message-Id: <20101123084213.12828560...@mws.zbmc.eu>
Date: Tue, 23 Nov 2010 06:45:52 + (GMT)
Status: RO
Content-Length: 983

Is that "08: 2:13" correct/allowed?

It's also a bit odd in that the mail actually arrived in my inbox (also
on the same LAN) at around 11:40 so it seems to have taken an awfully
long time for this E-Mail to get delivered.

Should I be worried?

-- 
Chris Green

Relayhost

[Angelo Amoruso]

Hi,

is there any way for postfix to change the outgoing relayhost using a 
sort of policy server?
So far I've seen in the docs it's possible to change the relay host 
using a sender dependant map, but it would be a static approach, not a 
dynamic one.


One idea that pops up in my mind is  using as relayhost an ad-hoc smtp 
server-like daemon, but this would get the email off postfix queue and 
then delegating another process for the delivery itself, which isn't 
what I'm looking for.


Thanks for any idea,

Angelo

Are subdomains in mydestination redundant?

[Chris G]

Currently my mydestination line looks like:-

zbmc.eu, dps.zbmc.eu, localhost

This is on the mail server which receives E-Mail from the 'outside
world' from whose point of view my system is zbmc.eu.  Is the
dps.zbmc.eu entry actually necessary or will mail for dps.zbmc.eu be
accepted anyway because mail for zbmc.eu is accepted?

There are likely *some* E-Mails sent to some...@dps.zbmc.eu from within
the LAN.

-- 
Chris Green

Re: Are subdomains in mydestination redundant?

[Noel Jones]

On 11/23/2010 6:00 AM, Chris G wrote:

Currently my mydestination line looks like:-

 zbmc.eu, dps.zbmc.eu, localhost

This is on the mail server which receives E-Mail from the 'outside
world' from whose point of view my system is zbmc.eu.  Is the
dps.zbmc.eu entry actually necessary or will mail for dps.zbmc.eu be
accepted anyway because mail for zbmc.eu is accepted?

There are likely *some* E-Mails sent to some...@dps.zbmc.eu from within
the LAN.



Subdomains are not automatically included in mydestination, 
they must be listed explicitly.


You must list dps.zbmc.eu in mydestination.


http://www.postfix.org/postconf.5.html#mydestination
http://www.postfix.org/postconf.5.html#relay_domains
http://www.postfix.org/ADDRESS_CLASS_README.html
http://www.postfix.org/STANDARD_CONFIGURATION_README.html
http://www.postfix.org/BASIC_CONFIGURATION_README.html

Re: Are subdomains in mydestination redundant?

[Chris G]

On Tue, Nov 23, 2010 at 06:42:11AM -0600, Noel Jones wrote:
> On 11/23/2010 6:00 AM, Chris G wrote:
> >Currently my mydestination line looks like:-
> >
> > zbmc.eu, dps.zbmc.eu, localhost
> >
> >This is on the mail server which receives E-Mail from the 'outside
> >world' from whose point of view my system is zbmc.eu.  Is the
> >dps.zbmc.eu entry actually necessary or will mail for dps.zbmc.eu be
> >accepted anyway because mail for zbmc.eu is accepted?
> >
> >There are likely *some* E-Mails sent to some...@dps.zbmc.eu from within
> >the LAN.
> >
> 
> Subdomains are not automatically included in mydestination, they
> must be listed explicitly.
> 
> You must list dps.zbmc.eu in mydestination.
> 
OK, thank you.

-- 
Chris Green (at server dps)

Data check before accepting a letter

[Неворотин Вадим]

I need to check letter's heading (subject) and if subject doesn't contain
some passphrase check this letter with greylisting or rbl. I can't
understand how can I solve such problem with postfix. My server should check
all smtpd_*_restrictions for SMTP headers, then start to receive data and if
data doesn't contain some passphrase, check smtpd_data_restrictions, in
which I want to add greylisting. So I need some option like
check_policy_service for smtpd_data_restrictions, which will send to
external service not only SMTP headers, but also a letter's body. And then
add to main.cf something like

smtpd_data_restrictions =
  reject_unauth_pipelining
  *check_data??? # What I should place here? *
  check_policy_service inet:localhost:6  # Greylisting
  reject_rbl_client bl.spamcop.net # RBL

My company has a lot of new clients and my managers should be able to say
some passphrase to client (by phone eg) for passing my spamfilter without
checking RBL and without Greylisting. I can't add clients to whitelist, so
passphrase is the easest solution for my users, but I can't understand how
to implement it with Postfix. In a few words: I need to check data
*before*accepting a letter to be able to reject a letter with SMTP
error code based
on its content. How can I do it with postfix?

Re: Oddly formatted date (malfomed even)

[Wietse Venema]

Chris G:
> I have a mail sent across my LAN (from postfix to postfix) whose headers
> start as follows:-
> 
> From r...@zbmc.eu  Tue Nov 23 08: 2:13 2010

Please verify that this line is from Postfix and not from procmail
or some other software.

In the case of Postfix, the time+date stamp is produced by the
asctime() system library function, as shown below. Please file a
bug with your vendor, and don't forget to include your locale
since the bug may be language-dependent.

vstream_fprintf(dst, "From %s  %.24s%s", *sender == 0 ?
MAIL_ADDR_MAIL_DAEMON : vstring_str(buf),
asctime(localtime(&now)), eol);

In the case of procmail or some other software, please file a bug
there.

Wietse

Re: Data check before accepting a letter

[Wietse Venema]

> I need to check letter's heading (subject) and if subject doesn't contain
> some passphrase check this letter with greylisting or rbl. I can't
> understand how can I solve such problem with postfix. My server should check

To inspect (and reject) content during the SMTP DATA command, see:
http://www.postfix.org/SMTPD_PROXY_README.html

Wietse

Re: Data check before accepting a letter

[Неворотин Вадим]

Thanks! I've read it but I still can't understand how to combine content
check and SMTP headers check. The task: if mail has some passphrase in
subject (eg [passme]), then do not apply greylistnig, RBL etc and transfer a
mail to LDA. If doesn't contain passphrase then check greylisting etc in a
usual way. PROXY, as I understand, can reject mails based on its content,
but how can I implement greylisting after proxy, but only if proxy hasn't
found passphrase in content?

2010/11/23 Wietse Venema 

> > I need to check letter's heading (subject) and if subject doesn't contain
> > some passphrase check this letter with greylisting or rbl. I can't
> > understand how can I solve such problem with postfix. My server should
> check
>
> To inspect (and reject) content during the SMTP DATA command, see:
> http://www.postfix.org/SMTPD_PROXY_README.html
>
>Wietse
>

Re: Data check before accepting a letter

[Wietse Venema]

> The task: if mail has some passphrase in subject (eg [passme]),
> then do not apply greylistnig, RBL etc and transfer a mail to LDA.

If you whitelist based on a string in the message, then why not
whitelist based on a string in the RECIPIENT ADDRESS? Postfix
supports address extensions, like this:

in main.cf:
recipient_delimiter = +
# Alternative: recipient_delimiter = -

smtpd_recipient_restrictions =
permit_mynetworks
reject_unauth_destination
check_policy_service x
reject_rbl_client y
...

in SMTP:
RCPT TO:

Then, the greylist daemon can reply with OK if the passphrase is
good and skip Postfix RBL checks.  You can make it as fancy as you
like, such as a passphrase that is valid for only a short time, a
passphrase that is valid only for with a specific sender IP address,
and so on.

Wietse

Re: Block A Sender in Postfix

[mouss]

Le 22/11/2010 20:04, Carlos Mennens a écrit :

On Mon, Nov 22, 2010 at 1:00 PM, Rich Shepard  wrote:

Carlos,

  I use a badaddr file that lists domains from whom I will not accept
messages. The content looks like these:

hostforreal.com 550 Rejected domain D23
nasty-mailings.com  550 Rejected domain D24


I've done the same and mine looks simular:

[r...@mail postfix]# cat client_access
bluehornet.com  REJECT Rejected Domain

But my confusion with Postfix has always been where to add then map
check under which specific smtpd_*_restriction(s). According to "The
Book of Postfix" I am still very confused:

- smtpd_client_restrictions = applies to the client's IP address or
its hostname or both.
- smtpd_recipient_restrictions = applies to the envelope recipient(s),
the envelope sender, the HELO/EHLO argument, and client IP / hostname
or both.
- smtpd_sender_restrictions = This is the 1st trigger set that
restricts parts of the envelope. Postfix applies to the envelope
sender, the HELO/EHLO argument, and the client.

So with that defined above, how am I to understand or determine where
I would add my 'client_access' check in my main.cf? According to the
definitions above, the 'smtpd_recipient_restirctions' looks like it
runs the specific map against every aspect of the sender rather than
the other two. It seems like the logical choice, no? I apologize if
I'm just dumb when it comes to Postfix but I'm really putting time and
effort in to trying to understand this so I wont have to annoy most
with my ignorance.




  In the UCE section of /etc/postfix/main.cf I have this line:

check_client_access hash:/etc/postfix/badaddr,

and it kicks back messages from the listed domains.

  Also, I use a badip file for specific IP addresses and address blocks.


Where do you have those listed under in your main.cf?

smtpd_recipient_restrictions = check_badaddrhash:/etc/postfix/badaddr ?



Put all your anti-spam checks under smtpd_recipient_restrictions, but be 
careful not become an open relay: for safety, put check_*_access after 
reject_unauth_destination.


here is an example to get you started.
Note 1: you need to understand before use. anything you do is under your 
own responsibility!


Note 2: order is important. order your checks as needed.


#db = hash:/etc/postfix/maps/hash
db = cdb:/etc/postfix/maps/cdb
pcre = pcre:/etc/postfix/maps/pcre
cidr = cidr:/etc/postfix/maps/cidr
sql = proxy:mysql:/etc/postfox/maps/mysql

smtpd_recipient_restrictions =
#reject_non_fqdn_sender
#reject_non_fqdn_recipient
#
#permit_sasl_authenticated
permit_mynetworks
reject_unauth_destination
#
reject_unlisted_recipient
reject_unlisted_sender
#
#reject_invalid_helo_hostname
#reject_non_fqdn_helo_hostname
#
#check_recipient_access${db}/access_recipient
#
#check_client_access   ${cidr}/access_client
#check_client_access   ${db}/access_client
#
#check_sender_access   ${db}/access_sender
#check_sender_access   ${pcre}/access_sender
#
#check_helo_access ${db}/access_helo
#check_helo_access ${db}/access_host
#check_reverse_client_hostname_access  ${db}/access_host
# DNSBL checks
#reject_rbl_client zen.spamhaus.org
#reject_rbl_clientbl.spamcop.net
#reject_rbl_clientpsbl.surriel.com
#reject_rbl_clientkorea.services.net
#reject_rbl_clientbb.barracudacentral.org


== access_recipient
ab...@example.com   OK
postmas...@example.com  OK
#opt-out from checks
spamlo...@example.com
#reject extension
joe+old...@example.com
#use a spcieific restriction class
example.net policy_dothat
.example.netpolicy_dothat


== access_sender
# reject a specific user
no...@gooddomain.exampleREJECT blah blah
# reject a full domain
evil.example  REJECT spammy domain
.evil.example REJECT spammy domain



== ${cidr}/access_client
#whitelist
192.0.2.0/24OK
#blacklist
192.168.1.2/32  OK
10.1.0.0/16 REJECT blah blah


== ${db}/access_client
# whitelist
good.exampleOK
.good.example   OK
# block
bad.example REJECT blah blah
.bad.exampleREJECT blah blah

Re: How to replace underscores in hostnames to a valid character?

[mouss]

Le 22/11/2010 08:55, J. Roeleveld a écrit :

[snip]

There is only 1 email address causing issues, eg. am only rewriting that
specific one.



if so, then simply use

u...@lists_example_org  u...@lists-example.org



better reject the address and let the sender fix his ratware...


That is the default behaviour, unfortunately, I am using fetchmail to get the
emails out of a mailbox elsewhere and injecting that into postfix.


with fetchmail, it is a good practice not send bounces (because that 
would cause backscatter).


Maybe you can tell cyrus to accept malformed sender adresses?



If postfix rejects the email, it tries to send the bounce out via the relay
smtp-server (which rejects the address) and I end up with error messages on my
side.
At least this way, I can still see the list-messages from this individual.

I don't have a guaranteed fixed IP-address which is a requirement for having
the MX-records point to my server directly.

--
Joost

Re: Relayhost

[Noel Jones]

On 11/23/2010 5:54 AM, Angelo Amoruso wrote:

Hi,

is there any way for postfix to change the outgoing relayhost
using a sort of policy server?


Postfix doesn't directly support that, although there may be 
workarounds for some situations.  What information will you 
use to determine the relayhost?  What's the application?


It would help if we have a full explanation of what you're 
trying to accomplish and/or what problem you're trying to 
solve (overview of the end result, not implementation details).



  -- Noel Jones

Re: Mysql queries being performed for domains not in virtual_alias_domains

[Andrew Beverley]

On Sun, 2010-11-21 at 16:50 -0500, Jerrale G wrote:
> On 11/21/2010 4:40 PM, Andrew Beverley wrote:
> > Hi,
> >
> > I have noticed that my Postfix (version 2.3.8) is performing a
> > virtual_alias_maps mysql database query for every email that it is
> > processing, even if the domain is not listed in virtual_alias_domains.
> >
> > So for example, I have andybev.com in virtual_alias_domains and a
> > database query set up for virtual_alias_maps. When I send an email
> > *from* the server to an external email address, then that recipient (eg
> > joeblo...@hotmail.com) gets looked up in the virtual_alias_maps database
> > query first before it is delivered.
> >
> > Is this normal? This doesn't seem right, so I am thinking that there is
> > something wrong with my configuration (all of the thousands of outbound
> > emails are causing database lookups).
> >
> > Thanks,
> >
> > Andy
> >
> >
> You have outgoing mail being processed by smtpd where it is suppose to 
> be processed by smtp.
> I would tell you how you did this but you didnt include your postconf -n

Thanks for the reply. I have copied the output from postconf -n as well
as master.cf below (a few things removed). I am using dkim-filter to
sign outgoing messages with DKIM - could this be the reason that I am
using smtpd?

postconf -n:

alias_database = hash:/etc/postfix/aliases
alias_maps = hash:/etc/postfix/aliases, regexp:/etc/postfix/aliases-regexp
allow_min_user = yes
alternate_config_directories = /etc/postfix-174
command_time_limit = 5000
config_directory = /etc/postfix
default_privs = simple
header_checks = regexp:/etc/postfix/header_checks
home_mailbox = Maildir/
html_directory = /usr/share/doc/postfix-2.3.3/html
mailq_path = /usr/bin/mailq
manpage_directory = /usr/share/man
message_size_limit = 2048
milter_connect_macros = i b j _ {daemon_name} {if_name} {if_addr}
milter_default_action = accept
milter_mail_macros = {auth_author} {auth_type} {auth_authen}
mydestination = $myhostname, localhost.$mydomain, localhost, mx1.$mydomain, 
ns1.$mydomain, www.$mydomain, root.$mydomain
newaliases_path = /usr/bin/newaliases
non_smtpd_milters = unix:/var/run/dkim-filter/dkim-filter.sock
parent_domain_matches_subdomains = 
queue_directory = /var/spool/postfix
readme_directory = /usr/share/doc/postfix-2.3.3/README
sample_directory = /etc/postfix
sendmail_path = /usr/sbin/sendmail
smtpd_authorized_verp_clients = $mynetworks
smtpd_client_restrictions = permit_sasl_authenticated reject_rbl_client 
zen.spamhaus.org
smtpd_helo_required = yes
smtpd_helo_restrictions = reject_invalid_helo_hostname 
reject_non_fqdn_helo_hostname
smtpd_milters = unix:/var/run/clamav/milter.ctl 
unix:/var/spool/postfix/spamass/spamass.sock 
unix:/var/run/dkim-filter/dkim-filter.sock
smtpd_recipient_restrictions = check_recipient_access hash:/etc/postfix/access, 
reject_unknown_sender_domain, permit_mynetworks, permit_sasl_authenticated, 
reject_unauth_destination,reject_unauth_pipelining
smtpd_restriction_classes = restrict_smtp_ip
smtpd_sasl_auth_enable = yes
smtpd_sasl_local_domain = $myhostname
transport_maps = hash:/etc/postfix/transport
unknown_local_recipient_reject_code = 550
virtual_alias_domains = /etc/postfix/virtual-domains
virtual_alias_maps = hash:/etc/postfix/virtual, 
regexp:/etc/postfix/virtual-regexp, mysql:/etc/postfix/mysql-global-aliases.cf, 
mysql:/etc/postfix/mysql-bounce-aliases.cf, 
mysql:/etc/postfix/mysql-majordomo-aliases.cf, 
mysql:/etc/postfix/mysql-aliases.cf


master.cf:

smtp  inet  n   -   n   -   -   smtpd
submission  inet  n -   n   -   -   smtpd
-o smtpd_etrn_restrictions=reject
-o smtpd_sasl_auth_enable=yes
-o 
smtpd_recipient_restrictions=permit_mynetworks,permit_sasl_authenticated,reject
pickupfifo  n   -   n   60  1   pickup
cleanup   unix  n   -   n   -   0   cleanup
qmgr  fifo  n   -   n   300 1   qmgr
tlsmgrunix  -   -   n   1000?   1   tlsmgr
rewrite   unix  -   -   n   -   -   trivial-rewrite
bounceunix  -   -   n   -   0   bounce
defer unix  -   -   n   -   0   bounce
trace unix  -   -   n   -   0   bounce
verifyunix  -   -   n   -   1   verify
flush unix  n   -   n   1000?   0   flush
proxymap  unix  -   -   n   -   -   proxymap
smtp  unix  -   -   n   -   -   smtp
relay unix  -   -   n   -   -   smtp
-o fallback_relay=
showq unix  n   -   n   -   -   showq
error unix  -   -   n   -   -   error
discard   unix  -   -   n   -   -   discard
local unix  -   n   n   -   -   local
virtual   unix  -   n   n   -   -   virtual
lmtp  unix  -   -   n   -   -   lmt

Re: Mysql queries being performed for domains not in virtual_alias_domains

[Andrew Beverley]

On Sun, 2010-11-21 at 17:13 -0500, Victor Duchovni wrote:
> On Sun, Nov 21, 2010 at 04:50:21PM -0500, Jerrale G wrote:
> 
> > On 11/21/2010 4:40 PM, Andrew Beverley wrote:
> >> Hi,
> >>
> >> I have noticed that my Postfix (version 2.3.8) is performing a
> >> virtual_alias_maps mysql database query for every email that it is
> >> processing, even if the domain is not listed in virtual_alias_domains.
> 
> This is correct behaviour. The rewriting performed by virtual(5) is
> documented and intended to apply to *all* addresses.

Is that all addresses including outbound SMTP? Or can I avoid it when
the final destination is external to the server?

> If you have an SQL or LDAP table that stores data for only a specific
> set of domains, you can use the "domain = ..." attribute of LDAP, or
> SQL tables to specify which domains the table applies to, and queries
> for unrelated domains will be skipped.

Thanks, I did have that originally (pointing
to /etc/postfix/virtual-domains). The problem is that I am trying to
move my virtual_alias_domains list to a SQL query as well (from the
virtual-domains file that it currently is).

> Note, that the above applies also to "bare" domain queries, so tables
> with "domain =" constraints cannot be used to store the list of
> virtual_alias_domains, virtual_mailbox_domains, ... Not a problem,
> since after hardcoding the candidate domains in the table definition,
> there is really no point in using a database at all, just list the
> domains in main.cf.

I would like to move the list of virtual domains to a SQL table, as
opposed to being hard-coded anywhere. Most of my email is outbound, so
if I can avoid database lookups for all outbound mail then that would be
preferable.

Thanks,

Andy

Re: How to replace underscores in hostnames to a valid character?

[J. Roeleveld]

On Tuesday 23 November 2010 16:58:34 mouss wrote:
> Le 22/11/2010 08:55, J. Roeleveld a écrit :
> > [snip]
> > 
> > There is only 1 email address causing issues, eg. am only rewriting that
> > specific one.
> 
> if so, then simply use
> 
> u...@lists_example_orgu...@lists-example.org

Already being done this way.

> >> better reject the address and let the sender fix his ratware...
> > 
> > That is the default behaviour, unfortunately, I am using fetchmail to get
> > the emails out of a mailbox elsewhere and injecting that into postfix.
> 
> with fetchmail, it is a good practice not send bounces (because that
> would cause backscatter).

True, but this is the only one it's trying to send bounce-mails for :)

> Maybe you can tell cyrus to accept malformed sender adresses?

Only option I did find was to manually patch cyrus, which I find not worth the 
effort for just one situation.
Rewriting the address works and is far easier to maintain.

The problem is is that underscores in hostnames are violating the standards 
and should actually be blocked. Because not all applications follow the 
standards properly and other do, situations like this occur.

--
Joost

Re: Data check before accepting a letter

[Неворотин Вадим]

Yes, now I use address extensions for this task. But all users, who are not
very friendly with computers, such as managers, has a lot of problems with
it. They can't understand extensions. There must be only one constant email
for each user without any variable extensions, or you will have a lot of
problems from your employees. Common user can understand some code in
subject (eg [for Dima]), but can't understand variable email addresses. I
have a lot of mail servers in different companies and passphrase in the mail
subject is almost the only good solution for delivery through the
spamfilter. And I can't understand why such simple task creates so many
problems in postfix. May be there is some hack to implement described
behavior? I only need to combine headers and body checks in one condition, I
don't believe that postfix isn't able to do it((

2010/11/23 Wietse Venema 

> > The task: if mail has some passphrase in subject (eg [passme]),
> > then do not apply greylistnig, RBL etc and transfer a mail to LDA.
>
> If you whitelist based on a string in the message, then why not
> whitelist based on a string in the RECIPIENT ADDRESS? Postfix
> supports address extensions, like this:
>
> in main.cf:
>recipient_delimiter = +
># Alternative: recipient_delimiter = -
>
>smtpd_recipient_restrictions =
>permit_mynetworks
>reject_unauth_destination
>check_policy_service x
>reject_rbl_client y
>...
>
> in SMTP:
>RCPT TO:>
>
> Then, the greylist daemon can reply with OK if the passphrase is
> good and skip Postfix RBL checks.  You can make it as fancy as you
> like, such as a passphrase that is valid for only a short time, a
> passphrase that is valid only for with a specific sender IP address,
> and so on.
>
>Wietse
>

Re: Mysql queries being performed for domains not in virtual_alias_domains

[Noel Jones]

On 11/23/2010 12:48 PM, Andrew Beverley wrote:

On Sun, 2010-11-21 at 16:50 -0500, Jerrale G wrote:

On 11/21/2010 4:40 PM, Andrew Beverley wrote:

Hi,

I have noticed that my Postfix (version 2.3.8) is performing a
virtual_alias_maps mysql database query for every email that it is
processing, even if the domain is not listed in virtual_alias_domains.

So for example, I have andybev.com in virtual_alias_domains and a
database query set up for virtual_alias_maps. When I send an email
*from* the server to an external email address, then that recipient (eg
joeblo...@hotmail.com) gets looked up in the virtual_alias_maps database
query first before it is delivered.

Is this normal? This doesn't seem right, so I am thinking that there is
something wrong with my configuration (all of the thousands of outbound
emails are causing database lookups).

Thanks,

Andy



You have outgoing mail being processed by smtpd where it is suppose to
be processed by smtp.
I would tell you how you did this but you didnt include your postconf -n


The above statement is nonsense.  Ignore it.




Thanks for the reply. I have copied the output from postconf -n as well
as master.cf below (a few things removed). I am using dkim-filter to
sign outgoing messages with DKIM - could this be the reason that I am
using smtpd?


There is nothing significant wrong with your config -- postfix 
is working correctly and performing the queries required.


My only suggestion is to be careful what you put in the 
"access" table under smtpd_recipient_restrictions so you don't 
inadvertently become an open relay.

http://www.postfix.org/SMTPD_ACCESS_README.html#danger


  -- Noel Jones




postconf -n:

alias_database = hash:/etc/postfix/aliases
alias_maps = hash:/etc/postfix/aliases, regexp:/etc/postfix/aliases-regexp
allow_min_user = yes
alternate_config_directories = /etc/postfix-174
command_time_limit = 5000
config_directory = /etc/postfix
default_privs = simple
header_checks = regexp:/etc/postfix/header_checks
home_mailbox = Maildir/
html_directory = /usr/share/doc/postfix-2.3.3/html
mailq_path = /usr/bin/mailq
manpage_directory = /usr/share/man
message_size_limit = 2048
milter_connect_macros = i b j _ {daemon_name} {if_name} {if_addr}
milter_default_action = accept
milter_mail_macros = {auth_author} {auth_type} {auth_authen}
mydestination = $myhostname, localhost.$mydomain, localhost, mx1.$mydomain, 
ns1.$mydomain, www.$mydomain, root.$mydomain
newaliases_path = /usr/bin/newaliases
non_smtpd_milters = unix:/var/run/dkim-filter/dkim-filter.sock
parent_domain_matches_subdomains =
queue_directory = /var/spool/postfix
readme_directory = /usr/share/doc/postfix-2.3.3/README
sample_directory = /etc/postfix
sendmail_path = /usr/sbin/sendmail
smtpd_authorized_verp_clients = $mynetworks
smtpd_client_restrictions = permit_sasl_authenticated reject_rbl_client 
zen.spamhaus.org
smtpd_helo_required = yes
smtpd_helo_restrictions = reject_invalid_helo_hostname 
reject_non_fqdn_helo_hostname
smtpd_milters = unix:/var/run/clamav/milter.ctl 
unix:/var/spool/postfix/spamass/spamass.sock 
unix:/var/run/dkim-filter/dkim-filter.sock
smtpd_recipient_restrictions = check_recipient_access hash:/etc/postfix/access, 
reject_unknown_sender_domain, permit_mynetworks, permit_sasl_authenticated, 
reject_unauth_destination,reject_unauth_pipelining
smtpd_restriction_classes = restrict_smtp_ip
smtpd_sasl_auth_enable = yes
smtpd_sasl_local_domain = $myhostname
transport_maps = hash:/etc/postfix/transport
unknown_local_recipient_reject_code = 550
virtual_alias_domains = /etc/postfix/virtual-domains
virtual_alias_maps = hash:/etc/postfix/virtual, 
regexp:/etc/postfix/virtual-regexp, mysql:/etc/postfix/mysql-global-aliases.cf, 
mysql:/etc/postfix/mysql-bounce-aliases.cf, 
mysql:/etc/postfix/mysql-majordomo-aliases.cf, 
mysql:/etc/postfix/mysql-aliases.cf


master.cf:

smtp  inet  n   -   n   -   -   smtpd
submission  inet  n -   n   -   -   smtpd
 -o smtpd_etrn_restrictions=reject
 -o smtpd_sasl_auth_enable=yes
 -o 
smtpd_recipient_restrictions=permit_mynetworks,permit_sasl_authenticated,reject
pickupfifo  n   -   n   60  1   pickup
cleanup   unix  n   -   n   -   0   cleanup
qmgr  fifo  n   -   n   300 1   qmgr
tlsmgrunix  -   -   n   1000?   1   tlsmgr
rewrite   unix  -   -   n   -   -   trivial-rewrite
bounceunix  -   -   n   -   0   bounce
defer unix  -   -   n   -   0   bounce
trace unix  -   -   n   -   0   bounce
verifyunix  -   -   n   -   1   verify
flush unix  n   -   n   1000?   0   flush
proxymap  unix  -   -   n   -   -   proxymap
smtp  unix  -   -   n   -   -   smtp
relay unix  -   -   n   -   -   smtp
-o fallback_re

Re: Mysql queries being performed for domains not in virtual_alias_domains

[Victor Duchovni]

On Tue, Nov 23, 2010 at 06:53:44PM +, Andrew Beverley wrote:

> > Note, that the above applies also to "bare" domain queries, so tables
> > with "domain =" constraints cannot be used to store the list of
> > virtual_alias_domains, virtual_mailbox_domains, ... Not a problem,
> > since after hardcoding the candidate domains in the table definition,
> > there is really no point in using a database at all, just list the
> > domains in main.cf.
> 
> I would like to move the list of virtual domains to a SQL table, as
> opposed to being hard-coded anywhere. Most of my email is outbound, so
> if I can avoid database lookups for all outbound mail then that would be
> preferable.

In that case, you have to tolerate the additional queries. Postfix
never loads the full list of domains into memory, if you store them
in a table, Postfix, at any time, only asks whether a particular
domain is of a given type or not.

The virtual(5) mapping applies to *all* addresses, if you want to
restrict it to a set of domains, to avoid extra queries, then
that list of domains must not require a query! So you either
declare it statically, or build a database beefy enough to
deal with the additional query load.

-- 
Viktor.

rewrite "from address" - one specific destination

[CT]

Postfix Version: postfix-2.5.1-1.rhel5

I have an internal relay with an external NAT.
The internal relay is not visible to the outside..

The receiving destination gives the error:
--
550 [PERMFAIL] xxx.xxx.net requires valid sender domain (in reply to 
RCPT TO command))

--
My question
What is the most efficient way to
"rewrite" the sender address with a valid smtp address
to a "specific" domain.

I only need to do this for 1 destination domain..

Thx
CT

How can I rewrite "from address" - one specific destination

[CT]

I accidentally replied to a thread instead of starting my own..oops..


Postfix Version: postfix-2.5.1-1.rhel5

I have an internal relay with an external NAT.
The internal relay is not visible to the outside..

The receiving destination gives the error:
--
550 [PERMFAIL] xxx.xxx.net requires valid sender domain (in reply to 
RCPT TO command))

--
My question
What is the most efficient way to
"rewrite" the sender address with a valid smtp address
to a "specific" domain.

I only need to do this for 1 destination domain..

Thx
CT

Re: Mysql queries being performed for domains not in virtual_alias_domains

[Andrew Beverley]

On Tue, 2010-11-23 at 13:21 -0600, Noel Jones wrote:
> On 11/23/2010 12:48 PM, Andrew Beverley wrote:
> > On Sun, 2010-11-21 at 16:50 -0500, Jerrale G wrote:
> >> On 11/21/2010 4:40 PM, Andrew Beverley wrote:
> >>> Hi,
> >>>
> >>> I have noticed that my Postfix (version 2.3.8) is performing a
> >>> virtual_alias_maps mysql database query for every email that it is
> >>> processing, even if the domain is not listed in virtual_alias_domains.
> >>>
> >>> So for example, I have andybev.com in virtual_alias_domains and a
> >>> database query set up for virtual_alias_maps. When I send an email
> >>> *from* the server to an external email address, then that recipient (eg
> >>> joeblo...@hotmail.com) gets looked up in the virtual_alias_maps database
> >>> query first before it is delivered.
> >>>
> >>> Is this normal? This doesn't seem right, so I am thinking that there is
> >>> something wrong with my configuration (all of the thousands of outbound
> >>> emails are causing database lookups).
> >>>
> >>> Thanks,
> >>>
> >>> Andy
> >>>
> >>>
> >> You have outgoing mail being processed by smtpd where it is suppose to
> >> be processed by smtp.
> >> I would tell you how you did this but you didnt include your postconf -n
> 
> The above statement is nonsense.  Ignore it.
> 

Thanks for the clarification.

> >
> > Thanks for the reply. I have copied the output from postconf -n as well
> > as master.cf below (a few things removed). I am using dkim-filter to
> > sign outgoing messages with DKIM - could this be the reason that I am
> > using smtpd?
> 
> There is nothing significant wrong with your config -- postfix 
> is working correctly and performing the queries required.
> 
> My only suggestion is to be careful what you put in the 
> "access" table under smtpd_recipient_restrictions so you don't 
> inadvertently become an open relay.
> http://www.postfix.org/SMTPD_ACCESS_README.html#danger
> 

Thanks for pointing that out. The access file is just used to restrict
the sending host for one particular email address, but I will move it to
the end of the smtpd_recipient_restrictions where it is less likely to
become a problem.

Andy

Re: Mysql queries being performed for domains not in virtual_alias_domains

[Andrew Beverley]

On Tue, 2010-11-23 at 14:22 -0500, Victor Duchovni wrote:
> On Tue, Nov 23, 2010 at 06:53:44PM +, Andrew Beverley wrote:
> 
> > > Note, that the above applies also to "bare" domain queries, so tables
> > > with "domain =" constraints cannot be used to store the list of
> > > virtual_alias_domains, virtual_mailbox_domains, ... Not a problem,
> > > since after hardcoding the candidate domains in the table definition,
> > > there is really no point in using a database at all, just list the
> > > domains in main.cf.
> > 
> > I would like to move the list of virtual domains to a SQL table, as
> > opposed to being hard-coded anywhere. Most of my email is outbound, so
> > if I can avoid database lookups for all outbound mail then that would be
> > preferable.
> 
> In that case, you have to tolerate the additional queries. Postfix
> never loads the full list of domains into memory, if you store them
> in a table, Postfix, at any time, only asks whether a particular
> domain is of a given type or not.
> 
> The virtual(5) mapping applies to *all* addresses, if you want to
> restrict it to a set of domains, to avoid extra queries, then
> that list of domains must not require a query! So you either
> declare it statically, or build a database beefy enough to
> deal with the additional query load.
> 

Thanks, I'm happy with that. I just thought that I had better check, as
it looked to me as if I had mis-configured something.

Andy

OT, but mail related

[Gary Smith]

I've been using the same IP's for years but I'm moving all my equipment to a 
new facility in the coming days.  Is there a checklist of things that you guys 
suggest when setting up mail servers on a brand new range?

I'm expecting to setup 5 IP's for outgoing email and as such the RNDS records 
for them as well.  I just want to make sure that there's little, if any, 
problems for myself and my customers that use my services.

Any advice would be greatly appreciated.

Gary Smith

Re: How can I rewrite "from address" - one specific destination

[Noel Jones]

On 11/23/2010 2:16 PM, CT wrote:

I accidentally replied to a thread instead of starting my
own..oops..


Postfix Version: postfix-2.5.1-1.rhel5

I have an internal relay with an external NAT.
The internal relay is not visible to the outside..

The receiving destination gives the error:



There is no easy way to rewrite the sender for just one 
destination.  If their claim is incorrect (the sender domain 
really is valid), you should contact their postmaster.


smtp_generic_maps is used to rewrite "internal" domains to 
valid "internet" domains.

http://www.postfix.org/ADDRESS_REWRITING_README.html#generic

If you need to do this for only one destination, you must
create a custom smtp transport in master.cf with appropriate 
-o smtp_generic_maps=... overrides, then add a transport_maps 
entry to point the recipient domain to the new transport.




  -- Noel Jones

Re: OT, but mail related

[Will Fong]

On Nov 23, 2010, at 12:17 PM, Gary Smith wrote:

> I'm expecting to setup 5 IP's for outgoing email and as such the RNDS records 
> for them as well.  I just want to make sure that there's little, if any, 
> problems for myself and my customers that use my services.


Hi Gary,

You'll probably want to warm up those IPs first. Generally ESPs do this by 
siphoning off  small percentage of emails from their usual IPs to deliver from 
this range, gradually increasing as deliverability improves. SPF may help as 
well here for those who check. And registering the IPs with as many large 
receivers will help (Yahoo in particular).

HTH,
-will

RE: OT, but mail related

[Gary Smith]

> > I'm expecting to setup 5 IP's for outgoing email and as such the RNDS
> records for them as well.  I just want to make sure that there's little,
> if any, problems for myself and my customers that use my services.
> 
> 
> Hi Gary,
> 
> You'll probably want to warm up those IPs first. Generally ESPs do this
> by siphoning off  small percentage of emails from their usual IPs to
> deliver from this range, gradually increasing as deliverability
> improves. SPF may help as well here for those who check. And registering
> the IPs with as many large receivers will help (Yahoo in particular).
> 
> HTH,
> -will


Makes sense.  I will keep a set of MX servers at the original COLO setup to 
accept emails from my new range and forward outgoing email through them until 
we verify the new one.  

Gary Smith

Re: OT, but mail related

[mouss]

Le 23/11/2010 21:17, Gary Smith a écrit :

I've been using the same IP's for years but I'm moving all my equipment to a 
new facility in the coming days.  Is there a checklist of things that you guys 
suggest when setting up mail servers on a brand new range?

I'm expecting to setup 5 IP's for outgoing email and as such the RNDS records 
for them as well.  I just want to make sure that there's little, if any, 
problems for myself and my customers that use my services.

Any advice would be greatly appreciated.

Gary Smith


you can try a lookup of these IPs in multiple RBLs, and lookup the 
"surrounding subnet" (/24 for example) on:

http://www.senderbase.org/

you may want to keep the old servers for some time.
(from experience, I've found MS to "cache" reputation for a long long 
time. That once forced me to keep one server just for hotmail! even 
registering the new IPs at their site, adding SPF, ... didn't help).

Re: Mysql queries being performed for domains not in virtual_alias_domains

[mouss]

Le 23/11/2010 21:19, Andrew Beverley a écrit :

On Tue, 2010-11-23 at 13:21 -0600, Noel Jones wrote:

There is nothing significant wrong with your config -- postfix
is working correctly and performing the queries required.

My only suggestion is to be careful what you put in the
"access" table under smtpd_recipient_restrictions so you don't
inadvertently become an open relay.
http://www.postfix.org/SMTPD_ACCESS_README.html#danger



Thanks for pointing that out. The access file is just used to restrict
the sending host for one particular email address, but I will move it to
the end of the smtpd_recipient_restrictions where it is less likely to
become a problem.



or if the check should apply to all mail (even from mynetworks), you can 
put it under smtpd_sender_restrictions

RE: OT, but mail related

[Gary Smith]

> you can try a lookup of these IPs in multiple RBLs, and lookup the
> "surrounding subnet" (/24 for example) on:
>   http://www.senderbase.org/
> 
> you may want to keep the old servers for some time.
> (from experience, I've found MS to "cache" reputation for a long long
> time. That once forced me to keep one server just for hotmail! even
> registering the new IPs at their site, adding SPF, ... didn't help).

I'd love to keep one around but I will only have 30 days for it.  The other 
problem is I won't get the IP's until tonight.  If they are bad/listed I'll 
have to do something alternative in the short term (like a vm over at slice 
host or something).

Re: OT, but mail related

[Will Fong]

On Nov 23, 2010, at 1:19 PM, Gary Smith wrote:
> 
> Makes sense.  I will keep a set of MX servers at the original COLO setup to 
> accept emails from my new range and forward outgoing email through them until 
> we verify the new one.  
> 
> Gary Smith

For receiving emails, you don't have too much to worry about. Just make sure to 
set your DNS TTL pretty low a few days before you make the actual switch. 

-will

RE: OT, but mail related

[Gary Smith]

> For receiving emails, you don't have too much to worry about. Just make
> sure to set your DNS TTL pretty low a few days before you make the
> actual switch.

Yeah, that's the easy part.  In the process of modifying TTL's as we speak.  
For my clients I have a backup MX that uses the same rules as the primary, so 
it should be transparent.  In fact for some of them they wont even notice the 
outage.  I'm just more worried about sites that I'll have to register the IP's 
at.  I guess I'll start doing that as soon as I get the IP's.  

Gary Smith

Re: OT, but mail related

[mouss]

Le 23/11/2010 22:28, Gary Smith a écrit :

you can try a lookup of these IPs in multiple RBLs, and lookup the
"surrounding subnet" (/24 for example) on:
http://www.senderbase.org/

you may want to keep the old servers for some time.
(from experience, I've found MS to "cache" reputation for a long long
time. That once forced me to keep one server just for hotmail! even
registering the new IPs at their site, adding SPF, ... didn't help).


I'd love to keep one around but I will only have 30 days for it.


then hope you'll get everything fixed by then:)


 The other problem is I won't get the IP's until tonight.  If they are 
bad/listed I'll have to do something alternative in the short term (like a vm 
over at slice host or something).



Once you're "hosted", check if the hoster offers failover IPs with a 
choice of the range. if so, you may select such IPs for outbound mail 
(there is no serious problem with reception, except for MX caching...).

Postfix stable/legacy release 2.7.2, 2.6.8, 2.5.11, and 2.4.15

[Wietse Venema]

An on-line version of this announcement will be available at
http://www.postfix.org/announcements/postfix-2.7.2.html

The stable release Postfix 2.7.2 addresses the defects described
below.  These defects are also addressed in the legacy releases
that are still maintained.

Note: Postfix 2.3 and earlier are no longer updated. Support for
Postfix 2.4 will end in 2011.

Defects fixed with Postfix 2.7.2, 2.6.8, 2.5.11, and 2.4.15:

- Postfix no longer automatically appends the system default CA
  (certificate authority) certificates, when it reads the CA
  certificates specified with {smtp, lmtp, smtpd}_tls_CAfile or
  with {smtp, lmtp, smtpd}_tls_CApath.  This prevents third-party
  certificates from getting mail relay permission with the
  permit_tls_all_clientcerts feature.  Unfortunately, this change
  may cause compatibility problems with configurations that rely
  on certificate verification for other purposes.  To get the old
  behavior, specify "tls_append_default_CA = yes".

- A prior fix for compatibility with Postfix < 2.3 was incomplete.
  When pipe-to-command delivery fails with a signal, mail is now
  correctly deferred, instead of being returned to sender.

- Poor smtpd_proxy_filter TCP performance over loopback (127.0.0.1)
  connections was fixed by adapting the output buffer size to the
  MTU.

- The SMTP server no longer applies the reject_rhsbl_helo feature
  to non-domain forms such as network addresses.  This would cause
  false positives with dbl.spamhaus.org.

- The Postfix SMTP server failed to deliver a "421" response and
  hang up the connection after Milter error.  Instead, the server
  delivered a "503 Access denied" response and left the connection
  open, due to some Postfix 1.1 workaround for RFC 2821.

Defects fixed with Postfix 2.7.2:

- The milter_header_checks parser failed to enable any of the actions
  that have no effect on message delivery (warn, replace, prepend,
  ignore, dunno, and ok).

You can find the updated Postfix versions at the mirrors listed at
http://www.postfix.org/

Wietse

proxymap performance with cidr, pcre, regexp, hash

[Stan Hoeppner]

Victor,

Would you please give us the run down on why these map types (and maybe
others) shouldn't be used with proxymap due to performance reasons?  You
mentioned something about this long ago but I can't seem to locate that
email in my archives.  IIRC you didn't go into much technical detail as
to why the performance would be lower using proxymap.

Thanks.

-- 
Stan

Re: proxymap performance with cidr, pcre, regexp, hash

[Victor Duchovni]

On Tue, Nov 23, 2010 at 05:20:11PM -0600, Stan Hoeppner wrote:

> Would you please give us the run down on why these map types (and maybe
> others) shouldn't be used with proxymap due to performance reasons?  You
> mentioned something about this long ago but I can't seem to locate that
> email in my archives.  IIRC you didn't go into much technical detail as
> to why the performance would be lower using proxymap.

There is no point in using IPC to ask a server to read a file, when the
client process can read it directly. Especially with CDB, since there
is no per-client page pool adding some per-client memory overhead.

-- 
Viktor.

Re: OT, but mail related

[Patrick Ben Koetter]

* Gary Smith :
> I've been using the same IP's for years but I'm moving all my equipment to a
> new facility in the coming days.  Is there a checklist of things that you
> guys suggest when setting up mail servers on a brand new range?

Get a reputation for your domain i.e. DKIM et al. If you don't have on yet,
you're probably too late to establish one. If you have one IPs are not THAT
important anymore.


> I'm expecting to setup 5 IP's for outgoing email and as such the RNDS
> records for them as well.  I just want to make sure that there's little, if
> any, problems for myself and my customers that use my services.
> 
> Any advice would be greatly appreciated.

We've had very bad experiences with reputation for "cloud IPs". IF you move
to a cloud, make sure senderbase.org, senderscore.org and such alike are happy
with your IP's.

The solution was to establish a DKIM identity and stick with RFCs close as
possible.

p...@rick

-- 
All technical questions asked privately will be automatically answered on the
list and archived for public access unless privacy is explicitely required and
justified.

saslfinger (debugging SMTP AUTH):

Re: proxymap performance with cidr, pcre, regexp, hash

[Wietse Venema]

Victor Duchovni:
> On Tue, Nov 23, 2010 at 05:20:11PM -0600, Stan Hoeppner wrote:
> 
> > Would you please give us the run down on why these map types (and maybe
> > others) shouldn't be used with proxymap due to performance reasons?  You
> > mentioned something about this long ago but I can't seem to locate that
> > email in my archives.  IIRC you didn't go into much technical detail as
> > to why the performance would be lower using proxymap.
> 
> There is no point in using IPC to ask a server to read a file, when the
> client process can read it directly. Especially with CDB, since there
> is no per-client page pool adding some per-client memory overhead.

With Stan's huge CIDR maps on a small machine, proxymap helps
to avoid running out of memory.

Wietse

Re: proxymap performance with cidr, pcre, regexp, hash

[Victor Duchovni]

On Tue, Nov 23, 2010 at 07:35:49PM -0500, Wietse Venema wrote:

> Victor Duchovni:
> > On Tue, Nov 23, 2010 at 05:20:11PM -0600, Stan Hoeppner wrote:
> > 
> > > Would you please give us the run down on why these map types (and maybe
> > > others) shouldn't be used with proxymap due to performance reasons?  You
> > > mentioned something about this long ago but I can't seem to locate that
> > > email in my archives.  IIRC you didn't go into much technical detail as
> > > to why the performance would be lower using proxymap.
> > 
> > There is no point in using IPC to ask a server to read a file, when the
> > client process can read it directly. Especially with CDB, since there
> > is no per-client page pool adding some per-client memory overhead.
> 
> With Stan's huge CIDR maps on a small machine, proxymap helps
> to avoid running out of memory.

CIDR maps are indeed a different kettle of fish, they are not
"read-a-file via IPC", rather they are "do a memory lookup via IPC", and
if the memory footprint is sufficiently high (lots Berkeley DB tables or
huge CIDR tables), then indeed one may want to use proxymap.

With "cdb" (the recommended read-only indexed table type for Postfix) direct
access is best.

-- 
Viktor.

Re: proxymap performance with cidr, pcre, regexp, hash

[Wietse Venema]

Victor Duchovni:
> On Tue, Nov 23, 2010 at 07:35:49PM -0500, Wietse Venema wrote:
> 
> > Victor Duchovni:
> > > On Tue, Nov 23, 2010 at 05:20:11PM -0600, Stan Hoeppner wrote:
> > > 
> > > > Would you please give us the run down on why these map types (and maybe
> > > > others) shouldn't be used with proxymap due to performance reasons?  You
> > > > mentioned something about this long ago but I can't seem to locate that
> > > > email in my archives.  IIRC you didn't go into much technical detail as
> > > > to why the performance would be lower using proxymap.
> > > 
> > > There is no point in using IPC to ask a server to read a file, when the
> > > client process can read it directly. Especially with CDB, since there
> > > is no per-client page pool adding some per-client memory overhead.
> > 
> > With Stan's huge CIDR maps on a small machine, proxymap helps
> > to avoid running out of memory.
> 
> CIDR maps are indeed a different kettle of fish, they are not
> "read-a-file via IPC", rather they are "do a memory lookup via IPC", and
> if the memory footprint is sufficiently high (lots Berkeley DB tables or
> huge CIDR tables), then indeed one may want to use proxymap.
> 
> With "cdb" (the recommended read-only indexed table type for Postfix) direct
> access is best.

proxymap is OK for moderate traffic to expensive or otherwise
inaccessible resources.  Opening a CDB map is not expensive, an
SQL database handle or memory-based map (regexp etc.) can be
expensive, and /etc/passwd can be inaccessible due to lazy opens.

proxymap is not the best solution if you need minimal latency at
all cost.

Wietse

Re: proxymap performance with cidr, pcre, regexp, hash

[Stan Hoeppner]

Wietse Venema put forth on 11/23/2010 6:57 PM:
> Victor Duchovni:
>> On Tue, Nov 23, 2010 at 07:35:49PM -0500, Wietse Venema wrote:
>>
>>> Victor Duchovni:
 On Tue, Nov 23, 2010 at 05:20:11PM -0600, Stan Hoeppner wrote:

> Would you please give us the run down on why these map types (and maybe
> others) shouldn't be used with proxymap due to performance reasons?  You
> mentioned something about this long ago but I can't seem to locate that
> email in my archives.  IIRC you didn't go into much technical detail as
> to why the performance would be lower using proxymap.

 There is no point in using IPC to ask a server to read a file, when the
 client process can read it directly. Especially with CDB, since there
 is no per-client page pool adding some per-client memory overhead.
>>>
>>> With Stan's huge CIDR maps on a small machine, proxymap helps
>>> to avoid running out of memory.
>>
>> CIDR maps are indeed a different kettle of fish, they are not
>> "read-a-file via IPC", rather they are "do a memory lookup via IPC", and
>> if the memory footprint is sufficiently high (lots Berkeley DB tables or
>> huge CIDR tables), then indeed one may want to use proxymap.
>>
>> With "cdb" (the recommended read-only indexed table type for Postfix) direct
>> access is best.
> 
> proxymap is OK for moderate traffic to expensive or otherwise
> inaccessible resources.  Opening a CDB map is not expensive, an
> SQL database handle or memory-based map (regexp etc.) can be
> expensive, and /etc/passwd can be inaccessible due to lazy opens.
> 
> proxymap is not the best solution if you need minimal latency at
> all cost.
> 
>   Wietse

Thank you gentlemen.  I relatively recently added the dnswl.org CIDR
whitelist table, which contains over 97K+ entries.  I use a perl script
to consolidate the CIDRs, cutting it down to around 67K entries.  Timing
postmap -q it takes on average .5 seconds to complete a lookup in the
consolidated file regardless of where the entry lies within the file.
Timing queries on the original file shows .75 seconds per lookup.  This
is after caching.  Testing cold is kind of irrelevant.  I'm guessing the
latency is actually higher when smtpd queries proxymap than when timing
postmap -q.  Is this the case?

An extra ~.5-1.0 seconds isn't a big deal regarding this MX.  However,
each new table or dns lookup I add to my anti-spam checks adds up, and I
don't want to reach a point where I'm taking 10 seconds per connection
to determine if I'm going to reject it.  So I try to continually
optimize, especially after adding something like the dnswl.org CIDR map.

-- 
Stan

Re: proxymap performance with cidr, pcre, regexp, hash

[Victor Duchovni]

On Tue, Nov 23, 2010 at 10:04:47PM -0600, Stan Hoeppner wrote:

> I'm guessing the
> latency is actually higher when smtpd queries proxymap than when timing
> postmap -q.  Is this the case?

Not necessarily, it depends on how memory-constrained your system is.
Paging in a large table in a large number of parallel smtpd processes
may be more expensive than querying a consolidated table via IPC.
The IPC overhead is fractions of a millisecond.

I would not use such a table, adding 0.5s of CPU for each client IP
lookup is to dear unless your traffic loads are *very* light. Surely
there's a better way! Have you considered rbldnd? The SpamHaus PBL
zone is a rather large CIDR table (in which all the reject messages
are one of two fixed templates in which the lookup key is embedded)
and it performs very nicely, since rbldnsd is able to query this
efficiently. You should consider an RBLDNS service instead of a large
CIDR table. Or find more effective data sources.

-- 
Viktor.

Re: OT, but mail related

[Stan Hoeppner]

Gary Smith put forth on 11/23/2010 2:17 PM:
> I've been using the same IP's for years but I'm moving all my equipment to a 
> new facility in the coming days.  Is there a checklist of things that you 
> guys suggest when setting up mail servers on a brand new range?
> 
> I'm expecting to setup 5 IP's for outgoing email and as such the RNDS records 
> for them as well.  I just want to make sure that there's little, if any, 
> problems for myself and my customers that use my services.
> 
> Any advice would be greatly appreciated.

>From a post I made to another list some time ago:

"My general advice WRT to VPS/colo/hosting outfits such as Softlayer,
Limestone, Sharktech, Hostnoc, Colocation America, Colo4, SingleHop,
Liquid Web, ServePath, GigeNet, WholeSale Internet, FDCservers,
CarolinaNet, Hurricane Electric, et al is to SMTP block their entire IP
space and then whitelist sender addresses if/when necessary."

That is by no means of a complete list of outfits that are persona non
grata at many sites.  Those listed above, and many others, have a
reputation for hosting snowshoe spammers, either willfully or by
ignorance or negligence.

What outfit are you moving to?  What are the IP addresses they have
assigned you?  Depending on your response to either of these questions I
may be able to tell you instantly whether you should pick another outfit
to host your gear.

For anyone planning to stick a rack of mail sending hosts into a colo
facility, you _must_ research the reputation of said facility long
before signing a contract and turning the key on the forklift.  This is
usually called "due diligence".

-- 
Stan

RE: OT, but mail related

[Gary Smith]

> "My general advice WRT to VPS/colo/hosting outfits such as Softlayer,
> Limestone, Sharktech, Hostnoc, Colocation America, Colo4, SingleHop,
> Liquid Web, ServePath, GigeNet, WholeSale Internet, FDCservers,
> CarolinaNet, Hurricane Electric, et al is to SMTP block their entire IP
> space and then whitelist sender addresses if/when necessary."

That pretty much lists everything available to me... The colo is Hurricane 
Electric in Fremont, but that's only because a) budget and b) locality.  I have 
a colo that I'm in right now that unfortunately isn't capable of providing "any 
service level".  So my options are limited.  The biggest problem is that this 
line of business really isn't a money make for us since large companies can 
simply undercut any price we set (or primary job is software architecture) and 
that we still have a need to support email for our custom clients.

I understand the blocking issue of IP's, which is why I think ideas such as 
SORBS has merit (though I was burned when the reported my current ISP as a 
dynamic block after being static for year -- some time ago).  Which is why I 
kind of started this thread.  I know that I have to register my primary MX out 
services with several of the larger providers, SORBS (probably through HE), 
setup RDNS, and whatever else.  It's that whatever else that I'm just making 
sure that I have done.

I was supposed to receive the IP's today but haven't yet (as we had to resend 
the wonderful justification again).  As for VPS services, I've haven't really 
had any problem with mine in terms of services.  I check pretty frequently to 
see if my VPS is on any blacklist and it hasn't popped up at this point.

Anyway, if you happen to know a cheap colo in that area, I'm willing to change 
already, but thus far I haven't found anything else in budget.

Re: proxymap performance with cidr, pcre, regexp, hash

[Stan Hoeppner]

Victor Duchovni put forth on 11/23/2010 11:05 PM:
> On Tue, Nov 23, 2010 at 10:04:47PM -0600, Stan Hoeppner wrote:
> 
>> I'm guessing the
>> latency is actually higher when smtpd queries proxymap than when timing
>> postmap -q.  Is this the case?
> 
> Not necessarily, it depends on how memory-constrained your system is.
> Paging in a large table in a large number of parallel smtpd processes
> may be more expensive than querying a consolidated table via IPC.
> The IPC overhead is fractions of a millisecond.
> 
> I would not use such a table, adding 0.5s of CPU for each client IP
> lookup is to dear unless your traffic loads are *very* light. Surely
> there's a better way! Have you considered rbldnd? The SpamHaus PBL
> zone is a rather large CIDR table (in which all the reject messages
> are one of two fixed templates in which the lookup key is embedded)
> and it performs very nicely, since rbldnsd is able to query this
> efficiently. You should consider an RBLDNS service instead of a large
> CIDR table. Or find more effective data sources.

I disabled this table lookup earlier during our discussions while I was
doing some other optimizations.  In reality it's not too great an impact
on my MX as the load is so low, but I just don't like that half second
hit per transaction.

This half second lookup delay is more a function of my hardware and the
precompiled Debian binaries than the size of the CIDR table.  The host
in question is an old dual 500 MHz 686 class machine.  On even
relatively modern hardware such as a 2 GHz Opty running 64bit binaries
these lookups would probably take less than 5 ms, instead of 500 ms.

I implemented dnswl.org's CIDR table strictly to prevent some of my
draconian A/S measures from blocking friendly mail, which has happened
on more than a few occasions.

Instead of using rbldnsd, I'm going to simply implement
permit_dnswl_client once I move to 2.8.  Timing digs to list.dnswl.org
shows an average of about 100-200 ms latency, which although not great,
is still 2.5 to 5 times quicker than my optimized version of their CIDR
table.

Speaking of upgrading to 2.8, I've always used the Debian Postfix
packages.  I'm guessing the next Debian release (Squeeze ETA Jan/Feb)
will contain 2.6.5 or 2.7 but not 2.8.  Can someone point me to a good
set of instructions on compiling and installing from source, while
keeping my current config files and updating them, and properly removing
or bypassing the currently installed Debian 2.5.5 in a way that doesn't
screw up the packaging system?

Also, can any of the Postfix code paths benefit from integer MMX
instructions if I can get GCC to use them?  I would think SIMD
instructions might be helpful with CIDR tables.  If so, it will be
interesting to compare before/after latencies.

Thanks for the tips Viktor, Wietse.

-- 
Stan