outgoing mail IP address?

[Ákos Maróy]

Hi,

I wonder how postfix decides on what IP address to bind to / initiation
IP connections from, in terms of sending outgoing mail?

I have a server with 5 IP addresses, where the IP address I'm using for
receiving mail is the main IP address for the machine. I've set the
following in main.cf to enforce this:

inet_interfaces = x.x.x.170,127.0.0.1


the range of IP addresses the machine is configured to ranges from .170
to .174.

interestingly, when my postfix sends mail, it initiates connections from
IP address .174, instead of .170. this is despite the fact the postfix
doesn't listed on anything but .170.

I wonder why this is happenning, and how one can force postfix to
initiate connections from an explicitly specified IP address.

thanks,


Akos

Re: outgoing mail IP address?

[Bas Mevissen]

On Tue, 2011-03-08 at 12:36 +0100, Ákos Maróy wrote: 
> Hi,
> 
> I wonder how postfix decides on what IP address to bind to / initiation
> IP connections from, in terms of sending outgoing mail?
> 
> I have a server with 5 IP addresses, where the IP address I'm using for
> receiving mail is the main IP address for the machine. I've set the
> following in main.cf to enforce this:
> 
> inet_interfaces = x.x.x.170,127.0.0.1
> 
> 
> the range of IP addresses the machine is configured to ranges from .170
> to .174.
> 
> interestingly, when my postfix sends mail, it initiates connections from
> IP address .174, instead of .170. this is despite the fact the postfix
> doesn't listed on anything but .170.
> 
> I wonder why this is happenning, and how one can force postfix to
> initiate connections from an explicitly specified IP address.
> 

If I'm not mistaken, the setting of inet_interfaces is for the
_listening_ port only and not for the outgoing one.

It depends on the the routing table which interface is used for outgoing
mail.

Regards,

-- 
Bas.

Re: outgoing mail IP address?

[Ákos Maróy]

On 08/03/11 12:57, Bas Mevissen wrote:
> If I'm not mistaken, the setting of inet_interfaces is for the
> _listening_ port only and not for the outgoing one.

I see.. is there no way to force the outgoing IP address?

> It depends on the the routing table which interface is used for outgoing
> mail.

interesting. the routing table allows for all configured IP addresses to
go through the default router - which is in fact the intended configuration.

how does postfix decide then on which interface to use, if they all seem
to be equal?

in my case, it seems to be the 'last' (or highest number) IP address.

Re: outgoing mail IP address?

[Patrick Ben Koetter]

* Bas Mevissen :
> On Tue, 2011-03-08 at 12:36 +0100, Ákos Maróy wrote: 
> > Hi,
> > 
> > I wonder how postfix decides on what IP address to bind to / initiation
> > IP connections from, in terms of sending outgoing mail?
> > 
> > I have a server with 5 IP addresses, where the IP address I'm using for
> > receiving mail is the main IP address for the machine. I've set the
> > following in main.cf to enforce this:
> > 
> > inet_interfaces = x.x.x.170,127.0.0.1
> > 
> > 
> > the range of IP addresses the machine is configured to ranges from .170
> > to .174.
> > 
> > interestingly, when my postfix sends mail, it initiates connections from
> > IP address .174, instead of .170. this is despite the fact the postfix
> > doesn't listed on anything but .170.
> > 
> > I wonder why this is happenning, and how one can force postfix to
> > initiate connections from an explicitly specified IP address.

$ man 5 postconf | less +/^smtp_bind_address

p@rick

-- 
All technical questions asked privately will be automatically answered on the
list and archived for public access unless privacy is explicitely required and
justified.

saslfinger (debugging SMTP AUTH):

Re: outgoing mail IP address?

[Ákos Maróy]

On 08/03/11 13:05, Patrick Ben Koetter wrote:
> $ man 5 postconf | less +/^smtp_bind_address

thanks, this does the trick!

Server-to-server TLS

[Raven]

Hi guys.
I would like to implement server-to-server TLS encryption between two
postfix instances I manage. One of the servers already has
TLS-capabilities but they are only used for SASL-AUTH clients.

Where do I start to have the entire stream between the servers to be
encrypted?

Thanks.
-RV

Re: Server-to-server TLS

[Victor Duchovni]

On Tue, Mar 08, 2011 at 01:38:28PM +0100, Raven wrote:

> I would like to implement server-to-server TLS encryption between two
> postfix instances I manage. One of the servers already has
> TLS-capabilities but they are only used for SASL-AUTH clients.
> 
> Where do I start to have the entire stream between the servers to be
> encrypted?

http://www.postfix.org/TLS_README.html#client_tls
http://www.postfix.org/TLS_README.html#client_tls_limits
http://www.postfix.org/TLS_README.html#client_tls_levels
http://www.postfix.org/TLS_README.html#client_tls_encrypt
http://www.postfix.org/TLS_README.html#client_tls_secure
http://www.postfix.org/TLS_README.html#client_tls_policy

main.cf:
indexed = ${default_database_type}:${config_directory}/
dynamic = btree:${data_directory}/
transport_maps = ${indexed}transport
smtp_tls_policy_maps = ${indexed}tls-policy
# Opportunistic TLS by default
smtp_tls_security_level = may
smtp_tls_session_cache_database = ${dynamic}smtp_tls_scache

transport:
example.com smtp:[mail.example.com]
example.org smtp:example.net
example.gov smtp:example.net

tls-policy:
# transport nexthop gateway for example.com mail
[mail.example.com]  secure match=nexthop

# transport nexthop domain for example.org and example.gov
example.net secure

# Domain routed via MX hosts to servers believed to support TLS
# with verifiable certificates
example.edu secure

-- 
Viktor.

Postfix x TMDA x transport

[Márcio Luciano Donada]

Hi list
I understand perfectly that postfix has nothing to do with tmda, but I'm
so use it to transport, as follows, but so the tmda returns code 99 [2].
Well, what I am now is much the same set up as a transport:

In the master.cf file:

# TMDA
tmdaunix-   n   n   -   -   spawn
  user=nobody argv=/bin/bash   
/usr/local/tmda/contrib/tmda-postfix-policy.sh

In the main.cf
mailbox_transport_maps  =
hash:/etc/postfix/maps/transport-tmda

smtpd_recipient_restrictions = ...
  check_policy_service
unix:private/tmda,


# cat /etc/postfix/maps/transport-tmda
usertmda

But, message not delivery to maildir. Someone uses the tmda can give
some idea to work correctly with this software?

[1]. http://tmda.sourceforge.net/cgi-bin/moin.cgi/ServerConfiguration
[2]. Mar  7 17:17:48 mx postfix/local[6202]: 3F01F8480A1:
to=, relay=local, delay=0.07, delays=0.01/0/0/0.06,
dsn=5.3.0, status=bounced (Command died with status 99:
"/usr/local/tmda/bin/tmda-filter")
 

-- 
Márcio Luciano Donada
Aurora Alimentos - T.I. Matriz
Coop. Central Oeste Catarinense

Re: Postfix x TMDA x transport

[Victor Duchovni]

On Tue, Mar 08, 2011 at 10:38:09AM -0300, M?rcio Luciano Donada wrote:

> I understand perfectly that postfix has nothing to do with tmda, but I'm
> so use it to transport, as follows, but so the tmda returns code 99 [2].
> Well, what I am now is much the same set up as a transport:

You can't use a policy service as a transport. They are completely
different beasts. If you try to use your sneakers as a hat, you'll
be disappointed.


> # TMDA
> tmdaunix-   n   n   -   -   spawn
>   user=nobody argv=/bin/bash /usr/local/tmda/contrib/tmda-postfix-policy.sh

This is a policy service script.

> In the main.cf
> mailbox_transport_maps = hash:/etc/postfix/maps/transport-tmda
> 
> smtpd_recipient_restrictions = ...
>   check_policy_service unix:private/tmda,

This is a policy service call, which is likely fine.

> # cat /etc/postfix/maps/transport-tmda
> usertmda

A pair of sneakers is not a hat, you can't use a policy service as a
transport.

> [1]. http://tmda.sourceforge.net/cgi-bin/moin.cgi/ServerConfiguration
> [2]. Mar  7 17:17:48 mx postfix/local[6202]: 3F01F8480A1:
> to=, relay=local, delay=0.07, delays=0.01/0/0/0.06,
> dsn=5.3.0, status=bounced (Command died with status 99:
> "/usr/local/tmda/bin/tmda-filter")

The exit status of "99" is I believe designed with qmail in mind. Also
this message did not get routed to the transport you configured, it got
handled by "local(8)", the command "tmda-filter" is not what you
show above. You appear to be rather confused.

-- 
Viktor.

Re: STARTTLS bug - background story

[Stan Hoeppner]

Wietse Venema put forth on 3/7/2011 2:08 PM:
> CERT/CC announces a flaw today in multiple STARTTLS implementations.
> This problem was silently fixed in Postfix 2.8 and 2.9. Updates
> for Postfix 2.[4-7] are made available via the usual channels.

Nice catch Wietse!  Normally I'd follow that with "nice save", but,
unfortunately...

It is so darn easy for miscreants to phish account credentials from
Joe/Jane sixpack that I doubt [m]any would even bother trying to exploit
this STARTTLS flaw--much lower ROI than phishing.

Code up a fix for the Joe/Jane sixpack flaw and the world's citizens
will commission a bronze statue in your honor.

-- 
Stan

Re: Postfix x TMDA x transport

[Márcio Luciano Donada]

Em 8/3/2011 10:54, Victor Duchovni escreveu:
> The exit status of "99" is I believe designed with qmail in mind. Also
> this message did not get routed to the transport you configured, it got
> handled by "local(8)", the command "tmda-filter" is not what you
> show above. You appear to be rather confused.

Note that in [1], is the setting that takes a tmda documentation, and
[2] is the error that returns me to setup, but I believe that
transportation is the most interesting for that. I'm not no little confused.

[1]. http://tmda.sourceforge.net/cgi-bin/moin.cgi/ServerConfiguration
[2]. Mar  7 17:17:48 mx postfix/local[6202]: 3F01F8480A1:
to=, relay=local, delay=0.07, delays=0.01/0/0/0.06,
dsn=5.3.0, status=bounced (Command died with status 99:
"/usr/local/tmda/bin/tmda-filter")



-- 
Márcio Luciano Donada
Aurora Alimentos - T.I. Matriz
Coop. Central Oeste Catarinense

Re: Thank you for great software

[jeffrey j donovan]

On Mar 7, 2011, at 5:47 PM, Reindl Harald wrote:

> Hi Wietse
> 
> I would like to say THANK YOU for postfix because i know
> developers are hearing this words not often enough as long
> their "baby" is running well and nobody takes notice about
> 
> Especially for the great documentation and crazy online-times
> on the mailing-list :-)

I would like to throw in my thanks to you and Victor. The documentation is 
amazing. 
your ascii art flow charts are like works of art.  " It's all about the 
plumbing " -w, sounds like the title of a good book.
-j

Re: Pcre header checks

[Stan Hoeppner]

Erik de Castro Lopo put forth on 3/7/2011 4:50 PM:

> Given the above data, I think I am justified in using the following 
> pcre rule:
> 
> /^Received-SPF:.*helo=[a-z0-9.-]+\.mail\\.yahoo\.com; 
> envelope-from=[^@]+@att.net/i REJECT

This is possibly less complex, totally effective, and provides an alternate 
path for any legit email that's rejected.

/etc/postfix/blacklist
..
yahoo.com   REJECT Please resend to postmas...@mydomain.tld if your email 
is not spam.
..

smtpd_recipient_restrictions =
...
check_reverse_client_hostname_access hash:/etc/postfix/blacklist
check_sender_access hash:/etc/postfix/blacklist
...

In something like 3 years since implementing this, I've not received a single 
Yahoo email in my postmaster mailbox.  I of course manually whitelisted some 
friends/family addresses, and my auto whitelist script handles yahoo addresses 
for folks I send mail to.

-- 
Stan

Re: STARTTLS bug - background story

[Wietse Venema]

Victor Duchovni:
> On Tue, Mar 08, 2011 at 12:59:15PM +1100, Brad Hards wrote:
> 
> > On Tue, 8 Mar 2011 07:08:09 am Wietse Venema wrote:
> > > This is a writeup about a flaw that I found recently, and that
> > > existed in multiple implementations of SMTP (Simple Mail Transfer
> > > Protocol) over TLS (Transport Layer Security) including my Postfix
> > > open source mailserver. I give an overview of the problem and its
> > > impact, technical background, how to find out if a server is affected,
> > > fixes, and draw lessons about where we can expect similar problems
> > > now or in the future. A time line is at the end.
> >
> > Thanks for the write-up. 
> 
> It is a bit disappointing that very few of the potentially impacted
> vendors, and some definitely impacted vendors are yet to respond to
> the vulnerability:
> 
> http://www.kb.cert.org/vuls/id/555316
> 
> Some email appliance vendors are not on the list. Apart from Postfix,
> Qmail, and some large mailbox hosting providers, which are already
> fixed, the issue will likely linger in less visible products for
> some time...

It's easy enough to make the one-line change to openssl source, so
that people can check for this now if they are concerned.

I would expect that penetration test toolkits will eventually look
for starttls plaintext injection vulnerabilities.  But that may
take a while.

Publishing "shame" lists on the web is better done by people who
work for organizations with no commercial interest in the issue.

Wietse

Re: smtpd_sasl_path tcp-socket?

[Wietse Venema]

Hajo Locke:
[ Charset ISO-8859-1 unsupported, converting... ]
> Hello,
> 
> >> smtpd_sasl_path = inet:localhost:1434
> >>
> >> seems to work but it is not documented.
> >
> > As documented:
> >
> > smtpd_sasl_path (default: smtpd)
> >   Implementation-specific information that the Postfix SMTP server 
> > passes
> >   through to the  SASL  plug-in  implementation  that  is  selected 
> > with
> >   smtpd_sasl_type.
> >
> > So, you are asking your question in the wrong place.
> 
> hmm, Timo Sirainen did sent me back.  ;)
> http://dovecot.org/list/dovecot/2011-March/057773.html
> src/xsasl/xsasl_dovecot_server.c was originally written by him but by now 
> heavily modified.
> 
> "when SASL type is "dovecot", SASL path "inet:localhost:1434" should be a 
> socket pathname"
> this warning is no longer required. may be an advice in readme that this 
> auth-traffic is unsecured...

RTFM, please.

The Postfix SASL_README file says:

Communication between the Postfix SMTP server and Dovecot
SASL happens via a UNIX-domain socket.

Support for inet: is NOT DOCUMENTED. It may disappear any time.
There is no promise nowhere that this actually works.

You use NOT DOCUMENTED settings at your own risk. I have no plans
to I write warnings for settings that are NOT DOCUMENTED.

Wietse

Change in bounce_template_file, postfix reload needed?

[Ralf Hildebrandt]

If I change the bounce_template_file, is a "postfix reload" for the
change to take (immediate) effect needed?

Looking at http://www.postfix.org/bounce.8.html I'm seeing

  max_idle (100s)
  max_use (100)

which are implying that after 100 bounces or 100s of no usage, Postfix
will let bounce die and be reborn.

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebra...@charite.de | http://www.charite.de

Re: regular expressions was: Kernel Oops

[Stan Hoeppner]

mouss put forth on 3/7/2011 5:45 PM:
> Le 07/03/2011 15:13, Stan Hoeppner a écrit :

>> Ok, so if I'm doing what I've heard called a "fully qualified regular
>> expression", WRT FQrDNS matching, should I use the anchors or not?
>> postmap -q says these all work (the actuals with action and text that is).

>> /^(\d{1,3}-){3}\d{1,3}\.dynamic\.chello\.sk$/
> .dynamic.chello.skREJECT blah blah
> 
>> /^(\d{1,3}\.){4}dsl\.dyn\.forthnet\.gr$/
> .dyn.forthnet.gr  REJECT blah blah
> 
>> /^(\d{1,3}-){4}adsl-dyn\.4u\.com\.gh$/
> /dyn\.4u.com\.gh$/REJECT blah
> assuming you get real mail from there. otherwise
> .4u.com.ghREJECT blah

Yes, these can all be done with a hash/cdb.  But these are being added
to my fqrdns.pcre file.  As the name implies the goal is to exactly
match fully qualified reverse DNS strings, at least, that's part of the
goal.  The other part is the exact opposite:  _not_ matching them.  I'll
explain that a little later.

>> /^[\d\w]{8}\.[\w]{2}-[\d]-[\d\w]{2}\.dynamic\.ziggo\.nl$/
> ahem? I fail to see what yoy're trying to match here. \d is a \w, so
> [\d\w] is the same as \w. do you mean \W (capital letter)? anyway:

I tried \d alone in those places and postmap -q wouldn't match it.  I
scoured my regex cheat sheet and it said \d is for digits, and \w is for
alphas.  I added \d\w and it worked.  I was trying to match this oddball
FQrDNS:

541ABE2E.cm-5-3c.dynamic.ziggo.nl

> well, that's what regular expressions are about by default:
> /foo/ means contains foo
> /^foo/ means starts with foo
> /foo$/ means ends with foo

Got it.  You (or Noel) already explained this, and it really helps
understanding.

> so
> /^bart.*homer.*marge$/ means: starts with "bart", ends with "marge" and
> somewhere between these contains "homer".

Also good to understand.


Ok, to explain the "not matching" goal.  The PCRE file is almost 1700
expressions, and growing.  In a couple of years it could be double that
size.  Over a longer period of time it could hit 5000 expressions.  For
users of this file, it is usually the first table checked against a
connecting smtp client.  That client rDNS will match 1 of 1700
expressions, or none.  Thus, we want the fastest processing of the "does
not match case, as this is the common case.

A match is "rare" from a mathematical and cycles consumed standpoint.
Modern processors are extremely fast.  But if our expressions aren't
speed optimized for the "does not match" case, we're slowing our system
down.  For most systems this is irrelevant.  But for an extremely high
volume MX gateway system, receiving say, 3000 connects/second,
consisting of 2700 spam bots and showshoe servers, with 300 legit mails
to be relayed to downstream mailbox servers, a few extra milliseconds of
table processing time per connection adds up quickly.  Assuming this
host is running the full gamut of anti spam checks, policy daemons,
content filters, etc, we need to keep each as lean as possible.

If this example MX gateway sees spikes of 5000 connections/second due to
a large botnet targeting multiple users, any extra delay this PCRE table
imposes may contribute to bogging the system down, and cause unwanted
delays.

So, the question is, which form of expression processes the "does not
match" case faster?  The fully qualified expression, or the simple
expression?  Noel mentioned that the fully qualified expressions will
tend to process faster.  Is this true?  Is it true for both the
"matches" and "does not match" case?

Thanks again for continuing my regex education guys. :)  This knew
knowledge and understanding is already paying dividends, mostly in time
savings and I'm knocking expressions out more easily without having to
reference help docs. :)

-- 
Stan

Re: regular expressions was: Kernel Oops

[Wietse Venema]

Stan Hoeppner:
> So, the question is, which form of expression processes the "does not
> match" case faster?  The fully qualified expression, or the simple
> expression?  Noel mentioned that the fully qualified expressions will
> tend to process faster.  Is this true?  Is it true for both the
> "matches" and "does not match" case?

I would expect better performance when patterns only match the text
that needs to be matched.

If you must match a very large numbers of patterns, you need an
implementation that transforms N patterns into one deterministic
automaton. This can match 1 pattern in the same time as N patterns.
Once the automaton is built (which takes some time) it is blindingly
fast. An example of such an implementation is flex.

Similar optimizations are needed for large CIDR maps. Right now,
Postfix's linear search does 10^8 patterns/s. With this, postscreen
can search the largest ipdeny.com file in 1ms on a modern CPU,
which is sufficient for the moment. To make it fast, the CIDR
entries need to be arranged into a tree that can be traversed in
log(N) time.

Wietse

Postix Newbie: Send all outbound mail to another postfix server

[Steve Angelo]

Hi all.. Please forgive me as I'm a postfix newbie.. I'm currently running
version 2.6.5 on an Ubuntu server. I have a codeignitor app that sends
outbound mail only. no inbound mail is processed/advertised.

I recently moved my production server from Rackspace to AWS and Amazon is
now having a PBL issue that may take days to resolve and my IP is listed so
most of the mail is not going out due to PBL from Spamhaus.

So I was thinking.. Since my old server at rackspace is still up and
running, would it be possible to have configure my AWS postfix server to
send all outbound mail to the Rackspace postfix server and have that server
send out the email? The originating IP would have to be the Rackspace server
though, as if the mail still shows the originating as the AWS server it will
fail due to PBL.

Is this possible and if so, any pointers/suggestions on how to go about
doing this? My other option is to move back to Rackspace which will be very
disruptive.

Thanks!

Re: Postfix 2.7.3, 2.6.9, 2.5.12 and 2.4.16 available

[fakessh @]

when is it the official rpm. 
Simon Mudd
the new rpms are still not present



Le lundi 07 mars 2011 à 15:18 -0500, Wietse Venema a écrit :
> [An on-line version of this announcement will be available at
> http://www.postfix.org/announcements/postfix-2.7.3.html]
> 
> Postfix legacy releases 2.7.3, 2.6.9, 2.5.12 and 2.4.16 are available.
> These releases contain a fix for CVE-2011-0411 which allows plaintext
> command injection with SMTP sessions over TLS. This defect was
> introduced with Postfix version 2.2. The same flaw exists in other
> implementations of the STARTTLS command.
> 
> Note: CVE-2011-0411 is an issue only for the minority of SMTP
> clients that actually verify server certificates. Without server
> certificate verification, clients are always vulnerable to
> man-in-the-middle attacks that allow attackers to inject
> plaintext commands or responses into SMTP sessions, and more.
> 
> Postfix 2.8 and 2.9 are not affected.
> 
> The following problems were fixed with the Postfix legacy releases:
> 
> * Fix for CVE-2011-0411: discard buffered plaintext input,
>   after reading the SMTP "STARTTLS" command or response.  
> 
> * Fix to the local delivery agent: look up the "unextended"
>   address in the local aliases database, when that address has
>   a malformed address extension.  
> 
> * Fix to virtual alias expansion: report a tempfail error,
>   instead of silently ignoring recipients that exceed the
>   virtual_alias_expansion_limit or the virtual_alias_recursion_limit.
> 
> * Fix for Solaris: the Postfix event engine was deaf for SIGHUP
>   and SIGALRM signals after the switch from select() to /dev/poll.
>   Symptoms were delayed "postfix reload" response, and killed
>   processes with watchdog timeout values under 100 seconds.
> 
> * Fix for HP-UX: the Postfix event engine was deaf for SIGALRM
>   signals. Symptoms were killed processes with watchdog timeout
>   values under 100 seconds.  
> 
> * Fix for BSD-ish mkdir() to prevent maildir directories from
>   inheriting their group ownership from the parent directory.
> 
> * Fix to the SMTP client: missing support for mail to
>   [ipv6:ipv6addr] address literal destinations.  
> 
> * FreeBSD back-ported closefrom() from FreeBSD 8x to 7x, breaking
>   Postfix builds retroactively.
> 
> Historical note:
> 
> Wietse Venema discovered the problem two weeks before the
> Postfix 2.8 release, and silently fixed it pending further
> investigation. While investigating the problem's scope and
> impact, Victor Duchovni found that many other TLS applications
> were also affected. At that point, CERT/CC was asked to coordinate
> with the problem's resolution.
> 
> You can find the updated Postfix source code at the mirrors listed
> at http://www.postfix.org/.
-- 
gpg --keyserver pgp.mit.edu --recv-key 092164A7
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x092164A7


signature.asc
Description: Ceci est une partie de message	numériquement signée

Re: Postix Newbie: Send all outbound mail to another postfix server

[Dennis Guhl]

On Tue, Mar 08, 2011 at 12:25:32PM -0500, Steve Angelo wrote:
> Hi all.. Please forgive me as I'm a postfix newbie.. I'm currently running
> version 2.6.5 on an Ubuntu server. I have a codeignitor app that sends
> outbound mail only. no inbound mail is processed/advertised.
> 
> I recently moved my production server from Rackspace to AWS and Amazon is
> now having a PBL issue that may take days to resolve and my IP is listed so
> most of the mail is not going out due to PBL from Spamhaus.

If you are blocked because of Spamhaus' PBL you are on an consumer
dial up (http://www.spamhaus.org/pbl/) and should not send mail
directly to MX. Instead use your providers SMTP relay.

http://www.postfix.org/BASIC_CONFIGURATION_README.html#relayhost
http://www.postfix.org/STANDARD_CONFIGURATION_README.html#dialup

> So I was thinking.. Since my old server at rackspace is still up and
> running, would it be possible to have configure my AWS postfix server to
> send all outbound mail to the Rackspace postfix server and have that server
> send out the email? The originating IP would have to be the Rackspace server
> though, as if the mail still shows the originating as the AWS server it will
> fail due to PBL.

Look at
 http://www.postfix.org/postconf.5.html#relayhost
or at
 http://www.postfix.org/postconf.5.html#sender_dependent_relayhost_maps
in case you don't want to relay all sender through your other server.

> Is this possible and if so, any pointers/suggestions on how to go about
> doing this? My other option is to move back to Rackspace which will be very
> disruptive.
> 
> Thanks!

Re: STARTTLS bug -- [Zimbra]

[Walter Smith]

Hi there!
 
How severe this bug is?
 
I'm running few Zimbra servers and seems like it's there:
 
 
% telnet 0 25
220 myzimbra ESMTP Postfix
starttls
220 2.0.0 Ready to start TLS
 
% telnet 0 587
220 myzimbra ESMTP Postfix
starttls
220 2.0.0 Ready to start TLS
 
 
Should I disable it for now - is there particular parameter(s) within Postfix - 
seems to me it's kind of blended with regular TSL settings...
 
Thanks,
W.S.

--- On Tue, 3/8/11, Wietse Venema  wrote:


From: Wietse Venema 
Subject: Re: STARTTLS bug - background story
To: postfix-users@postfix.org
Date: Tuesday, March 8, 2011, 6:45 AM


Victor Duchovni:
> On Tue, Mar 08, 2011 at 12:59:15PM +1100, Brad Hards wrote:
> 
> > On Tue, 8 Mar 2011 07:08:09 am Wietse Venema wrote:
> > > This is a writeup about a flaw that I found recently, and that
> > > existed in multiple implementations of SMTP (Simple Mail Transfer
> > > Protocol) over TLS (Transport Layer Security) including my Postfix
> > > open source mailserver. I give an overview of the problem and its
> > > impact, technical background, how to find out if a server is affected,
> > > fixes, and draw lessons about where we can expect similar problems
> > > now or in the future. A time line is at the end.
> >
> > Thanks for the write-up. 
> 
> It is a bit disappointing that very few of the potentially impacted
> vendors, and some definitely impacted vendors are yet to respond to
> the vulnerability:
> 
>     http://www.kb.cert.org/vuls/id/555316
> 
> Some email appliance vendors are not on the list. Apart from Postfix,
> Qmail, and some large mailbox hosting providers, which are already
> fixed, the issue will likely linger in less visible products for
> some time...

It's easy enough to make the one-line change to openssl source, so
that people can check for this now if they are concerned.

I would expect that penetration test toolkits will eventually look
for starttls plaintext injection vulnerabilities.  But that may
take a while.

Publishing "shame" lists on the web is better done by people who
work for organizations with no commercial interest in the issue.

    Wietse



  

Re: STARTTLS bug -- [Zimbra]

[Wietse Venema]

Walter Smith:
> Hi there!
> ?
> How severe this bug is?

Please read the announcement, section "overview and impact".
http://www.postfix.org/CVE-2011-0411.html

"This is not as big a problem as it may appear to be. The reason
is that many SMTP client applications don't verify server TLS
certificates. These SMTP clients are always vulnerable to command
injection and other attacks. Their TLS sessions are only encrypted
but not protected."

Wietse

Re: Change in bounce_template_file, postfix reload needed?

[Victor Duchovni]

On Tue, Mar 08, 2011 at 04:27:20PM +0100, Ralf Hildebrandt wrote:

> If I change the bounce_template_file, is a "postfix reload" for the
> change to take (immediate) effect needed?

Yes, if you want the effect to be "immediate".

> Looking at http://www.postfix.org/bounce.8.html I'm seeing
> 
>   max_idle (100s)
>   max_use (100)
> 
> which are implying that after 100 bounces or 100s of no usage, Postfix
> will let bounce die and be reborn.

Correct.

-- 
Viktor.

Re: Change in bounce_template_file, postfix reload needed?

[Ralf Hildebrandt]

* Victor Duchovni :
> On Tue, Mar 08, 2011 at 04:27:20PM +0100, Ralf Hildebrandt wrote:
> 
> > If I change the bounce_template_file, is a "postfix reload" for the
> > change to take (immediate) effect needed?
> 
> Yes, if you want the effect to be "immediate".
> 
> > Looking at http://www.postfix.org/bounce.8.html I'm seeing
> > 
> >   max_idle (100s)
> >   max_use (100)
> > 
> > which are implying that after 100 bounces or 100s of no usage, Postfix
> > will let bounce die and be reborn.
> 
> Correct.

OK! As suspected then...

Re: Postix Newbie: Send all outbound mail to another postfix server

[Steve Angelo]

> > So I was thinking.. Since my old server at rackspace is still up and
> > running, would it be possible to have configure my AWS postfix server to
> > send all outbound mail to the Rackspace postfix server and have that
> server
> > send out the email? The originating IP would have to be the Rackspace
> server
> > though, as if the mail still shows the originating as the AWS server it
> will
> > fail due to PBL.
>
> Look at
>  http://www.postfix.org/postconf.5.html#relayhost
> or at
>  http://www.postfix.org/postconf.5.html#sender_dependent_relayhost_maps
> in case you don't want to relay all sender through your other server.
>
>
Thanks.. by doing this will this change the source IP address of the mail to
the IP address of the email server I'm relaying the email to?

Header checks and HOLD queue release

[Michael]

This might be a stupid question.

I have a header_check rule that puts certain mails into the HOLD queue
for inspection by senior staff, and when its released into the deferred
queue for delivery, it gets chucked back into the HOLD queue due to
Postfix reapplying the Header_checks. 

It does go through on the second release and flush however. Is there
away to bypass the Header_checks for deferred mail?

Re: Header checks and HOLD queue release

[Ralf Hildebrandt]

* Michael :
> This might be a stupid question.
> 
> I have a header_check rule that puts certain mails into the HOLD queue
> for inspection by senior staff, and when its released into the deferred
> queue for delivery, it gets chucked back into the HOLD queue due to
> Postfix reapplying the Header_checks. 

Header_checks are only reapplied when another smtpd is involved.

> It does go through on the second release and flush however. Is there
> away to bypass the Header_checks for deferred mail?

You seem to have a content_filter setup. Could that be?

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebra...@charite.de | http://www.charite.de

Re: Header checks and HOLD queue release

[Michael]

On Tue, 2011-03-08 at 21:13 +0100, Ralf Hildebrandt wrote:
> 
> You seem to have a content_filter setup. Could that be? 

Yes, SpamAssassin is in the chain. I did'nt realise this may be the
issue.

Re: Header checks and HOLD queue release

[Ralf Hildebrandt]

* Michael :
> On Tue, 2011-03-08 at 21:13 +0100, Ralf Hildebrandt wrote:
> > 
> > You seem to have a content_filter setup. Could that be? 
> 
> Yes, SpamAssassin is in the chain. I did'nt realise this may be the
> issue.

Please show master.cf

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebra...@charite.de | http://www.charite.de

Re: Header checks and HOLD queue release

[Michael]

On Tue, 2011-03-08 at 21:18 +0100, Ralf Hildebrandt wrote:
> * Michael :
> > On Tue, 2011-03-08 at 21:13 +0100, Ralf Hildebrandt wrote:
> > > 
> > > You seem to have a content_filter setup. Could that be? 
> > 
> > Yes, SpamAssassin is in the chain. I did'nt realise this may be the
> > issue.
> 
> Please show master.cf
> 

smtp  inet  n   -   n   -   -   smtpd
   -o content_filter=spamassassin
pickupfifo  n   -   n   60  1   pickup
cleanup   unix  n   -   n   -   0   cleanup
qmgr  fifo  n   -   n   300 1   qmgr
tlsmgrunix  -   -   -   1000?   1   tlsmgr
rewrite   unix  -   -   n   -   -   trivial-rewrite
bounceunix  -   -   n   -   0   bounce
defer unix  -   -   n   -   0   bounce
trace unix  -   -   n   -   0   bounce
verifyunix  -   -   -   -   1   verify
flush unix  n   -   -   1000?   0   flush
proxymap  unix  -   -   n   -   -   proxymap
smtp  unix  -   -   -   -   -   smtp
relay unix  -   -   -   -   -   smtp
-o fallback_relay=
showq unix  n   -   -   -   -   showq
error unix  -   -   -   -   -   error
discard   unix  -   -   -   -   -   discard
local unix  -   n   n   -   -   local
virtual   unix  -   n   n   -   -   virtual
lmtp  unix  -   -   -   -   -   lmtp
anvil unix  -   -   -   -   1   anvil
scacheunix  -   -   -   -   1   scache

receive_override_options=no_unknown_recipient_checks,no_header_body_checks
maildrop  unix  -   n   n   -   -   pipe
  flags=DRhu user=vmail argv=/usr/bin/maildrop -d ${recipient}
uucp  unix  -   n   n   -   -   pipe
  flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail
($recipient)
ifmailunix  -   n   n   -   -   pipe
  flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient)
bsmtp unix  -   n   n   -   -   pipe
  flags=Fq. user=bsmtp argv=/usr/lib/bsmtp/bsmtp -t$nexthop -f$sender
$recipient
scalemail-backend unix  -   n   n   -   2   pipe
  flags=R user=scalemail argv=/usr/lib/scalemail/bin/scalemail-store
${nexthop} ${user} ${extension}
mailman   unix  -   n   n   -   -   pipe
  flags=FR user=list argv=/usr/lib/mailman/bin/postfix-to-mailman.py
  ${nexthop} ${user}

retry unix  -   -   -   -   -   error

spamassassin unix - n   n   -   -   pipe
user=spamd argv=/usr/bin/spamc -f -e
/usr/sbin/sendmail -oi -f ${sender} ${recipient}

Re: regular expressions was: Kernel Oops

[Stan Hoeppner]

Wietse Venema put forth on 3/8/2011 10:39 AM:
> Stan Hoeppner:
>> So, the question is, which form of expression processes the "does not
>> match" case faster?  The fully qualified expression, or the simple
>> expression?  Noel mentioned that the fully qualified expressions will
>> tend to process faster.  Is this true?  Is it true for both the
>> "matches" and "does not match" case?
> 
> I would expect better performance when patterns only match the text
> that needs to be matched.

So this would mean the simpler expressions would be faster?  That makes
me wonder why Enemies List[1] uses complex expressions, each one
precisely matching a specific rDNS pattern, given EL matches 65k+
patterns total.  Likewise, the original author of my fqrdns.pcre table
also used mostly expressions that exactly match a specific rDNS pattern,
although in this case we have only 1600+ expressions so speed isn't as
critical.

I've not made 1 to 1 equivalent simpler expressions and run timing
tests.  It would be rather time consuming to copy the current table and
simplify the expressions in the copy.  I'm wondering now if execution
times would show any meaningful difference.  I wonder if testing just a
small subset, say 100 expressions, would be sufficient to show
meaningful execution time differences.

> If you must match a very large numbers of patterns, you need an
> implementation that transforms N patterns into one deterministic
> automaton. This can match 1 pattern in the same time as N patterns.
> Once the automaton is built (which takes some time) it is blindingly
> fast. An example of such an implementation is flex.

This sounds really interesting.  Do you have a link to info about this
flex software?  I'd like to read about it.

> Similar optimizations are needed for large CIDR maps. Right now,
> Postfix's linear search does 10^8 patterns/s. With this, postscreen
> can search the largest ipdeny.com file in 1ms on a modern CPU,
> which is sufficient for the moment. To make it fast, the CIDR
> entries need to be arranged into a tree that can be traversed in
> log(N) time.

I recall you and Viktor discussing this a while ago.  I don't really
understand how an OP (myself) would go about creating a tree of our CIDR
tables.  Or is this something that the Postfix CIDR code would handle?


[1] Enemies List is not available for Postfix, yet, and the intelligence
dataset is not free, although the source code is open.  EL is integrated
in some commercial AS appliances and commercial mail software.  I
mention it frequently here because it is the only antispam tool I'm
aware of that makes almost exclusive use of regexes to identify likely
spam sources, and it uses 10s of thousands of regexes.

-- 
Stan

Re: Header checks and HOLD queue release

[Ralf Hildebrandt]

* Michael :

> smtp  inet  n   -   n   -   -   smtpd
>-o content_filter=spamassassin

...

> spamassassin unix - n   n   -   -   pipe
> user=spamd argv=/usr/bin/spamc -f -e
> /usr/sbin/sendmail -oi -f ${sender} ${recipient}

So the reinjection (using sendmail) goes through cleanup again (which
does the header_checks).

One solution would be to use mini_sendmail instead of
/usr/sbin/sendmail and let mini_sendmail inject into another smtpd
instance on localhost, Port 10026, e.g.:

localhost:10026
  inet  n   -   -   --  smtpd -o 
receive_override_options=no_header_body_checks

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebra...@charite.de | http://www.charite.de

Re: Postix Newbie: Send all outbound mail to another postfix server

[Stan Hoeppner]

Steve Angelo put forth on 3/8/2011 11:25 AM:

> I recently moved my production server from Rackspace to AWS and Amazon is
> now having a PBL issue that may take days to resolve and my IP is listed so
> most of the mail is not going out due to PBL from Spamhaus.

You should have asked for advice before switching.  Everyone and his
grandmother would have told you that sending mail from a cloud,
especially Amazon, will always be fraught with delivery issues.

Clouds and VPS are cheap.  Snowshoe spammers tend to gravitate to the
cheapest hosting they can find.  They pay for a month with stolen credit
cards, give false identification/contact info, spam for a month while
the complaints are still coming in, then, right before the provider
makes the decision to boot them, whoosh, they vanish without a trace,
already having their next operation up and spamming.  Rinse, repeat.

Thus, unfortunately, legit senders who jump at the cheapest hosting are
invariably treated the same as spammers by receivers, and, in some
cases, by dnsbls.

BTW, if that Amazon IP space is on the PBL as you mention, it very
likely won't be coming off any time soon.  SBL listings expire.  PBL
listings tend to last a very long time, as it is usually the netblock
owner that puts it in the PBL.

-- 
Stan

Re: regular expressions was: Kernel Oops

[Victor Duchovni]

On Tue, Mar 08, 2011 at 02:29:23PM -0600, Stan Hoeppner wrote:

> So this would mean the simpler expressions would be faster?  That makes
> me wonder why Enemies List[1] uses complex expressions, each one
> precisely matching a specific rDNS pattern,

To avoid false positives by matching in the wrong context.
The performance can be improved by grouping:

/^\d+\.\d+\.\d+\.\d+$/  DUNNO only hostnames matched below

if /\.net$/
# patterns for ".net" hosts
...
/^/ DUNNO done with .net[
endif

if /\.net\.au$/
# patterns for ".net.au" hosts
...
/^/ DUNNO done with .net.au
endif

if /\.com$/
# patterns for ".com" hosts
...
/^/ DUNNO done with .com
endif

if /\.edu$/
# patterns for ".edu" hosts
...
/^/ DUNNO done with .edu
endif

-- 
Viktor.

Re: Postix Newbie: Send all outbound mail to another postfix server

[Dennis Guhl]

On Tue, Mar 08, 2011 at 02:58:50PM -0500, Steve Angelo wrote:
> > > So I was thinking.. Since my old server at rackspace is still up and
> > > running, would it be possible to have configure my AWS postfix server to
> > > send all outbound mail to the Rackspace postfix server and have that
> > server
> > > send out the email? The originating IP would have to be the Rackspace
> > server
> > > though, as if the mail still shows the originating as the AWS server it
> > will
> > > fail due to PBL.
> >
> > Look at
> >  http://www.postfix.org/postconf.5.html#relayhost
> > or at
> >  http://www.postfix.org/postconf.5.html#sender_dependent_relayhost_maps
> > in case you don't want to relay all sender through your other server.
> >
> >
> Thanks.. by doing this will this change the source IP address of the mail to
> the IP address of the email server I'm relaying the email to?

Yes. You send your emails to the relayhost which queues them and send
them to the receiving MX.

You should read about the fundamental concepts of email in genral and
smtp in particular.

Dennis

Re: Postix Newbie: Send all outbound mail to another postfix server

[Stan Hoeppner]

Dennis Guhl put forth on 3/8/2011 11:52 AM:

> If you are blocked because of Spamhaus' PBL you are on an consumer
> dial up (http://www.spamhaus.org/pbl/) 

FYI, the PBL isn't limited to dynamic listings.  Many corporations add
their unused IP space to the PBL, along with other IPs within their
netblocks that shouldn't be sending direct mail.  They do this as part
of a multi-layered approach to network security, in addition to egress
filtering at the edge firewalls.  One errant mouse click by an
apprentice/junior SA can accidentally disable an egress filter, as can a
botched firmware update on a firewall or router, etc, etc.  If, when
such a thing occurs, you already have an internal spambot outbreak that
the firewalls/routers were containing...

I would have never considered this until one day the chief of network
security at Nortel informed me they do precisely what I described above.

Dorothy, you're not in Kansas anymore.

-- 
Stan

Message is modified after after-queue filter

[Ulrich Mierendorff]

Hi,
I am using an after queue-content filter that signs outgoing emails 
(dkimproxy). My problem is that these messages are modified AFTER the 
signature was added which breaks the signature. As far as I can see, 
"\n" is replaced by "\r\n" and "Content-Transfer-Encoding: 8bit" is 
replaced by "Content-Transfer-Encoding: quoted-printable". I have no 
problem with these modifications in general, but I would like if postfix 
could do these modifications BEFORE passing the mail to my filter. And I 
currently do not know how I can enable this.
I have tried postfix versions 2.5.5 and 2.7.1 (Debian) with no luck. I 
am also using a similar configuration on another server (with an older 
version of postfix) which works as expected (messages are still 
modified, but before filtering).
My current work-around is to correctly format my emails in my software 
before they are sent to postfix so that the messages are not modified at 
all. But that is not the best solution.



-Ulrich



Relevant parts of master.cf:

smtp  inet  n   -   -   -   -   smtpd
submission inet n   -   -   -   -   smtpd
  -o receive_override_options=no_address_mappings
  -o content_filter=dksign:[127.0.0.1]:10028


dksignunix  -   -   n   -   6   smtp
  -o smtp_send_xforward_command=yes
  -o smtp_discard_ehlo_keywords=8bitmime,starttls

127.0.0.1:10029 inet  n  -  n   -   10  smtpd
  -o content_filter=
  -o 
receive_override_options=no_unknown_recipient_checks,no_header_body_checks

  -o smtpd_helo_restrictions=
  -o smtpd_client_restrictions=
  -o smtpd_sender_restrictions=
  -o smtpd_recipient_restrictions=permit_mynetworks,reject
  -o mynetworks=127.0.0.0/8
  -o smtpd_authorized_xforward_hosts=127.0.0.0/8

Re: Postix Newbie: Send all outbound mail to another postfix server

[Randy Ramsdell]

Stan Hoeppner wrote:

Dennis Guhl put forth on 3/8/2011 11:52 AM:


If you are blocked because of Spamhaus' PBL you are on an consumer
dial up (http://www.spamhaus.org/pbl/) 


FYI, the PBL isn't limited to dynamic listings.  Many corporations add
their unused IP space to the PBL, along with other IPs within their
netblocks that shouldn't be sending direct mail.  They do this as part
of a multi-layered approach to network security, in addition to egress
filtering at the edge firewalls.  One errant mouse click by an
apprentice/junior SA can accidentally disable an egress filter, as can a
botched firmware update on a firewall or router, etc, etc.  If, when
such a thing occurs, you already have an internal spambot outbreak that
the firewalls/routers were containing...

I would have never considered this until one day the chief of network
security at Nortel informed me they do precisely what I described above.

Dorothy, you're not in Kansas anymore.



If the firewall is blocking an outbreak of spam bots from sending mail 
to the outside, why did they not know and fix this? I mean is it so bad 
that the whole network team can't contain it? And then someone botched 
the firewall which allowed the spam to be sent? Nortel hmmm.

Re: Message is modified after after-queue filter

[Victor Duchovni]

On Tue, Mar 08, 2011 at 10:55:25PM +0100, Ulrich Mierendorff wrote:

> I am using an after queue-content filter that signs outgoing emails 
> (dkimproxy). My problem is that these messages are modified AFTER the 
> signature was added which breaks the signature. As far as I can see, "\n" 
> is replaced by "\r\n"

As required by SMTP standards. SMTP messages lines are separated by CRLF
not "CR" or "LF" alone.

> and "Content-Transfer-Encoding: 8bit" is replaced by 
> "Content-Transfer-Encoding: quoted-printable".

This happens when the destination does not support 8bitmime.

> I have no problem with these 
> modifications in general, but I would like if postfix could do these 
> modifications BEFORE passing the mail to my filter. And I currently do not 
> know how I can enable this.

Remove "8BITMIME" from the EHLO features of the proxy or the Postfix
SMTP server behind it (if the proxie's EHLO response is just copied
from downstream).

> My current work-around is to correctly format my emails in my software 
> before they are sent to postfix so that the messages are not modified at 
> all. But that is not the best solution.

Actually that *is* the best solution. Send 7-bit encoded mail with
correct line endings.

-- 
Viktor.

Re: submission port : "Client host rejected: Access denied"

[/dev/rob0]

On Sun, Mar 06, 2011 at 02:20:19PM +0100, David Touzeau wrote:
> Thanks Jeroen
> 
> Here it is information requested
> 
> 
> postconf -n

Yikes. I didn't read through all of that, but I'm sure you have gone 
way overboard in changing (or perhaps, restating) default settings. 
Jeroen would say, if your postconf -n is more than 30 lines, you are 
either an expert, or you have done it all wrong. :)

Consider starting over with a simple config.
-- 
Offlist mail to this address is discarded unless
"/dev/rob0" or "not-spam" is in Subject: header

rewrite the from based on a client hostname or ip

[Katzir, Igal]

Hi Postfix Users,

We are running a couple of postfix servers version 2.3.x which mainly relay 
notification messages to our customers.
The various application servers which trigger notifications are not consistent 
so we are rewriting the mail  field with these two tables:
sender_canonical_maps = hash:/etc/postfix/mail_rewrite
smtp_generic_maps = hash:/etc/postfix/generic

Now the problem:
We have one customer which host the application for his partners so he wishes 
that all mail coming from this specific app. Server will have a fixed address.
Can I create a rule where all mail coming from one / two servers will have a 
specific  address?
Thanks,
i...@hp.com

Re: regular expressions was: Kernel Oops

[Erik de Castro Lopo]

Wietse Venema wrote:

> If you must match a very large numbers of patterns, you need an
> implementation that transforms N patterns into one deterministic
> automaton. This can match 1 pattern in the same time as N patterns.
> Once the automaton is built (which takes some time) it is blindingly
> fast. An example of such an implementation is flex.

Is there a limit the the pattern length in the pcre tables?

If not, it would be possible to convert this (3 only, but could be
hundreds or even thousands):

   /^([0-9]{1,3}\.){4}\.dsl\.dynamic\.eranet\.pl$/
   /^([0-9]{1,3}\.){4}\.dynamic\.snap\.net\.nz$/
   /^([0-9]{1,3}\.){4}\.nat\.umts\.dynamic\.eranet\.pl$/

to this:

   
/^([0-9]{1,3}\.){4}\.(dsl\.dynamic\.eranet\.pl|dynamic\.snap\.net\.nz|nat\.umts\.dynamic\.eranet\.pl)$/

and that should reject "1.1.1.1.not-found" in 1/3 the time of the
three original regexes while also matching quicker than the original.

Obviously, a conversion from the first three to the optimised version
has to be done mechanistically to avoid errors.

Cheers,
Erik
-- 
--
Erik de Castro Lopo
http://www.mega-nerd.com/

Re: regular expressions was: Kernel Oops

[mouss]

[WARNING: Steven CC'd]

Le 08/03/2011 21:29, Stan Hoeppner a écrit :
> Wietse Venema put forth on 3/8/2011 10:39 AM:
>> Stan Hoeppner:
>>> So, the question is, which form of expression processes the "does not
>>> match" case faster?  The fully qualified expression, or the simple
>>> expression?  Noel mentioned that the fully qualified expressions will
>>> tend to process faster.  Is this true?  Is it true for both the
>>> "matches" and "does not match" case?
>>
>> I would expect better performance when patterns only match the text
>> that needs to be matched.
> 

to get better performance, one would use patterns that fail to match as
soon as possible. I mean if you have /^a/, then the check would stop
as soon as the first char isn't an "a". but the expressions we would
like to match and the expressions we see are completely different
things. so I'd say, do not consider performances as a primary target. go
for catching spammers first. only tune after you get the irght rules,
and only if needed (I personally don't tune anything here. I'm happy to
focus on catching spammers).

> So this would mean the simpler expressions would be faster?

No. /^a(complex blah)/ is faster than /joe/ because the first will stop
if the first char sin't "a" whatever is the rest of the expression.

>  That makes
> me wonder why Enemies List[1] uses complex expressions, each one
> precisely matching a specific rDNS pattern, given EL matches 65k+
> patterns total. 

as said above, the goal isn't performance (to improve performance, buy
better hardware or run multiple instances). The goal of Steven is to
maximize hit rate while minimizing false positives. many of us have
created rules to block generic/dynamic/silly senders. when doing so, you
can start by being precise at the risk of doing a lot of work because
your rules minimise FPs, or going the other side by using expressions
that block a lot of senders inclusing legitimate ones, that is
increasing the FP rate. it takes time and efforts to get a good balance,
and that's what Steven work is about.

>[snip]
> 
>> If you must match a very large numbers of patterns, you need an
>> implementation that transforms N patterns into one deterministic
>> automaton. This can match 1 pattern in the same time as N patterns.
>> Once the automaton is built (which takes some time) it is blindingly
>> fast. An example of such an implementation is flex.
> 
> This sounds really interesting.  Do you have a link to info about this
> flex software?  I'd like to read about it.
> 

[note: it wasn't me who said the text above. I however studied the
problem, in a completely different context. I can tell you one thing:
forget about optimizing your pcre rules. optimisation is useful in DNA
matching problems and the like. and even then...).

>> Similar optimizations are needed for large CIDR maps. Right now,
>> Postfix's linear search does 10^8 patterns/s. With this, postscreen
>> can search the largest ipdeny.com file in 1ms on a modern CPU,
>> which is sufficient for the moment. To make it fast, the CIDR
>> entries need to be arranged into a tree that can be traversed in
>> log(N) time.
> 
> I recall you and Viktor discussing this a while ago.  I don't really
> understand how an OP (myself) would go about creating a tree of our CIDR
> tables.  Or is this something that the Postfix CIDR code would handle?
> 

if cidr is to be enhanced, then it would be done inside cidr
implementation. the problem is the usual one: algorithms are often said
to be k*O(f(n)). so you generally prefer f(n)=log(n) over f(n)=n^2. but
this is only good for large n, and n is never large, so you need to
remember about the k constant. said otherwise: k1 * n^2 < k2 < log(n)
for small n under some conditions.

> 
> [1] Enemies List is not available for Postfix, yet, and the intelligence
> dataset is not free, although the source code is open.  EL is integrated
> in some commercial AS appliances and commercial mail software.  I
> mention it frequently here because it is the only antispam tool I'm
> aware of that makes almost exclusive use of regexes to identify likely
> spam sources, and it uses 10s of thousands of regexes.
> 

I don't use EL, but I think it is usable with postfix. Steven, can you
confirm this? (some of the features may be sendmail oriented, but it
would be easy to "generalize" them).

Re: regular expressions was: Kernel Oops

[Steve]

 Original-Nachricht 
> Datum: Wed, 9 Mar 2011 09:49:21 +1100
> Von: Erik de Castro Lopo 
> An: postfix-users@postfix.org
> Betreff: Re: regular expressions  was:  Kernel Oops

> Wietse Venema wrote:
> 
> > If you must match a very large numbers of patterns, you need an
> > implementation that transforms N patterns into one deterministic
> > automaton. This can match 1 pattern in the same time as N patterns.
> > Once the automaton is built (which takes some time) it is blindingly
> > fast. An example of such an implementation is flex.
> 
> Is there a limit the the pattern length in the pcre tables?
> 
I think there is one (if memory does not fool me then it is somewhere around 
1000 characters). But I am not 100% sure.


> If not, it would be possible to convert this (3 only, but could be
> hundreds or even thousands):
> 
>/^([0-9]{1,3}\.){4}\.dsl\.dynamic\.eranet\.pl$/
>/^([0-9]{1,3}\.){4}\.dynamic\.snap\.net\.nz$/
>/^([0-9]{1,3}\.){4}\.nat\.umts\.dynamic\.eranet\.pl$/
> 
Are you sure the above is correct? You have there a double dot and I think that 
is not correct.


> to this:
> 
>   
> /^([0-9]{1,3}\.){4}\.(dsl\.dynamic\.eranet\.pl|dynamic\.snap\.net\.nz|nat\.umts\.dynamic\.eranet\.pl)$/
> 
Or even shorter:
/^([0-9]{1,3}\.){4}((dsl\.dynamic|nat\.umts)\.dynamic\.eranet\.pl|dynamic\.snap\.net\.nz)$/

Maybe using if/endif conditions like Stan Hoeppner has done on his pcre map 
could speedup things even more? -> http://www.hardwarefreak.com/fqrdns.pcre


> and that should reject "1.1.1.1.not-found" in 1/3 the time of the
> three original regexes while also matching quicker than the original.
> 
> Obviously, a conversion from the first three to the optimised version
> has to be done mechanistically to avoid errors.
> 
Well... if the source is already buggy (double dot issue) then automating that 
transformation is not going to help you much.



> Cheers,
> Erik
> -- 
// Steve


> --
> Erik de Castro Lopo
> http://www.mega-nerd.com/

-- 
GMX DSL Doppel-Flat ab 19,99 Euro/mtl.! Jetzt mit 
gratis Handy-Flat! http://portal.gmx.net/de/go/dsl

Re: regular expressions was: Kernel Oops

[Noel Jones]

On 3/8/2011 4:49 PM, Erik de Castro Lopo wrote:

Wietse Venema wrote:


If you must match a very large numbers of patterns, you need an
implementation that transforms N patterns into one deterministic
automaton. This can match 1 pattern in the same time as N patterns.
Once the automaton is built (which takes some time) it is blindingly
fast. An example of such an implementation is flex.


Is there a limit the the pattern length in the pcre tables?


The pattern length limit is controlled by the pcre library 
you're using.  I think most implementations limit single 
expressions to 64k characters.


It's unclear to me if a single huge complex expression will 
evaluate faster that multiple less complex expressions.




If not, it would be possible to convert this (3 only, but could be
hundreds or even thousands):

/^([0-9]{1,3}\.){4}\.dsl\.dynamic\.eranet\.pl$/
/^([0-9]{1,3}\.){4}\.dynamic\.snap\.net\.nz$/
/^([0-9]{1,3}\.){4}\.nat\.umts\.dynamic\.eranet\.pl$/

to this:


/^([0-9]{1,3}\.){4}\.(dsl\.dynamic\.eranet\.pl|dynamic\.snap\.net\.nz|nat\.umts\.dynamic\.eranet\.pl)$/

and that should reject "1.1.1.1.not-found" in 1/3 the time of the
three original regexes while also matching quicker than the original.


(your sample expression looks a little wonky to me.  You sure 
it works?)


Improving performance would be better accomplished by 
enclosing the similar lines in an IF..ENDIF statement. 
Performance should be improved for non-matching input, 
readability and maintainability is dramatically improved.


Skipping rules always beats evaluating rules.
Unreadable rules should be avoided.


  -- Noel Jones

Re: regular expressions was: Kernel Oops

[mouss]

Le 08/03/2011 23:49, Erik de Castro Lopo a écrit :
> Wietse Venema wrote:
> 
>> If you must match a very large numbers of patterns, you need an
>> implementation that transforms N patterns into one deterministic
>> automaton. This can match 1 pattern in the same time as N patterns.
>> Once the automaton is built (which takes some time) it is blindingly
>> fast. An example of such an implementation is flex.
> 
> Is there a limit the the pattern length in the pcre tables?
> 
> If not, it would be possible to convert this (3 only, but could be
> hundreds or even thousands):
> 
>/^([0-9]{1,3}\.){4}\.dsl\.dynamic\.eranet\.pl$/
>/^([0-9]{1,3}\.){4}\.dynamic\.snap\.net\.nz$/
>/^([0-9]{1,3}\.){4}\.nat\.umts\.dynamic\.eranet\.pl$/
> 
> to this:
> 
>
> /^([0-9]{1,3}\.){4}\.(dsl\.dynamic\.eranet\.pl|dynamic\.snap\.net\.nz|nat\.umts\.dynamic\.eranet\.pl)$/
> 
> and that should reject "1.1.1.1.not-found" in 1/3 the time of the
> three original regexes while also matching quicker than the original.


your speculations are wrong. /(joe|foo|bar)/ isn't /3 times faster than
individual tests. but before all, "premature optimisation is the root of
all evil". one should not convert readable stuff to unmaintainable
hieroglyph without measuring the real benefits.


> 
> Obviously, a conversion from the first three to the optimised version
> has to be done mechanistically to avoid errors.
> 

if it should be done, then it should be done inside the implementation.

> Cheers,
> Erik

Re: regular expressions was: Kernel Oops

[Erik de Castro Lopo]

Steve wrote:

> > If not, it would be possible to convert this (3 only, but could be
> > hundreds or even thousands):
> > 
> >/^([0-9]{1,3}\.){4}\.dsl\.dynamic\.eranet\.pl$/
> >/^([0-9]{1,3}\.){4}\.dynamic\.snap\.net\.nz$/
> >/^([0-9]{1,3}\.){4}\.nat\.umts\.dynamic\.eranet\.pl$/
> > 
> Are you sure the above is correct? You have there a double dot and
> I think that is not correct.

Yeah, there is a mistake there. This was a paper exercise :-).


> > /^([0-9]{1,3}\.){4}\.(dsl\.dynamic\.eranet\.pl|dynamic\.snap\.net\.nz|nat\.umts\.dynamic\.eranet\.pl)$/
> > 
> Or even shorter:
> /^([0-9]{1,3}\.){4}((dsl\.dynamic|nat\.umts)\.dynamic\.eranet\.pl|dynamic\.snap\.net\.nz)$/

Agreed.

> Maybe using if/endif conditions like Stan Hoeppner has done on his
> pcre map could speedup things even more? -> 
> http://www.hardwarefreak.com/fqrdns.pcre

I'm actually working with Stan already.

> > and that should reject "1.1.1.1.not-found" in 1/3 the time of the
> > three original regexes while also matching quicker than the original.
> > 
> > Obviously, a conversion from the first three to the optimised version
> > has to be done mechanistically to avoid errors.
> > 
> Well... if the source is already buggy (double dot issue) then automating
> that transformation is not going to help you much.

True, but my plan was to automate converting a list of rDNS names like:

178.183.237.0.dsl.dynamic.eranet.pl
183.246.69.111.dynamic.snap.net.nz
188.146.109.136.nat.umts.dynamic.eranet.pl

Into an optimal regex, assuming long pattern lines are accepted.

Erik
-- 
--
Erik de Castro Lopo
http://www.mega-nerd.com/

Re: regular expressions was: Kernel Oops

[Wietse Venema]

mouss:
[ Charset ISO-8859-1 unsupported, converting... ]
> Le 08/03/2011 23:49, Erik de Castro Lopo a ?crit :
> > Wietse Venema wrote:
> > 
> >> If you must match a very large numbers of patterns, you need an
> >> implementation that transforms N patterns into one deterministic
> >> automaton. This can match 1 pattern in the same time as N patterns.
> >> Once the automaton is built (which takes some time) it is blindingly
> >> fast. An example of such an implementation is flex.
> > 
> > Is there a limit the the pattern length in the pcre tables?
> > 
> > If not, it would be possible to convert this (3 only, but could be
> > hundreds or even thousands):
> > 
> >/^([0-9]{1,3}\.){4}\.dsl\.dynamic\.eranet\.pl$/
> >/^([0-9]{1,3}\.){4}\.dynamic\.snap\.net\.nz$/
> >/^([0-9]{1,3}\.){4}\.nat\.umts\.dynamic\.eranet\.pl$/
> > 
> > to this:
> > 
> >
> > /^([0-9]{1,3}\.){4}\.(dsl\.dynamic\.eranet\.pl|dynamic\.snap\.net\.nz|nat\.umts\.dynamic\.eranet\.pl)$/
> > 
> > and that should reject "1.1.1.1.not-found" in 1/3 the time of the
> > three original regexes while also matching quicker than the original.
> 
> 
> your speculations are wrong. /(joe|foo|bar)/ isn't /3 times faster than
> individual tests. but before all, "premature optimisation is the root of
> all evil". one should not convert readable stuff to unmaintainable
> hieroglyph without measuring the real benefits.

In the Postfix implementation, each regexp/pcre pattern is executed
separately, therefore (a|b|c) is faster than separate rules for a,
b and c. The savings are noticeable only in body_checks.

As for large numbers of CIDR patterns, I was referring to files
with 100,000 patterns. That is a non-trivial number, and I took
care to implement this such that postscreen could handle them.

I do agree with all the comments about skipping patterns with
IF/ENDIF or terminating matches early (which PCRE is very good at
if you use look-ahead and look-behind).

Wietse

Re: regular expressions was: Kernel Oops

[Erik de Castro Lopo]

Noel Jones wrote:

> The pattern length limit is controlled by the pcre library 
> you're using.  I think most implementations limit single 
> expressions to 64k characters.

Obviously something that needs testing.

> It's unclear to me if a single huge complex expression will 
> evaluate faster that multiple less complex expressions.

I'm not exactly sure how the pcre regex engine works in Postfix.
My assumptions below is that each pattern is matched individually
which is why I am suggesting that patterns can be combined for
speed improvements.

If the multiple complex expressions have the same prefix, then
combining the prefix test into a single expression will definitely
be faster to fail some non matching strings than using multiple
less complex expressions.

Consider the input string '123-234-32-12.whatever' and now compare
matching against three rules:

 /^([0-9]{1,3}\.){4}foo$/
 /^([0-9]{1,3}\.){4}bar$/
 /^([0-9]{1,3}\.){4}baz$/

In this ase, there will be three attempts (one on each pattern)
that fail on the fourth character ('-') of the input pattern. That
means that to fail all three patterns, there will be 12 character
comparisions.

Now compare that against:

 /^([0-9]{1,3}\.){4}(foo|bar|baz)$/

which will again fail on the fourth character, but there is only one
pattern which matches the same strings as the 3 patterns above.

> (your sample expression looks a little wonky to me.  You sure 
> it works?)

No, this was a poorly checked paper example.

> Improving performance would be better accomplished by 
> enclosing the similar lines in an IF..ENDIF statement. 
> Performance should be improved for non-matching input, 
> readability and maintainability is dramatically improved.

Personally I find reading regexes a pita even though I've been 
doing it for about 2 decades.

My idea was to autogenerate the complex regexes using
something like this:

178.183.237.0.dsl.dynamic.eranet.pl
183.246.69.111.dynamic.snap.net.nz
188.146.109.136.nat.umts.dynamic.eranet.pl

as input.

> Skipping rules always beats evaluating rules.

Agreed.

> Unreadable rules should be avoided.

Unless those rules were never intended to me read or modified
by hand.

Erik
-- 
--
Erik de Castro Lopo
http://www.mega-nerd.com/

Re: Thank you for great software

[Benny Pedersen]

 
> Especially for the great documentation and crazy online-times
> on the mailing-list :-)

agre 215% with you :=)

postfix on gentoo add defaults to main.cf when new version comes out

[Benny Pedersen]

subject says it all imho

added lines is exact same as found on postconf -d

why ?

main.cf is just a initial example conf mostly better to start with 100%
empty main.cf and add lines that need to be changed from defaults of
postconf -d

if this changes are really needed, then its okay, but i like to hear if
its not

sadly to see so many examples main.cf with pure defaults :(

Re: rewrite the from based on a client hostname or ip

[Wietse Venema]

Katzir, Igal:
> Hi Postfix Users,
> 
> We are running a couple of postfix servers version 2.3.x which
> mainly relay notification messages to our customers.
> The various application servers which trigger notifications are
> not consistent so we are rewriting the mail  field with
> these two tables:
> sender_canonical_maps = hash:/etc/postfix/mail_rewrite
> smtp_generic_maps = hash:/etc/postfix/generic
> 
> Now the problem:
> We have one customer which host the application for his partners
> so he wishes that all mail coming from this specific app. Server
> will have a fixed address.
> Can I create a rule where all mail coming from one / two servers
> will have a specific  address?

The simplest is to set up a dedicated Postfix instance on a dedicated
IP address, with sender canonical mapping to a fixed sender address.

http://www.postfix.org/MULTI_INSTANCE_README.html makes the first
part easy. For the second part, use "sender_canonical_maps =
regexp:/some/where/mail_rewrite.regexp", with contents like:

/./ REPLACE usern...@example.com

Wietse

Re: postfix on gentoo add defaults to main.cf when new version comes out

[Wietse Venema]

Benny Pedersen:
> subject says it all imho
> 
> added lines is exact same as found on postconf -d

Postfix always sets the "installation" parameters with the patnames
of commands and directories. This is much more reliable than trying
to be clever and setting only the ones that aren't at the default.

Wietse

> main.cf is just a initial example conf mostly better to start with 100%
> empty main.cf and add lines that need to be changed from defaults of
> postconf -d
> 
> if this changes are really needed, then its okay, but i like to hear if
> its not
> 
> sadly to see so many examples main.cf with pure defaults :(
> 
> 
> 
> 

Re: Thank you for great software

[Andy Wright]

On Mon, 2011-03-07 at 23:47 +0100, Reindl Harald wrote:
> Hi Wietse
> 
> I would like to say THANK YOU for postfix because i know
> developers are hearing this words not often enough as long
> their "baby" is running well and nobody takes notice about
> 
> Especially for the great documentation and crazy online-times
> on the mailing-list :-)
> 

my rant...

More than ten years ago I was playing with qmail, Sendmail (I used this
back in the day of course.), and Postfix deciding which I would put my
marbles in.  I chose Postfix, thanks Wietse and contributors!  Though I
don't agree with all of your decisions, the team has made this product
prosper.

Andy


signature.asc
Description: This is a digitally signed message part

Re: Postix Newbie: Send all outbound mail to another postfix server

[Dennis Guhl]

On Tue, Mar 08, 2011 at 03:19:53PM -0600, Stan Hoeppner wrote:
> Dennis Guhl put forth on 3/8/2011 11:52 AM:
> 
> > If you are blocked because of Spamhaus' PBL you are on an consumer
> > dial up (http://www.spamhaus.org/pbl/) 
> 
> FYI, the PBL isn't limited to dynamic listings.  Many corporations add
> their unused IP space to the PBL, along with other IPs within their
> netblocks that shouldn't be sending direct mail.  They do this as part

I'm aware of this. But I wasn't aware that the OP switched to AWS -- I
should have read the initial posting more closely.

[..]

> Dorothy, you're not in Kansas anymore.

What does this saying mean?

Re: regular expressions was: Kernel Oops

[Steven Champeon]

on Wed, Mar 09, 2011 at 12:03:27AM +0100, mouss wrote:
> [WARNING: Steven CC'd]

:-)
 
> Le 08/03/2011 21:29, Stan Hoeppner a écrit :
> > That makes me wonder why Enemies List[1] uses complex expressions,
> > each one precisely matching a specific rDNS pattern, given EL
> > matches 65k+ patterns total.

Eh, it varies quite a bit, some of them are complex groups like this:

[0-9]+\-[0-9]+\-[0-9]+\-[0-9]+\.dynamic\.(brasov|craiova|fagaras|resita|sfantugheorghe|victoria|zarnesti)\.rdsnet\.ro

because for whatever reason I can't just use a [0-9a-z\-]+ in place of
the group, or because they just grew over time as I saw more hosts. But
some are relatively simple:

[0-9a-z\-]+\-[0-9]+\.fiberlink\.[a-z]+\.rdsnet\.ro

wherever I can get away with it. You have to be careful with blanket
"alphanumeric token" host parts, because sometimes you're matching a
city or town or state or abbreviation and everything's fine, and then
the ISP starts putting 'mail' or 'static' in that token's position in
a similar hostname and suddenly you're blocking more than residential
dynamic cable modems. :-/

eg [0-9]+\-[0-9]+\-[0-9]+\-[0-9]+\.mail[0-9]+\.fft\.com\.au
   [0-9]+\-[0-9]+\-[0-9]+\-[0-9]+\.mail\.eletti\.com\.br
   [0-9]+\.[0-9]+\.[0-9]+\.[0-9]+\.mail\.sistemairis\.com\.br

I haven't really tried to optimize the regular expressions, because of
the way our library processes them - by walking down a tree from '.'
(so, '.' -> ro -> rdsnet -> all the patterns for rdsnet.ro) - so perf is
acceptable (several hundred thousand matches/sec on decent hardware;
~225K lookups/s on my old Macbook via C program).

Oh, and we're long past 65K - last build was 74494 patterns. I keep
forgetting to update the Web site. :-)

> as said above, the goal isn't performance (to improve performance, buy
> better hardware or run multiple instances).

Well, no, the goal is acceptable performance, but also managable update
mechanisms that allow for rapid correction of FP classifications. 

> The goal of Steven is to maximize hit rate while minimizing false
> positives. many of us have created rules to block
> generic/dynamic/silly senders. when doing so, you can start by being
> precise at the risk of doing a lot of work because your rules minimise
> FPs, or going the other side by using expressions that block a lot of
> senders inclusing legitimate ones, that is increasing the FP rate. it
> takes time and efforts to get a good balance, and that's what Steven
> work is about.

Yup. And it took me a few months to really understand that the useful
concept of a 'generic' hostname also unfortunately also applied to large
mail farms that we wanted mail from. (Now we track 'outmx' patterns,
too, and they account for around an eighth of all the patterns we have.
Same goes for 'webhost' - we mostly just see phishing scams from most of
them, but when you're analyzing someone's mailflow it helps to be able
to tell them which of their mail is coming from legit or quasi-legit
mail sources.)

I used to have a few hundred "compact" expressions, like this, which were
left-anchored but not fully qualified:

%compact = (
   "duN" => 'du[0-9]+',
  "dynN" => 'dyn[0-9]+',
  "pppN" => 'ppp[0-9]+',
 "N-N-N" => '[0-9]+\-[0-9]+\-[0-9]+',
 "dhcpH" => 'dhcp[0-9a-f]+',
 "dhcpN" => 'dhcp[0-9]+',
 "dialN" => 'dial[0-9]+',
 "duN-N" => 'du[0-9]+\-[0-9]+',
 "dyn-N" => 'dyn\-[0-9]+',
 "portN" => 'port[0-9]+',
 "ppp-N" => 'ppp\-[0-9]+',
"dhcp-N" => 'dhcp\-[0-9]+',
"dial-N" => 'dial\-[0-9]+',
"dialup" => 'dialup',
"du-N-N" => 'du\-[0-9]+\-[0-9]+',
"dynN-N" => 'dyn[0-9]+\-[0-9]+',
"port-N" => 'port\-[0-9]+',

[...]

but frankly the FP rate was so awful I ditched them. And not just
because of silly people like whoever set up Marriott's reservations
transactional servers with names like host184.marriott.com, but they
were one very big reason why I ditched them.
 
> >[snip]
> > 
> >> If you must match a very large numbers of patterns, you need an
> >> implementation that transforms N patterns into one deterministic
> >> automaton. This can match 1 pattern in the same time as N patterns.
> >> Once the automaton is built (which takes some time) it is blindingly
> >> fast. An example of such an implementation is flex.
> > 
> > This sounds really interesting.  Do you have a link to info about this
> > flex software?  I'd like to read about it.

Oh, that was what we tried first. Matt Sergeant wrote a perl wrapper
around a hunk of C object code that we generated using re2c. Worked
fine, you feed it regexes, it generates C code, you compile it into
an object and call it from a simple perl DNS server, voila. That was
how I provided the first instance of the Enemieslist via DNSBL, for
a year or so, on a Mac Mini. As far as the code went, it worked great.

Unfortunately, it took almost an hour to compil

Re: regular expressions was: Kernel Oops

[Noel Jones]

On 3/8/2011 6:00 PM, Erik de Castro Lopo wrote:

Noel Jones wrote:


The pattern length limit is controlled by the pcre library
you're using.  I think most implementations limit single
expressions to 64k characters.


Obviously something that needs testing.


Many years ago I worked on a system with a 32k limit on pcre 
expressions.  Ever since then, everything I've checked has 
been 64k, and then I gave up checking.  I expect any 
non-ancient system will support 64k, and some maybe even more. 
 (To clarify for others following along, this is a characters 
per single expression limit, not a filesize or number of 
expressions per file limit)



Consider the input string '123-234-32-12.whatever' and now compare
matching against three rules:

  /^([0-9]{1,3}\.){4}foo$/
  /^([0-9]{1,3}\.){4}bar$/
  /^([0-9]{1,3}\.){4}baz$/

In this ase, there will be three attempts (one on each pattern)
that fail on the fourth character ('-') of the input pattern. That
means that to fail all three patterns, there will be 12 character
comparisions.

Now compare that against:

  /^([0-9]{1,3}\.){4}(foo|bar|baz)$/

which will again fail on the fourth character, but there is only one
pattern which matches the same strings as the 3 patterns above.


This example is pretty easy to see that combining is better. 
It's not so clear if you create 32k of complex gibberish if it 
will actually operate faster as there may be significant 
startup times.  YMMV and all that.


BTW, with pcre you should use the the non-greedy flag inside 
parenthesis if you're not doing $n substitutions.  This saves 
another smidgen of time and memory.

/^(?:[0-9]{1,3}\.){4}(?:foo|bar|baz)$/


 -- Noel Jones

Re: STARTTLS bug -- [Zimbra]

[Quanah Gibson-Mount]

--On March 8, 2011 10:20:21 AM -0800 Walter Smith  
wrote:




Hi there!

How severe this bug is?


The CVE gave it 1.39 out of a possible 180 points.  Or < 2%.

It will of course be addressed in a later Zimbra release.

--Quanah

--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc

Zimbra ::  the leader in open source messaging and collaboration

Re: regular expressions was: Kernel Oops

[Erik de Castro Lopo]

Noel Jones wrote:

> Many years ago I worked on a system with a 32k limit on pcre 
> expressions.  Ever since then, everything I've checked has 
> been 64k, and then I gave up checking.  I expect any 
> non-ancient system will support 64k, and some maybe even more. 
>   (To clarify for others following along, this is a characters 
> per single expression limit, not a filesize or number of 
> expressions per file limit)

Thanks for the info.

> > Now compare that against:
> >
> >   /^([0-9]{1,3}\.){4}(foo|bar|baz)$/
> >
> > which will again fail on the fourth character, but there is only one
> > pattern which matches the same strings as the 3 patterns above.
> 
> This example is pretty easy to see that combining is better. 

Exactly. Fortunately this is the very common example that will
very easily lend itself to this optimisation.

> It's not so clear if you create 32k of complex gibberish if it 
> will actually operate faster as there may be significant 
> startup times.  YMMV and all that.

I agree completely.

> BTW, with pcre you should use the the non-greedy flag inside 
> parenthesis if you're not doing $n substitutions.  This saves 
> another smidgen of time and memory.
> /^(?:[0-9]{1,3}\.){4}(?:foo|bar|baz)$/

Good tip, thanks.

Erik
-- 
--
Erik de Castro Lopo
http://www.mega-nerd.com/

Re: STARTTLS bug -- [Zimbra]

[Victor Duchovni]

On Tue, Mar 08, 2011 at 06:26:47PM -0800, Quanah Gibson-Mount wrote:

>
>
> --On March 8, 2011 10:20:21 AM -0800 Walter Smith  
> wrote:
>
>>
>> Hi there!
>>
>> How severe this bug is?
>
> The CVE gave it 1.39 out of a possible 180 points.  Or < 2%.
>
> It will of course be addressed in a later Zimbra release.

Sites operating a STARTTLS submission service for MSAs that use AUTH
PLAIN, may want to upgrade if their users frequent public wifi hotspots,
... where MITM attacks are relatively easy to stage. There may well be
softer targets than port 587, but I think it is prudent to close this
gap. At least one large free mailbox provider has already fixed their
previously vulnerable submission service.

-- 
Viktor.