Re: any api to read logs ?

[Tom Hendrikx]

Hi,

I have a set of grok patterns for logstash. You can send the postfix
logs to logstash, have them parsed into something more or less
structured by the pattersn, then expose the logstash data through some api.

https://github.com/whyscream/postfix-grok-patterns/

Kind regards,
Tom


On 01-10-18 07:47, Илья Шипицин wrote:
> 
> 
> вс, 30 сент. 2018 г. в 4:40, Wietse Venema  >:
> 
> Wietse:
> > > Open a web search engine, ask for for 'logfile analysis tools'.
> 
>  ???:
> > logfile analys is good for human, it is not rest api.
> > I did search already
> 
> Here is an idea: combine log analysis with a web API. End of problem.
> 
> 
> 
> that was what I was going to implement.
> however, I do not like to implement thing that are implemented already.
> so, I did a google search and I asked mailing list.
> 
> seems, I need to implement rest api myself.
> 
> 
> one more question.
> I like text logs. they are fast, and available 100% (comparing, for
> example to some network logging).
> 
> is there a way (I did a seach already) to make logs structured ?
> something like apache access log (field with separator), json, xml, csv ?
> whatever to be machine readable (to make rest api actually read it)
>  
> 
> 
>         Wietse
> 

Re: any api to read logs ?

[Patrick Ben Koetter]

* Илья Шипицин :
> > Here is an idea: combine log analysis with a web API. End of problem.
> 
> that was what I was going to implement.
> however, I do not like to implement thing that are implemented already.
> so, I did a google search and I asked mailing list.
> 
> seems, I need to implement rest api myself.
> 
> 
> one more question.
> I like text logs. they are fast, and available 100% (comparing, for example
> to some network logging).
> 
> is there a way (I did a seach already) to make logs structured ? something
> like apache access log (field with separator), json, xml, csv ?
> whatever to be machine readable (to make rest api actually read it)

There isn't and it is one of the (very few) shortcomings of Postfix. AFAIK
logging – as it is today – was hard wired into the code, which means if you
would like to add a new way to output it, i.e. alternate format, structure,
you would have to work your way through all the code.

p@rick

-- 
[*] sys4 AG
 
https://sys4.de, +49 (89) 30 90 46 64
Schleißheimer Straße 26/MG,80333 München
 
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer, Wolfgang Stief
Aufsichtsratsvorsitzender: Florian Kirstein
 

Re: any api to read logs ?

[Илья Шипицин]

пн, 1 окт. 2018 г. в 12:30, Tom Hendrikx :

> Hi,
>
> I have a set of grok patterns for logstash. You can send the postfix
> logs to logstash, have them parsed into something more or less
> structured by the pattersn, then expose the logstash data through some api.
>
> https://github.com/whyscream/postfix-grok-patterns/


nice.
you saved me a lot of time.


>
>
> Kind regards,
> Tom
>
>
> On 01-10-18 07:47, Илья Шипицин wrote:
> >
> >
> > вс, 30 сент. 2018 г. в 4:40, Wietse Venema  > >:
> >
> > Wietse:
> > > > Open a web search engine, ask for for 'logfile analysis tools'.
> >
> >  ???:
> > > logfile analys is good for human, it is not rest api.
> > > I did search already
> >
> > Here is an idea: combine log analysis with a web API. End of problem.
> >
> >
> >
> > that was what I was going to implement.
> > however, I do not like to implement thing that are implemented already.
> > so, I did a google search and I asked mailing list.
> >
> > seems, I need to implement rest api myself.
> >
> >
> > one more question.
> > I like text logs. they are fast, and available 100% (comparing, for
> > example to some network logging).
> >
> > is there a way (I did a seach already) to make logs structured ?
> > something like apache access log (field with separator), json, xml, csv ?
> > whatever to be machine readable (to make rest api actually read it)
> >
> >
> >
> > Wietse
> >
>

macOS X, Operation not permitted - rename sendmail

[James Brown]

I’ve just tired to install Postfix 3.3.1 on macOS X 10.13.6 High Sierra.

Sudo make install finishes with:

Updating /usr/sbin/sendmail...
mv: rename /Users/jlbrown/Downloads/postfix-3.3.1/junk to /usr/sbin/sendmail: 
Operation not permitted
make: *** [install] Error 1

My make command was:

make -f Makefile.init makefiles CCARGS='-DUSE_TLS -DUSE_SASL_AUTH \
-DDEF_SERVER_SASL_TYPE=\"dovecot\" \
-DDEF_COMMAND_DIR=\"/usr/local/sbin\" \
-DDEF_CONFIG_DIR=\"/usr/local/etc/postfix\" \
-DDEF_DAEMON_DIR=\"/usr/local/libexec/postfix\" \
-DHAS_PCRE -I/usr/local/opt//include \
-DHAS_SSL -I/usr/local/opt/openssl@1.1 \
-DHAS_MYSQL -I/usr/local/opt/mysql@5.7/include/mysql' AUXLIBS='-L/usr/local/lib 
-lpcre -lssl -lcrypto -L/usr/local/opt/mysql@5.7/lib \


Is this a SIP thing?

Anyone come across this? How to fix?

(I’ve been using Homebrew to install the other bits, but there is no formula 
for Postfix)

Thanks,

James.

Re: macOS X, Operation not permitted - rename sendmail

[Viktor Dukhovni]

On Mon, Oct 01, 2018 at 05:56:57PM +1000, James Brown wrote:

> I’ve just tired to install Postfix 3.3.1 on macOS X 10.13.6 High Sierra.
> 
> Sudo make install finishes with:
> 
> Updating /usr/sbin/sendmail...

In MacOSX /usr is immutable, except during upgrade reboots.  You
can't install Postfix in /usr.  You need to build it for installation
in /usr/local.  This also means you can't replace /usr/sbin/sendmail,
but that should not be a problem, since the system-provided sendmail
will write compatible queue files, and the Postfix you build in
/usr/local can use the same queue-directory (owned by the "_postfix"
user and group-writable by "_postdrop").

However, you'll also need to disable the MacOS launchd agent for
the built-in Postfix, which watches the queue directory and starts
Postfix on demand and stops it when the queue is drained.  You'll
need a separate job to run your own Postfix.

MacOS/X is no longer a good platform for running your own Postfix
builds, the other major obstacle is that getting usable logs is is
painfully different.  You're running Postfix on a system that is
not designed to be a server.

-- 
Viktor.

pickup performance

[Raymond Sellars]

Hi


Looking for some hints on a performance problem i have with postfix and looping 
mail through a content filter and it slowly feedback back out via the maildrop.


It would seem that pickup process is trickle feeding the maildrop back into the 
active queue. I conscious that pickup is single thread but for comparison on 
reboot of the match postfix will seemingly take the maildrop items real fast 
and smtp send them out. But if I batch send the items in via postfix input 
queue (from an external smtp server) it goes slow.


Scenario:

Stress test were I send 30-60k of messages to a single domain (not the best 
test but its suits my requirements).

Jame SMTP server -> sends to postfix (25 connections) -> postfix setup with a 
content filter that is a python script (see below) -> reinjects using send mail 
-> pickup trickles messages into active -> the relay (smtp) to another mail 
server but very slowly, i.e 1 a second or there about.

Again. If I reboot the box (restarting postfix doesn't seem to do it) the relay 
sending goes out at 20 connections and really fast.

Snippet from the python script:

# pass message to python script which calls webservice
/usr/bin/python /etc/postfix/custom_scripts/audit/mime.py 
/etc/postfix/custom_scripts/audit/messages/rawmessage_$$.msg "$@"
# || { echo Python script did not execute; exit $EX_UNAVAILABLE; }

# send message back to queue - strip the postfix instance name
/usr/sbin/sendmail -G -i -f "$1" "${@:3}" 

Re: MTA-STS when?

[yarmak]

I have implemented such policy server: it lookups MTA-STS policy, caches and
updates it as RFC 8461 defines.

Github: https://github.com/Snawoot/postfix-mta-sts-resolver
PyPI: https://pypi.org/project/postfix-mta-sts-resolver/

Daemon lacks some features required by standard like proactive policy fetch,
reporting and ratelimit, but it serves its main purpose - TLS policy
discovery. I use it for my personal mailserver. Hope it'll be useful for
someone.




--
Sent from: http://postfix.1071664.n5.nabble.com/Postfix-Users-f2.html

postfix functional testing

[Илья Шипицин]

Hello,

we have pretty complicated setup. when we change something, we can break
something else.
however, we can describe "what must work".

is there a way of describing configuration testing like
https://openresty.gitbooks.io/programming-openresty/content/testing/test-nginx.html
?

cheers,
Ilya Shipitsin

Re: Invalid address is accepted by postfix

[jcdole]

Human guru perhaps not, but programs , spammers, or others may.

So how to reject mail with bad sender addresses that postfix accept.

Any help is welcome.



-
Thank you for helping

Opensuse Leap 15
--
Sent from: http://postfix.1071664.n5.nabble.com/Postfix-Users-f2.html

Re: Invalid address is accepted by postfix

[Wietse Venema]

jcdole:
> Human guru perhaps not, but programs , spammers, or others may.
> 
> So how to reject mail with bad sender addresses that postfix accept.
> 
> Any help is welcome.

Use check_sender_access with PCRE patterns, to retrict the subset
of address syntax that you want to allow.

http://www.postfix.org/postconf.5.html#smtpd_sender_restrictions
http://www.postfix.org/postconf.5.html#check_sender_access
http://www.postfix.org/access.5.html
http://www.postfix.org/pcre_table.5.html
http://www.postfix.org/DATABASE_README.html

Wietse

Re: Invalid address is accepted by postfix

[Ralph Seichter]

On 01.10.18 12:48, jcdole wrote:

> So how to reject mail with bad sender addresses that postfix accept.

A PCRE-based access restriction "/[@!%].*[@!%]/ REJECT" should do it
(untested).

-Ralph

Re: SMTP SNI Support

[vrubiella]

Hello!,

I'm can't see references about SNI in 3.4 release notes, this feature has
been discarded for next release or is WIP?

Thks!





--
Sent from: http://postfix.1071664.n5.nabble.com/Postfix-Users-f2.html

Re: SMTP SNI Support

[Wietse Venema]

vrubiella:
> Hello!,
> 
> I'm can't see references about SNI in 3.4 release notes, this feature has
> been discarded for next release or is WIP?

It's in progress. We don't announce things until they exist.

Wietse

smptd_tls_security_level = encrypt

[Virtual Xmas]

Hello,

Running Postfix 2.10.1.

I am setting up an internal mail relay to receive mail from other 
internal clients.  I have a requirement that all email be received via 
TLS only.


I have configured TLS using our internal PKI and set the appropriate 
settings in main.cf and mail is being received via TLS according to the 
headers.


I have set smptd_tls_security_level = encrypt.  According to the 
documentation:


encrypt: Mandatory TLS encryption: announce STARTTLS support to remote 
SMTP clients, and require that clients use TLS encryption.


However, the server is still willing to accept non TLS unencrypted 
emails from smtp clients.


Am I still missing a setting?

Getting quotes for MTA-STS implementation (was: MTA-STS when?)

[Paul Menzel]

Dear Postfix folks,


On 02/19/18 20:11, Wietse Venema wrote:
> Jonathan Sélea:
 [...].  One can of course automate periodic SMTP TLS policy
 updates from the STS URIs of a handful of providers, and let the
 usual outbound TLS policy take care of the rest:

http://www.postfix.org/TLS_README.html#client_tls_policy
>>> I'm much in favor of reusing the Postfix SMTP client's TLS policy
>>> lookup mechanism for this, for example
>>>
>>> smtp_policy_maps = socketmap:inet:host:port:name
>>>
>>> and to extend the policy map feature set as needed.
>>>
>>> If the (key, value) interface turns out to be too restrictive, this
>>> interface could be generalized towards something like the SMTP
>>> server access policy delegation protocol (possibly with multiple
>>> commands, multiple request attributes, or multiple reply attributes).
>>>
>>> Like DKIM/DMARC I do not think that complex policies like STS should
>>> be built into core Postfix SMTP components.
>>>
>>
>> It sounds like it is a fairly "easy" implementation? If so, when can
>> expect a testing version for this?
> 
> By my estimate this would involve multiple weeks of sustained effort
> by a highly-skilled person. The elapsed time would be considerably
> longer because Postfix maintainers have real jobs, don't take time
> off to do work on the side, and STS development would compete with
> other Postfix development. I would not even estimate the year of
> completion.
> 
> The bulk of Postfix SMTPUTF8 support was done by a developer who
> acquired sponsorship from CNNIC (I spent some time to make it nice).
> If you have 10 grand lying around, maybe you can find someone.

$10.000 doesn’t seem a lot judging from all the companies and
organizations using Postfix in their critical infrastructure.

Are Postfix developers and companies listed somewhere, which could
give a quote for the implementation?

If not, could interested people please reply with their contact
detail?


Kind regards,

Paul



smime.p7s
Description: S/MIME Cryptographic Signature

Re: MTA-STS when?

[Wietse Venema]

yarmak:
> I have implemented such policy server: it lookups MTA-STS policy, caches and
> updates it as RFC 8461 defines.
> 
> Github: https://github.com/Snawoot/postfix-mta-sts-resolver
> PyPI: https://pypi.org/project/postfix-mta-sts-resolver/
> 
> Daemon lacks some features required by standard like proactive policy fetch,
> reporting and ratelimit, but it serves its main purpose - TLS policy
> discovery. I use it for my personal mailserver. Hope it'll be useful for
> someone.

This may be a good start. Thanks.

Wietse

Re: Getting quotes for MTA-STS implementation (was: MTA-STS when?)

[Wietse Venema]

Paul Menzel:
> Dear Postfix folks,
> 
> 
> On 02/19/18 20:11, Wietse Venema wrote:
> > Jonathan S?lea:
>  [...].  One can of course automate periodic SMTP TLS policy
>  updates from the STS URIs of a handful of providers, and let the
>  usual outbound TLS policy take care of the rest:
> 
> http://www.postfix.org/TLS_README.html#client_tls_policy
> >>> I'm much in favor of reusing the Postfix SMTP client's TLS policy
> >>> lookup mechanism for this, for example
> >>>
> >>> smtp_policy_maps = socketmap:inet:host:port:name
> >>>
> >>> and to extend the policy map feature set as needed.
> >>>
> >>> If the (key, value) interface turns out to be too restrictive, this
> >>> interface could be generalized towards something like the SMTP
> >>> server access policy delegation protocol (possibly with multiple
> >>> commands, multiple request attributes, or multiple reply attributes).
> >>>
> >>> Like DKIM/DMARC I do not think that complex policies like STS should
> >>> be built into core Postfix SMTP components.
> >>>
> >>
> >> It sounds like it is a fairly "easy" implementation? If so, when can
> >> expect a testing version for this?
> > 
> > By my estimate this would involve multiple weeks of sustained effort
> > by a highly-skilled person. The elapsed time would be considerably
> > longer because Postfix maintainers have real jobs, don't take time
> > off to do work on the side, and STS development would compete with
> > other Postfix development. I would not even estimate the year of
> > completion.
> > 
> > The bulk of Postfix SMTPUTF8 support was done by a developer who
> > acquired sponsorship from CNNIC (I spent some time to make it nice).
> > If you have 10 grand lying around, maybe you can find someone.
> 
> $10.000 doesn?t seem a lot judging from all the companies and
> organizations using Postfix in their critical infrastructure.
> 
> Are Postfix developers and companies listed somewhere, which could
> give a quote for the implementation?

The two developers are fully employed and can't take money; if
someone can provide a viable design, then I think that we would
consider it. This could be hashed out on the postfix-devel list.

> If not, could interested people please reply with their contact
> detail?

Wietse

Re: smptd_tls_security_level = encrypt

[Wietse Venema]

Virtual Xmas:
> Hello,
> 
> Running Postfix 2.10.1.
> 
> I am setting up an internal mail relay to receive mail from other 
> internal clients.? I have a requirement that all email be received via 
> TLS only.
> 
> I have configured TLS using our internal PKI and set the appropriate 
> settings in main.cf and mail is being received via TLS according to the 
> headers.
> 
> I have set smptd_tls_security_level = encrypt.? According to the 
> documentation:
> 
> encrypt: Mandatory TLS encryption: announce STARTTLS support to remote 
> SMTP clients, and require that clients use TLS encryption.
> 
> However, the server is still willing to accept non TLS unencrypted 
> emails from smtp clients.
> 
> Am I still missing a setting?

TO REPORT A PROBLEM see http://www.postfix.org/DEBUG_README.html#mail

TO (UN)SUBSCRIBE see http://www.postfix.org/lists.html

Thank you for using Postfix.

Re: macOS X, Operation not permitted - rename sendmail

[Larry Stone]

> On Oct 1, 2018, at 3:13 AM, Viktor Dukhovni  
> wrote:
> 
> On Mon, Oct 01, 2018 at 05:56:57PM +1000, James Brown wrote:
> 
>> I’ve just tired to install Postfix 3.3.1 on macOS X 10.13.6 High Sierra.
>> 
>> Sudo make install finishes with:
>> 
>> Updating /usr/sbin/sendmail...
> 
> In MacOSX /usr is immutable, except during upgrade reboots.  You
> can't install Postfix in /usr.  You need to build it for installation
> in /usr/local.  This also means you can't replace /usr/sbin/sendmail,

Not quite. If you turn off SIP (System Integrity Protection), you can modify 
/usr. I’ve been running with SIP off since shortly after Apple added that 
feature. So far, they haven’t added anything that gets upset with you for doing 
so. Although when Apple had their hands on my MacBookPro to replace the 
battery, I found they turned it back on.

> MacOS/X is no longer a good platform for running your own Postfix
> builds, the other major obstacle is that getting usable logs is is
> painfully different.  You're running Postfix on a system that is
> not designed to be a server.

Agree. As I like to say, Apple thinks they know best how you should be using 
their products - there’s the “Apple Way” and the “wrong way” with nothing in 
between.

I build Postfix (which I use only for outbound system messages) on an old MacOS 
10.9 system and then transfer the build. That keeps logging working the “right” 
way but is obviously not a long-term viable solution. Not concerned about 
having the latest and greatest Postfix since it’s not externally accessible.

-- 
Larry Stone
lston...@stonejongleux.com

Re: Invalid address is accepted by postfix

[jcdole]

That does not work with the data I have use to make a test.
This is the reason I open a thread.

email-address :: local-p...@example.com
but if local-part :: local-p...@example.com ==>
local-p...@example.com@example.com

Now the lookup table

local-p...@example.comOK

*postmap -q "local-p...@example.com@example.com"
type_of_table:/etc/postfix/lookup_table
return null*

postmap -q "local-p...@example2.com@example.com"
type_of_table:/etc/postfix/lookup_table
return null

postmap -q "local-p...@example.com@example2.com"
type_of_table:/etc/postfix/lookup_table
return null

but postfix lookup table
*return OK for local-p...@example.com@example.com*
return not found for local-p...@example2.com@example.com
return not found for local-p...@example.com@example2.com

I have made 2 other tests

One with :
local-p...@example.com*.*@example.com ( see the dot before the second @ )
postmap -q return null
but postfix lookup return OK

The second one with :
local-p...@example.com*.a*@example.com ( see  ".a" before the second @ )
postmap -q return null
and postfix lookup return not found.

During ma test, I noticed that  a@b@cc (from the sender address)
was tagged  as "a@b"@cc.

If the `c` part is different from the domain part in the loookup table,
the sender address is rejected
If the `` part is different from the domain part in the loookup table,
the sender address is rejected

Any help is welcome.




-
Thank you for helping

Opensuse Leap 15
--
Sent from: http://postfix.1071664.n5.nabble.com/Postfix-Users-f2.html

Re: macOS X, Operation not permitted - rename sendmail

[Bill Cole]

On 1 Oct 2018, at 3:56, James Brown wrote:


Is this a SIP thing?


Yes. As Viktor said, Apple-populated paths under /usr are immutable 
while SIP is enabled.



Anyone come across this? How to fix?


Don't fight it. As Viktor said, you can adjust your build to NOT replace 
the system 'sendmail' (which is Apple's custom-built Postfix 3.2.2.) 
Even if you disable SIP to do that replacement, you will not be informed 
when a future minor system update re-installs Apple's binary.


If you don't feel like cobbling up your own build under the Homebrew 
model, MacPorts has a mature flexible port for Postfix, along with all 
of its dependencies and possible dependencies (e.g. Dovecot, if you're 
using it's SASL layer.)


And of course you CAN just try to work with Apple's build and buy their 
additions (not sure what those are exactly any more...) at the cost of 
their compile-time config choices. Everything you need to configure it 
is right there in /etc/postfix/...

Re: smptd_tls_security_level = encrypt

[Matus UHLAR - fantomas]

On 01.10.18 10:21, Virtual Xmas wrote:

Running Postfix 2.10.1.

I am setting up an internal mail relay to receive mail from other 
internal clients.  I have a requirement that all email be received via 
TLS only.


I have configured TLS using our internal PKI and set the appropriate 
settings in main.cf and mail is being received via TLS according to 
the headers.


I have set smptd_tls_security_level = encrypt.  According to the 
documentation:


encrypt: Mandatory TLS encryption: announce STARTTLS support to remote 
SMTP clients, and require that clients use TLS encryption.


However, the server is still willing to accept non TLS unencrypted 
emails from smtp clients.


Am I still missing a setting?


something overridden in master.cf?


--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Eagles may soar, but weasels don't get sucked into jet engines. 

Re: postfix functional testing

[Wietse Venema]

 ???:
> Hello,
> 
> we have pretty complicated setup. when we change something, we can break
> something else.  however, we can describe "what must work".

In the case of email, this is usually tested by sending email and
monitoring one or more destination mailboxes, to determine if the
message is delivered in the expected time and with the expected
content.

> is there a way of describing configuration testing like
> https://openresty.gitbooks.io/programming-openresty/content/testing/test-nginx.html
> ?

SMTP is a store-and-forward protocol, therefore server responses
alone cover only a small part of a complete email transaction.

Wietse