filtering locally submitted emails / tidying up the config
Hello, I've been using postfix with great success and delight for many years now, but my config has become quite messy over time, with additions like dkim/spf/dmarc and various filtering options. I'm currently having a problem with some locally generated emails (eg. sendmail command). I would like to filter them with amavisd-new but my current setup does not allow that as I'm filtering only incoming emails. I'm also running fetchmail for some user on the server, and I don't want this mail flow to be filtered. I really feel like my setup needs some cleanup and I'm not sure were to start. Some of you have suggested to me, in the past, that I could move to amavisd-milter instead of smtpd_proxy_filter for antispam/antivirus filtering. It might be the way but I don't know if it can help me with my new need. I've tried to force locally generated email filtering with an after-queue Amavisd setup using something really ugly (I guess): __ pickupfifo n - n 60 1 pickup -o content_filter=smtp-amavis:[127.0.0.1]:10024 smtp-amavis unix- - - - 2 smtp -o smtp_data_done_timeout=1200 -o smtp_send_xforward_command=yes -o disable_dns_lookups=yes -o max_use=20 __ It worked but emails reentering the after-filter SMTP server 127.0.0.1:10025 triggered OpenDMARC failures, sending reports to me and adding a failed authentication header to email messages. It looks like an unhealthy hack :/ Here is my setup: __ master.cf __ # # Postfix master process configuration file. For details on the format # of the file, see the master(5) manual page (command: "man 5 master"). # # == # service type private unpriv chroot wakeup maxproc command + args # (yes) (yes) (yes) (never) (100) # == # Before-filter SMTP server. Receive mail from the network and # pass it to the content filter on localhost port 10025. # smtpd pass - - n - - smtpd -o smtpd_proxy_filter=127.0.0.1:10024 -o smtpd_client_connection_count_limit=10 -o smtpd_proxy_ehlo=amavis-filtering -o disable_mime_output_conversion=yes -o smtpd_milters=unix:/var/milter-greylist/milter-greylist.sock -o non_smtpd_milters= IP.AD.DR.ESS:smtp inet n - n - 1 postscreen dnsblog unix - - n - 0 dnsblog tlsproxy unix - - n - 0 tlsproxy policyd-spf unix - n n - 0 spawn user=nobody argv=/usr/local/bin/policyd-spf # inside smtp without before-queue (because of milter) # 127.0.0.1:smtp inet n - n - 5 smtpd -o smtpd_client_connection_count_limit=10 -o disable_mime_output_conversion=yes -o smtpd_milters=inet:127.0.0.1:8891 -o non_smtpd_milters=inet:127.0.0.1:8891 # # After-filter SMTP server. Receive mail from the content filter # on localhost port 10025. # 127.0.0.1:10025 inet n - n -- smtpd -o smtpd_authorized_xforward_hosts=127.0.0.0/8 -o smtpd_client_restrictions= -o smtpd_helo_restrictions= -o smtpd_sender_restrictions= -o smtpd_recipient_restrictions=permit_mynetworks,reject -o smtpd_data_restrictions= -o mynetworks=127.0.0.0/8 -o receive_override_options=no_unknown_recipient_checks -o smtpd_milters=inet:localhost:8893 -o non_smtpd_milters=inet:localhost:8893 submission inet n - n - - smtpd -o smtpd_sasl_auth_enable=yes -o smtpd_sasl_type=dovecot -o smtpd_tls_security_level=encrypt -o smtpd_tls_auth_only=yes -o smtpd_client_restrictions=permit_sasl_authenticated,reject -o smtpd_sasl_path=inet:127.0.0.1:12345 -o smtpd_sasl_tls_security_options=noanonymous -o smtpd_client_auth_rate_limit=2 -o smtpd_tls_cert_file=/usr/local/etc/letsencrypt/live/host.name/fullchain.pem -o smtpd_tls_key_file=/usr/local/etc/letsencrypt/live/host.name/privkey.pem -o smtpd_milters=inet:127.0.0.1:8891 -o non_smtpd_milters=inet:127.0.0.1:8891 -o smtpd_client_restrictions=permit_sasl_authenticated,reject -o smtpd_recipient_restrictions=permit_sasl_authenticated,reject -o syslog_name=postfix/submission #smtpsinet n - n - - smtpd # -o smtpd_tls_wrappermode=yes # -o smtpd_sasl_auth_enable=yes # -o smtpd_client_restrictions=permit_sasl_authenticated,reject #628 inet n - n - - qmqpd pickupfifo n - n 60 1 pickup cleanup unix n - n - 0 cleanup qmgr fifo n - n 300 1 qmgr #qmgr fifo n - n 300 1 oqmgr tlsmgr
Re: filtering locally submitted emails / tidying up the config
Patrick Proniewski: > I really feel like my setup needs some cleanup and I'm not sure were to > start. Some of you have suggested to me, in the past, that I could move to > amavisd-milter instead of smtpd_proxy_filter for antispam/antivirus > filtering. It might be the way but I don't know if it can help me with my new > need. It may well be: - You can use amavisd-milter to filter inbound SMTP mail (with smtpd_milters) and /usr/sbin/sendmail local submissions (with non_smtpd_milters). The two settings can be different if needed. - Assuming that amavisd supports per-user configuration, it can be told not to touch email for your fetchmail user. - Otherwise, maybe you can configure fetchmail to talk to a dedicated smtpd entry in master.cf that has "-o smtpd_milters=". i.e. no filter. Wietse
Re: filtering locally submitted emails / tidying up the config
Thanks Wietse, > On 01 mai 2020, at 15:37, Wietse Venema wrote: > > Patrick Proniewski: >> I really feel like my setup needs some cleanup and I'm not sure were to >> start. Some of you have suggested to me, in the past, that I could move to >> amavisd-milter instead of smtpd_proxy_filter for antispam/antivirus >> filtering. It might be the way but I don't know if it can help me with my >> new need. > > It may well be: > > - You can use amavisd-milter to filter inbound SMTP mail (with > smtpd_milters) and /usr/sbin/sendmail local submissions (with > non_smtpd_milters). The two settings can be different if needed. would there be a performance/functionality penalty switching from smtpd_proxy_filter to smtpd_milters/non_smtpd_milters? > > - Assuming that amavisd supports per-user configuration, it > can be told not to touch email for your fetchmail user. > > - Otherwise, maybe you can configure fetchmail to talk to a dedicated > smtpd entry in master.cf that has "-o smtpd_milters=". i.e. no filter. not sure about the per-user, but I can definitely do a lot of plumbing inside amavisd, or apparently tune fetchmail to deliver into a dedicated smtp. thanks, patpro
Re: filtering locally submitted emails / tidying up the config
Patrick Proniewski: > Thanks Wietse, > > > On 01 mai 2020, at 15:37, Wietse Venema wrote: > > > > Patrick Proniewski: > >> I really feel like my setup needs some cleanup and I'm not sure were to > >> start. Some of you have suggested to me, in the past, that I could move to > >> amavisd-milter instead of smtpd_proxy_filter for antispam/antivirus > >> filtering. It might be the way but I don't know if it can help me with my > >> new need. > > > > It may well be: > > > > - You can use amavisd-milter to filter inbound SMTP mail (with > > smtpd_milters) and /usr/sbin/sendmail local submissions (with > > non_smtpd_milters). The two settings can be different if needed. > > would there be a performance/functionality penalty switching from > smtpd_proxy_filter to smtpd_milters/non_smtpd_milters? It will be faster, because it avoids the need to deliver the whole message outside of Postfix, and then to receive the whole message again into Postfix. Also, the logging will be easier to follow because there is only one queue file instead of two (one before amavisd and one after amavisd). > > - Assuming that amavisd supports per-user configuration, it > > can be told not to touch email for your fetchmail user. > > > > - Otherwise, maybe you can configure fetchmail to talk to a dedicated > > smtpd entry in master.cf that has "-o smtpd_milters=". i.e. no filter. > > not sure about the per-user, but I can definitely do a lot of > plumbing inside amavisd, or apparently tune fetchmail to deliver > into a dedicated smtp. Wietse
RE: AUTH Messages in log
On 30 Apr 2020, at 10:52, Keith wrote: Have enabled postscreen as per docs, one thing as this machine does not have TLS or any type of auth enabled, so I did not enable tlsproxy in master.cf. Doesn't seem to have affected things and I had to manually null a .eu IP that would not stop auth after the reload. So far looks ok and have not seen any AUTH attempts since then, or at least for the last hour. > Using version 3.2.2 under FreeBSD 11. >The current version of 3.2 is 3.2.12, so update at least to that. Will look at that in the next few days. > There are a lot of this in the log as bots etc try to AUTH on port 25. Is > there a way to turn this off or at least not have it scattered in the logs? >99.9% of these have no hostname associated with the IP. >Two things you should do >1) Stop allowing auth connections on port 25 at all This is controlled via master.cf using -o options under smtp? >2) use postscreen >Also, when you saw there are a lot of these attempts, how many are we talking >about? >Is it preventing legitimate access and making the server load so high you are >losing mail, or is >this just a reaction to looking at the logs? Yesterday's log file counted 1500+. Server load is negligible and has lots of room and mostly a knee jerk to seeing a crap to in logs scrolling by. Sometimes for a few mins that’s all I see. One last, is pflogsumm still a valid tool for log stats? Thanks very much for the pointers. So far this appears to have cleaned things up. Keith
Re: AUTH Messages in log
Keith: > >1) Stop allowing auth connections on port 25 at all > > This is controlled via master.cf using -o options under smtp? It would go under the smtpd that serves port 25, as smtp pass ... ... ... ... ... smtpd -o smtpd_sasl_auth_enable=no > >2) use postscreen > > One last, is pflogsumm still a valid tool for log stats? There is a version that supports postscreen (for example debian). Wietse
RE: AUTH Messages in log
Keith: > >1) Stop allowing auth connections on port 25 at all > > This is controlled via master.cf using -o options under smtp? >>It would go under the smtpd that serves port 25, as >>smtp pass ... ... ... ... ... smtpd >> -o smtpd_sasl_auth_enable=no That's what I thought. > >2) use postscreen > > One last, is pflogsumm still a valid tool for log stats? >>There is a version that supports postscreen (for example debian). Will take a look. I'm just wondering if all the changes I made are working. But overall it doesn't seem to be blocking what appears to be legit email and the logs look cleaner. Thanks. Keith
Re: Replace null sender addresses?
Wietse Venema wrote: > Jason Bailey: > > I've got notification emails from a legacy system passing through a > > Postfix install I'm using to relay messages to the proper outbound Are the notification mails coming from an internal system? That's okay. But why are the recipients undeliverable? If it is valid and the internal system is generating a bounce message from the MAILER-DAEMON (which uses <> as the MAIL FROM address) then that is operating correctly too. Who is getting the bounce messages? That admin receiving the bounce messages should then know that something is broken and go fix it so that those notification messages can be delivered to the address of an admin designated to receive those notifications. Or those notifications should be disabled to prevent the bounces. Bounce messages internally on the LAN are okay. It's all people living in the same house. They are simply notifications. Bounce messages to the outside world to innocent 3rd parties are problematic however. > > server. Things are working great except occasionally messages are sent > > from the legacy system with a null sender address (e.g. "MAIL > > FROM: <>"). > > These are normally sent when an email address was undeliverable. To clarify a little bit... It would be an error to map <> to a non-bounce address as that would likely create conditions for an infinite loop. Loop avoidance depends upon a bounce of an address from the MAILER-DAEMON <> being dropped in order to prevent mail loops continuing infinitely. The problem description. A mail relay accepts an undeliverable message and tries to relay it onward. Can't. Generates a bounce message. Bounce messages are from MAILER-DAEMON <> by design. If *that* message, the bounce message, cannot be delivered then we don't want it to generate *another* bounce message. That would be bad. It would be infinite. Therefore if delivery from MAILER-DAEMON <> fails the message is simply discarded in order to avoid an infinite loop. In general the situation of generating bounce messages should be avoided whenever possible. Because most spam uses forged from addresses and any bounce message would become "backscatter" spam to innocent 3rd parties. They should validly report you as a spammer for generating the backscatter spam. Therefore messages from the outside world inbound to your network that are undeliverable should be rejected at SMTP time. That avoids the creation of a later bounce message and avoids the possibility of becoming a backscatter spam source. It is the spammer that is connecting at SMTP time and rejecting the message then rejects the spam at the spammer directly. > The correct fix is to configure the inbound mail relay to not accept > mail for undeliverable recipients. This is what the Postfix features > relay_recipient_maps or reject_unverified_recipient are used for. > > > Is there a way to get Postfix to replace those null sender addresses > > with a valid, predefined address before Postfix forwards the messages > > to its smart host? > > No. Fix the right problem: don't accept mail for an invalid address. For example if you must have a machine A accept mail for another internally connected machine B then on machine A one needs to have a list of every valid email address for which mail can be delivered. Let's say I have an internal system foo.example.net and it has the following valid recipients. ab...@example.net postmas...@example.net al...@example.net b...@example.net I would convert that into a relay_recipient_maps like this: ab...@example.net OK postmas...@example.net OK al...@example.net OK b...@example.net OK Then update the associated relay_recipient.foo.example.net.db file. # postmap relay_recipient.foo.example.net Then configure main.cf like this: relay_recipient_maps = hash:/etc/postfix/relay_recipient.foo.example.net Then when spammers try to send spam to mall...@example.net to the system the system looks up the address in the relay_recipient_maps to see if it is valid. If so then it would accept the message. If not, and here it is not a valid addres, then it is rejected at SMTP time. No bounce message is created. Hope that helps clarify things. Bob
Re: Replace null sender addresses?
On May 1, 2020 8:13 PM, Bob Proulx wrote: Wietse Venema wrote: > Jason Bailey: > > I've got notification emails from a legacy system passing through a > > Postfix install I'm using to relay messages to the proper outbound Are the notification mails coming from an internal system? That's okay. But why are the recipients undeliverable? If it is valid and the internal system is generating a bounce message from the MAILER-DAEMON (which uses <> as the MAIL FROM address) then that is operating correctly too. Who is getting the bounce messages? That admin receiving the bounce messages should then know that something is broken and go fix it so that those notification messages can be delivered to the address of an admin designated to receive those notifications. Or those notifications should be disabled to prevent the bounces. Bounce messages internally on the LAN are okay. It's all people living in the same house. They are simply notifications. Bounce messages to the outside world to innocent 3rd parties are problematic however. > > server. Things are working great except occasionally messages are sent > > from the legacy system with a null sender address (e.g. "MAIL > > FROM: <>"). > > These are normally sent when an email address was undeliverable. To clarify a little bit... It would be an error to map <> to a non-bounce address as that would likely create conditions for an infinite loop. Loop avoidance depends upon a bounce of an address from the MAILER-DAEMON <> being dropped in order to prevent mail loops continuing infinitely. The problem description. A mail relay accepts an undeliverable message and tries to relay it onward. Can't. Generates a bounce message. Bounce messages are from MAILER-DAEMON <> by design. If *that* message, the bounce message, cannot be delivered then we don't want it to generate *another* bounce message. That would be bad. It would be infinite. Therefore if delivery from MAILER-DAEMON <> fails the message is simply discarded in order to avoid an infinite loop. In general the situation of generating bounce messages should be avoided whenever possible. Because most spam uses forged from addresses and any bounce message would become "backscatter" spam to innocent 3rd parties. They should validly report you as a spammer for generating the backscatter spam. Therefore messages from the outside world inbound to your network that are undeliverable should be rejected at SMTP time. That avoids the creation of a later bounce message and avoids the possibility of becoming a backscatter spam source. It is the spammer that is connecting at SMTP time and rejecting the message then rejects the spam at the spammer directly. > The correct fix is to configure the inbound mail relay to not accept > mail for undeliverable recipients. This is what the Postfix features > relay_recipient_maps or reject_unverified_recipient are used for. > > > Is there a way to get Postfix to replace those null sender addresses > > with a valid, predefined address before Postfix forwards the messages > > to its smart host? > > No. Fix the right problem: don't accept mail for an invalid address. For example if you must have a machine A accept mail for another internally connected machine B then on machine A one needs to have a list of every valid email address for which mail can be delivered. Let's say I have an internal system foo.example.net and it has the following valid recipients. ab...@example.net postmas...@example.net al...@example.net b...@example.net I would convert that into a relay_recipient_maps like this: ab...@example.net OK postmas...@example.net OK al...@example.net OK b...@example.net OK Then update the associated relay_recipient.foo.example.net.db file. # postmap relay_recipient.foo.example.net Then configure main.cf like this: relay_recipient_maps = hash:/etc/postfix/relay_recipient.foo.example.net Then when spammers try to send spam to mall...@example.net to the system the system looks up the address in the relay_recipient_maps to see if it is valid. If so then it would accept the message. If not, and here it is not a valid addres, then it is rejected at SMTP time. No bounce message is created. Hope that helps clarify things. Bob On May 1, 2020 8:13 PM, Bob Proulx wrote: Wietse Venema wrote: > Jason Bailey: > > I've got notification emails from a legacy system passing through a > > Postfix install I'm using to relay messages to the proper outbound Are the notification mails coming from an internal system? That's okay. But why are the recipients undeliverable? If it is valid and the internal system is generating a bounce message from the MAILER-DAEMON (which uses <> as the MAIL FROM address) then that is operating correctly too. Who is getting the bounce messages? That admin receiving the bounce messages should then know that something is broken and go fix it so that those notification messages can be delivered to the address of an
Re: Replace null sender addresses?
Hello Jason, I do not know why but you sent out three copies of my email message in full unquoted as a reply. And then in between the 2nd and 3rd copies you wrote your own response. That made the message rather unreadable. Please in the future double check the message before hitting send. Jason Bailey wrote: > It is indeed being generated internally. The RCPT TO is there, but > because it lacks a MAIL FROM, we are seeing some email providers > drop the message, presumably because it looks like UCE/spam. "some email providers"? That makes it sound like you are generating bounces to random places on the net. That makes it sound like you have an open relay problem. > We are trying to get the manufacturer of the system to acknowledge > the problem and address it, but they're currently insisting there's > no problem. In the mean time I've got important emails that aren't > getting delivered. Your description is too vague to be useful. Details, or it didn't happen. > I was hoping to get Postfix to fill it in so that the resulting > email traversing the public Internet was standards compliant and > less likely to get filtered by someone else's UCE solution. Bounce messages from the MAILER-DAEMON using <> *are* standards compliant. That isn't the problem. It is definitely not the first problem to be solved. The first problem to be solved is to debug and fix why you are generating those bounce messages. > To be clear, this system sends out two sets of email. The first > includes all the proper headers and is not an issue. The second set > is missing the MAIL FROM all together. I am glad you are happy with the first set. That's great. That second set you as you have described them so far match the description of a bounce message. Bounce messages will contain a body part with the original message. And there will be an explanation of the action which generated it. Reading those messages will say what is happening. It is also possible, and perhaps even likely given what we know now, that you have a system that is an open relay and has been found by spammers who are exploiting it. Spammers often use forged from addresses. Which if so would explain why you are getting rejects from those email providers. Bob
