"Delayed Mail" notification
Hello, when I receive "Delayed Mail" notification, the message only has the header of the original email. Is it possible to have the full email body included in the notification message, so that I can see which email has not been delivered? I am dealing with a remote mail server, which uses some kind of greylisting, and when I send more emails within given period, it blocks my IP address temporarily. These emails are queued on my side and eventually get delivered. But I think it would be useful for me, when I receive the notification, to see which email exactly was not delivered. thank you,
Re: "Delayed Mail" notification
Fourhundred Thecat: > Hello, > > when I receive "Delayed Mail" notification, the message only has the > header of the original email. > > Is it possible to have the full email body included in the notification > message, so that I can see which email has not been delivered? > > I am dealing with a remote mail server, which uses some kind of > greylisting, and when I send more emails within given period, it blocks > my IP address temporarily. > > These emails are queued on my side and eventually get delivered. > > But I think it would be useful for me, when I receive the notification, > to see which email exactly was not delivered. This scenario, where Postfix receives one message and delivers two, is called message multiplication. The headers-only notification is a safety mechanism that avoids doubling the amount of mail in the queue. For an example of a multiplication disaster see: https://video.disney.com/watch/sorcerer-s-apprentice-fantasia-4ea9ebc01a74ea59a5867853 The delay notification contains a lot of infomation: The envelope recipient and server respnse (twice) The envelope sender The message header including the message ID, date You can correlate that with information in the maillog file. Since the messages are in your local queue, you can see the above information with the command mailq (postqueue -p), and more with "postcat -q" if you must see the content. Wietse
Re: Spammer succeeded in relaying through my server
It's good that you're willing to learn, but perhaps you can do some experiments on a "internal" environment before exposing it to the full internet. Wietse
Re: Spammer succeeded in relaying through my server
Actually I would appreciate advice on how to do this on an internal environment. Is there a way to do this, like tools? The challenge is that I need an external email client to check IP addresses through the proxy, do the TLS communication, etc. My plan is to completely cut off relaying by setting mynetworks to localhost and mynetworks_style to host, then only launch the containers for short periods of time while I do the experiments. The containers will be running in foreground, which means I'll be seeing what's going on constantly. Finally, if I succeed with all that, I'll set the DNS records of my main domain, and wait for them to update. Once they're updated, I'll go to mxtoolbox and ensure that I don't have an open relay, confirm that that SPF and other things are fine. Best regards, Sam On 22/12/2022 6:48 PM, Wietse Venema wrote: It's good that you're willing to learn, but perhaps you can do some experiments on a "internal" environment before exposing it to the full internet. Wietse
Re: Spammer succeeded in relaying through my server
Samer Afach: > Actually I would appreciate advice on how to do this on an internal > environment. Is there a way to do this, like tools? The challenge is > that I need an external email client to check IP addresses through the > proxy, do the TLS communication, etc. My plan is to completely cut off > relaying by setting mynetworks to localhost and mynetworks_style to > host, then only launch the containers for short periods of time while I > do the experiments. The containers will be running in foreground, which > means I'll be seeing what's going on constantly. Finally, if I succeed > with all that, I'll set the DNS records of my main domain, and wait for > them to update. Once they're updated, I'll go to mxtoolbox and ensure > that I don't have an open relay, confirm that that SPF and other things > are fine. With Docker containers, you could run a "non-production" environment anywhere, even in a VM on a laptop. Wietse
Re: Spammer succeeded in relaying through my server
Actually I would appreciate advice on how to do this on an internal environment. google's your friend on this one https://www.google.com/search?q=postfix+docker+testing first link, https://fabianlee.org/2019/10/23/docker-running-a-postfix-container-for-testing-mail-during-development/ looks like a reasonable place to start start simple, demonstrate basic setup works inbound & outbound, then worry about adding proxies, etc, and finally opening yourself up to the 'net -- and the 'net to you.
Re: Spammer succeeded in relaying through my server
Am 22.12.22 um 01:47 schrieb Samer Afach: Thank you, Matthias, for your opinion. Ironically, the proxy part in this whole story is not only the simple part that I fully understand, but also the part that's very easily testable with my current knowledge and understanding of networking. It's easy to look into the IP addresses of incoming connections and cross-check them with the senders. It's easy to see whether a PROXY protocol with HAProxy works. I'm looking into that setup as we speak. All that is easy. I'm actually surprised you're using that part as argument to why I should shutdown my server. Thanks to the wonderful people in this email list, now I understand my mistake with that regard. I learned a lot. If I were you, I'd focus on my lack of understanding of the email protocol. Now that, is a part that I still cannot fully understand, embarrassingly so. I still don't know what ehlo means, except that it's the first message. I don't know why it matters what address we put after it. That does make me look like an idiot, doesn't it? :-) I fully understand your opinion and why you'd want me to shutdown my server. But if I were you, I'd try to encourage people to learn more or explain more to them. This whole incident reminds me of people who fall for scams and pay scammers money. They can be ashamed of it and never talk about it, or try to learn how to not fall for it again. Just my humble opinion. Sam, your proxy is what underminded Postfix's relay guards, by your own assessments and descriptions - because it was the proxy that connected with its own IP address to Postfix's smtpd, and $mynetworks subnets are usually permitted to relay. So while you may have understood how the proxy itself works, you have not identified the implications for operating a mail server proofed against unintended relaying. Reading Postfix's documentation thoroughly would have brought you across XCLIENT (and I was saying application-level or -layer proxy because it talks SMTP not TCP), and you would have asked specific questions. And why does your server have to go online BEFORE you have learned the ropes? I did not ask you to shut down permanently (that should have been clear, but you are getting defensive so I may have stepped on your toes after you stepped on the public's), but implied to first learn and then go online with a safe and possibly reviewed configuration. The public constantly has to put up with people who experiment in public and ask for help *after* accidents. Not sure where such dangerous approaches are taught. Postfix has been around for a quarter century, and it has always been *the* example of good design, documentation and compatibility with predecessor versions. I would really appreciate if people in general learned BEFORE putting systems live. Cheers, Matthias Cheers, Sam On 21/12/2022 10:21 PM, Matthias Andree wrote: Am 21.12.22 um 09:45 schrieb Samer Afach: Thank you for these hints, Benny. I wanna point out that I'm, in no way, an expert in any of this, and my configuration is based on online research and some copy/paste. Then with all due respect, please shut down your mail server and do not start it again until you have fully understood what your services are doing, and need to do instead in order to be operated securely. We don't need more public accidents. Postfix has ways to let application-level proxies convey actual client IP (XCLIENT). You will want those in the logs, and not your proxy's IP address. Sooner or later you will find it necessary to at least temporarily block offending subnets. If your proxy can't do that, or you don't know how, or you don't know how to do that with containers, perhaps use postscreen instead and run your Postfix on a bridged networked container of VM instead. I do not believe you (personally - no offense, but after your statement above) are currently fit for setting up and operating such a complex setup as you are using with several proxy and NAT layers and all that. Containers may seem like a good idea, but if they dispose of crucial information such as the client address, then revise your setup THOROUGHLY. Sorry to be blunt without asking, but we've all had our shares of eating unsolicited e-mail. Thank you. S
Re: Spammer succeeded in relaying through my server
Dnia 22.12.2022 o godz. 19:26:26 Samer Afach pisze: > My plan is to completely > cut off relaying by setting mynetworks to localhost and > mynetworks_style to host, BTW. You only need mynetworks_style if you don't set mynetworks explicitly. If you do set it explicitly, mynetworks_style is just ignored. -- Regards, Jaroslaw Rafa r...@rafa.eu.org -- "In a million years, when kids go to school, they're gonna know: once there was a Hushpuppy, and she lived with her daddy in the Bathtub."
Re: Spammer succeeded in relaying through my server
sorry to have to burst your bubble, but postfix does not have documentation at least not in the way we call documentation these days maybe you'd call them "notes" or a "reference guide" but not real documentation it is helpful to people who already know everything, but not helpful to people who are learning On Thu, 22 Dec 2022 19:05:47 +0100 Matthias Andree wrote: > Postfix has been around for a quarter century, and it has always been > *the* example of good design, documentation and compatibility with > predecessor versions.
Re: Spammer succeeded in relaying through my server
On 12-22-2022 2:18 pm, mailm...@ionos.gr wrote: sorry to have to burst your bubble, but postfix does not have documentation at least not in the way we call documentation these days maybe you'd call them "notes" or a "reference guide" but not real documentation it is helpful to people who already know everything, but not helpful to people who are learning I would disagree. Postfix has documentation, however it is not a tutorial which is what i think you are describing. Documentation isn't meant to be a how-to or tutorial.
Re: Spammer succeeded in relaying through my server
On Thu, Dec 22, 2022 at 09:18:07PM +0200, mailm...@ionos.gr wrote: > On Thu, 22 Dec 2022 19:05:47 +0100 Matthias Andree > wrote: > > > Postfix has been around for a quarter century, and it has always been > > *the* example of good design, documentation and compatibility with > > predecessor versions. > > sorry to have to burst your bubble, but postfix does not have documentation > at least not in the way we call documentation these days > maybe you'd call them "notes" or a "reference guide" but not real > documentation > it is helpful to people who already know everything, but not helpful to > people who are learning Postfix's documentation consists of about 129 documents. I'd call it a heroic effort of documentation. They do a fairly good job of explaining in precise detail what you need to know. It can be easy to miss some important detail in some cases, but once you realise that and go back to the documentation, what you needed to know is usually there. Once you realise that, you know to read them more carefully. cheers, raf
Re: Spammer succeeded in relaying through my server
On Thu, Dec 22, 2022 at 02:23:53PM -0500, post...@ptld.com wrote: > > On 12-22-2022 2:18 pm, mailm...@ionos.gr wrote: > > sorry to have to burst your bubble, but postfix does not have documentation > > > > at least not in the way we call documentation these days > > maybe you'd call them "notes" or a "reference guide" but not real > > documentation > > > > it is helpful to people who already know everything, but not helpful to > > people who are learning > > I would disagree. > Postfix has documentation, however it is not a tutorial which is what i think > you are describing. > Documentation isn't meant to be a how-to or tutorial. I disagree with both. I found that reading all of Postfix's documentation did help me to learn a lot about it, and to finally understand the configuration that I'd inherited from someone else that I'd been using for years only tweaking slightly for my needs. But that probably just suits my preferred way of learning. But not everyone has time to do that. There is a lot of documentation to read. But also, some of a system's documentation can definitely take the form of how-to guides and tutorials. Postfix has many How-To Guides. My favourite theory of documentation is explained here: https://documentation.divio.com/ There are four forms of documentation: Tutorials: Learning-oriented How-To Guides: Problem-oriented Explanation: Understanding-oriented Reference: Information-oriented Postfix itself provides Reference and How-To Guides, but not so much in the way of Tutorials (which others have written and they are available elsewhere online), or Explanation (hundreds of RFCs don't count unless you've really got a lot of time to spare). :-) Postfix's documentation could be improved with more "Explanation" documents, and "Tutorials" (not all that are out there give good advice), but someone has to have the time and desire and ability to write them. Actually, the "Explanation" part of the documentation is probably handled best by the Postfix book: Postfix: The Definitive Guide by Kyle D. Dent https://www.seaglass.com/postfix/ And there are good tutorials/how-tos out there. So maybe all four forms of documentation are well represented, they're just scattered around the internet. If there isn't a How-To for Postfix in Docker and/or behind HAProxy yet, then it would be great if someone who was setting that up and encountering and solving any problems along the way were able to write and publish a How-To Guide to benefit others. cheers, raf
Re: Spammer succeeded in relaying through my server
On 12/22/22 14:23, post...@ptld.com wrote: On 12-22-2022 2:18 pm, mailm...@ionos.gr wrote: sorry to have to burst your bubble, but postfix does not have documentation at least not in the way we call documentation these days maybe you'd call them "notes" or a "reference guide" but not real documentation it is helpful to people who already know everything, but not helpful to people who are learning I would disagree. Postfix has documentation, however it is not a tutorial which is what i think you are describing. Documentation isn't meant to be a how-to or tutorial. The word 'documentation' covers a lot of ground. Documentation in the context in which we use it here should not be a *tutorial*; it should, however, be able to serve as a *guide*, not merely a simple reference. That said, there are limits in how deep of a guide it can possible be to a complex tool such as Postfix. Perl's documentation, for example, is extraordinarily deep and complete; but there are reasons why the camel and llama books exist as well. -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958
Re: Spammer succeeded in relaying through my server
On Thu, Dec 22, 2022 at 04:47:57AM +0400, Samer Afach wrote: > If I were you, I'd focus on my lack of understanding of the email protocol. > Now that, is a part that I still cannot fully understand, embarrassingly so. > I still don't know what ehlo means, except that it's the first message. I > don't know why it matters what address we put after it. That does make me > look like an idiot, doesn't it? :-) There are hundreds of RFCs (at least!) involved in all the facets of email. Nobody is born having already read them all. Learning anything takes time. And email is famously complex. There's an old quote from Sendmail's documentation that says: The world is complex, and the mail configuration reflects that. But you don't have to read all the RFCs, they are mostly for implementers. Although I recommend it whenever you really want to understand the details of formats and protocols that you are interested in. It might seem daunting, but it's worth the effort. Reading a fair amount of Postfix documentation is needed, though, if using Postfix to handle your email, and your requirements aren't simple. The http://www.postfix.org website has links to lots of documentation. Read what seems relevant to your needs. And look for tutorials relevant to your needs elsewhere on the internet as well. Back to EHLO... EHLO is the first client message in the SMTP protocol. Originally, the first message was HELO and it includes the hostname of the client (so the server knows who is saying hello). That matters because a lot of mail servers will check that that hostname is sensible. They do that because (presumably) a lot of spam comes from clients that don't provide a sensible hostname. Ideally, the hostname would be one that matches the public IP address of the mail server. EHLO is an extension to the SMTP protocol that causes the server to send back a list of features that it supports, so that the client knows what it can do with that server (e.g., STARTTLS, SMTPUTF8, 8BITMIME, etc.). cheers, raf
Re: Spammer succeeded in relaying through my server
Dear Matthias, I think there's a misunderstanding here. The server is already shutdown. I thought you meant that I should shutdown the server permanently and move on with my life because I'm incapable of running a server, which seems to have been the context (I'll explain next why that's obvious). But please allow me to elaborate that usually the cost of that is extremely high, and I'm willing to pay it. Luckily I'm on vacation now so I'm just spending my own time to do this. I've spent so far no less than 100 hours on this fiasco. Zero regrets. Btw, the relays happened because I actively changed mynetworks_style to subnet, forgetting and not checking that all incoming connections will come from the gateway of docker subnet. Still under research to identify how that works. There have been suggestions on the email list that I could start the containers locally to do the experiments. Maybe I'm missing something, but isn't the primary problem that I need to identify connection sources with the PROXY protocol? How will I do that if I can't produce remote connections? There's no way to guarantee that unless I do that remotely. That's why I'll have to turn on the server before it's fully tested... because the ultimate test will be communicating with the server remotely and getting the desired results. And again, let's not forget that disabling email relaying is the primary solutions (as many have pointed out already). So none of this is that bad. I think we have a fair assessment of the risk, all things considered. Now about learning, people have shared interesting opinions and I read the email chain (readers, please read them first before commenting on what I'm saying) and I agree with many of them and disagree with others, and yes, it's not as simple as it seems. There's no point in anyone's life where one said "now I'm educated, let's run an email server". Not really. Even people graduating university don't know what they're doing after having spent years grinding information. The reason for this phenomenon is complex and has to do with human nature and upbringing, but it's what it's. You either inherit a system that works and you're taught the caveats in it and how to test it and what to look for, or you start from somewhere (some configuration online?) and learn as you go. The Linux ecosystem in general is based on testing things manually, which is why configuration mistakes happen all the time, and which is why there are tons of online tools to test the obvious vulnerabilities (OSS, etc, most of which use public interfaces, because no one knows everything). Email is very sensitive, granted, but I'm sorry, you're asking for the impossible. You're asking for people to "learn first", well, I have news for you... THIS is what learning looks like :-) There's a saying I like: Experience is the number of mistakes you've done so far. You're asking for "safe and possibly reviewed configuration", I'm sorry, but that doesn't really mean anything in practice unless you have an army of experts to support you. It's just words that sound nice. Even with all the wonderful people here and all the help I got, when I shared my configuration, I barely got commentary on my configuration, and when I asked further I got your email asking me to "learn". It didn't take more than one email admitting my ignorance to start getting requests to shutdown my server (which I had already done), instead of providing more information. That's not odd, it's just human nature, which leads to where we are right now. Please don't misunderstand what I'm saying, I'm in NO WAY entitled to anything, and I'm already very grateful for everything I learned here, but please be realistic in what you're asking for. I'm trying to make the best of the current situation. Thank you all, lovely people. I wanna emphasize that I'm learning from every email I'm reading here. Best regards, Sam On 22/12/2022 10:05 PM, Matthias Andree wrote: Am 22.12.22 um 01:47 schrieb Samer Afach: Thank you, Matthias, for your opinion. Ironically, the proxy part in this whole story is not only the simple part that I fully understand, but also the part that's very easily testable with my current knowledge and understanding of networking. It's easy to look into the IP addresses of incoming connections and cross-check them with the senders. It's easy to see whether a PROXY protocol with HAProxy works. I'm looking into that setup as we speak. All that is easy. I'm actually surprised you're using that part as argument to why I should shutdown my server. Thanks to the wonderful people in this email list, now I understand my mistake with that regard. I learned a lot. If I were you, I'd focus on my lack of understanding of the email protocol. Now that, is a part that I still cannot fully understand, embarrassingly so. I still don't know what ehlo means, except that it's the first message. I don't know why it matte
Re: Spammer succeeded in relaying through my server
Thank you very much, Raf. I really appreciate your empathy and all this help. I'm reading more and more every day. I really appreciate your support. The challenge is knowing what matters. And about your HELO definition, thanks for explaining it. What confuses me is that clients usually don't have hostnames (like my laptop with its Thunderbird installation or my smartphone, don't have a hostname or FQDN to say HELO with), hence making it a requirement is what I don't understand. It even got me into trouble when requiring it on postfix. I understand it's valid among public servers, but when we say "client", that sounds like any client, and not all clients share having a global hostname/FQDN. Thanks for explaining EHLO, I think I understand it now. Cheers, Sam On 23/12/2022 5:01 AM, raf wrote: On Thu, Dec 22, 2022 at 04:47:57AM +0400, Samer Afach wrote: If I were you, I'd focus on my lack of understanding of the email protocol. Now that, is a part that I still cannot fully understand, embarrassingly so. I still don't know what ehlo means, except that it's the first message. I don't know why it matters what address we put after it. That does make me look like an idiot, doesn't it? :-) There are hundreds of RFCs (at least!) involved in all the facets of email. Nobody is born having already read them all. Learning anything takes time. And email is famously complex. There's an old quote from Sendmail's documentation that says: The world is complex, and the mail configuration reflects that. But you don't have to read all the RFCs, they are mostly for implementers. Although I recommend it whenever you really want to understand the details of formats and protocols that you are interested in. It might seem daunting, but it's worth the effort. Reading a fair amount of Postfix documentation is needed, though, if using Postfix to handle your email, and your requirements aren't simple. The http://www.postfix.org website has links to lots of documentation. Read what seems relevant to your needs. And look for tutorials relevant to your needs elsewhere on the internet as well. Back to EHLO... EHLO is the first client message in the SMTP protocol. Originally, the first message was HELO and it includes the hostname of the client (so the server knows who is saying hello). That matters because a lot of mail servers will check that that hostname is sensible. They do that because (presumably) a lot of spam comes from clients that don't provide a sensible hostname. Ideally, the hostname would be one that matches the public IP address of the mail server. EHLO is an extension to the SMTP protocol that causes the server to send back a list of features that it supports, so that the client knows what it can do with that server (e.g., STARTTLS, SMTPUTF8, 8BITMIME, etc.). cheers, raf
Re: Spammer succeeded in relaying through my server
Samer Afach: > Thank you very much, Raf. I really appreciate your empathy and all this > help. > > I'm reading more and more every day. I really appreciate your support. > The challenge is knowing what matters. > > And about your HELO definition, thanks for explaining it. What confuses > me is that clients usually don't have hostnames (like my laptop with its > Thunderbird installation or my smartphone, don't have a hostname or FQDN > to say HELO with), hence making it a requirement is what I don't That is OK, MUAs (mail user agents that connect to port 587 etc) use SASL or some other form of authentication. On the other hand, MTAs (mail transfer agents that connect to port 25) better have a proper hostname, or the odds that their mail will be seen as spam will be high. Wietse
Re: Spammer succeeded in relaying through my server
Got it. So HELO/EHLO is specifically for MTAs. Thank you very much for explaining! All the best, Sam On 23/12/2022 6:37 AM, Wietse Venema wrote: Samer Afach: Thank you very much, Raf. I really appreciate your empathy and all this help. I'm reading more and more every day. I really appreciate your support. The challenge is knowing what matters. And about your HELO definition, thanks for explaining it. What confuses me is that clients usually don't have hostnames (like my laptop with its Thunderbird installation or my smartphone, don't have a hostname or FQDN to say HELO with), hence making it a requirement is what I don't That is OK, MUAs (mail user agents that connect to port 587 etc) use SASL or some other form of authentication. On the other hand, MTAs (mail transfer agents that connect to port 25) better have a proper hostname, or the odds that their mail will be seen as spam will be high. Wietse
Re: Spammer succeeded in relaying through my server
On 12/22/22 21:19, Samer Afach wrote: There's a saying I like: Experience is the number of mistakes you've done so far. There is truth in that. We are each the sum of our own experiences, including our mistakes. -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958
Re: Spammer succeeded in relaying through my server
On 23/12/22 15:48, Samer Afach wrote: Got it. So HELO/EHLO is specifically for MTAs. Thank you very much for explaining! It still needs to be sent for submission connections, but the hostname does not have to be a legitimate FQDN. What is important for submission is that it passes some other form of authentication, such as SASL AUTH. Peter
Re: Spammer succeeded in relaying through my server
I see. Thank you for the explanation. So the right way to state this is that HELO/EHLO requires a valid FQDN/hostname only for MTAs, and for MUAs it's just ignored because authentication is what matters. Cheers, Sam On 23/12/2022 9:34 AM, Peter wrote: On 23/12/22 15:48, Samer Afach wrote: Got it. So HELO/EHLO is specifically for MTAs. Thank you very much for explaining! It still needs to be sent for submission connections, but the hostname does not have to be a legitimate FQDN. What is important for submission is that it passes some other form of authentication, such as SASL AUTH. Peter
Re: Spammer succeeded in relaying through my server
In my case, I reject invalid HELO names with: reject_invalid_helo_hostname reject_non_fqdn_helo_hostname I've run postfix like this for about 10 years now without any problems. I don't expect others to use such restrictions but it works for me. On Fri, 23 Dec 2022 09:51:48 +0400 Samer Afach wrote: > I see. Thank you for the explanation. So the right way to state this is > that HELO/EHLO requires a valid FQDN/hostname only for MTAs, and for > MUAs it's just ignored because authentication is what matters. > > Cheers, > Sam
Re: Spammer succeeded in relaying through my server
Thank you. I will re-visit this next after the proxy problem is solved. Btw, thank you very much for suggesting HAProxy's proxy protocol. I've succeeded in getting it partially to work (I'm not done yet, there's more to configure), and I can see external IP addresses in postfix and dovecot being recognized. I guess the primary issue is solved. Best regards, Sam On 23/12/2022 11:09 AM, mailm...@ionos.gr wrote: In my case, I reject invalid HELO names with: reject_invalid_helo_hostname reject_non_fqdn_helo_hostname I've run postfix like this for about 10 years now without any problems. I don't expect others to use such restrictions but it works for me. On Fri, 23 Dec 2022 09:51:48 +0400 Samer Afach wrote: I see. Thank you for the explanation. So the right way to state this is that HELO/EHLO requires a valid FQDN/hostname only for MTAs, and for MUAs it's just ignored because authentication is what matters. Cheers, Sam
