syn flood generated by a postfix transaction
Hello, we recently had some situation of full bandwidth usage through our firewall, and by investigation we discovered that this was caused by a specific mail to a specific destination sent via Postifx 3.1.6. We found an ACK SYN DUP flood during the problem, many many packets sent, and this on the postfix log regarding that transaction: enabling PIX workarounds: disable_esmtp delay_dotcrlf for... conversation with ... timed out while sending message body We had to block that destination on the firewall and remove the message on the queue, or it would restart the problem when retried by the queue. The same mail sent via Postfix 2.7.1 on a different data center with the same firewall was sent without problems and without the postfix logs above. I can reproduce the problem by sending again the same email. Other different smaller mails to the same desintation are sent correctly. What may be happening? Gabriele Sonicle S.r.l. : http://www.sonicle.com Music: http://www.gabrielebulfon.com Quantum Mechanics : http://www.cdbaby.com/cd/gabrielebulfon
Re: syn flood generated by a postfix transaction
Gabriele Bulfon: > Hello, > we recently had some situation of full bandwidth usage through our > firewall, and by investigation we discovered that this was caused > by a specific mail to a specific destination sent via Postifx 3.1.6. > We found an ACK SYN DUP flood during the problem, many many packets > sent, and this on the postfix log regarding that transaction: FYI, Postfix does not send SYN packets. SYN packets are sent by the network stack in your kernel when a program opens a TCP connection. The remote server is then expected to respond with a SYN|ACK packet. client SYN -> server client <- SYN|ACK server client ACK -> server A properly working network stack will send a SYN packet for each new outbound TCP connection, and will retransmit the SYN packet after several seconds until it gives up. A repeated SYN packet will result in a SYN|ACK response (with perhaps a different server initial sequence number). With delays of seconds it is hard to achieve a SYN flood. Now, it is possible that you have configured Postfix with huge limits on process counts and destination concurrency. In that case Postfix will attempt to make a huge number of connections to the same site, which effectively results in large numbers of SYN and SYN|ACK packets. That would be a problem of your own making. Wietse
Re: syn flood generated by a postfix transaction
Thanks so much for the deep explanation :) Kernel is illumos, I exclude it can be a bug in the kernel stack. I will check Postfix config, but I don't think there is any huge limit as you suggested. What about the "PIX workaround"? Can it be something causing this? It did not happen on a previous version on another site with same firewall, same message. Or, we also find answers from the destination host saying "write error" on their side: can it be an unhandled situation somewhere? Also keep in mind it happens only with this message (or maybe some others I don't know) any time I retry a send, not with any other simple message I try. Gabriele Sonicle S.r.l. : http://www.sonicle.com Music: http://www.gabrielebulfon.com Quantum Mechanics : http://www.cdbaby.com/cd/gabrielebulfon -- Da: Wietse Venema A: Postfix users Cc: postfix-users@postfix.org u...@porcupine.org Data: 10 settembre 2020 15.52.39 CEST Oggetto: Re: syn flood generated by a postfix transaction Gabriele Bulfon: Hello, we recently had some situation of full bandwidth usage through our firewall, and by investigation we discovered that this was caused by a specific mail to a specific destination sent via Postifx 3.1.6. We found an ACK SYN DUP flood during the problem, many many packets sent, and this on the postfix log regarding that transaction: FYI, Postfix does not send SYN packets. SYN packets are sent by the network stack in your kernel when a program opens a TCP connection. The remote server is then expected to respond with a SYN|ACK packet. client SYN -server client client ACK -server A properly working network stack will send a SYN packet for each new outbound TCP connection, and will retransmit the SYN packet after several seconds until it gives up. A repeated SYN packet will result in a SYN|ACK response (with perhaps a different server initial sequence number). With delays of seconds it is hard to achieve a SYN flood. Now, it is possible that you have configured Postfix with huge limits on process counts and destination concurrency. In that case Postfix will attempt to make a huge number of connections to the same site, which effectively results in large numbers of SYN and SYN|ACK packets. That would be a problem of your own making. Wietse
Re: syn flood generated by a postfix transaction
Gabriele Bulfon: > Thanks so much for the deep explanation :) Considering the questions asked, I must keep the conversation at a basic level. > Kernel is illumos, I exclude it can be a bug in the kernel stack. > I will check Postfix config, but I don't think there is any huge > limit as you suggested. Don't overlook the firewall (yours or theirs). or other boxes in the middle that may manipulate TCP packets. > What about the "PIX workaround"? That causes Postfix to speak SMTP slower. If the data sent over an established TCP connection can trigger a TCP SYN flood, then SOMETHING AT THE NETWORK LEVEL is massively screwed up. Wietse
