timeout problem on inbound and outbound SMTP
Hi, I would appreciate any suggestions anyone can offer on the following problem that I'm having with postfix... I'm running postfix+pgsql-2.3.3-2.1.el5_2 on a CentOS 5.4 server. I see what looks likes a server in stress mode as described in http://www.postfix.org/STRESS_README.html except the odd think about it is that the server is not heavily loaded and I sure can't see where it's exceeding any process limits. What's even odder is it doesn't appear that the stress code is implemented in this version. If I telnet to port 25 I get an immediate SMTP greeting followed in 10 seconds by 421 4.4.2 mymail.com Error: timeout exceeded and the connection being closed. The following maillog entry is logged: May 3 16:44:06 mymail postfix/smtpd[22573]: timeout after CONNECT from 173-12-149-200.client.myisp.com[173.12.149.200] This is like this constantly. I see 0-4 smtpd processes on the server at any one time (I'm not sure if it's limited at 4, I just haven't seen more). There are a similar number of policy daemons. There is a "-" in the maxproc field for smtpd in master.cf. From everything I can tell the default is a limit of 100. I do run a policy daemon (vpostmaster). I've changed its maxproc field to 0 per the recommendation in the STRESS_README (and restarted postfix). It's master.cf entry looks like this... vpm-pfpolicy unix - n n - 0 spawn user=vpostmaster argv=/usr/lib/vpostmaster/postfix/vpm-pfpolicy I also get lots of log entries like this for timeouts on the policy daemon: May 2 05:36:20 aspen postfix/spawn[6003]: warning: /usr/lib/vpostmaster/postfix/vpm-pfpolicy: process id 6004: command time limit exceeded and occasionally similar timeouts on the transport daemon (which is part of vpostmaster as well). My system load (which is running on a Vmware ESXi virtual machine) is: 16:59:36 up 2 days, 22:17, 3 users, load average: 0.08, 0.02, 0.01 Then, on outbound mail, I found this one site that adds delays of 18 seconds before the helo message to their SMTP server. My server cannot get a message through to that server at all. I tried adding -o stress= for the smtpd and nothing changed. The system does not automatically add the stress parameter to smtpd if I don't add it myself, so I'm not inclined to believe that Centos/Redhat 5.4 has the stress patch applied. I do not see any slowness in the DNS servers. I have at most 2 RBL lists that I check. I did have limits set on the number of connections and timeouts etc, but I've restored them all to the defaults for purposes of debugging. I get these timeouts even when there's only one smtpd process. Some of my config parameters right now are: ipc_timeout = 3600s command_time_limit = 1000s smtpd_error_sleep_time = 1s smtpd_policy_service_timeout = 100s smtpd_proxy_timeout = 100s smtpd_starttls_timeout = 300s smtpd_timeout = 300s smtpd_tls_session_cache_timeout = 3600s smtpd_client_connection_count_limit = 50 smtpd_client_connection_rate_limit = 0 smtpd_client_message_rate_limit = 0 smtpd_client_new_tls_session_rate_limit = 0 smtpd_client_recipient_rate_limit = 0 smtpd_hard_error_limit = 20 smtpd_junk_command_limit = 100 smtpd_recipient_limit = 1000 smtpd_recipient_overshoot_limit = 1000 smtpd_soft_error_limit = 10 THank you, Nataraj
Re: timeout problem on inbound and outbound SMTP
lst_ho...@kwsoft.de wrote: Zitat von Nataraj : Hi, I would appreciate any suggestions anyone can offer on the following problem that I'm having with postfix... I'm running postfix+pgsql-2.3.3-2.1.el5_2 on a CentOS 5.4 server. I see what looks likes a server in stress mode as described in http://www.postfix.org/STRESS_README.html except the odd think about it is that the server is not heavily loaded and I sure can't see where it's exceeding any process limits. What's even odder is it doesn't appear that the stress code is implemented in this version. If I telnet to port 25 I get an immediate SMTP greeting followed in 10 seconds by 421 4.4.2 mymail.com Error: timeout exceeded and the connection being closed. The following maillog entry is logged: May 3 16:44:06 mymail postfix/smtpd[22573]: timeout after CONNECT from 173-12-149-200.client.myisp.com[173.12.149.200] Why do you think your Postfix server is "stressed"?? The automatic stress-dependant features are introduced in version 2.5 as far as i know so your Postfix does not support -o stress at all. If you only have 4 smtpd running and your server show greeting immediately when telneting to port 25 all should be fine. I don't think it is a stress condition. The problem is that it is timing everything out. Further checking shows, it times out inbound SMTP connections in like 3-4 seconds and fails outbound deliveries to slow servers. The transports also timeout and bounce messages. Regards Andreas
Re: timeout problem on inbound and outbound SMTP
lst_ho...@kwsoft.de wrote: Zitat von Nataraj : lst_ho...@kwsoft.de wrote: Zitat von Nataraj : Hi, I would appreciate any suggestions anyone can offer on the following problem that I'm having with postfix... I'm running postfix+pgsql-2.3.3-2.1.el5_2 on a CentOS 5.4 server. I see what looks likes a server in stress mode as described in http://www.postfix.org/STRESS_README.html except the odd think about it is that the server is not heavily loaded and I sure can't see where it's exceeding any process limits. What's even odder is it doesn't appear that the stress code is implemented in this version. If I telnet to port 25 I get an immediate SMTP greeting followed in 10 seconds by 421 4.4.2 mymail.com Error: timeout exceeded and the connection being closed. The following maillog entry is logged: May 3 16:44:06 mymail postfix/smtpd[22573]: timeout after CONNECT from 173-12-149-200.client.myisp.com[173.12.149.200] Why do you think your Postfix server is "stressed"?? The automatic stress-dependant features are introduced in version 2.5 as far as i know so your Postfix does not support -o stress at all. If you only have 4 smtpd running and your server show greeting immediately when telneting to port 25 all should be fine. I don't think it is a stress condition. The problem is that it is timing everything out. Further checking shows, it times out inbound SMTP connections in like 3-4 seconds and fails outbound deliveries to slow servers. The transports also timeout and bounce messages. Sorry i still don't understand what the problem is... Are you concerned about "timeout after SOMETHING" messages in the logs? Do you have problems with mails not arriving? Do you have problems with mails not leaving? Please try to explain, maybe with log data for a problematic mail. Regards Andreas "fails outbound deliveries to slow servers." means there are problems with outbound mail deliveries. Yes there are complaints of people sending mail from the outside whose delivery is either delayed or does not get through. This means there are problems with inbound mail. An SMTP server that times out after 4 seconds of inactivity is not reasonable given the possibility of delays in others mail systems and on the internet. The question that I am asking is what else can cause these timeouts if there is no stress code in my current version of postfix? Nataraj
Re: timeout problem on inbound and outbound SMTP
lst_ho...@kwsoft.de wrote: Zitat von Nataraj : lst_ho...@kwsoft.de wrote: Zitat von Nataraj : lst_ho...@kwsoft.de wrote: Zitat von Nataraj : Hi, I would appreciate any suggestions anyone can offer on the following problem that I'm having with postfix... I'm running postfix+pgsql-2.3.3-2.1.el5_2 on a CentOS 5.4 server. I see what looks likes a server in stress mode as described in http://www.postfix.org/STRESS_README.html except the odd think about it is that the server is not heavily loaded and I sure can't see where it's exceeding any process limits. What's even odder is it doesn't appear that the stress code is implemented in this version. If I telnet to port 25 I get an immediate SMTP greeting followed in 10 seconds by 421 4.4.2 mymail.com Error: timeout exceeded and the connection being closed. The following maillog entry is logged: May 3 16:44:06 mymail postfix/smtpd[22573]: timeout after CONNECT from 173-12-149-200.client.myisp.com[173.12.149.200] Why do you think your Postfix server is "stressed"?? The automatic stress-dependant features are introduced in version 2.5 as far as i know so your Postfix does not support -o stress at all. If you only have 4 smtpd running and your server show greeting immediately when telneting to port 25 all should be fine. I don't think it is a stress condition. The problem is that it is timing everything out. Further checking shows, it times out inbound SMTP connections in like 3-4 seconds and fails outbound deliveries to slow servers. The transports also timeout and bounce messages. Sorry i still don't understand what the problem is... Are you concerned about "timeout after SOMETHING" messages in the logs? Do you have problems with mails not arriving? Do you have problems with mails not leaving? Please try to explain, maybe with log data for a problematic mail. Regards Andreas "fails outbound deliveries to slow servers." means there are problems with outbound mail deliveries. Yes there are complaints of people sending mail from the outside whose delivery is either delayed or does not get through. This means there are problems with inbound mail. An SMTP server that times out after 4 seconds of inactivity is not reasonable given the possibility of delays in others mail systems and on the internet. Please show the relevant log entries for a failed delivery attempt from connect until disconnect. Postfix has reasonable defaults and for sure no 4s timeout. The question that I am asking is what else can cause these timeouts if there is no stress code in my current version of postfix? The smtpd__timeout settings are relevant on postfix side. The default for smtpd is 300s. If your settings are at default or far from the values you are seeing you should check if some stateful firewall in between drops "inactive" connections earlier. If this does not help you should capture a tcpdump to see what is going on. Regards Andreas Hi Andreas, I did post my configuration parameters in my first message. I mentioned that I had changed them, but had restored the defaults to see if that would solve the problem. The smtpd_timeout was never changed from 300s. Here is a log of a failed outbound delivery. As I mentioned, the sysadmin for this server told me they have an 18 second delay before sending the greeting, which they've found effective at stopping spambots. I've done crazier things myself, and I think that a properly working mailserver should be able to deliver to them even if it is a bit unfriendly. Certainly I was able to deliver to them before this problem started. May 2 05:22:32 aspen postfix/smtp[6575]: 4CF471D0200: to=, relay=mail.cs.rutgers.edu[128.6.4.3]:25, delay=124076, delays=124072/0.05/4\ .3/0, dsn=4.4.2, status=deferred (conversation with mail.cs.rutgers.edu[128.6.4.3] timed out while receiving the initial server greeting) It most definitely is valid to telnet to an smtp server and type commands to test it. I've been doing this for years. It was only until this problem came up that I am no longer able to do it because I can't type fast enough before it times me out (unless I enable pipelining and paste the commands). I maintain the firewall and there are no logs of problems there. I'm fairly certain this is the SMTP server. If a firewall was timing out the connection postfix would not know that it was a timeout and would not be able to log a timeout error. penguin> telnet aspen 25 Trying 66.35.48.14... Connected to aspen. Escape character is '^]'. 220 aspen.rjl.com ESMTP vPostMaster 421 4.4.2 aspen.rjl.com Error: timeout exceeded Connection closed by foreign host. From this log of the telnet session, the 421 error comes from the smtp server and running tcpdump is useless because the problem is at the SMTP level and not at the TCP or
Re: server stops responding / smtpd client count
P.A wrote: Hi during times of high mail load, spam attacks usually, I sometimes run into an issue where postfix will stop responding or becomes extremely slow on the stmp port. In turn this causes my pop/imap server (dovecot) to stop responding or to become extremely slow as well. When I stop postfix, the pop/imap server go back to normal. I have 3 email filter servers that scan the email before delivering it to the postfix server. When the problem occurred I did notice with netstat that there was a huge number of established connections on port 25 with the mail filter servers on the postfix server. The thing that I don’t understand is that before the problem occurred I had “smtpd_client_connection_count_limit = 30” which was working with no issues . When the problem started to occur I saw exceed errors on the mail log, basically connection numbers over that limit of 30. I was seeing upwards of 70 connections per email filter host. When this started happening ports 25/110/143 became extremely slow. My question is if I have a smtp client limit why do still see so many established connections with netstat. Shouldn’t postfix stop the extra connections? (the email filter servers are not part of $mynetworks) Why is postfix slowing down my pop/imap server when this occurs? This is extremely busy server with plenty of CPU and memory, what is a reasonable smtpd count limit that will not overwhelm the server? FYi, when I changed that smtpd client connection to 100, the problem went away and all was good again. mail_version = 2.6.5 250-PIPELINING 250-SIZE 2600 250-VRFY 250-ETRN 250-AUTH LOGIN CRAM-MD5 PLAIN DIGEST-MD5 250-ENHANCEDSTATUSCODES 250-8BITMIME 250 DSN Thanks Paul You might start here: http://www.postfix.org/STRESS_README.html There are other parameters affecting this as well. I don't remember what they all are, but simply not allowing so many smtpd's when there are spam attacks with many attempted incoming connections, will not be enough to alleviate the load of the attack and may worsen the situtation unless used in conjunction with other measures. Your other services are slow because your server is obviously under heavy load, including the TCP stack. You see all of the connections because they are coming in at a high rate and even though postfix may have closed them, they are still waiting for the final tcp handshake which closes the connection and for the kernel tcp stack to clear them out. Nataraj
Re: timeout problem on inbound and outbound SMTP
Here is the complete output of postconf -n. Thanks... alias_database = hash:/etc/aliases alias_maps = hash:/etc/aliases broken_sasl_auth_clients = yes command_directory = /usr/sbin config_directory = /etc/postfix daemon_directory = /usr/libexec/postfix debug_peer_level = 2 disable_vrfy_command = yes html_directory = no inet_interfaces = all mail_name = vPostMaster mail_owner = postfix mailq_path = /usr/bin/mailq.postfix manpage_directory = /usr/share/man mydestination = $myhostname, localhost.$mydomain, localhost mydomain = aspen.rjl.com mynetworks = 127.0.0.0/8 newaliases_path = /usr/bin/newaliases.postfix proxy_read_maps = proxy:pgsql:/etc/postfix/vpm_recipient_access $local_recipient_maps $mydestination $virtual_alias_maps $virtual_alias_domains $virtual_mailbox_maps $virtual_mailbox_domains $relay_recipient_maps $relay_domains $canonical_maps $sender_canonical_maps $recipient_canonical_maps $relocated_maps $transport_maps $mynetworks queue_directory = /var/spool/postfix readme_directory = /usr/share/doc/postfix+pgsql-2.3.3/README_FILES sample_directory = /usr/share/doc/postfix+pgsql-2.3.3/samples sendmail_path = /usr/sbin/sendmail.postfix setgid_group = postdrop smtp_tls_loglevel = 1 smtp_tls_note_starttls_offer = yes smtp_tls_security_level = may smtp_tls_session_cache_database = btree:/var/run/smtp_tls_session_cache smtpd_client_event_limit_exceptions = 127.0.0.1 smtpd_client_restrictions = check_client_access hash:/etc/postfix/smtpd_client_accessreject_unknown_client_hostname smtpd_data_restrictions = reject_unauth_pipelining smtpd_helo_restrictions = check_helo_access hash:/etc/postfix/smtpd_helo_namesreject_invalid_helo_hostname reject_non_fqdn_helo_hostname smtpd_recipient_restrictions = permit_sasl_authenticated permit_mynetworkscheck_recipient_access proxy:pgsql:/etc/postfix/vpm_recipient_accesscheck_sender_access hash:/etc/postfix/smtpd_sender_accesscheck_recipient_access hash:/etc/postfix/smtpd_recipient_accessreject_rbl_client dul.dnsbl.sorbs.netcheck_policy_service unix:private/vpm-pfpolicy reject_unauth_destination smtpd_restriction_classes = restrictive, permissive smtpd_sasl_auth_enable = yes smtpd_sasl_path = private/auth smtpd_sasl_security_options = noanonymous noplaintext smtpd_sasl_type = dovecot smtpd_sender_restrictions = permit_mynetworks reject_unknown_sender_domainreject_unknown_recipient_domain smtpd_tls_CAfile = /etc/postfix/certs/CAcert.crt smtpd_tls_CApath = /etc/postfix/certs smtpd_tls_cert_file = /etc/postfix/certs/tls.crt smtpd_tls_key_file = /etc/postfix/certs/tls.key smtpd_tls_loglevel = 1 smtpd_tls_security_level = may smtpd_tls_session_cache_timeout = 3600s tls_random_source = dev:/dev/urandom unknown_address_reject_code = 550 unknown_client_reject_code = 550 unknown_local_recipient_reject_code = 550 virtual_mailbox_domains = proxy:pgsql:/etc/postfix/vpm-domains virtual_transport = vpm-pftransport
Re: timeout problem on inbound and outbound SMTP
Noel Jones wrote:
Please show the contents of your master.cf
See the enclosed attachment.
Thank You,
Nataraj
#
# Postfix master process configuration file. For details on the format
# of the file, see the master(5) manual page (command: "man 5 master").
#
# ==
# service type private unpriv chroot wakeup maxproc command + args
# (yes) (yes) (yes) (never) (100)
# ==
smtp inet n - n - - smtpd
#submission inet n - n - - smtpd
# -o smtpd_enforce_tls=yes
# -o smtpd_sasl_auth_enable=yes
# -o smtpd_client_restrictions=permit_sasl_authenticated,reject
#smtps inet n - n - - smtpd
# -o smtpd_tls_wrappermode=yes
# -o smtpd_sasl_auth_enable=yes
# -o smtpd_client_restrictions=permit_sasl_authenticated,reject
#628 inet n - n - - qmqpd
pickupfifo n - n 60 1 pickup
cleanup unix n - n - 0 cleanup
qmgr fifo n - n 300 1 qmgr
#qmgr fifo n - n 300 1 oqmgr
tlsmgrunix - - n 1000? 1 tlsmgr
rewrite unix - - n - - trivial-rewrite
bounceunix - - n - 0 bounce
defer unix - - n - 0 bounce
trace unix - - n - 0 bounce
verifyunix - - n - 1 verify
flush unix n - n 1000? 0 flush
proxymap unix - - n - - proxymap
smtp unix - - n - - smtp
# When relaying mail as backup MX, disable fallback_relay to avoid MX loops
relay unix - - n - - smtp
-o fallback_relay=
# -o smtp_helo_timeout=5 -o smtp_connect_timeout=5
showq unix n - n - - showq
error unix - - n - - error
discard unix - - n - - discard
local unix - n n - - local
virtual unix - n n - - virtual
lmtp unix - - n - - lmtp
anvil unix - - n - 1 anvil
scacheunix - - n - 1 scache
#
#
# Interfaces to non-Postfix software. Be sure to examine the manual
# pages of the non-Postfix software to find out what options it wants.
#
# Many of the following services use the Postfix pipe(8) delivery
# agent. See the pipe(8) man page for information about ${recipient}
# and other message envelope options.
#
#
# maildrop. See the Postfix MAILDROP_README file for details.
# Also specify in main.cf: maildrop_destination_recipient_limit=1
#
maildrop unix - n n - - pipe
flags=DRhu user=vmail argv=/usr/local/bin/maildrop -d ${recipient}
#
# The Cyrus deliver program has changed incompatibly, multiple times.
#
old-cyrus unix - n n - - pipe
flags=R user=cyrus argv=/usr/lib/cyrus-imapd/deliver -e -m ${extension}
${user}
# Cyrus 2.1.5 (Amos Gouaux)
# Also specify in main.cf: cyrus_destination_recipient_limit=1
cyrus unix - n n - - pipe
user=cyrus argv=/usr/lib/cyrus-imapd/deliver -e -r ${sender} -m ${extension}
${user}
#
# See the Postfix UUCP_README file for configuration details.
#
uucp unix - n n - - pipe
flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail ($recipient)
#
# Other external delivery methods.
#
ifmailunix - n n - - pipe
flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient)
bsmtp unix - n n - - pipe
flags=Fq. user=foo argv=/usr/local/sbin/bsmtp -f $sender $nexthop $recipient
# vPostMaster setup
vpm-pfpolicy unix - n n - 0 spawn
user=vpostmaster argv=/usr/lib/vpostmaster/postfix/vpm-pfpolicy
vpm-pftransport unix - n n - 5 pipe
flags=Fqhu user=vpostmaster null_sender=
argv=/usr/lib/vpostmaster/postfix/vpm-pftransport $sender $recipient
Re: timeout problem on inbound and outbound SMTP
Nataraj wrote: Noel Jones wrote: Please show the contents of your master.cf See the enclosed attachment. Thank You, Nataraj Enclosed is a tcpdump of a telnet connection where nothing was typed, i.e. I telnetted to the smtp server and 5 seconds later the server closed the connection. Thanks, Nataraj 11:33:16.455614 IP 173-16-166-221.client.myisp.com.59695 > mymail.com.smtp: S 5024978:5024978(0) win 5840 11:33:16.512877 IP mymail.com.smtp > 173-16-166-221.client.myisp.com.59695: S 3162720878:3162720878(0) ack 5024979 win 5792 11:33:16.512818 IP 173-16-166-221.client.myisp.com.59695 > mymail.com.smtp: . ack 1 win 92 11:33:16.566384 IP mymail.com.smtp > 173-16-166-221.client.myisp.com.59695: S 3162720878:3162720878(0) ack 5024979 win 5792 11:33:16.566325 IP 173-16-166-221.client.myisp.com.59695 > mymail.com.smtp: . ack 1 win 92 11:33:16.570860 IP mymail.com.smtp > 173-16-166-221.client.myisp.com.59695: P 1:38(37) ack 1 win 91 11:33:16.570808 IP 173-16-166-221.client.myisp.com.59695 > mymail.com.smtp: . ack 38 win 92 11:33:21.052384 IP mymail.com.smtp > 173-16-166-221.client.myisp.com.59695: P 38:87(49) ack 1 win 91 11:33:21.052302 IP 173-16-166-221.client.myisp.com.59695 > mymail.com.smtp: . ack 87 win 92 11:33:21.052407 IP mymail.com.smtp > 173-16-166-221.client.myisp.com.59695: F 87:87(0) ack 1 win 91 11:33:21.052774 IP 173-16-166-221.client.myisp.com.59695 > mymail.com.smtp: F 1:1(0) ack 88 win 92 11:33:21.110988 IP mymail.com.smtp > 173-16-166-221.client.myisp.com.59695: . ack 2 win 91
Re: timeout problem on inbound and outbound SMTP
Charles Gregory wrote: On Tue, 4 May 2010, Nataraj wrote: Enclosed is a tcpdump of a telnet connection where nothing was typed, i.e. I telnetted to the smtp server and 5 seconds later the server closed the connection. THIS IS NORMAL. As I said previously, type the MAIL FROM, RCPT TO, and DATA commands, send a couple of ilnes, THEN wait and time the timeout. How about those logs showing a complete mail 'life cycle'? - C I have attached tcpdump-with-commands.txt where I pasted with the mouse helo mymail.com mail from: I then waited and it still timed out in 5 seconds. I think the timeout should be whatever the smtpd_timout parameter is set to (300s in my case), unless the stress code is enabled and operating, in which case it should be set to the stress config timeout parameter). 5 seconds is not a normal timeout for my configuration. I am enclosing a tcpdump of a telnet session to the mail server which serves this mailing list (I hope it's not considered abusive to use it as a reference:-)), which times out after 21 seconds (probably because it is configured that way). 5 seconds is just too short, and it should change based on my configuration, which it is not doing. As I've mentioned, I can't type fast enough to my server to prevent timeouts, but if I reenable pipelining, I can paste smtp commands and submit messages, only if I paste them all at once (and pipelining is enabled). Also, please don't loose sight of the fact that all of my timeouts are screwed up, i.e. inbound smtp, outbound smtp as well as transport and policy. Maybe this is not a postfix problem, or postfix is having some strange interaction with something going on in the OS (or a vmware clocking problem or something). Nataraj 12:29:26.066127 IP 173-16-199-243.client.mchsi.com.60783 > russian-caravan.cloud9.net.smtp: S 1292340171:1292340171(0) win 5840 12:29:26.151117 IP russian-caravan.cloud9.net.smtp > 173-16-199-243.client.mchsi.com.60783: S 3576036070:3576036070(0) ack 1292340172 win 65535 12:29:26.151070 IP 173-16-199-243.client.mchsi.com.60783 > russian-caravan.cloud9.net.smtp: . ack 1 win 92 12:29:26.309925 IP russian-caravan.cloud9.net.smtp > 173-16-199-243.client.mchsi.com.60783: P 1:47(46) ack 1 win 8326 12:29:26.309868 IP 173-16-199-243.client.mchsi.com.60783 > russian-caravan.cloud9.net.smtp: . ack 47 win 92 12:29:47.022834 IP russian-caravan.cloud9.net.smtp > 173-16-199-243.client.mchsi.com.60783: P 47:109(62) ack 1 win 8326 12:29:47.022662 IP 173-16-199-243.client.mchsi.com.60783 > russian-caravan.cloud9.net.smtp: . ack 109 win 92 12:29:47.022857 IP russian-caravan.cloud9.net.smtp > 173-16-199-243.client.mchsi.com.60783: F 109:109(0) ack 1 win 8326 12:29:47.023074 IP 173-16-199-243.client.mchsi.com.60783 > russian-caravan.cloud9.net.smtp: F 1:1(0) ack 110 win 92 12:29:47.109467 IP russian-caravan.cloud9.net.smtp > 173-16-199-243.client.mchsi.com.60783: . ack 2 win 8325 12:41:43.075178 IP 173-16-66-55.client.myisp.com.39370 > mymail.com.smtp: S 4275458803:4275458803(0) win 5840 12:41:43.132516 IP mymail.com.smtp > 173-16-66-55.client.myisp.com.39370: S 3873633767:3873633767(0) ack 4275458804 win 5792 12:41:43.132459 IP 173-16-66-55.client.myisp.com.39370 > mymail.com.smtp: . ack 1 win 92 12:41:43.234825 IP mymail.com.smtp > 173-16-66-55.client.myisp.com.39370: P 1:38(37) ack 1 win 91 12:41:43.234838 IP 173-16-66-55.client.myisp.com.39370 > mymail.com.smtp: . ack 38 win 92 12:41:43.793026 IP 173-16-66-55.client.myisp.com.39370 > mymail.com.smtp: P 1:15(14) ack 38 win 92 12:41:43.850611 IP mymail.com.smtp > 173-16-66-55.client.myisp.com.39370: . ack 15 win 91 12:41:43.850503 IP 173-16-66-55.client.myisp.com.39370 > mymail.com.smtp: P 15:44(29) ack 38 win 92 12:41:43.850636 IP mymail.com.smtp > 173-16-66-55.client.myisp.com.39370: P 38:57(19) ack 15 win 91 12:41:43.850547 IP 173-16-66-55.client.myisp.com.39370 > mymail.com.smtp: . ack 57 win 92 12:41:43.908466 IP mymail.com.smtp > 173-16-66-55.client.myisp.com.39370: P 57:71(14) ack 44 win 91 12:41:43.908326 IP 173-16-66-55.client.myisp.com.39370 > mymail.com.smtp: . ack 71 win 92 12:41:48.796465 IP mymail.com.smtp > 173-16-66-55.client.myisp.com.39370: P 71:120(49) ack 44 win 91 12:41:48.796479 IP 173-16-66-55.client.myisp.com.39370 > mymail.com.smtp: . ack 120 win 92 12:41:48.796544 IP mymail.com.smtp > 173-16-66-55.client.myisp.com.39370: F 120:120(0) ack 44 win 91 12:41:48.796886 IP 173-16-66-55.client.myisp.com.39370 > mymail.com.smtp: F 44:44(0) ack 121 win 92 12:41:48.854679 IP mymail.com.smtp > 173-16-66-55.client.myisp.com.39370: . ack 45 win 91
Re: timeout problem on inbound and outbound SMTP
Charles Gregory wrote: On Tue, 4 May 2010, Noel Jones wrote: The described behavior suggests smtpd_timeout is set for 4s, but that parameter isn't in the postconf or master.cf shown to the list. Or the poster has a front-end on his mail server, and that is why I asked for a complete log of the 'lifecycle' of the mail, so we can spot if he is getting the timeout from some other piece of software. - C I do not have a front end on my mailserver. All incoming mail comes directly into postfix. I am running a vpostmaster install which is not a front end, but simply provides policy and transport agents that interface to postfix. Postfix is the standard install that comes with CentOS, though vpostmaster and I have changed the config files. Here are the 3 lines that get logged for a timeout. May 4 12:56:07 aspen postfix/smtpd[1277]: connect from 173-16-199-243.client.mchsi.com[173.16.199.243] May 4 12:56:11 aspen postfix/smtpd[1277]: timeout after CONNECT from 173-16-199-243.client.mchsi.com[173.16.199.243] May 4 12:56:11 aspen postfix/smtpd[1277]: disconnect from 173-16-199-243.client.mchsi.com[173.16.199.243] Nataraj
Re: timeout problem on inbound and outbound SMTP
Noel Jones wrote: You can try setting smtpd_timeout and smtp_connect_timeout to riduclously high values (maybe 3000s for testing) to see if it makes any difference. If it doesn't change anything, then this problem is outside postfix somewhere and you'll need to examine other parts of your system. I suppose this could be some sort of a clock issue, but I've never heard of one this extreme. Still, the problem and solution would both be outside postfix. -- Noel Jones Thank you Noel. You have been helpful in convincing me that I have reviewed the obvious configuration issues. I guess one option at this point is to try to fire up the debugger on an SMTPD process and see what's really going on (or add some debugging code which logs the details). Nataraj
Re: timeout problem on inbound and outbound SMTP
N. Yaakov Ziskind wrote: Noel Jones wrote (on Tue, May 04, 2010 at 02:33:48PM -0500): On 5/4/2010 2:16 PM, Charles Gregory wrote: On Tue, 4 May 2010, Nataraj wrote: Enclosed is a tcpdump of a telnet connection where nothing was typed, i.e. I telnetted to the smtp server and 5 seconds later the server closed the connection. THIS IS NORMAL. As I said previously, type the MAIL FROM, RCPT TO, and DATA commands, send a couple of ilnes, THEN wait and time the timeout. How about those logs showing a complete mail 'life cycle'? - C No, it's not normal. When you telnet to a postfix smtpd, postfix will sit there patiently for $smtpd_timeout before it disconnects if you don't type anything. The described behavior suggests smtpd_timeout is set for 4s, but that parameter isn't in the postconf or master.cf shown to the list. I don't think there's anything else we can do for the OP. -- Noel Jones If the timeout is really set to 4s, can it be overriden in master.cf? Wouldn't that be a useful workaround, or at least a diagnostic? Thanks. I have tried already to change timeout parameters with no change in the behaviour. Thank You, Nataraj
Re: timeout problem on inbound and outbound SMTP
Charles Gregory wrote: On Tue, 4 May 2010, Nataraj wrote: I do not have a front end on my mailserver. All incoming mail comes directly into postfix. I am running a vpostmaster install which is not a front end, I'm not an expert on this, but your machine ID's with 'vpostmaster' in the greeting, so that means it is pretty tightly integrated, if not in fact a front-end. So I would go through your vpostmaster settings. You may have specified something as seconds when it is meant to be hundredths, etc... - C It means that the mail_name parameter is set in /etc/postfix/main.cf See: http://www.postfix.org/postconf.5.html#mail_name I am extensively familiar with my main.cf file and am aware of exactly what changes have been made. This mail server has been running stably for years and I am not yet aware of changes to the mail configuration files surrounding the time when this problem came up. There are quite a number of mailserver packages out their that use postfix as the MTA. I'm extensively familiar with the architectures of both the postfix and vpostmaster systems and I assure you that there is no front end. The port 25 service on my mailserver is the postfix smtpd. Vpostmaster, is simply a policy and transport agent with spam management and a web based GUI for management. It's quite nice and I highly recommend it to people. Nataraj
Re: server stops responding / smtpd client count
P.A wrote: Nataraj, thanks for the reply, below is the postconf -n output. As far as your explanation as to why the other services are slow, pop/imap, it may be that the TCP stack is under heavy load and might slow down these connections but the server CPU/MEM are fine and the connections are in est. state not time_wait. Off the top of you head do you have any idea what else I can use to eliviate the issue, sorry for not posting the config. [r...@pop ~]# postconf -n alias_database = hash:/etc/postfix/aliases alias_maps = hash:/etc/postfix/aliases bounce_queue_lifetime = 0 bounce_template_file = /etc/postfix/bounce.cf command_directory = /usr/sbin config_directory = /etc/postfix daemon_directory = /usr/libexec/postfix data_directory = /var/lib/postfix debug_peer_level = 1 default_destination_concurrency_limit = 15 default_process_limit = 200 fast_flush_domains = $relay_domains home_mailbox = Maildir/ html_directory = no inet_interfaces = all mail_owner = postfix mail_spool_directory = /var/spool/mail mailbox_size_limit = 0 mailq_path = /usr/bin/mailq manpage_directory = /usr/local/man maximal_queue_lifetime = 1 message_size_limit = 2600 mydestination = $myhostname, localhost, hash:/etc/postfix/domain-accept myhostname = pop.cape.com mynetworks = hash:/etc/postfix/ip-relay newaliases_path = /usr/bin/newaliases queue_directory = /var/spool/postfix readme_directory = no relay_domains = hash:/etc/postfix/domain-relay sample_directory = /etc/postfix sendmail_path = /usr/sbin/sendmail setgid_group = postdrop smtp_helo_timeout = 100 smtp_rset_timeout = 22s smtp_sasl_security_options = noanonymous, nodictionary, noactive smtpd_banner = $myhostname ESMTP $mail_name Networks that SPAM will be BLOCKED smtpd_client_connection_count_limit = 100 smtpd_error_sleep_time = 0 smtpd_hard_error_limit = 8 smtpd_helo_required = yes smtpd_helo_restrictions = permit_mynetworks, reject_invalid_hostname, reject_non_fqdn_hostname smtpd_recipient_restrictions = regexp:/etc/postfix/recipient_regexp, permit_sasl_authenticated, reject_non_fqdn_recipient, reject_unknown_recipient_domain, permit_mynetworks, reject_unauth_destination smtpd_sasl_auth_enable = yes smtpd_sender_restrictions = permit_mynetworks, reject_unauth_pipelining, reject_non_fqdn_sender, reject_unknown_sender_domain, permit smtpd_soft_error_limit = 5 smtpd_timeout = 160 strict_rfc821_envelopes = yes transport_maps = hash:/etc/postfix/transport unknown_local_recipient_reject_code = 550 unverified_recipient_reject_code = 550 virtual_alias_domains = hash:/etc/postfix/virtdoms virtual_alias_maps = hash:/etc/postfix/virtmaps I would take a quick scan through the various smtpd_*_limit and smtpd time related parameters. I don't have a great memory for these, but I do something like postconf | grep -i smtpd | grep -i limit to find them and then look them up in http://www.postfix.org/postconf.5.html The first ones that I can remember are ones like smtpd_client_connection_rate_limit smtpd_client_connection_count_limit . Keep in mind that reducing some limits can make things worse in certain cases if the machine is being hit harder with failed connection attempts. I can't be more specific cause it's been a while since I've played with these. You also need to check your log files, run tcpdump etc and determine the patterns of spam attacks that are taking you down and target those initially in your strategy. Are you getting hundreds of connections from a small number of servers or just lots of different servers? Also in your various restriction policies, try to find the least costly mechanisms for rejecting things as early on as possible. For example, rejecting messages that: 1) do not have local recipients on your system 2) do not have valid domainnames in the smtp sender, or smtp hello name 3) do not have valid reverse mappings in the dns of the mail server 4) also optionally checking for matching of the reverse and forward dns 5) rbl Note that, python or perl written policy daemon is usually way more costly than the builtin postfix checks, so try to weed out as much as possible with least resource intensive checks. The rbls are useful, but watchout for too many dns based checks and make sure your dns setup is strong before relying on it too heavily. There are lots of options like the ones I list above in the postfix configuration parameters. If you read through the postconf web page link above, you'll find ones that are interesting to you. Unfortunately one of the costs is that some legitimate companies and ISP's out there have their mailservers misconfigured and can get some complaints from your users that their mail is not getting through, in which case I end up both notifying admins about misconfigured servers and also adding exceptions to my checking, usually through a simple berkeley db based exception file. Here are some of the check that I run. Be careful with smtpd_r
Re: timeout problem on inbound and outbound SMTP
Wietse Venema wrote:
Nataraj:
I am extensively familiar with my main.cf file and am aware of exactly
what changes have been made. This mail server has been running stably
for years and I am not yet aware of changes to the mail configuration
files surrounding the time when this problem came up.
There are quite a number of mailserver packages out their that use
postfix as the MTA. I'm extensively familiar with the architectures of
both the postfix and vpostmaster systems and I assure you that there is
no front end. The port 25 service on my mailserver is the postfix
smtpd. Vpostmaster, is simply a policy and transport agent with spam
management and a web based GUI for management. It's quite nice and I
highly recommend it to people.
If the Postfix SMTP server hangs after 10s and no such smtpd_timeout
is in main.cf, then either the smtpd_timeout value is taken from
master.cf, or it is taken from a different main.cf file.
It is worthwhile at this point to do
find / -name main.cf -ls
and see what shows up.
Wietse
Thank you everyone for your helpful responses. I've narrowed the problem
down further, though it is not solved yet. It does not appear to be
specific to postfix. I've written a simple program with a select
statement that delays 10 seconds when run on any of my own computers,
but when run on the virtual machine hosting my mailserver it returns
immediately. I'm suspecting a vmware related problem, but I don't know
yet...
Here's the program
#include
/* #include */
#include
#include
#include
int main() {
fd_set set;
struct timeval timeout;
int filedes = STDIN_FILENO;
FD_ZERO (&set);
FD_SET (filedes, &set);
timeout.tv_sec = 10;
timeout.tv_usec = 0;
select(FD_SETSIZE, &set, NULL, NULL, &timeout);
}
Re: timeout problem on inbound and outbound SMTP
Wietse Venema wrote: Nataraj: Thank you everyone for your helpful responses. I've narrowed the problem down further, though it is not solved yet. It does not appear to be specific to postfix. I've written a simple program with a select statement that delays 10 seconds when run on any of my own computers, but when run on the virtual machine hosting my mailserver it returns immediately. I'm suspecting a vmware related problem, but I don't know yet... Accurate time keeping is one of the dirty secrets of virtualization. Wietse http://www.vmware.com/pdf/vmware_timekeeping.pdf Thank you Wietse, I needed something to lighten my day... Perhaps I'm having an experience of time converging to 0. Now, if I could only get everything done in that timeframe Nataraj
Re: Allowing e-mails to be relayed from a dynamic IP
Mike A. Leonetti wrote: I want to relay messages coming through a server with a dynamic IP (Exchange) through my postfix. My postfix my smtpd_recipient_restrictions already has a "hash:/etc/postfix/allowed_relays" option on it, and I've tried adding the Dynamic DNS name that resolves to that IP address and put it in the list but it still gave me an "relay access denied" error. Is there another way to do it? Thanks. Commonly this is done us using SASL authentication, ideally with TLS for added security. See the relevant documentation in http://www.postfix.org/docs.html The first round I used Cyrus SASL, but I went to dovecot with my last upgrade and it was much less work to get running and has performed reliably. With dovecot, you also install it with the dovecot pop/imap server since they are integrated, but the SASL functionality is available for other purposes. Nataraj
Re: Stopping spammers extreme
Noel Jones wrote: On 5/5/2010 11:40 AM, Appliantologist wrote: It seems pretty straight forward to me. If you dont have any non-local users sending mail using this server you could just shut down port 25. For those virtual-file id users use port 587 with smtp authentication.Forwarding for those users is not relevant here. Hello, I was assuming this would be pretty easy, I'm a little surprised and now messing with this amavid-wtf I can't shut down port 25, since we accept mail for sites and forward it offsite. I don't have any users sending mail via our SMTP, they all use gmail boxes and their servers. I only want to accept mail that is to an address listed in some file somewhere, like /etc/postfix/virtual OR is sent by the local host. What's really interesting is my spam fests are normally preceeded by a /var with no free inodes left due to disk errors. 100% usage till I run fsck on it. I have to figure who ever is doing this crap obviously is targeting postfix and probably reads these same lists. It's unclear weather you might have a relaying problem or not. Are there large numbers of messages destined for delivery to other sites in your mailq? Is the spam that you are seeing addressed to local delivery addresses at your site? Are there bounce messages in your mailq? I realize you are forwarding local mail to gmail. So is your mailq filled up with spam getting forwarded to gmail? nataraj
Re: Allowing e-mails to be relayed from a dynamic IP
Mike A. Leonetti wrote: Thanks for the reply, Nataraj. I did see that online and the server does have SASL Auth working, but we are having a difficult time getting it to try and provide a username/password on the Exchange server so I was wondering if there was a way to get around that. Personally, I have more experience with replacing exchange servers than with making them work, however I would think that recent versions of exchange would support SASL authentication. If you really can't implement authentication, I can think of a few other ideas. I'm guessing that this situation might be internal to your organization. If you DNS is fairly secure and the exchange server updates dynamic DNS reliably, you could check that somehow. Alternatively you could run an SMTP submission server on another port and protect it with a dynamically updated firewall list, or something like fwknop, but you would probably be challenged with getting the client to authenticate with FWKNOP. These later approaches could have tiny holes in them, depending in the freqency with which your dynamic ip address changes and how quickly you verify on the new ip address, but if this is within your corporate network it might not be too bad. Nataraj
Re: Allowing e-mails to be relayed from a dynamic IP
Noel Jones wrote: On 5/5/2010 1:06 PM, Nataraj wrote: Mike A. Leonetti wrote: Thanks for the reply, Nataraj. I did see that online and the server does have SASL Auth working, but we are having a difficult time getting it to try and provide a username/password on the Exchange server so I was wondering if there was a way to get around that. Personally, I have more experience with replacing exchange servers than with making them work, however I would think that recent versions of exchange would support SASL authentication. If you really can't implement authentication, I can think of a few other ideas. I'm guessing that this situation might be internal to your organization. If you DNS is fairly secure and the exchange server updates dynamic DNS reliably, you could check that somehow. Alternatively you could run an SMTP submission server on another port and protect it with a dynamically updated firewall list, or something like fwknop, but you would probably be challenged with getting the client to authenticate with FWKNOP. These later approaches could have tiny holes in them, depending in the freqency with which your dynamic ip address changes and how quickly you verify on the new ip address, but if this is within your corporate network it might not be too bad. Nataraj Also, if the exchange server is a box under your control you can use a VPN. OpenVPN is pretty easy to set up and works under Windows and virtually every flavor of *nix. http://openvpn.net/index.php/open-source.html The vpn idea is a good one, however I would want to make sure to either have iptables access lists or something to protect my mailserver from unrestricted access from the exchange server. This is fairly easy if you know how to use iptables or one of the user friendly front ends to it. Nataraj
Re: timeout problem on inbound and outbound SMTP
Nataraj wrote: Hi, I would appreciate any suggestions anyone can offer on the following problem that I'm having with postfix... I'm running postfix+pgsql-2.3.3-2.1.el5_2 on a CentOS 5.4 server. I see what looks likes a server in stress mode as described in http://www.postfix.org/STRESS_README.html except the odd think about it is that the server is not heavily loaded and I sure can't see where it's exceeding any process limits. What's even odder is it doesn't appear that the stress code is implemented in this version. If I telnet to port 25 I get an immediate SMTP greeting followed in 10 seconds by 421 4.4.2 mymail.com Error: timeout exceeded and the connection being closed. The following maillog entry is logged: May 3 16:44:06 mymail postfix/smtpd[22573]: timeout after CONNECT from 173-12-149-200.client.myisp.com[173.12.149.200] This is like this constantly. I see 0-4 smtpd processes on the server at any one time (I'm not sure if it's limited at 4, I just haven't seen more). There are a similar number of policy daemons. There is a "-" in the maxproc field for smtpd in master.cf. From everything I can tell the default is a limit of 100. I do run a policy daemon (vpostmaster). I've changed its maxproc field to 0 per the recommendation in the STRESS_README (and restarted postfix). It's master.cf entry looks like this... vpm-pfpolicy unix - n n - 0 spawn user=vpostmaster argv=/usr/lib/vpostmaster/postfix/vpm-pfpolicy I also get lots of log entries like this for timeouts on the policy daemon: May 2 05:36:20 aspen postfix/spawn[6003]: warning: /usr/lib/vpostmaster/postfix/vpm-pfpolicy: process id 6004: command time limit exceeded This problem is solved. The problem was caused when my hosted VM was migrated from a VMware 3.5 to a VMware 4 server and the vmware config for my VM had the OS configured to Ubuntu instead of redhat. Well in vmware ESXI 4.0 it appears that they are more clever about how the clocking interfaces with options that are set in the kernel (Centos 5.4 does not use a TICKLESS kernel, but current UBUNTU releases do). After the vmware config was changed my mail server problems disappeared. Thank you, nataraj
Re: Allowing only certain From:... to send email to a specific To:... possible?
Patrick Lists wrote: On 05/06/2010 04:07 PM, Noel Jones wrote: [snip] You can use an external policy service such as postfwd to compare envelope sender and recipient. But it sounds as if you really need to compare the From: header with the envelope recipient. You'll need a content_filter or milter to do that. -- Noel Jones Thanks Noel. I was hoping I could do it within Postfix to keep things lean & mean. Will check out your suggestions. Kind regards, Patrick I use vpostmaster (which uses postfix as the mta) for this (though you'd probably want to do a clean install, or at least save your postfix config before installing). vpostmaster (which uses virtual mailboxes, i.e. not users in /etc/password) allows you to define an extension character, usually '-'. So if fred is a vpostmaster mailbox, you can also send email to fred-string1 and fred-string2 and then use the vpostmaster GUI to define white/blacklist rules to reject or accept fred-string1 etc. So if I give my online banking service the name fred-bank1 and then I start receiving viagra spam addressed to fred-bank1, I know that bank1 has leaked my email address. (There have been large class action law suits against presumably secure finanacial institutions who have had their customers email addreses compromised). Anyway, you just block fred-bank1 when this happens and vpostmaster will cause postfix to reject fred-bank1 during the recipient phase of the SMTP session. The nice thing abouit this approach is you don't have to edit any configuration to create new email addresses, you only have to do something when you want to block one. The white/blacklists supports checking on sender, recipient, helo name, remote hostname and remote ip address and you can accept, reject, continue. Nataraj
Re: Newsletter server setup questions
Dragan Zubac wrote:
Hello
I need to setup Postfix mail server that will be used only
occasionally for sending out newsletters and other automated emails.
There are 4 boxes,1. is the box where Postfix is installed,boxes 2.,3.
and 4. are boxes that have various scripts that will use SMTP to
connect to box 1. in order to send emails.
The requirements are as follows:
1. All Postfix mail logs must be able to check via some kind of web
interface,where one will be able to see the MessageID,Subject,To,
Date,Time and status of sent message,the similar can be seen on the
following URL:
http://www.kyapanel.com/images/rsgallery/original/kp8.png
(although not necessarily using this software)
The purpose of this requirement is for somebody to be able to find
out if any of the emails sent out was not delivered,and if not,
what was the reason.
2. The scripts will send 'important' and 'less important' emails. If
script is programmed to send 'important' ones,the copy of email
must be sent to a separate account that will archive all sent
emails (automatically BCC or something similar).
If script will send 'less important' email,there is no need to keep
a copy of sent email.
The purpose of this request is for somebody to be able to find out
the same copy of email if a recipient confirm that he has not
received that very same email.
3. Some emails will have kind of 'no-r...@domain.com' email address in
'From' field. If recipient of this email by accident or so
does send a reply back to 'no-re...@domain.com',he should receive
an error email ('User does not exist' or similar error) and
also certain local user at 'domai.com' should be alerted that an
attempt of email delivery to 'no-re...@domain.com' has been
occurred.
Could You please share Your ideas/thoughts how this can be achieved or
so ?
Sincerely
Dragan Zubac
I'm not sure if this is useful or not, but the two most common open
source pieces of software for managing mailing lists are mailman and
majordomo (the server used for this list). Both can be used with
postfix. mailman is written in python and has a web based interface to
allow users to subscribe as well as for management purposes. Majordomo
is written in perl, and I believe the administration as well as
subscription lists is still managed in email. Personally, I like the
web interface of mailman as well as the way that it handles headers.,
though I have not managed a list myself with mailman. I don't know
what kind of reporting these packages provide, but everything is in log
files and python/perl code can be easily customized, so if your skilled
at working with python/perl you could add report pages and features as
needed. Both packages have been used quite extensively for large
mailing lists.
http://www.greatcircle.com/majordomo/
http://www.gnu.org/software/mailman/index.html
Outside of these options, there are commercial services that are
relatively inexpensive that provide mailing list managers with reporting
functions that marketing types tend to want to see. I'm not personally
a big fan of these things, and many are used for what I consider
borderline spamming, but sometimes it's easier to farm things out than
implement everything yourself. If it interests you, I can send you the
name of one that one of my clients likes, though I have no personal
experience.
Nataraj
Re: Empty 'local_recipient_maps =' and security
Noel Jones wrote: On 5/12/2010 1:56 AM, Aniruddha wrote: Hi, I have set up postfix with a mail_transport to Zarafa. To fix an ' Recipient address rejected: User unknown in local recipient table' error I have to put an empty 'local_recipient_maps =' in postfix's main.cf. The correct solution is to point that parameter at a map containing all your valid users. Often this is caused by listing a virtual_mailbox_domain in mydestination. Don't do that. I do wonder about the security implications of setting this option. If I understand the documentation correctly it isn't wise to set this option to empty. Is this correct? Besides the information below I can't find much information about this option.Thanks in advance! Accepting mail for undeliverable recipients will cause postfix to send non-delivery notices -- bounces -- to the reported envelope sender. The envelope sender on spam is frequently either a non-working address or an innocent third party. This has two results; your queue is filled with undeliverable bounces, and you send bounces to innocent third parties. The full queue will badly affect delivery of legit mail, and the backscatter you send to innocent people will get you blacklisted. Rejecting the mail during the initial SMTP session avoids these problems. -- Noel Jones Postfix provides many different mechanisms to access different formats of tables and/or define policy agents that can check things like this, even if the database is part of another software package. For example, I have my user database in the vpostmaster package and my smtpd_recipient_restrictions include (directly after the permit_sasl_authenticated and permit_mynetworks), check_recipient_access proxy:pgsql:/etc/postfix/vpm_recipient_access The file vpm_recipient_access contains a single rather complex nested postgres sql statement which checks the data base and verifies both the existance of the domain and username on the local mail server. It then returns DUNNO if the recipient address is valid or "REJECT No such domain %d" or "REJECT No such user %u in domain %d" You could also, of course, implement this with simple berkely db files, or by writing a policy agent. Previously the vpostmaster policy daemon was validating the recipients, however I moved this into the postfix sql interface because it is much faster and causes sooner rejection of bad reciepients in the smtpd session, increasing the performance of my mail server. Nataraj Nataraj
Re: I've inherited a botnet target
brian wrote: On 10-05-26 03:55 PM, Noel Jones wrote: Some random suggestions... Use a bogus MX record for the old domain if that domain has no valid mail recipients. Of course, some bots will connect to your A record anyway... OK, I like the sound of that. Per your other email, I think I did, a long time ago, learn about A being used in the absence of an MX. That seems familiar now. Thanks for the tip. You can use "reject_unlisted_recipient" early in your smtpd_recipient_restrictions to dump connections to bad users early. A later RBL check will only apply to valid recipients. Set smtpd_hard_error_limit to a low number, such as 2, to disconnect clients after just a few errors. Set smtpd_error_sleep_time to 0 to get rid of bad clients without delay. I'll give all that a try. Does this order seem alright? smtpd_recipient_restrictions = permit_mynetworks, reject_unlisted_recipient, reject_invalid_hostname, reject_non_fqdn_hostname, reject_non_fqdn_recipient, reject_non_fqdn_sender, reject_unauth_destination, reject_unknown_recipient_domain, reject_unauth_pipelining I'll bet the postfix 2.7 "postscreen" feature will get rid of 1/2 or more of the bots before they every talk to you. Postfix 2.7 allows you to specify 521 for the various *_reject_code parameters to signal a disconnect. I've just been having a look at that. It does seem to be something very useful in this situation. But, maybe the bogus MX will solve my problems. Increase the max number of smtpd listeners in master.cf to the highest number your memory will allow. What's the best way of determining that? I have had success with lowering these 3 parameters as well, if even temporarily during an attack. The values should be chosen based on how many legitamite incoming connections you receive (so as not to limit those). Also beware that in some cases rate limiting can cause a buildup of incoming connection requests which could cause problems with your tcp/ip stack, but generally these have worked for me. smtpd_client_connection_count_limit = 6 smtpd_client_connection_rate_limit = 6 smtpd_client_message_rate_limit = 6 If you never want mail destined for the domain example.com, you can use setup a rule to reject it. I would think this would be quite fast. I would also use the bogus MX record to prevent as much traffic as possible form hitting the server. smtpd_recipient_restrictions = permit_mynetworks check_recipient_access hash:/etc/postfix/smtpd_recipient_access REST OF YOUR RESTRICTIONS In the file smtpd_recipient_access example.comREJECT "NO THANK YOU, WE ARE NOT EXCEPTING MAIL" Here's a simple script to build the hash file from smtpd_recipient_access #! /bin/bash /usr/sbin/postmap hash:/etc/postfix/smtpd_sender_access /bin/chgrp postfix smtpd_sender_access* /bin/chmod g=r,o-rwx smtpd_sender_access* Nataraj
Re: I've inherited a botnet target
Nataraj wrote: brian wrote: On 10-05-26 03:55 PM, Noel Jones wrote: Some random suggestions... Use a bogus MX record for the old domain if that domain has no valid mail recipients. Of course, some bots will connect to your A record anyway... OK, I like the sound of that. Per your other email, I think I did, a long time ago, learn about A being used in the absence of an MX. That seems familiar now. Thanks for the tip. You can use "reject_unlisted_recipient" early in your smtpd_recipient_restrictions to dump connections to bad users early. A later RBL check will only apply to valid recipients. Set smtpd_hard_error_limit to a low number, such as 2, to disconnect clients after just a few errors. Set smtpd_error_sleep_time to 0 to get rid of bad clients without delay. I'll give all that a try. Does this order seem alright? smtpd_recipient_restrictions = permit_mynetworks, reject_unlisted_recipient, reject_invalid_hostname, reject_non_fqdn_hostname, reject_non_fqdn_recipient, reject_non_fqdn_sender, reject_unauth_destination, reject_unknown_recipient_domain, reject_unauth_pipelining I'll bet the postfix 2.7 "postscreen" feature will get rid of 1/2 or more of the bots before they every talk to you. Postfix 2.7 allows you to specify 521 for the various *_reject_code parameters to signal a disconnect. I've just been having a look at that. It does seem to be something very useful in this situation. But, maybe the bogus MX will solve my problems. Increase the max number of smtpd listeners in master.cf to the highest number your memory will allow. What's the best way of determining that? I have had success with lowering these 3 parameters as well, if even temporarily during an attack. The values should be chosen based on how many legitamite incoming connections you receive (so as not to limit those). Also beware that in some cases rate limiting can cause a buildup of incoming connection requests which could cause problems with your tcp/ip stack, but generally these have worked for me. smtpd_client_connection_count_limit = 6 smtpd_client_connection_rate_limit = 6 smtpd_client_message_rate_limit = 6 If you never want mail destined for the domain example.com, you can use setup a rule to reject it. I would think this would be quite fast. I would also use the bogus MX record to prevent as much traffic as possible form hitting the server. smtpd_recipient_restrictions = permit_mynetworks check_recipient_access hash:/etc/postfix/smtpd_recipient_access REST OF YOUR RESTRICTIONS In the file smtpd_recipient_access example.comREJECT "NO THANK YOU, WE ARE NOT EXCEPTING MAIL" Here's a simple script to build the hash file from smtpd_recipient_access #! /bin/bash /usr/sbin/postmap hash:/etc/postfix/smtpd_sender_access /bin/chgrp postfix smtpd_sender_access* /bin/chmod g=r,o-rwx smtpd_sender_access* Nataraj You won't really need the check_recipient_access if example.com is not configured as a local domain. Nataraj
Re: I've inherited a botnet target
Stan Hoeppner wrote: brian put forth on 5/26/2010 8:28 PM: On 10-05-26 09:03 PM, Stan Hoeppner wrote: brian put forth on 5/26/2010 1:53 PM: FWIW, aside from aliases for the usual postmaster, abuse, and webmaster addresses, this domain has just 2 actual addresses to be maintained. So, might a whitelist approach be the way to go? Or, is this something i should leave to iptables/fail2ban? Care to share some of the spammer IP address info? Is this botnet traffic or snowshoe? If snowshoe, I might be able to provide you with a complete list of netblocks to blacklist, solving your problem with a simple edit or two. Here you go: http://pastebin.com/DMgZsNCc I dunno about snowshoe. That was the first I'd seen the term. But it looks like it could be, as I understand it. I'm really not knowledgable enough to say. I checked out a sampling of those IPs. They're a combination of bot and snowshoe, mostly bot. Typical spam stream, but apparently at a higher rate than what your VPS can effectively handle via standard Postfix smtpd restrictions. As others have stated, Postscreen should be a big help to you given that most of this is bot spam--exactly what Postscreen was designed to address. How does rate limiting work in conjunction with postscreen? Can the various rate limits be applied to postscreen or would rate limiting no longer be necessary? I run in a vmware virtual machine which used to fall on its knees from both bot and snowshoe attacks and since I added the rate limits that I previously posted, I haven't had any major problems (been running with the rate limits for several years). I often see attacks like the one below where it will log these rate limit exceeded messages over the course of several minutes before the attackers go away. And yes, I do see attacks that come from multiple IP's. May 26 15:55:42 aspen postfix/smtpd[19600]: warning: Connection rate limit exceeded: 22 from 74-218-134-95.pool.ukrtel.net[95.134.218.74] for service smtp May 26 15:55:42 aspen postfix/smtpd[19600]: disconnect from 74-218-134-95.pool.ukrtel.net[95.134.218.74] May 26 15:55:42 aspen postfix/smtpd[17267]: connect from 74-218-134-95.pool.ukrtel.net[95.134.218.74] May 26 15:55:42 aspen postfix/smtpd[17267]: warning: Connection rate limit exceeded: 23 from 74-218-134-95.pool.ukrtel.net[95.134.218.74] for service smtp May 26 15:55:42 aspen postfix/smtpd[17267]: disconnect from 74-218-134-95.pool.ukrtel.net[95.134.218.74] May 26 15:56:17 aspen postfix/smtpd[21694]: connect from 114-26-181-192.dynamic.hinet.net[114.26.181.192] Nataraj
Re: postscreen questions
Andy Dills wrote: I've been investigating postscreen, as we've been address probed/bombed for years, as we have a few domains that are very old (well, early 90s) that had a lot of users back in the dialup days. Our approach was to just throw hardware at the problem, and we've had a whole cluster of servers just sending out 550s all day long for years now. We don't do any RBL checks at the postfix level; we have amavisd-new handle all of that via spamassassin. I'm hesitant to allow a single blacklist to determine the fate of mail acceptance, especially when we have a very low false negative rate with amavisd/SA. Essentially, we'd rather throw hardware at the problem than potentially reject legit mail. My primary question is, would we see significant improvement by using postscreen if we don't use RBLs? Also, would postscreen_cache_map work with a mysql backend? Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- Using things like amavisd and spamassasin besides being very costly in terms of performance, is far more vulnerable to security exploits than rejecting as many connections as possible at an earlier time. I have used the various checks for valid domain names, helo names, etc, in conjunction with the RBL's to minimize scanning with spamassasin. I use restriction classes to define more and less conservative policys for different domains and even specific users when necessary. smtpd_restriction_classes = restrictive, permissive restrictive = reject_rbl_client zen.spamhaus.org reject_rbl_client dul.dnsbl.sorbs.net reject_rbl_client bl.spamcop.net permissive = reject_rbl_client pbl.spamhaus.org reject_rbl_client dul.dnsbl.sorbs.net check_recipient_access hash:/etc/postfix/smtpd_recipient_access smtpd_recipient_access contains: mydomain.com restrictive # I get the abuse mail and don't want to see alot of spam ab...@otherdomain.com restrictive otherdomain.org permissive 127.0.0.1 OK The permissive class is very conservative and should cause practically no false positives. Even my restrictive class includes rbls known to have extremely low false positive rates. Spamhaus is known to be one of the most accurate rbl's with an excellent hit rate and low false positives. If you have a large site, check their web pages, since they do charge for high volume query rates and will block your access if you exceed the free limit. Nataraj
Re: permit_mynetworks in smtpd_helo_restrictions
p...@alt-ctrl-del.org wrote: Hello postfix admins, I have always placed all restrictions in smtpd_recipient_restrictions. Over the last few days, I have been experimenting with breaking the restrictions up into client, helo, sender, etc. I ran into something odd (to me), when permit_mynetworks is in smtpd_helo_restrictions. --- My pretend config: Version 2.6 host ip: 10.123.45.37 mynetworks = 127.0.0.0/8, 10.123.45.0/24, 10.123.46.0/24 relay_domains = $mynetworks, $transport_maps smtpd_helo_restrictions = permit_mynetworks, reject_non_fqdn_helo_hostname smtpd_client_restrictions = permit_mynetworks, reject_unknown_reverse_client_hostname, check_reverse_client_hostname_access regexp:/etc/postfix/rhv1, reject_rbl_client bla.bla.org smtpd_sender_restrictions = reject_non_fqdn_sender, reject_unknown_sender_domain --- So I notice that the logs show that when a evil client sends a helo name of 10.123.45.37 (my ip), they sometimes get stopped by the reject_unknown_reverse_client_hostname, other times by the check_reverse_client_hostname_access map, and other times by one of the rbl checks. So I whip up a check_helo_access map with 10.123.45.37 521 Go Away (postmap -q shows that it works). Then change smtpd_helo_restrictions to smtpd_helo_restrictions = permit_mynetworks, check_helo_access /etc/postfix/heloaccess, reject_non_fqdn_helo_hostname But clients that send a helo of 10.123.45.37, still get as far as reject_unknown_reverse_client_hostname, or check_reverse_client_hostname_access map, or one of the rbl checks. p...@alt-ctrl-del.org Then I try the check_helo_access in smtpd_client_restrictions. smtpd_client_restrictions = permit_mynetworks, check_helo_access ..., etc. But clients that send a helo of 10.123.45.37, still get as far as reject_unknown_reverse_client_hostname, or check_reverse_client_hostname_access map, or one of the rbl checks. If I remove permit_mynetworks from smtpd_helo_restrictions, the rules in my check_helo_access map "hit" and are applied. --- In my line of thinking, $mynetworks is a list of IP addresses. The client hostname is a string. I would think that having permit_mynetworks in smtpd_helo_restrictions, would be applied as "accept any helo, from hosts in mynetworks". But it appears that permit_mynetworks is testing the helo string, against the list of IP's in $mynetworks (as strings), then allowing it to pass. Is this the way it's supposed to behave? It seems wrong to me. If this is the way it's supposed to behave, then what about permit_mynetworks in smtpd_client_restrictions? Let's say an evil client sets the reverse dns for their IP to "10.123.45.37". Would permit_mynetworks in smtpd_client_restrictions, then permit the client to pass? I would be inclined to agree with you on what the desired behavior should be. What are your smtpd_recipient_restrictions? Also, what happens if you remove the permit_mynetworks from smtpd_helo_restrictions and then try the hello command that matches an address in mynetworks? What I'm asking, is if the helo restrictions is really where the permit is happening? Nataraj
Re: permit_mynetworks in smtpd_helo_restrictions
Nataraj wrote: p...@alt-ctrl-del.org wrote: Hello postfix admins, I have always placed all restrictions in smtpd_recipient_restrictions. Over the last few days, I have been experimenting with breaking the restrictions up into client, helo, sender, etc. I ran into something odd (to me), when permit_mynetworks is in smtpd_helo_restrictions. --- My pretend config: Version 2.6 host ip: 10.123.45.37 mynetworks = 127.0.0.0/8, 10.123.45.0/24, 10.123.46.0/24 relay_domains = $mynetworks, $transport_maps smtpd_helo_restrictions = permit_mynetworks, reject_non_fqdn_helo_hostname smtpd_client_restrictions = permit_mynetworks, reject_unknown_reverse_client_hostname, check_reverse_client_hostname_access regexp:/etc/postfix/rhv1, reject_rbl_client bla.bla.org smtpd_sender_restrictions = reject_non_fqdn_sender, reject_unknown_sender_domain --- So I notice that the logs show that when a evil client sends a helo name of 10.123.45.37 (my ip), they sometimes get stopped by the reject_unknown_reverse_client_hostname, other times by the check_reverse_client_hostname_access map, and other times by one of the rbl checks. So I whip up a check_helo_access map with 10.123.45.37 521 Go Away (postmap -q shows that it works). Then change smtpd_helo_restrictions to smtpd_helo_restrictions = permit_mynetworks, check_helo_access /etc/postfix/heloaccess, reject_non_fqdn_helo_hostname But clients that send a helo of 10.123.45.37, still get as far as reject_unknown_reverse_client_hostname, or check_reverse_client_hostname_access map, or one of the rbl checks. p...@alt-ctrl-del.org Then I try the check_helo_access in smtpd_client_restrictions. smtpd_client_restrictions = permit_mynetworks, check_helo_access ..., etc. But clients that send a helo of 10.123.45.37, still get as far as reject_unknown_reverse_client_hostname, or check_reverse_client_hostname_access map, or one of the rbl checks. If I remove permit_mynetworks from smtpd_helo_restrictions, the rules in my check_helo_access map "hit" and are applied. Whoops, I missed that you already tried removing permit_mynetworks. --- In my line of thinking, $mynetworks is a list of IP addresses. The client hostname is a string. I would think that having permit_mynetworks in smtpd_helo_restrictions, would be applied as "accept any helo, from hosts in mynetworks". But it appears that permit_mynetworks is testing the helo string, against the list of IP's in $mynetworks (as strings), then allowing it to pass. Is this the way it's supposed to behave? It seems wrong to me. If this is the way it's supposed to behave, then what about permit_mynetworks in smtpd_client_restrictions? Let's say an evil client sets the reverse dns for their IP to "10.123.45.37". Would permit_mynetworks in smtpd_client_restrictions, then permit the client to pass? I would be inclined to agree with you on what the desired behavior should be. What are your smtpd_recipient_restrictions? Also, what happens if you remove the permit_mynetworks from smtpd_helo_restrictions and then try the hello command that matches an address in mynetworks? What I'm asking, is if the helo restrictions is really where the permit is happening? Nataraj
