Re: Mail server recently became an open relay

[Rich Wales]

John Fawcett wrote:

> One thing I would suggest looking at is if there is a web server running
> on the same host it may be allowing email to be injected into postfix
> via smtp on the loopback interface using some scripting language like
> php or others.

I suppose that's possible.

I spent some time last night cleaning up old stuff from the server in
question -- and also rebooting the box for good measure -- so the
problem *might* just go away at this point.

Before I can say anything more about this, unfortunately, I'll probably
need to wait for another incident similar to the preceding ones, and try
to capture more evidence while the problem is ongoing.  If it never
happens again, then maybe it was the fault of an old PHP web page which
I have removed.

If the problem were in fact due to a hijacked PHP page, btw, would this
necessarily require the page to be using e-mail or TCP connections
already for its own legitimate purposes, but being co-opted by a hacker
to nefarious ends?  Or could *any* PHP script theoretically be infected
in a way that would cause this misbehaviour?

Rich Wales
ri...@richw.org

Re: Mail server recently became an open relay

[John Fawcett]

On 19/10/2020 20:50, Rich Wales wrote:
> John Fawcett wrote:
>
>> One thing I would suggest looking at is if there is a web server running
>> on the same host it may be allowing email to be injected into postfix
>> via smtp on the loopback interface using some scripting language like
>> php or others.
> I suppose that's possible.
>
> I spent some time last night cleaning up old stuff from the server in
> question -- and also rebooting the box for good measure -- so the
> problem *might* just go away at this point.
>
> Before I can say anything more about this, unfortunately, I'll probably
> need to wait for another incident similar to the preceding ones, and try
> to capture more evidence while the problem is ongoing.  If it never
> happens again, then maybe it was the fault of an old PHP web page which
> I have removed.
>
> If the problem were in fact due to a hijacked PHP page, btw, would this
> necessarily require the page to be using e-mail or TCP connections
> already for its own legitimate purposes, but being co-opted by a hacker
> to nefarious ends?  Or could *any* PHP script theoretically be infected
> in a way that would cause this misbehaviour?
>
> Rich Wales
> ri...@richw.org

Sorry not to be able to give a definitive answer. Typical mail injection
via php will use a script that already calls the php mail function or
similar functions that open the smtp connection. But there are other
attack vectors that are possible that allow hackers to gain the
privileges of the web server user. The web server user is often allowed
to inject mail to localhost without any authentication (under the
permit_mynetworks syntax in postfix main.cf).

John

Re: Host not found?

[Bob Proulx]

Joey J wrote:
> I'm trying to understand why this is telling me host not found.
> On that same server if I nslookup the ip it does resolve.
> 
> Oct 18 16:00:51 mgw postfix/smtpd[24119]: NOQUEUE: reject: RCPT from
> unknown[199.5.50.180]: 450 4.7.1 : Helo command rejected: Host
> not found; from= to= proto=ESMTP
> helo=

In addition to what has already been said...  I look up the SPF record
for Audi.com and find:

$ host -t txt Audi.com | grep spf
Audi.com descriptive text "v=spf1 include:cust-spf.exacttarget.com 
ip4:199.5.47.0/24 ip4:91.198.139.136/31 -all"

$ host -t txt cust-spf.exacttarget.com | grep spf
cust-spf.exacttarget.com descriptive text "v=spf1 ip4:64.132.92.0/24 
ip4:64.132.88.0/23 ip4:66.231.80.0/20 ip4:68.232.192.0/20 ip4:199.122.120.0/21 
ip4:207.67.38.0/24 " "ip4:207.67.98.192/27 ip4:207.250.68.0/24 
ip4:209.43.22.0/28 ip4:198.245.80.0/20 ip4:136.147.128.0/20 
ip4:136.147.176.0/20 ip4:13.111.0.0/16 ip4:161.71.32.0/17 -all"
$ host -t txt cust-spf.exacttarget.com | grep spf | grep 199.5
...nothing...

Since 199.5.50.180 does not appear in the allowance for the SPF
records that I can see (I inspected by eye, did I miss something?)
then the final "-all" would have caused the message to be rejected by
SPF policy *if* it had been not been already rejected by the policy of
using "reject_unknown_client_hostname" .

Which means this SMTP transaction had multiple problems making it very
likely a spammer as the most simple explanation.  Or simply very deeply
misconfigured if not.

Bob

Re: Mail server recently became an open relay

[Jaroslaw Rafa]

Dnia 19.10.2020 o godz. 21:12:20 John Fawcett pisze:
> Sorry not to be able to give a definitive answer. Typical mail injection
> via php will use a script that already calls the php mail function or
> similar functions that open the smtp connection. But there are other
> attack vectors that are possible that allow hackers to gain the
> privileges of the web server user.

Very often hackers abuse web pages that allow users to upload files to the
web server. If the input is not correctly sanitized, it may be possible to
upload an arbitrary php script and get it executed.

There were multiple attacks based on this scenario.
-- 
Regards,
   Jaroslaw Rafa
   r...@rafa.eu.org
--
"In a million years, when kids go to school, they're gonna know: once there
was a Hushpuppy, and she lived with her daddy in the Bathtub."

Re: Mail server recently became an open relay

[Bob Proulx]

Rich Wales wrote:
> If the problem were in fact due to a hijacked PHP page, btw, would this
> necessarily require the page to be using e-mail or TCP connections
> already for its own legitimate purposes, but being co-opted by a hacker
> to nefarious ends?  Or could *any* PHP script theoretically be infected
> in a way that would cause this misbehaviour?

*If* the host is running a site, such as Wordpress but there are also
many other possibilities, and if it is not absolutely up to date with
security upgrades current as of *TODAY*, then it is very likely that
the site has been compromised.

That's just the history of WP and other similar frameworks!  They are
allowed to do brain surgery on themselves without restriction and they
consist of a community of thousands and thousands of inexperienced
developers all submitting modules without a security focus.  When I
read John Fawcett's suggestion that it might be a web server
compromise I thought immediately, "Oh good suggestion!", since that is
so often a typical compromise case!

The default PHP "mail()" method sends mail by using the system's
/usr/sbin/sendmail interface rather than SMTP.

https://www.php.net/manual/en/mail.requirements.php
https://www.php.net/manual/en/function.mail.php

Bob

Limiting HELO spoofing in Postfix?

[Rich Wales]

This question is kind-of related to my recent open relay problem, which
at the moment seems most likely caused by a hacker invading my mail
server (possibly through a web service running on the same box) and
injecting fraudulent messages into my server directly via the localhost
interface.

Postfix "sort-of" recognizes this, because each of the fake messages
appears to contain a "Received:" header line (created by my Postfix)
saying the sender sent out a host name or address in its HELO, but my
Postfix knew that the message was really coming from 127.0.0.1.

My question is, is there any configuration option for Postfix to reject
mail in a situation where the sender is clearly spoofing its identity
like this during the SMTP conversation?

I suppose it might possibly be good enough in this situation to use an
option that rejects inbound mail claiming to be from some outside host
when in fact it's really coming from localhost.  I've looked at the
various smtpd_helo_restrictions, but it's not obvious to me which (if
any) of these can do what I want.

Obviously, this option (if it exists) would need to be used judiciously
so as not to block legitimate e-mail passing through milters and such.
But what I want to know is if any such option exists at all.

Rich Wales
ri...@richw.org

Re: Mail server recently became an open relay

[IL Ka]

If someone hacked your PHP script, he or she may add any code to it,
including code that connects to your smtpd and sends email.
In PHP one can use mail() function (which can use TCP connection to the
localhost:25 according to the settings in php.ini) or establish connection
directly.

As we can see from
>Oct 15 14:48:06 memoryalpha postfix/postscreen[18030]: CONNECT from
[127.0.0.1]:52138 to [127.0.0.1]:25

Some locally running process just connected to your server and send spam.
I think you can use iptables to log all connections to this port to catch
the pid of the culprit.

As a quick-and-dirty solution you can move your smtpd to the different port
(see master.cf).
Script would not know about the new port, hence it wouldn't be able to
connect to postfix.

Or you can move smtpd to the unix domain socket or even disable it and
use the "sendmail" interface instead.

Greping your scripts for "socket_connect" and "mail" is also worth doing.

Btw, this is not an "open relay": relaying mail from the localhost
(127.0.0.1) is the default postfix behavior because "mynewtorks =
127.0.0.0/8" in may installations,
and "smtpd_relay_restrictions" includes $mynetworks by default.

If you were running each website in the separate docker instance for
example, then you would be able to create the separate port for each
container in master.cf, and know for sure which one was hacked.





<#m_-2421472136311685306_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

On Mon, Oct 19, 2020 at 9:51 PM Rich Wales  wrote:

> John Fawcett wrote:
>
> > One thing I would suggest looking at is if there is a web server running
> > on the same host it may be allowing email to be injected into postfix
> > via smtp on the loopback interface using some scripting language like
> > php or others.
>
> I suppose that's possible.
>
> I spent some time last night cleaning up old stuff from the server in
> question -- and also rebooting the box for good measure -- so the
> problem *might* just go away at this point.
>
> Before I can say anything more about this, unfortunately, I'll probably
> need to wait for another incident similar to the preceding ones, and try
> to capture more evidence while the problem is ongoing.  If it never
> happens again, then maybe it was the fault of an old PHP web page which
> I have removed.
>
> If the problem were in fact due to a hijacked PHP page, btw, would this
> necessarily require the page to be using e-mail or TCP connections
> already for its own legitimate purposes, but being co-opted by a hacker
> to nefarious ends?  Or could *any* PHP script theoretically be infected
> in a way that would cause this misbehaviour?
>
> Rich Wales
> ri...@richw.org
>


Без
вирусов. www.avg.com

<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Re: Mail server recently became an open relay

[Bob Proulx]

Bob Proulx wrote:
> The default PHP "mail()" method sends mail by using the system's
> /usr/sbin/sendmail interface rather than SMTP.
>
> https://www.php.net/manual/en/mail.requirements.php
> https://www.php.net/manual/en/function.mail.php

Oh!  It depends upon the system's php.ini configuration.  For which
there might be many.  Here is an (older) 7.0 system for example.

/etc$ find php* -name php.ini
php/7.0/cli/php.ini
php/7.0/apache2/php.ini
php/7.0/fpm/php.ini

And the php.ini for both FPM and Apache contains:

[mail function]
; For Win32 only.
; http://php.net/smtp
SMTP = localhost
; http://php.net/smtp-port
smtp_port = 25

My bad for posting my previous misguided information.  Sorry about that.

Bob

Re: Limiting HELO spoofing in Postfix?

[Wietse Venema]

Rich Wales:
> This question is kind-of related to my recent open relay problem, which
> at the moment seems most likely caused by a hacker invading my mail
> server (possibly through a web service running on the same box) and
> injecting fraudulent messages into my server directly via the localhost
> interface.
> 
> Postfix "sort-of" recognizes this, because each of the fake messages
> appears to contain a "Received:" header line (created by my Postfix)
> saying the sender sent out a host name or address in its HELO, but my
> Postfix knew that the message was really coming from 127.0.0.1.
> 
> My question is, is there any configuration option for Postfix to reject
> mail in a situation where the sender is clearly spoofing its identity
> like this during the SMTP conversation?
> 
> I suppose it might possibly be good enough in this situation to use an
> option that rejects inbound mail claiming to be from some outside host
> when in fact it's really coming from localhost.  I've looked at the
> various smtpd_helo_restrictions, but it's not obvious to me which (if
> any) of these can do what I want.
> 
> Obviously, this option (if it exists) would need to be used judiciously
> so as not to block legitimate e-mail passing through milters and such.
> But what I want to know is if any such option exists at all.

I would suggest that you fix the exploited script. Look for time
stamps that appear in both web server logging and Postfix logging.

1) Use awk or perl to extract time stamps from web server logs and
normalize them so that they look the same. Produce sorted unqiue
timestamps with "sort -u" and save the result to file. Ditto for
time stamps from Postfix logs.

2) Look for time stamps that appear in both files.

comm -12 webserver-timestamps postfix-timestamps

3) Look up the corresponding records in webserver logs and Postfix logs.

Wietse

Re: Host not found?

[@lbutlr]

On 19 Oct 2020, at 13:13, Bob Proulx  wrote:
> Since 199.5.50.180 does not appear in the allowance for the SPF
> records that I can see 

dig -x 199.5.50.180 +short
br2.vw.com.

VW does own Audi, so… mystery deepens?

-- 
'They're the cream!' Rincewind sighed. 'Cohen, they're the cheese.'

Re: Host not found?

[Viktor Dukhovni]

> On Oct 19, 2020, at 7:19 PM, @lbutlr  wrote:
> 
> dig -x 199.5.50.180 +short
> br2.vw.com.
> 
> VW does own Audi, so… mystery deepens?

Anyone can spoof PTR records, but in this case the address really
does appear to be VW:

NetRange:   199.5.32.0 - 199.5.63.255
CIDR:   199.5.32.0/19
NetName:NETBLK-NET-VWNA
NetHandle:  NET-199-5-32-0-1
Parent: NET199 (NET-199-0-0-0-0)
NetType:Direct Assignment
OriginAS:   
Organization:   Volkswagen Group of America, Inc. (VOLKSW-1)
RegDate:1994-01-05
Updated:2007-03-22
Ref:https://rdap.arin.net/registry/ip/199.5.32.0


OrgName:Volkswagen Group of America, Inc.
OrgId:  VOLKSW-1
Address:3800 Hamlin Rd
City:   Auburn Hills
StateProv:  MI
PostalCode: 48326
Country:US
RegDate:1994-01-05
Updated:2018-05-01
Ref:https://rdap.arin.net/registry/entity/VOLKSW-1

-- 
Viktor.

Re: Host not found?

[Bob Proulx]

@lbutlr wrote:
> Bob Proulx wrote:
> > Since 199.5.50.180 does not appear in the allowance for the SPF
> > records that I can see 
> 
> dig -x 199.5.50.180 +short
> br2.vw.com.
> 
> VW does own Audi, so... mystery deepens?

That's simply the reverse DNS PTR record.  Anyone can set their own
PTR records to be anything they wish them to be.  That's why for the
full circle test the br2.vw.com would need to resolve back to
199.5.50.180 in order to have any trust in it at all.  Because that
would show that vw.com lists 199.5.50.180.  The PTR record is only
just so ever slightly more trustworthy than the HELO hostname.  Ever
so slightly since it means the hosting provider supported the
setting.  Most VPS providers do for example.

However I did look one step deeper and queried the whois records for
that IP address allocation.  It does have an address allocation to
Volkswagen Group of America, Inc. and therefore may actually be part
of them.

NetRange:   199.5.32.0 - 199.5.63.255
CIDR:   199.5.32.0/19
NetName:NETBLK-NET-VWNA
NetHandle:  NET-199-5-32-0-1
Parent: NET199 (NET-199-0-0-0-0)
NetType:Direct Assignment
OriginAS:   
Organization:   Volkswagen Group of America, Inc. (VOLKSW-1)

Which would swing my opinion over to the deeply misconfigured side of
things.

Bob

Re: Mail server recently became an open relay

[Demi M. Obenour]

On 10/19/20 3:29 PM, Jaroslaw Rafa wrote:
> Dnia 19.10.2020 o godz. 21:12:20 John Fawcett pisze:
>> Sorry not to be able to give a definitive answer. Typical mail injection
>> via php will use a script that already calls the php mail function or
>> similar functions that open the smtp connection. But there are other
>> attack vectors that are possible that allow hackers to gain the
>> privileges of the web server user.
> 
> Very often hackers abuse web pages that allow users to upload files to the
> web server. If the input is not correctly sanitized, it may be possible to
> upload an arbitrary php script and get it executed.
> 
> There were multiple attacks based on this scenario.

Can this be mitigated by denying the PHP user write permission on
any directory where PHP files will be executed?

Demi



OpenPGP_0xB288B55FFF9C22C1.asc
Description: application/pgp-keys


OpenPGP_signature
Description: OpenPGP digital signature

Re: Mail server recently became an open relay

[Jaroslaw Rafa]

Dnia 19.10.2020 o godz. 18:26:28 Demi M. Obenour pisze:
> Can this be mitigated by denying the PHP user write permission on
> any directory where PHP files will be executed?

There are multiple methods to mitigate this, this may be one of them.

But unsecured scripts that allow such behaviour are still being found here
and there.
-- 
Regards,
   Jaroslaw Rafa
   r...@rafa.eu.org
--
"In a million years, when kids go to school, they're gonna know: once there
was a Hushpuppy, and she lived with her daddy in the Bathtub."

Re: Mail server recently became an open relay

[IL Ka]

Rock solid solution is to separate htdocs (a folder that is accessible via
web) from the code folder (the one with scripts).
I do not know how that could be done with PHP (I believe you can serve
static files with nginx and run php as FPM connected to the nginx with
FastCGI) but in Python world we have separate process (uwsgi or gunicorn)
that is connected to the nginx and it runs under the different user.
All user files are uploaded to the folder accessible by nginx, far away
from the folder with python scripts.
Even if you upload a .py file, nginx will serve it as a static plain text
file.

On Tue, Oct 20, 2020 at 1:27 AM Demi M. Obenour 
wrote:

> On 10/19/20 3:29 PM, Jaroslaw Rafa wrote:
> > Dnia 19.10.2020 o godz. 21:12:20 John Fawcett pisze:
> >> Sorry not to be able to give a definitive answer. Typical mail injection
> >> via php will use a script that already calls the php mail function or
> >> similar functions that open the smtp connection. But there are other
> >> attack vectors that are possible that allow hackers to gain the
> >> privileges of the web server user.
> >
> > Very often hackers abuse web pages that allow users to upload files to
> the
> > web server. If the input is not correctly sanitized, it may be possible
> to
> > upload an arbitrary php script and get it executed.
> >
> > There were multiple attacks based on this scenario.
>
> Can this be mitigated by denying the PHP user write permission on
> any directory where PHP files will be executed?
>
> Demi
>
>

Re: Limiting HELO spoofing in Postfix?

[Rich Wales]

> I would suggest that you fix the exploited script. Look for time
> stamps that appear in both web server logging and Postfix logging.

Thanks, Wietse.  That would obviously be the best approach, if it
worked, but I tried it and (so far at least) haven't been able to find
any matching entries.

I did find some generally suspicious things in my web server logs --
including lots of clients looking for the following item:

/nette.micro?callback=shell_exec&cmd=ifconfig

but there isn't any /nette.micro anywhere on my server, and all these
GET requests failed with 404 or 302 SMTP response codes.  I don't
currently think these have anything to do with the recent attacks on my
server, and I'm mentioning them just in case anyone on this list might
recognize this kind of thing.

As I said earlier, I did clean out a bunch of old, obsolete, possibly
buggy content from my web server's directories, and I rebooted the box
just for good measure.  I haven't seen any more occurrences of the
problem so far, but I am keeping a close eye on the server, and if I do
see another flood of apparent open-relay messages, I hope I can capture
more (and more useful) information about the problem this time.

In the meantime, I would still like to know if there is any way to make
Postfix check for and reject messages which are coming from localhost
but which are falsely claiming (in the HELO / EHLO) to be originating
from some outside host.  I realize this wouldn't solve every problem,
but it seems to me like a very useful thing for Postfix to be able to
do.  If this option is intentionally not and most likely never will be
part of Postfix, I would be grateful for an explanation of why it is not
actually helpful, even if it might appear to be at first glance.

Rich Wales
ri...@richw.org

Re: Limiting HELO spoofing in Postfix?

[Bob Proulx]

Rich Wales wrote:
> I did find some generally suspicious things in my web server logs --
> including lots of clients looking for the following item:
> 
> /nette.micro?callback=shell_exec&cmd=ifconfig
> 
> but there isn't any /nette.micro anywhere on my server, and all these
> GET requests failed with 404 or 302 SMTP response codes.

If the web server logs said it was 404 then that is an HTTP 404 return
code and not an SMTP 404 return code since it is in the web server
logs not the smtp server logs.

https://en.wikipedia.org/wiki/HTTP_404

That's the normal response to a probe.  There are a lot of probes of
web servers on the net.

rwp@havoc:/var/log$ awk '$9=="404"' /var/log/nginx/access.log | wc -l
1124

That's one day of log file on one of my systems.  The probes are
endless.  Script kiddies.  It's why we can't have nice things.

Examples of the types of probes.  None of these are a problem on my
system but malicious attackers probe to see if there are any
vulnerabilities.  These are probes.

GET /data/admin/allowurl.txt
GET /html/public/index.php
GET /phpinfo.php
GET /wp-config.php
GET /wp-content/plugins/wp-file-manager/readme.txt
GET /wp-login.php
POST /wp-admins.php

The best thing you can do for these is to configure 'fail2ban' to
watch those log files and to drop the malicious probes at the firewall
level.  I think most of us would not do without fail2ban!

The HTTP 302 responses need more investigation before an analyses
could be made however.  Because for example they might be simply a
normal http redirection from http to https.  Or similar.  So they all
might turn out to be perfectly normal.  You would have to look and
see.

Bob

Re: Limiting HELO spoofing in Postfix?

[Rich Wales]

Bob Proulx wrote:

> If the web server logs said it was 404 then that is an HTTP 404 return
> code and not an SMTP 404 return code since it is in the web server
> logs not the smtp server logs

I know that.  (Sorry that I accidentally said they were SMTP response
codes, I do know better, they were HTTP response codes, mea culpa, it's
late at night and I'm dealing with pain from a kidney stone right now.)

Again, for clarity, I was asked by Wietse to examine both the web logs
and the Postfix logs on my server (both services are running on a single
box, I'm not using Docker), in order to find correlations.  I wasn't
able to find any correlations.  I did see various extraneous HTTP GET
requests in my web logs, but none of them related to my mail incidents.

I singled out the "GET /nette.micro" requests in particular because they
stood out as likely instances of attempted reverse tunneling attacks.
Also (something I didn't mention before), one of these corresponded in
time pretty closely to one of the fake messages I received -- though I
am inclined to dismiss this as mere coincidence, since the GET request
failed with a 404 HTTP return code.

The HTTP 302 responses to "GET /nette.micro" requests appear, as best I
can tell, to have all been simple redirections from HTTP to HTTPS.  The
corresponding HTTPS GET requests were all rejected with 404 codes.

Rich Wales
ri...@richw.org