On Sun, Jun 29, 2003 at 01:00:57PM +0800, Jason Lim wrote:
>
> One of our servers was hacked (woody)... badly, from what I can see. A
> whole bunch of binaries have been modified, and strange processes are
> running on the server. The hack date appears to be jun 6.
>
> Is there a document somewhere, or procedure, to recover after this? This
> is a working and running system, so somehow need to be able to recover
> from this with minimal impact to end-users.
>
> Some things like:
>
> www-data 17451 0.0 0.0 2164 928 ? S 02:31 0:00 /bin/sh
> www-data 21550 0.0 0.0 1232 236 ? S 05:02 0:00 ./x
> www-data 21551 0.0 0.0 0 0 ? Z 05:02 0:00 [x
> <defunct>]
> root 21552 0.0 0.0 0 0 ? Z 05:02 0:00 [modprobe
> <defunc
> root 21554 0.0 0.0 2148 912 ? S 05:02 0:00 /bin/sh
> root 21755 0.0 0.0 2164 948 ? S 05:02 0:00 /bin/sh
> root 21801 0.0 0.0 2180 964 ? S 05:03 0:00 /bin/bash
> ./troja
> root 22010 0.0 0.0 1244 204 ? S 05:03 0:00 ./siz
> ifconfigx /
> root 12267 0.0 0.0 0 0 ? Z 07:15 0:00 [date
> <defunct>]
> root 12266 0.0 0.0 1264 252 ? T 07:15 0:00 date +%d
Hi!
I'm no expert in this at all...
Here are some basic try-to-solve-it hints.
In most cases its not possible to reinstall the whole system, as in this
case. I mean. A home-server/workstation is no problem to reinstall but
a high SLA 60k -user cluster is quite boring and time consuming.
I'd do it like this.
First. We need some fresh & clean tools;
kill, killall, ps, more, netstat, ls, dpkg, apt-tools, chattr, lsattr, bash
(or whatever shell you prefer).
Replace your shell with the clean one (the /etc/passwd -race).
Killing the procs right off is almost
impossible unless you find the master process (often protected and hidden in a
patched
ps or proctable and chattr'ed away on your filesystem).
since you're using the >2.4.20 kernels modprobe-bug exists, so get rid
of that bug first.
echo "blah" > /proc/sys/kernel/modprobe
then I'd; lsattr -a /*|more to see weither we have some hidden and/or
write protected files that we dont know.
I bet you'll get some interesting output here.
chattr this files and move them to some secret place so you can check
them out later... Dont forget to check .history-files, logs etc. Most
hacks are done in a rush and there are always pices of information left
every here and there.
After the filesystem looks nice and clean I would try find and
kill the processes.
As stated abowe it's quite hard to kill processes that are not meant to
be killed. netstat -anp is a good tool here aswell as kill and ps.
I've seen cases when the mastersprocess is hidden within sshd, init,
various daemons such as ftp, telnet, ldap, gpm etc. Kill all processes you dont
need. Look for respawning ones.
A reboot might help, but dont reboot until you've checked the
startup rc-files, stuff needed to boot etc...
then i'd apt-getted the base system and then all debian-packages.
Now try find out how he did it and try fix it before it happens again.
Hope this helps...
--
__
Yours sincerely,
Christofer Algotsson - [EMAIL PROTECTED]
