-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello World,
I've got a severe problem. It looks like my webserver has been compromised. I have a webserver running apache2 (Debian Sarge). My webserver's load is always remaining around 1.5 and the cpu utilization is 95%. My webserver is not accepting web connections at the moment. The top reports show that a perl process is eating up all the cpu cycles which is executed by www-data user. Following is a result of `ps aux`: www-data 15855 0.0 2.8 22564 14808 ? S 05:05 0:00 /usr/sbin/apache2 -k start -DSSL www-data 15919 0.0 2.8 22592 14840 ? S 05:06 0:00 /usr/sbin/apache2 -k start -DSSL www-data 15929 0.0 2.3 20076 12176 ? S 05:07 0:00 /usr/sbin/apache2 -k start -DSSL www-data 15959 0.0 0.0 0 0 ? Z 05:07 0:00 [sh] <defunct> www-data 15963 0.0 0.6 5352 3408 ? S 05:07 0:01 /usr/sbin/httpd www-data 15964 0.0 0.6 5352 3408 ? S 05:07 0:01 /usr/sbin/httpd www-data 15994 0.0 0.0 0 0 ? Z 05:07 0:00 [sh] <defunct> www-data 15998 0.0 0.6 5352 3408 ? S 05:07 0:01 /usr/sbin/httpd www-data 15999 0.0 0.6 5352 3408 ? S 05:07 0:01 /usr/sbin/httpd www-data 16229 0.0 2.8 22596 14828 ? S 05:09 0:00 /usr/sbin/apache2 -k start -DSSL www-data 16302 0.0 0.0 0 0 ? Z 05:10 0:00 [sh] <defunct> www-data 16306 0.0 0.6 5352 3404 ? S 05:10 0:01 /usr/sbin/httpd www-data 16307 0.0 0.6 5352 3408 ? S 05:10 0:01 /usr/sbin/httpd www-data 16375 0.0 2.3 20124 12224 ? S 05:13 0:00 /usr/sbin/apache2 -k start -DSSL www-data 16411 0.0 0.0 0 0 ? Z 05:13 0:00 [sh] <defunct> www-data 16415 0.0 0.6 5352 3404 ? S 05:13 0:01 /usr/sbin/httpd www-data 16416 0.0 0.6 5352 3404 ? S 05:13 0:01 /usr/sbin/httpd www-data 16629 0.0 0.0 0 0 ? Z 05:14 0:00 [sh] <defunct> www-data 16633 0.0 0.6 5352 3400 ? S 05:14 0:01 /usr/sbin/httpd www-data 16634 0.0 0.6 5352 3400 ? S 05:14 0:01 /usr/sbin/httpd www-data 16930 0.0 2.3 20100 12200 ? S 05:15 0:00 /usr/sbin/apache2 -k start -DSSL www-data 16963 0.0 0.0 0 0 ? Z 05:15 0:00 [sh] <defunct> www-data 16967 0.0 0.6 5352 3400 ? S 05:15 0:01 /usr/sbin/httpd www-data 16968 0.0 0.6 5352 3400 ? S 05:15 0:01 /usr/sbin/httpd www-data 17087 0.0 2.8 22564 14800 ? S 05:16 0:00 /usr/sbin/apache2 -k start -DSSL www-data 17089 0.0 2.3 20100 12204 ? S 05:17 0:00 /usr/sbin/apache2 -k start -DSSL www-data 17121 0.0 0.0 0 0 ? Z 05:17 0:00 [sh] <defunct> www-data 17125 0.0 0.6 5352 3400 ? S 05:17 0:01 /usr/sbin/httpd www-data 17126 0.0 0.6 5352 3400 ? S 05:17 0:01 /usr/sbin/httpd www-data 17176 0.0 2.3 20148 12192 ? S 05:17 0:00 /usr/sbin/apache2 -k start -DSSL www-data 17179 0.0 2.8 22588 14832 ? S 05:17 0:00 /usr/sbin/apache2 -k start -DSSL www-data 17267 0.0 0.0 0 0 ? Z 05:17 0:00 [sh] <defunct> www-data 17271 0.0 0.6 5352 3396 ? S 05:17 0:01 /usr/sbin/httpd www-data 17272 0.0 0.6 5352 3400 ? S 05:17 0:01 /usr/sbin/httpd www-data 17362 0.0 0.0 0 0 ? Z 05:18 0:00 [sh] <defunct> www-data 17366 0.0 0.6 5352 3396 ? S 05:18 0:01 /usr/sbin/httpd www-data 17367 0.0 0.6 5352 3396 ? S 05:18 0:01 /usr/sbin/httpd www-data 17599 0.0 0.0 0 0 ? Z 05:18 0:00 [sh] <defunct> www-data 17604 0.0 0.6 5352 3400 ? S 05:18 0:01 /usr/sbin/httpd www-data 17605 0.0 0.6 5352 3396 ? S 05:18 0:01 /usr/sbin/httpd www-data 18252 0.0 2.8 22588 14836 ? S 05:21 0:00 /usr/sbin/apache2 -k start -DSSL www-data 18356 0.0 0.0 0 0 ? Z 05:22 0:00 [sh] <defunct> www-data 18360 0.0 0.6 5232 3392 ? S 05:22 0:01 /usr/sbin/httpd www-data 18361 0.0 0.6 5232 3392 ? S 05:22 0:01 /usr/sbin/httpd www-data 18742 0.0 2.7 21820 13960 ? S 05:23 0:00 /usr/sbin/apache2 -k start -DSSL www-data 18956 0.0 0.0 0 0 ? Z 05:24 0:00 [sh] <defunct> www-data 18960 0.0 0.6 5232 3388 ? S 05:24 0:01 /usr/sbin/httpd www-data 18961 0.0 0.6 5232 3392 ? S 05:24 0:01 /usr/sbin/httpd www-data 18979 0.0 2.8 22564 14800 ? S 05:24 0:00 /usr/sbin/apache2 -k start -DSSL www-data 19855 0.0 2.8 22544 14776 ? S 05:31 0:00 /usr/sbin/apache2 -k start -DSSL www-data 19982 0.0 0.0 0 0 ? Z 05:31 0:00 [sh] <defunct> www-data 19987 0.0 0.6 5232 3380 ? S 05:31 0:01 /usr/sbin/httpd www-data 19988 0.0 0.6 5232 3384 ? S 05:31 0:01 /usr/sbin/httpd www-data 20021 0.0 2.8 22604 14852 ? S 05:31 0:00 /usr/sbin/apache2 -k start -DSSL www-data 20496 0.0 2.8 22572 14804 ? S 05:32 0:00 /usr/sbin/apache2 -k start -DSSL www-data 20497 0.0 2.8 22564 14804 ? S 05:32 0:00 /usr/sbin/apache2 -k start -DSSL www-data 20856 0.0 0.0 0 0 ? Z 05:33 0:00 [sh] <defunct> www-data 20860 0.0 0.6 5232 3380 ? S 05:33 0:01 /usr/sbin/httpd www-data 20861 0.0 0.6 5232 3380 ? S 05:33 0:01 /usr/sbin/httpd www-data 20922 0.0 2.3 20148 12184 ? S 05:33 0:00 /usr/sbin/apache2 -k start -DSSL www-data 21049 0.0 0.0 0 0 ? Z 05:33 0:00 [sh] <defunct> www-data 21057 0.0 0.6 5232 3380 ? S 05:33 0:01 /usr/sbin/httpd www-data 21058 0.0 0.6 5232 3380 ? S 05:33 0:01 /usr/sbin/httpd www-data 21494 0.0 0.0 0 0 ? Z 05:33 0:00 [sh] <defunct> www-data 21498 0.0 0.6 5232 3380 ? S 05:33 0:01 /usr/sbin/httpd www-data 21499 0.0 0.6 5232 3380 ? S 05:33 0:01 /usr/sbin/httpd www-data 21589 0.0 0.0 0 0 ? Z 05:34 0:00 [sh] <defunct> www-data 21596 0.0 0.6 5232 3380 ? S 05:34 0:01 /usr/sbin/httpd www-data 21597 0.0 0.6 5232 3380 ? S 05:34 0:01 /usr/sbin/httpd www-data 22509 0.0 2.8 22564 14800 ? S 05:35 0:00 /usr/sbin/apache2 -k start -DSSL www-data 22545 0.0 0.0 0 0 ? Z 05:36 0:00 [sh] <defunct> www-data 22549 0.0 0.6 5232 3376 ? S 05:36 0:01 /usr/sbin/httpd www-data 22550 0.0 0.6 5232 3376 ? S 05:36 0:01 /usr/sbin/httpd www-data 22554 0.0 2.8 22564 14812 ? S 05:36 0:00 /usr/sbin/apache2 -k start -DSSL www-data 22705 0.0 2.8 22560 14800 ? S 05:37 0:00 /usr/sbin/apache2 -k start -DSSL www-data 22789 0.0 0.0 0 0 ? Z 05:39 0:00 [sh] <defunct> www-data 22793 0.0 0.6 5232 3372 ? S 05:39 0:01 /usr/sbin/httpd www-data 22794 0.0 0.6 5232 3372 ? S 05:39 0:01 /usr/sbin/httpd www-data 23037 0.0 2.3 20048 12144 ? S 05:40 0:00 /usr/sbin/apache2 -k start -DSSL www-data 23042 0.0 0.0 0 0 ? Z 05:40 0:00 [sh] <defunct> www-data 23047 0.0 0.6 5232 3376 ? S 05:40 0:01 /usr/sbin/httpd www-data 23048 0.0 0.6 5232 3372 ? S 05:40 0:01 /usr/sbin/httpd www-data 23072 0.0 0.0 0 0 ? Z 05:42 0:00 [sh] <defunct> www-data 23076 0.0 0.6 5232 3372 ? S 05:42 0:01 /usr/sbin/httpd www-data 23077 0.0 0.6 5232 3372 ? S 05:42 0:01 /usr/sbin/httpd www-data 23079 0.0 2.8 22564 14792 ? S 05:42 0:00 /usr/sbin/apache2 -k start -DSSL www-data 23088 0.0 2.8 22564 14788 ? S 05:42 0:00 /usr/sbin/apache2 -k start -DSSL www-data 23107 0.0 2.8 22564 14804 ? S 05:42 0:00 /usr/sbin/apache2 -k start -DSSL www-data 23240 0.0 0.0 0 0 ? Z 05:42 0:00 [sh] <defunct> www-data 23244 0.0 0.6 5232 3372 ? S 05:42 0:01 /usr/sbin/httpd www-data 23245 0.0 0.6 5232 3372 ? S 05:42 0:01 /usr/sbin/httpd www-data 23259 0.0 0.0 0 0 ? Z 05:42 0:00 [sh] <defunct> www-data 23263 0.0 0.6 5232 3368 ? S 05:42 0:01 /usr/sbin/httpd www-data 23264 0.0 0.6 5232 3372 ? S 05:42 0:01 /usr/sbin/httpd www-data 23265 0.0 2.3 20064 12160 ? S 05:42 0:00 /usr/sbin/apache2 -k start -DSSL www-data 23366 0.0 0.0 0 0 ? Z 05:42 0:00 [sh] <defunct> www-data 23373 0.0 0.6 5232 3372 ? S 05:42 0:01 /usr/sbin/httpd www-data 23374 0.0 0.6 5232 3368 ? S 05:42 0:01 /usr/sbin/httpd www-data 23907 0.0 2.8 22564 14784 ? S 05:43 0:00 /usr/sbin/apache2 -k start -DSSL www-data 23971 0.0 0.0 0 0 ? Z 05:45 0:00 [sh] <defunct> www-data 23975 0.0 0.6 5232 3368 ? S 05:45 0:01 /usr/sbin/httpd www-data 23976 0.0 0.6 5232 3368 ? S 05:45 0:01 /usr/sbin/httpd www-data 24006 0.0 2.8 22564 14796 ? S 05:45 0:00 /usr/sbin/apache2 -k start -DSSL www-data 24093 0.0 2.3 20100 12200 ? S 05:45 0:00 /usr/sbin/apache2 -k start -DSSL www-data 24153 0.0 0.0 0 0 ? Z 05:45 0:00 [sh] <defunct> www-data 24157 0.0 0.6 5232 3368 ? S 05:45 0:01 /usr/sbin/httpd www-data 24158 0.0 0.6 5232 3364 ? S 05:45 0:01 /usr/sbin/httpd www-data 24244 0.0 0.0 0 0 ? Z 05:46 0:00 [sh] <defunct> www-data 24248 0.0 0.6 5232 3368 ? S 05:46 0:01 /usr/sbin/httpd www-data 24249 0.0 0.6 5232 3368 ? S 05:46 0:01 /usr/sbin/httpd www-data 24417 0.0 0.0 0 0 ? Z 05:46 0:00 [sh] <defunct> www-data 24425 0.0 0.6 5232 3364 ? S 05:46 0:01 /usr/sbin/httpd www-data 24426 0.0 0.6 5232 3364 ? S 05:46 0:01 /usr/sbin/httpd www-data 24633 0.0 0.0 0 0 ? Z 05:46 0:00 [sh] <defunct> www-data 24637 0.0 0.6 5232 3364 ? S 05:46 0:01 /usr/sbin/httpd www-data 24638 0.0 0.6 5232 3364 ? S 05:46 0:01 /usr/sbin/httpd www-data 24650 0.0 2.3 20120 12232 ? S 05:47 0:00 /usr/sbin/apache2 -k start -DSSL www-data 24783 0.0 0.0 0 0 ? Z 05:47 0:00 [sh] <defunct> www-data 24789 0.0 0.6 5232 3364 ? S 05:47 0:01 /usr/sbin/httpd www-data 24790 0.0 0.6 5232 3364 ? S 05:47 0:01 /usr/sbin/httpd www-data 24880 0.0 2.8 22564 14788 ? S 05:48 0:00 /usr/sbin/apache2 -k start -DSSL www-data 24924 0.0 0.0 0 0 ? Z 05:49 0:00 [sh] <defunct> www-data 24929 0.0 0.6 5232 3360 ? S 05:49 0:01 /usr/sbin/httpd www-data 24930 0.0 0.6 5232 3364 ? S 05:49 0:01 /usr/sbin/httpd www-data 24932 0.0 2.3 20196 12288 ? S 05:49 0:00 /usr/sbin/apache2 -k start -DSSL www-data 25096 0.0 0.0 0 0 ? Z 05:49 0:00 [sh] <defunct> www-data 25100 0.0 0.6 5232 3364 ? S 05:49 0:01 /usr/sbin/httpd www-data 25101 0.0 0.6 5232 3360 ? S 05:49 0:01 /usr/sbin/httpd www-data 25415 0.0 2.8 22608 14884 ? S 05:50 0:00 /usr/sbin/apache2 -k start -DSSL www-data 25478 0.0 0.0 0 0 ? Z 05:51 0:00 [sh] <defunct> www-data 25482 0.0 0.6 5232 3360 ? S 05:51 0:01 /usr/sbin/httpd www-data 25483 0.0 0.6 5232 3360 ? S 05:51 0:01 /usr/sbin/httpd www-data 25598 0.0 2.8 22580 14784 ? S 05:51 0:00 /usr/sbin/apache2 -k start -DSSL www-data 25748 0.0 0.0 0 0 ? Z 05:52 0:00 [sh] <defunct> www-data 25752 0.0 0.6 5232 3360 ? S 05:52 0:01 /usr/sbin/httpd www-data 25753 0.0 0.6 5232 3360 ? S 05:52 0:01 /usr/sbin/httpd www-data 25795 0.0 2.3 20140 12236 ? S 05:52 0:00 /usr/sbin/apache2 -k start -DSSL www-data 26003 0.0 0.0 0 0 ? Z 05:52 0:00 [sh] <defunct> www-data 26008 0.0 0.6 5232 3360 ? S 05:52 0:01 /usr/sbin/httpd www-data 26009 0.0 0.6 5232 3356 ? S 05:52 0:01 /usr/sbin/httpd www-data 26306 0.0 2.8 22564 14788 ? S 05:53 0:00 /usr/sbin/apache2 -k start -DSSL www-data 26319 0.0 2.3 20080 12192 ? S 05:54 0:00 /usr/sbin/apache2 -k start -DSSL www-data 26325 0.0 0.0 0 0 ? Z 05:54 0:00 [sh] <defunct> www-data 26329 0.0 0.6 5232 3356 ? S 05:54 0:01 /usr/sbin/httpd www-data 26330 0.0 0.6 5232 3356 ? S 05:54 0:01 /usr/sbin/httpd www-data 26351 0.0 2.3 20184 12280 ? S 05:55 0:00 /usr/sbin/apache2 -k start -DSSL www-data 26669 0.0 0.0 0 0 ? Z 05:55 0:00 [sh] <defunct> www-data 26673 0.0 0.6 5232 3356 ? S 05:55 0:01 /usr/sbin/httpd www-data 26674 0.0 0.6 5232 3356 ? S 05:55 0:01 /usr/sbin/httpd www-data 26694 0.0 0.0 0 0 ? Z 05:55 0:00 [sh] <defunct> www-data 26698 0.0 0.6 5232 3356 ? S 05:55 0:01 /usr/sbin/httpd www-data 26699 0.0 0.6 5232 3356 ? S 05:55 0:01 /usr/sbin/httpd www-data 27008 0.0 2.8 22564 14776 ? S 05:55 0:00 /usr/sbin/apache2 -k start -DSSL www-data 27066 0.0 0.0 0 0 ? Z 05:55 0:00 [sh] <defunct> www-data 27070 0.0 0.6 5232 3356 ? S 05:55 0:01 /usr/sbin/httpd www-data 27071 0.0 0.6 5232 3356 ? S 05:55 0:01 /usr/sbin/httpd www-data 28373 0.0 2.3 20196 12304 ? S 06:08 0:00 /usr/sbin/apache2 -k start -DSSL www-data 28375 0.0 0.0 0 0 ? Z 06:08 0:00 [sh] <defunct> www-data 28379 0.0 0.6 5232 3344 ? S 06:08 0:01 /usr/sbin/httpd www-data 28380 0.0 0.6 5232 3340 ? S 06:08 0:01 /usr/sbin/httpd www-data 28382 0.0 2.3 20260 12308 ? S 06:08 0:00 /usr/sbin/apache2 -k start -DSSL www-data 28384 0.0 0.0 0 0 ? Z 06:08 0:00 [sh] <defunct> www-data 28390 27.7 0.7 5760 3808 ? R 06:08 23:19 /usr/sbin/httpd identd 28391 0.0 0.1 51948 1032 ? S 06:08 0:00 identd www-data 32753 0.0 2.7 22240 14348 ? S 07:20 0:00 /usr/sbin/apache2 -k start -DSSL root 307 0.0 0.3 6224 1956 ? Ss 07:21 0:00 sshd: rrs [priv] rrs 310 0.0 0.3 6388 2060 ? S 07:21 0:00 sshd: [EMAIL PROTECTED]/0 rrs 311 0.0 0.4 3728 2356 pts/0 Ss 07:21 0:00 -bash root 348 0.0 0.2 2592 1476 pts/0 S 07:22 0:00 -su www-data 368 0.1 2.7 22240 14340 ? S 07:23 0:00 /usr/sbin/apache2 -k start -DSSL www-data 376 0.0 2.7 22240 14348 ? S 07:24 0:00 /usr/sbin/apache2 -k start -DSSL www-data 394 0.1 2.7 22428 14412 ? S 07:26 0:00 /usr/sbin/apache2 -k start -DSSL www-data 420 0.1 2.7 21836 13968 ? S 07:29 0:00 /usr/sbin/apache2 -k start -DSSL www-data 426 0.0 0.0 0 0 ? Z 07:29 0:00 [perl] <defunct> www-data 453 0.1 2.7 22420 14396 ? S 07:30 0:00 /usr/sbin/apache2 -k start -DSSL root 462 0.0 0.1 2052 932 ? S 07:30 0:00 /USR/SBIN/CRON root 463 0.0 0.2 2696 1200 ? Ss 07:30 0:00 /bin/sh - -c /usr/local/bin/update-data.sh > /dev/null root 464 0.0 0.2 2696 1248 ? S 07:30 0:00 /bin/sh /usr/local/bin/update-data.sh root 493 0.0 0.2 3376 1496 ? S 07:30 0:00 wget -q - -O /etc/tinydns/root/data.srv-1 http://127.0.0.1/veg root 495 0.0 0.1 1512 624 ? Ss 07:30 0:00 /usr/sbin/anacron -s www-data 546 5.2 2.7 22440 14424 ? S 07:31 0:00 /usr/sbin/apache2 -k start -DSSL www-data 548 37.5 2.7 22464 14448 ? S 07:32 0:00 /usr/sbin/apache2 -k start -DSSL www-data 552 0.0 2.7 22240 14348 ? S 07:32 0:00 /usr/sbin/apache2 -k start -DSSL root 553 0.0 0.1 2496 848 pts/0 R+ 07:32 0:00 ps aux ns1:~# cd /etc/cron.d Interesting part is that it shows that "/usr/sbin/httpd" process is being run where as there's no "/usr/sbin/httpd" on my machine. ns1:/etc/cron.d# file /usr/sbin/httpd /usr/sbin/httpd: ERROR: cannot open `/usr/sbin/httpd' (No such file or directory) I installed "chkrootkit" to see if any rootkit was installed but chkrootkit reports that the system is not infected. Can anyone help me if my system is compromised or is it a system related issue ? What steps should I follow to get my webserver usable again ? It's a machine under production usage. Regards, rrs - -- Ritesh Raj Sarraf RESEARCHUT -- http://www.researchut.com Gnupg Key ID: 04F130BC "Stealing logic from one person is plagiarism, stealing from many is research." "Necessity is the mother of invention." -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (GNU/Linux) iD8DBQFDhzP44Rhi6gTxMLwRAlDXAJ99xQFDV7r0uKEYpSfz6TgFbO91rwCgjGV1 JH2u2yK6pcmnbPwy6KfZh1Y= =14iX -----END PGP SIGNATURE----- -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
