thank you so much all of you guys, but still we are facing the same problem.
we recompiled new php source, upgraded httpd to 2.2.11 but still getting same
backtrace. nothing more. we removed .htaccess rules and tested but no use.
can you guys suggest me any other ways of debugging pls? actually i am not a
system administration guy. I am a php developer with little knowledge on httpd.
But we have system administration group who are working on this problem.
just pls check out again the segfault statement that we are getting:
httpd[5250]: segfault at 00007fff84114ff8 rip 00002aaab1bd47ac rsp
00007fff84115000 error
6
that bold line is repeating many times randomly on our log file and always it
is same on
specific server. we have 6 load balanced servers and all are giving the same
error randomly.
address locations are different from each server but for a specific server it
is same always
and every day.
you guys think any kind of attack thats happening over our systems? actually we
do not accept
much input from users other than a keyword(for search) which can be an unicode
and we have some
POST requests which are purely browser generated data(no user data here).
im again pasting our gdb backtraces of two different core dumps for reference:
> #0 0x000055555557dfee in ap_merge_per_dir_configs () from /usr/sbin/httpd
> No symbol table info available.
> #1 0x000055555557b121 in ap_directory_walk () from /usr/sbin/httpd
> No symbol table info
available.
> #2 0x00005555555765b9 in ap_is_recursion_limit_exceeded () from
> /usr/sbin/httpd
> No symbol table info available.
> #3 0x0000555555578b42 in ap_run_map_to_storage () from /usr/sbin/httpd
> No symbol table info available.
> #4 0x0000555555579cbc in ap_process_request_internal () from /usr/sbin/httpd
> No symbol table info available.
> #5 0x000055555558b668 in ap_process_request () from /usr/sbin/httpd
> No symbol table info available.
> #6 0x0000555555588900 in ap_register_input_filter () from /usr/sbin/httpd
> No symbol table info available.
above thing is happening many times say 10 to 15 times a minute(actualy
depends on the traffic at that time)
> #0 0x00002aaab1c46688 in ZEND_FETCH_DIM_RW_SPEC_VAR_UNUSED_HANDLER (
> execute_data=0x5555714ea6c8)
> at
/usr/local/src/php/php-5.2.9/Zend/zend_vm_execute.h:13204
> opline = (zend_op *) 0x5555714e8798
> free_op1 = {var = 0x2aaaac1454fd}
> #1 0x00005555714e86b8 in ?? ()
> No symbol table info available.
> #2 0x00002aaaac145afb in apr_pool_destroy () from /usr/lib64/libapr-1.so.0
> No symbol table info available.
> #3 0x000055555556a27b in suck_in_APR () from /usr/sbin/httpd
> No
> symbol table info available.
> #4 0x000055555556adf6 in main () from /usr/sbin/httpd
> No symbol table info available.
above thing is happening in very lesser frequency compared to the above.
--thank you so much guys, rats.
jvlad wrote:
>>
>>> #2 0x00002aaaac145afb in apr_pool_destroy () from
>>> /usr/lib64/libapr-1.so.0
>>> No symbol table info available.
>> See http://httpd.apache.org/dev/debugging.html on how to grab the
>> -win32-symbols.zip package. Unpack it over httpd and you will probably
>> have more legible backtraces. Ensure you have all the debugging symbols
>> for php you'll have something completely legible.
>
> aha, and /usr/local/src/php/php-5.2.9/Zend/zend_vm_execute.h is a very
> common path under Win32 :)
Here? Yea, it is.
But looking at the libapr-1.so.0 module, agreed that
I've misread this :)
> @Rathnakar Konda, is it possible that php was compiled against wrong Apache
> headers? Did you install something like OS-supplied httpd-devel package to
> compile php or grabbed apache sources from their web site?
I'm not used to seeing bad stack unwinds on unix, most people don't optimize
that aggressively.
If this is system-httpd, that's fine; if you installed an httpd package,
there is likely an httpd-devel package that also includes the debugging
symbols you need (unstripped).
If not and you've built this, you can simply try -O0, or -g, or for httpd
--enable-maintainer-mode, but offhand I can't think of a trivial way to
simply avoid -strip, which is what you want for your build when things
go wonky in your gdb where output.
Bill
Original E-Mail:
> Hi Guys,
>
> We are have a problem with apache segfault on our production server. Please
> read bellow for description.
>
> Its
> a web application written in php5 and implemented most of the oop
> concepts and lot of regular expressions, curl, mcrypt, simplexml,
> mssql, exceptions and user defined error handlers. When we run this app
> on our test server, we had no problems, but when we moved it on to the
> production(here we used have big amount of traffic), initially we saw
> no problems from our end user testing but from system log, we saw lots
> of 'segfaults' and thus requests were being dropped(difference in
> traffic).
>
> Weird thing is that, on the
same apache httpd, there
> is an another application running successfully which is having lesser
> oop concepts but with same libraries. We are running these two
> applications with virtual host concept. We see 'segfaults' only when
> the traffic is very high on the first application.
>
> We have upgraded our php module from 5.2.6 to 5.2.9 but with no result. We
> have the core dump of the apache below:
>
> #0 0x000055555557dfee in ap_merge_per_dir_configs () from /usr/sbin/httpd
> No symbol table info available.
> #1 0x000055555557b121 in ap_directory_walk () from /usr/sbin/httpd
> No symbol table info available.
> #2 0x00005555555765b9 in ap_is_recursion_limit_exceeded () from
> /usr/sbin/httpd
> No symbol table info available.
> #3 0x0000555555578b42 in ap_run_map_to_storage () from /usr/sbin/httpd
> No symbol table info available.
>
#4 0x0000555555579cbc in ap_process_request_internal () from /usr/sbin/httpd
> No symbol table info available.
> #5 0x000055555558b668 in ap_process_request () from /usr/sbin/httpd
> No symbol table info available.
> #6 0x0000555555588900 in ap_register_input_filter () from /usr/sbin/httpd
> No symbol table info available.
________________________________
Add more friends to your messenger and enjoy! Invite them now.
Add more friends to your messenger and enjoy! Go to
http://messenger.yahoo.com/invite/
