[
https://issues.apache.org/jira/browse/TS-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14335371#comment-14335371
]
Susan Hinrichs edited comment on TS-3405 at 2/24/15 9:01 PM:
-------------------------------------------------------------
Saw a similar stack after a seg fault in my dev build. In that case contp
(Http2ClientSession?) had been freed but FetchSM still had a reference to it.
It was processing EVENT_EOS.
The GDB prints are below. I'll give Ryo's patch a try.
{code}
(gdb) bt
#0 0x000000000050caf8 in Continuation::handleEvent (this=0x7fffc82842d0,
event=-4, data=0x7fffc8039650)
at ../iocore/eventsystem/I_Continuation.h:146
#1 0x000000000050ac43 in FetchSM::InvokePluginExt (this=0x7fffc8039650,
fetch_event=104) at FetchSM.cc:301
#2 0x000000000050b4e6 in FetchSM::process_fetch_read (this=0x7fffc8039650,
event=104) at FetchSM.cc:465
#3 0x000000000050b7c5 in FetchSM::fetch_handler (this=0x7fffc8039650,
event=104, edata=0x7fffdc0727a8) at FetchSM.cc:514
#4 0x000000000050cafa in Continuation::handleEvent (this=0x7fffc8039650,
event=104, data=0x7fffdc0727a8)
at ../iocore/eventsystem/I_Continuation.h:146
#5 0x000000000054bd42 in PluginVC::process_read_side (this=0x7fffdc072698,
other_side_call=false) at PluginVC.cc:640
#6 0x000000000054a220 in PluginVC::main_handler (this=0x7fffdc072698,
event=1, data=0x7fffe01e8580) at PluginVC.cc:206
#7 0x000000000050cafa in Continuation::handleEvent (this=0x7fffdc072698,
event=1, data=0x7fffe01e8580) at ../iocore/eventsystem/I_Continuation.h:146
#8 0x00000000007a876e in EThread::process_event (this=0x7ffff69e9010,
e=0x7fffe01e8580, calling_code=1) at UnixEThread.cc:144
#9 0x00000000007a8c5d in EThread::execute (this=0x7ffff69e9010)
at UnixEThread.cc:238
#10 0x00000000007a7c74 in spawn_thread_internal (a=0x11280d0) at Thread.cc:88
#11 0x000000351e807851 in start_thread () from /lib64/libpthread.so.0
#12 0x000000351e4e890d in clone () from /lib64/libc.so.6
(gdb) frame 1
#1 0x000000000050ac43 in FetchSM::InvokePluginExt (this=0x7fffc8039650,
fetch_event=104) at FetchSM.cc:301
301 contp->handleEvent(TS_FETCH_EVENT_EXT_BODY_DONE, this);
(gdb) print this->contp[0]
$14 = {<force_VFPT_to_top> = {_vptr.force_VFPT_to_top = 0x7fffc8283c51},
handler = (int (Continuation::*)(Continuation *, int,
void *)) 0xefbeaddeefbeadde, this adjustment -1171307680053154338,
handler_name = 0xefbeaddeefbeadde <Address 0xefbeaddeefbeadde out of bounds>,
mutex = {m_ptr = 0xefbeaddeefbeadde}, link = {<SLink<Continuation>> = {
next = 0xefbeaddeefbeadde}, prev = 0xefbeaddeefbeadde}}
(gdb)
{code}
was (Author: shinrich):
Saw a similar stack after a seg fault in my dev build. In that case contp
(Http2ClientSession?) had been freed but FetchSM still had a reference to it.
It was processing EVENT_EOS.
The GDB prints are below. I'll give Ryo's patch a try.
(gdb) bt
#0 0x000000000050caf8 in Continuation::handleEvent (this=0x7fffc82842d0,
event=-4, data=0x7fffc8039650)
at ../iocore/eventsystem/I_Continuation.h:146
#1 0x000000000050ac43 in FetchSM::InvokePluginExt (this=0x7fffc8039650,
fetch_event=104) at FetchSM.cc:301
#2 0x000000000050b4e6 in FetchSM::process_fetch_read (this=0x7fffc8039650,
event=104) at FetchSM.cc:465
#3 0x000000000050b7c5 in FetchSM::fetch_handler (this=0x7fffc8039650,
event=104, edata=0x7fffdc0727a8) at FetchSM.cc:514
#4 0x000000000050cafa in Continuation::handleEvent (this=0x7fffc8039650,
event=104, data=0x7fffdc0727a8)
at ../iocore/eventsystem/I_Continuation.h:146
#5 0x000000000054bd42 in PluginVC::process_read_side (this=0x7fffdc072698,
other_side_call=false) at PluginVC.cc:640
#6 0x000000000054a220 in PluginVC::main_handler (this=0x7fffdc072698,
event=1, data=0x7fffe01e8580) at PluginVC.cc:206
#7 0x000000000050cafa in Continuation::handleEvent (this=0x7fffdc072698,
event=1, data=0x7fffe01e8580) at ../iocore/eventsystem/I_Continuation.h:146
#8 0x00000000007a876e in EThread::process_event (this=0x7ffff69e9010,
e=0x7fffe01e8580, calling_code=1) at UnixEThread.cc:144
#9 0x00000000007a8c5d in EThread::execute (this=0x7ffff69e9010)
at UnixEThread.cc:238
#10 0x00000000007a7c74 in spawn_thread_internal (a=0x11280d0) at Thread.cc:88
#11 0x000000351e807851 in start_thread () from /lib64/libpthread.so.0
#12 0x000000351e4e890d in clone () from /lib64/libc.so.6
(gdb) frame 1
#1 0x000000000050ac43 in FetchSM::InvokePluginExt (this=0x7fffc8039650,
fetch_event=104) at FetchSM.cc:301
301 contp->handleEvent(TS_FETCH_EVENT_EXT_BODY_DONE, this);
(gdb) print this->contp[0]
$14 = {<force_VFPT_to_top> = {_vptr.force_VFPT_to_top = 0x7fffc8283c51},
handler = (int (Continuation::*)(Continuation *, int,
void *)) 0xefbeaddeefbeadde, this adjustment -1171307680053154338,
handler_name = 0xefbeaddeefbeadde <Address 0xefbeaddeefbeadde out of bounds>,
mutex = {m_ptr = 0xefbeaddeefbeadde}, link = {<SLink<Continuation>> = {
next = 0xefbeaddeefbeadde}, prev = 0xefbeaddeefbeadde}}
(gdb)
> Memory use after free in HTTP/2
> -------------------------------
>
> Key: TS-3405
> URL: https://issues.apache.org/jira/browse/TS-3405
> Project: Traffic Server
> Issue Type: Bug
> Components: HTTP/2
> Reporter: Bryan Call
> Fix For: 5.3.0
>
> Attachments: fix-h2.patch
>
>
> From Leif running on docs.trafficserver.apache.org:
>
> {code}
> traffic_server: using root directory '/opt/ats'
> =================================================================
> ==31101==ERROR: AddressSanitizer: heap-use-after-free on address
> 0x61800000c888 at pc 0x4f3558 bp 0x2aaf10c88930 sp 0x2aaf10c88928
> READ of size 8 at 0x61800000c888 thread T2 ([ET_NET 1])
> #0 0x4f3557 in Continuation::handleEvent(int, void*)
> ../iocore/eventsystem/I_Continuation.h:146
> #1 0x4f3557 in FetchSM::InvokePluginExt(int)
> /usr/local/src/trafficserver/proxy/FetchSM.cc:301
> #2 0x4f3a7a in FetchSM::process_fetch_read(int)
> /usr/local/src/trafficserver/proxy/FetchSM.cc:465
> #3 0x4f5112 in FetchSM::fetch_handler(int, void*)
> /usr/local/src/trafficserver/proxy/FetchSM.cc:514
> #4 0x59f1b7 in Continuation::handleEvent(int, void*)
> ../iocore/eventsystem/I_Continuation.h:146
> #5 0x59f1b7 in PluginVC::process_read_side(bool)
> /usr/local/src/trafficserver/proxy/PluginVC.cc:640
> #6 0x5abcb9 in PluginVC::main_handler(int, void*)
> /usr/local/src/trafficserver/proxy/PluginVC.cc:206
> #7 0xc821fe in Continuation::handleEvent(int, void*)
> /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:146
> #8 0xc821fe in EThread::process_event(Event*, int)
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:144
> #9 0xc84819 in EThread::execute()
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:238
> #10 0xc80e18 in spawn_thread_internal
> /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:88
> #11 0x2aaf0b083df2 in start_thread (/lib64/libpthread.so.0+0x7df2)
> #12 0x2aaf0c8ec1ac in clone (/lib64/libc.so.6+0xf61ac)
> 0x61800000c888 is located 8 bytes inside of 816-byte region
> [0x61800000c880,0x61800000cbb0)
> freed by thread T0 ([ET_NET 0]) here:
> #0 0x2aaf08c131c7 in __interceptor_free
> ../../.././libsanitizer/asan/asan_malloc_linux.cc:62
> #1 0x7b7d42 in Http2ClientSession::do_io_close(int)
> /usr/local/src/trafficserver/proxy/http2/Http2ClientSession.cc:194
> #2 0x7b7d42 in Http2ClientSession::main_event_handler(int, void*)
> /usr/local/src/trafficserver/proxy/http2/Http2ClientSession.cc:237
> #3 0xc1351f in Continuation::handleEvent(int, void*)
> ../../iocore/eventsystem/I_Continuation.h:146
> #4 0xc1351f in read_signal_and_update
> /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:140
> #5 0xc1351f in read_signal_done
> /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:185
> #6 0xc1351f in UnixNetVConnection::readSignalDone(int, NetHandler*)
> /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:939
> #7 0xbbabf8 in SSLNetVConnection::net_read_io(NetHandler*, EThread*)
> /usr/local/src/trafficserver/iocore/net/SSLNetVConnection.cc:596
> #8 0xbda09c in NetHandler::mainNetEvent(int, Event*)
> /usr/local/src/trafficserver/iocore/net/UnixNet.cc:513
> #9 0xc85089 in Continuation::handleEvent(int, void*)
> /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:146
> #10 0xc85089 in EThread::process_event(Event*, int)
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:144
> #11 0xc85089 in EThread::execute()
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:268
> #12 0x498f96 in main /usr/local/src/trafficserver/proxy/Main.cc:1826
> #13 0x2aaf0c817af4 in __libc_start_main (/lib64/libc.so.6+0x21af4)
> previously allocated by thread T0 ([ET_NET 0]) here:
> #0 0x2aaf08c1393b in __interceptor_posix_memalign
> ../../.././libsanitizer/asan/asan_malloc_linux.cc:130
> #1 0x2aaf09afd2f9 in ats_memalign
> /usr/local/src/trafficserver/lib/ts/ink_memory.cc:96
> #2 0x7cd804 in ClassAllocator<Http2ClientSession>::alloc()
> ../../lib/ts/Allocator.h:124
> #3 0x7cd804 in Http2SessionAccept::accept(NetVConnection*, MIOBuffer*,
> IOBufferReader*)
> /usr/local/src/trafficserver/proxy/http2/Http2SessionAccept.cc:57
> #4 0x7cd3c4 in Http2SessionAccept::mainEvent(int, void*)
> /usr/local/src/trafficserver/proxy/http2/Http2SessionAccept.cc:69
> #5 0xbc2fae in SSLNextProtocolTrampoline::ioCompletionEvent(int, void*)
> /usr/local/src/trafficserver/iocore/net/SSLNextProtocolAccept.cc:101
> #6 0xc1351f in Continuation::handleEvent(int, void*)
> ../../iocore/eventsystem/I_Continuation.h:146
> #7 0xc1351f in read_signal_and_update
> /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:140
> #8 0xc1351f in read_signal_done
> /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:185
> #9 0xc1351f in UnixNetVConnection::readSignalDone(int, NetHandler*)
> /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:939
> #10 0xbbba59 in SSLNetVConnection::net_read_io(NetHandler*, EThread*)
> /usr/local/src/trafficserver/iocore/net/SSLNetVConnection.cc:489
> #11 0xbda09c in NetHandler::mainNetEvent(int, Event*)
> /usr/local/src/trafficserver/iocore/net/UnixNet.cc:513
> #12 0xc85089 in Continuation::handleEvent(int, void*)
> /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:146
> #13 0xc85089 in EThread::process_event(Event*, int)
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:144
> #14 0xc85089 in EThread::execute()
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:268
> #15 0x498f96 in main /usr/local/src/trafficserver/proxy/Main.cc:1826
> #16 0x2aaf0c817af4 in __libc_start_main (/lib64/libc.so.6+0x21af4)
> Thread T2 ([ET_NET 1]) created by T0 ([ET_NET 0]) here:
> #0 0x2aaf08be286a in __interceptor_pthread_create
> ../../.././libsanitizer/asan/asan_interceptors.cc:183
> #1 0xc81aa5 in ink_thread_create ../../lib/ts/ink_thread.h:148
> #2 0xc81aa5 in Thread::start(char const*, unsigned long, void*
> (*)(void*), void*)
> /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:103
> #3 0xc8a026 in EventProcessor::start(int, unsigned long)
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEventProcessor.cc:140
> #4 0x498d0b in main /usr/local/src/trafficserver/proxy/Main.cc:1636
> #5 0x2aaf0c817af4 in __libc_start_main (/lib64/libc.so.6+0x21af4)
> SUMMARY: AddressSanitizer: heap-use-after-free
> ../iocore/eventsystem/I_Continuation.h:146 Continuation::handleEvent(int,
> void*)
> Shadow bytes around the buggy address:
> 0x0c307fff98c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff98d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff98e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff98f0: fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa
> 0x0c307fff9900: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> =>0x0c307fff9910: fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff9920: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff9930: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff9940: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff9950: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff9960: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> Shadow byte legend (one shadow byte represents 8 application bytes):
> Addressable: 00
> Partially addressable: 01 02 03 04 05 06 07
> Heap left redzone: fa
> Heap right redzone: fb
> Freed heap region: fd
> Stack left redzone: f1
> Stack mid redzone: f2
> Stack right redzone: f3
> Stack partial redzone: f4
> Stack after return: f5
> Stack use after scope: f8
> Global redzone: f9
> Global init order: f6
> Poisoned by user: f7
> Contiguous container OOB:fc
> ASan internal: fe
> ==31101==ABORTING
> traffic_server: using root directory '/opt/ats'
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
