Running on Linux with pthreads. One thread (CBTcpProxyListenerThread below) adds bufferevents (with option BEV_OPT_THREADSAFE) to an event_base. A second thread (CBTcpProxySenderThread) dispatches on the event_base.
bufferevents are removed from the event_base either by a third thread or by the CBTcpProxySenderThread by calling bufferevent_free (without calling bufferevent_disable first - is this a misuse?). The deadlock happens on pretty high load: ~6000 bufferevents are added and removed per second. Each one is triggered for write ~10 times per seconds (which gives ~60,000 triggeres per-second). Here's the deadlock stack: Thread 1 (CBTcpProxyListenerThread) #0 0x00000030c1c0c758 in __lll_mutex_lock_wait () from /opt/breach-proxy/lib64/libpthread.so.0 #1 0x00000030c1c087fa in _L_mutex_lock_908 () from /opt/breach-proxy/lib64/libpthread.so.0 #2 0x00000030c1c08682 in pthread_mutex_lock () from /opt/breach-proxy/lib64/libpthread.so.0 #3 0x00002aaab13039c8 in evthread_posix_lock (mode=0, _lock=0x15f27b20) at evthread_pthread.c:71 #4 0x00002aaab10e0f05 in event_add (ev=0x15f27d58, tv=0x0) at event.c:1815 #5 0x00002aaab10ed356 in _bufferevent_add_event (ev=0x15f27d58, tv=0x15f27e40) at bufferevent.c:824 #6 0x00002aaab10edf3e in be_socket_enable (bufev=0x15f27cc0, event=4) at bufferevent_sock.c:548 #7 0x00002aaab10ec296 in bufferevent_enable (bufev=0x15f27cc0, event=4) at bufferevent.c:418 #8 0x00002aaaaaac398d in CBTcpProxySenderThread::AddEvents (this=0x15f27490, px...@0x15e649d0) at sender_thread.cpp:56 #9 0x00002aaaaaac4a83 in CBTcpProxy::AddEvents (this=0x7fff061d3a70, px...@0x15e649d0) at tcpproxy.cpp:465 #10 0x00002aaaaaabc920 in complete_pxcn_after_server_connect (fd=236, what=4, arg=0x15e649d0) at listener_thread.cpp:234 #11 0x00002aaab10dedca in event_process_active_single_queue (base=0x15c04650, activeq=0x15c066f0) at event.c:1232 #12 0x00002aaab10df35d in event_process_active (base=0x15c04650) at event.c:1290 #13 0x00002aaab10df98c in event_base_loop (base=0x15c04650, flags=0) at event.c:1483 #14 0x00002aaab10df3c6 in event_base_dispatch (event_base=0x15c04650) at event.c:1317 #15 0x00002aaaaaabdae3 in CBTcpProxyListenerThread::run (this=0x15c04510) at listener_thread.cpp:41 ... Thread 2 (CBTcpProxySenderThread) #0 0x00000030c1c0c758 in __lll_mutex_lock_wait () from /opt/breach-proxy/lib64/libpthread.so.0 #1 0x00000030c1c087fa in _L_mutex_lock_908 () from /opt/breach-proxy/lib64/libpthread.so.0 #2 0x00000030c1c08682 in pthread_mutex_lock () from /opt/breach-proxy/lib64/libpthread.so.0 #3 0x00002aaab13039c8 in evthread_posix_lock (mode=0, _lock=0x15f27b20) at evthread_pthread.c:71 #4 0x00002aaab10e188e in event_del (ev=0x2aaabc0678a0) at event.c:2015 #5 0x00002aaab10ee095 in be_socket_destruct (bufev=0x2aaabc067890) at bufferevent_sock.c:581 #6 0x00002aaab10ec956 in _bufferevent_decref_and_unlock (bufev=0x2aaabc067890) at bufferevent.c:600 #7 0x00002aaab10ed85f in bufferevent_writecb (fd=321, event=4, arg=0x2aaabc067890) at bufferevent_sock.c:306 #8 0x00002aaab10df1a7 in event_persist_closure (base=0x15f275e0, ev=0x2aaabc067928) at event.c:1184 #9 0x00002aaab10ded6d in event_process_active_single_queue (base=0x15f275e0, activeq=0x15f27b00) at event.c:1227 #10 0x00002aaab10df35d in event_process_active (base=0x15f275e0) at event.c:1290 #11 0x00002aaab10df98c in event_base_loop (base=0x15f275e0, flags=0) at event.c:1483 #12 0x00002aaab10df3c6 in event_base_dispatch (event_base=0x15f275e0) at event.c:1317 #13 0x00002aaaaaac3ab2 in CBTcpProxySenderThread::run (this=0x15f27490) at sender_thread.cpp:35 ... Thanks, Avi
