Hi, We are using ATS 7.1.4 and recently have started noticing trafficserver crashes randomly with the following assertion. We are trying to analyze what could be causing this. A quick google search did not yield promising results. Hence, reaching out for pointers to debug and understand the cause for the crash. Any log/metric that can be enabled which can help in identifying the root cause?
Fatal: HttpTunnel.cc:905: failed assertion `p->num_consumers > 0` StackTrace: Nov 17 04:56:48 ftest03 traffic_cop[6416]: spawning traffic_manager Nov 17 04:56:48 ftest03traffic_manager[6425]: NOTE: --- Manager Starting ---Nov 17 17:33:17 ftest03traffic_server[6848]: Fatal: InkAPI.cc:4865: failed assertion `sdk_sanity_check_mbuffer(*bufp) == TS_SUCCESS` Nov 17 17:33:21 ftest03 traffic_server[11999]: NOTE: --- traffic_server Starting --- Nov 17 17:33:21 ftest03 traffic_server[11999]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 7.1.4 - (build # 060114 on Jun 1 2020 at 14:24:51)Nov 19 14:16:41 ftest03 traffic_server[11999]: Fatal: HttpTunnel.cc:905: failed assertion `p->num_consumers > 0` Nov 19 14:16:44 ftest03 traffic_server[7566]: NOTE: --- traffic_server Starting --- Nov 19 14:16:44 ftest03 traffic_server[7566]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 7.1.4 - (build # 060114 on Jun 1 2020 at 14:24:51)Nov 22 19:47:50 ftest03 traffic_server[2144]: Fatal: InkAPI.cc:4865: failed assertion `sdk_sanity_check_mbuffer(*bufp) == TS_SUCCESS` Nov 22 19:47:51 ftest03 traffic_cop[6416]: (test) read failed [104 'Connection reset by peer'] Nov 22 19:47:51 ftest03 traffic_cop[6416]: server heartbeat failed [1] Nov 22 19:47:54 ftest03 traffic_server[46729]: NOTE: --- traffic_server Starting --- Nov 22 19:47:54 ftest03 traffic_server[46729]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 7.1.4 - (build # 060114 on Jun 1 2020 at 14:24:51) Nov 22 19:48:01 ftest03 traffic_cop[6416]: server heartbeat succeeded Nov 23 05:11:26 ftest03 systemd: Starting Cleanup of Temporary Directories... Nov 23 05:11:26 ftest03 systemd: Started Cleanup of Temporary Directories. Traffic.out logs Fatal: HttpTunnel.cc:905: failed assertion `p->num_consumers > 0` traffic_server: received signal 6 (Aborted) traffic_server - STACK TRACE: /opt/CCURcache/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0x8e)[0x4b063e] /lib64/libpthread.so.0(+0xf5f0)[0x2b65110585f0] /lib64/libc.so.6(gsignal+0x37)[0x2b6511cbe337] /lib64/libc.so.6(abort+0x148)[0x2b6511cbfa28] /opt/CCURcache/lib64/libtsutil.so.7(+0x2debb)[0x2b650f739ebb] /opt/CCURcache/lib64/libtsutil.so.7(+0x2bf45)[0x2b650f737f45] /opt/CCURcache/bin/traffic_server(_ZN10HttpTunnel12producer_runEP18HttpTunnelProducer+0x10b6)[0x5ccfc6] /opt/CCURcache/bin/traffic_server(_ZN10HttpTunnel10tunnel_runEP18HttpTunnelProducer+0x29)[0x5cd0f9] /opt/CCURcache/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x416)[0x591b96] /opt/CCURcache/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x32)[0x57f2a2] /opt/CCURcache/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x8a)[0x5913ba] /opt/CCURcache/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x43e)[0x5896be] /opt/CCURcache/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x135)[0x590ca5] /opt/CCURcache/bin/traffic_server(TSHttpTxnReenable+0x249)[0x4da759] /opt/CCURcache/libexec/trafficserver/collapsed_forwarding.so(+0x1595)[0x2b6644240595] /opt/CCURcache/bin/traffic_server(_ZN15INKContInternal12handle_eventEiPv+0x8e)[0x4c5c0e] /opt/CCURcache/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x234)[0x5894b4] /opt/CCURcache/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x135)[0x590ca5] /opt/CCURcache/bin/traffic_server(TSHttpTxnReenable+0x249)[0x4da759] /opt/CCURcache/libexec/trafficserver/RouterPlugin.so(_ZN12RouterPlugin25handleReadResponseHeadersERN9atscppapi11TransactionE+0x60a)[0x2b6648a9ed26] /opt/CCURcache/lib64/libatscppapi.so.7(+0x1300d)[0x2b664467400d] /opt/CCURcache/bin/traffic_server(_ZN15INKContInternal12handle_eventEiPv+0x8e)[0x4c5c0e] /opt/CCURcache/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x234)[0x5894b4] /opt/CCURcache/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x135)[0x590ca5] /opt/CCURcache/bin/traffic_server(TSHttpTxnReenable+0x249)[0x4da759] /opt/CCURcache/lib64/libatscppapi.so.7(+0x1f305)[0x2b6644680305] /opt/CCURcache/bin/traffic_server(_ZN15INKContInternal12handle_eventEiPv+0x8e)[0x4c5c0e] /opt/CCURcache/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x234)[0x5894b4] /opt/CCURcache/bin/traffic_server(_ZN6HttpSM33state_read_server_response_headerEiPv+0x350)[0x58aa10] /opt/CCURcache/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xbd)[0x590e5d] /opt/CCURcache/bin/traffic_server[0x73298e] /opt/CCURcache/bin/traffic_server(_ZN10NetHandler15waitForActivityEl+0x2c0)[0x7223c0] /opt/CCURcache/bin/traffic_server(_ZN7EThread15execute_regularEv+0x3f2)[0x7552b2] /opt/CCURcache/bin/traffic_server[0x753fda] /lib64/libpthread.so.0(+0x7e65)[0x2b6511050e65] /lib64/libc.so.6(clone+0x6d)[0x2b6511d8688d] Thanks in advance for the help. Raji
