> On Dec 13, 2016, at 12:45 AM, Di Li <di...@apple.com> wrote: > > Hey Guys, > > When I doing some benchmark for outbound proxy, and has http_cache enabled, > well, first of all, the performance are pretty low, I guess I didn’t do it > right with the cache enabled, 2nd when I use wrk to have 512 connection with > 40 thread to go through proxy with http, it cause a core dump, here’s the > trace > > And when I disable the http.cache, the performance has went up a lot, and no > more coredump at all. > > > FATAL: CacheRead.cc:249: failed assert `w->alternate.valid()` > traffic_server: using root directory '/ngs/app/oproxy/trafficserver' > traffic_server: Aborted (Signal sent by tkill() 20136 1001) > traffic_server - STACK TRACE: > /ngs/app/oproxy/trafficserver/bin/traffic_server(_Z19crash_logger_invokeiP7siginfoPv+0xc3)[0x50730d] > /lib64/libc.so.6(+0x35670)[0x2aaaad284670] > /lib64/libc.so.6(gsignal+0x37)[0x2aaaad2845f7] > /lib64/libc.so.6(abort+0x148)[0x2aaaad285ce8] > /ngs/app/oproxy/trafficserver/lib/trafficserver/libtsutil.so.6(_Z12ink_fatal_vaPKcP13__va_list_tag+0x0)[0x2aaaaad0cffd]
This is an assert, there should be a corresponding message in logs somewhere. > /ngs/app/oproxy/trafficserver/lib/trafficserver/libtsutil.so.6(_Z9ink_fatalPKcz+0x0)[0x2aaaaad0d0b4] > /ngs/app/oproxy/trafficserver/lib/trafficserver/libtsutil.so.6(_Z10ink_pfatalPKcz+0x0)[0x2aaaaad0d179] > /ngs/app/oproxy/trafficserver/lib/trafficserver/libtsutil.so.6(+0x3be7a)[0x2aaaaad0ae7a] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN7CacheVC20openReadChooseWriterEiP5Event+0x3be)[0x74292c] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN7CacheVC18openReadFromWriterEiP5Event+0x390)[0x743032] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN12Continuation11handleEventEiPv+0x6c)[0x50a34e] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN5Cache9open_readEP12ContinuationPKN3ats10CryptoHashEP7HTTPHdrP21CacheLookupHttpConfig13CacheFragTypePKci+0x5b0)[0x7422aa] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN14CacheProcessor9open_readEP12ContinuationPK12HttpCacheKeybP7HTTPHdrP21CacheLookupHttpConfigl13CacheFragType+0xbf)[0x728a83] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN11HttpCacheSM18do_cache_open_readERK12HttpCacheKey+0xf7)[0x5c86b5] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN11HttpCacheSM9open_readEPK12HttpCacheKeyP3URLP7HTTPHdrP21CacheLookupHttpConfigl+0x118)[0x5c884a] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM24do_cache_lookup_and_readEv+0x276)[0x5e6956] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xc3a)[0x5f11e6] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x1ae)[0x5f05a4] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0xfd)[0x5db00f] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x95c)[0x5dae1c] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x1ca)[0x5e8f00] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM14do_api_calloutEv+0x27)[0x5f7979] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x5e)[0x5f060a] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x1ae)[0x5f05a4] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x1a2)[0x5f074e] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x1ae)[0x5f05a4] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0xfd)[0x5db00f] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x95c)[0x5dae1c] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x1ca)[0x5e8f00] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM14do_api_calloutEv+0x27)[0x5f7979] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x5e)[0x5f060a] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x1ae)[0x5f05a4] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0xfd)[0x5db00f] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x95c)[0x5dae1c] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x1c0)[0x5da444] > /ngs/app/oproxy/trafficserver/bin/traffic_server(TSHttpTxnReenable+0x198)[0x52b339] > /ngs/app/oproxy/trafficserver/lib/trafficserver/plugins/stats_over_http.so(+0x1c86)[0x2aadfef71c86] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN15INKContInternal12handle_eventEiPv+0xe8)[0x51eff6] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN12Continuation11handleEventEiPv+0x6c)[0x50a34e] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN7APIHook6invokeEiPv+0x73)[0x51f9ab] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x671)[0x5dab31] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x1ca)[0x5e8f00] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM14do_api_calloutEv+0x27)[0x5f7979] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x5e)[0x5f060a] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x1ae)[0x5f05a4] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM32state_read_client_request_headerEiPv+0x124b)[0x5d88d9] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x270)[0x5df19c] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN12Continuation11handleEventEiPv+0x6c)[0x50a34e] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM32setup_client_read_request_headerEv+0xe9)[0x5d7667] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x5e)[0x5daf70] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x95c)[0x5dae1c] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x1ca)[0x5e8f00] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM14do_api_calloutEv+0x27)[0x5f7979] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17state_add_to_listEiPv+0x27e)[0x5d6c86] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM21attach_client_sessionEP22ProxyClientTransactionP14IOBufferReader+0x484)[0x5d7506] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN22ProxyClientTransaction15new_transactionEv+0x2d0)[0x552138] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN18Http1ClientSession15new_transactionEv+0xc0)[0x5cabfe] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN18Http1ClientSession16state_keep_aliveEiPv+0x1d3)[0x5ca715] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN12Continuation11handleEventEiPv+0x6c)[0x50a34e] > /ngs/app/oproxy/trafficserver/bin/traffic_server[0x78973b] > /ngs/app/oproxy/trafficserver/bin/traffic_server[0x78a494] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN18UnixNetVConnection11net_read_ioEP10NetHandlerP7EThread+0x2b)[0x78c5ef] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x6f0)[0x7816fc] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN12Continuation11handleEventEiPv+0x6c)[0x50a34e] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x136)[0x7ac1ec] > /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN7EThread7executeEv+0x4ae)[0x7ac818] > /ngs/app/oproxy/trafficserver/bin/traffic_server[0x7ab784] > /lib64/libpthread.so.0(+0x7dc5)[0x2aaaad615dc5] > /lib64/libc.so.6(clone+0x6d)[0x2aaaad345c4d] > > > Thanks, > Di Li > > > > >
