ckdarby opened a new issue #15: URL: https://github.com/apache/pulsar-helm-chart/issues/15
**Describe the bug** ```text [conf/proxy.conf] Applying config authenticationEnabled = true [conf/proxy.conf] Applying config authenticationProviders = org.apache.pulsar.broker.authentication.AuthenticationProviderToken [conf/proxy.conf] Applying config brokerClientAuthenticationParameters = file:///pulsar/tokens/proxy/token [conf/proxy.conf] Applying config brokerClientAuthenticationPlugin = org.apache.pulsar.client.impl.auth.AuthenticationToken [conf/proxy.conf] Applying config brokerServiceURL = pulsar://dga-detection-pulsar-broker:6650 [conf/proxy.conf] Applying config brokerWebServiceURL = http://dga-detection-pulsar-broker:8080 [conf/proxy.conf] Applying config httpNumThreads = 8 [conf/proxy.conf] Applying config servicePort = 6650 [conf/proxy.conf] Applying config statusFilePath = /pulsar/status [conf/proxy.conf] Applying config tokenPublicKey = file:///pulsar/keys/token/public.key [conf/proxy.conf] Applying config webServicePort = 80 [conf/pulsar_env.sh] Applying config PULSAR_GC = " -XX:+UseG1GC -XX:MaxGCPauseMillis=10 " [conf/pulsar_env.sh] Applying config PULSAR_MEM = " -Xms1024m -Xmx4096m -XX:MaxDirectMemorySize=4096m -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=4 -XX:ConcGCThreads=4 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem " [conf/pulsar_env.sh] Applying config PULSAR_GC = " -XX:+UseG1GC -XX:MaxGCPauseMillis=10 " [conf/pulsar_env.sh] Applying config PULSAR_MEM = " -Xms1024m -Xmx4096m -XX:MaxDirectMemorySize=4096m -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=4 -XX:ConcGCThreads=4 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem " 03:23:26.784 [main] INFO org.apache.pulsar.broker.authentication.AuthenticationService - org.apache.pulsar.broker.authentication.AuthenticationProviderToken has been loaded. 03:23:26.894 [main] INFO org.eclipse.jetty.util.log - Logging initialized @1494ms to org.eclipse.jetty.util.log.Slf4jLog 03:23:27.015 [main] INFO org.apache.pulsar.proxy.server.ProxyService - Started Pulsar Proxy at /0.0.0.0:6650 03:23:27.110 [main] INFO org.eclipse.jetty.server.Server - jetty-9.4.20.v20190813; built: 2019-08-13T21:28:18.144Z; git: 84700530e645e812b336747464d6fbbf370c9a20; jvm 1.8.0_232-b09 03:23:27.134 [main] INFO org.eclipse.jetty.server.session - DefaultSessionIdManager workerName=node0 03:23:27.134 [main] INFO org.eclipse.jetty.server.session - No SessionScavenger set, using defaults 03:23:27.136 [main] INFO org.eclipse.jetty.server.session - node0 Scavenging every 600000ms 03:23:27.144 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@1ac85b0c{/metrics,null,AVAILABLE} 03:23:27.649 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@3dd69f5a{/,null,AVAILABLE} 03:23:27.675 [main] INFO org.eclipse.jetty.util.thread.ThreadPoolBudget - ReservedThreadExecutor@409986fe{s=0/1,p=0} requires 1 threads from WebExecutorThreadPool[etp425015667]@19553973{STARTED,8<=8<=8,i=8,q=0,ReservedThreadExecutor@409986fe{s=0/1,p=0}} 03:23:27.675 [main] INFO org.eclipse.jetty.util.thread.ThreadPoolBudget - ClientSelectorManager@19b047fe{STARTING} requires 8 threads from WebExecutorThreadPool[etp425015667]@19553973{STARTED,8<=8<=8,i=8,q=0,ReservedThreadExecutor@409986fe{s=0/1,p=0}} 03:23:27.676 [main] WARN org.eclipse.jetty.server.handler.ContextHandler.admin - unavailable javax.servlet.ServletException: java.lang.IllegalStateException: Insufficient configured threads: required=9 < max=8 for WebExecutorThreadPool[etp425015667]@19553973{STARTED,8<=8<=8,i=8,q=0,ReservedThreadExecutor@409986fe{s=0/1,p=0}} at org.apache.pulsar.proxy.server.AdminProxyHandler.createHttpClient(AdminProxyHandler.java:138) ~[org.apache.pulsar-pulsar-proxy-2.5.0.jar:2.5.0] at org.eclipse.jetty.proxy.AbstractProxyServlet.init(AbstractProxyServlet.java:130) ~[org.eclipse.jetty-jetty-proxy-9.4.20.v20190813.jar:9.4.20.v20190813] at javax.servlet.GenericServlet.init(GenericServlet.java:244) ~[javax.servlet-javax.servlet-api-3.1.0.jar:3.1.0] at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:656) ~[org.eclipse.jetty-jetty-servlet-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:421) ~[org.eclipse.jetty-jetty-servlet-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.servlet.ServletHandler.lambda$initialize$0(ServletHandler.java:746) ~[org.eclipse.jetty-jetty-servlet-9.4.20.v20190813.jar:9.4.20.v20190813] at java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:357) [?:1.8.0_232] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:483) [?:1.8.0_232] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) [?:1.8.0_232] at java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:313) [?:1.8.0_232] at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:743) [?:1.8.0_232] at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742) [?:1.8.0_232] at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647) [?:1.8.0_232] at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:739) [org.eclipse.jetty-jetty-servlet-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:361) [org.eclipse.jetty-jetty-servlet-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:821) [org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:276) [org.eclipse.jetty-jetty-servlet-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) [org.eclipse.jetty-jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) [org.eclipse.jetty-jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) [org.eclipse.jetty-jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:106) [org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) [org.eclipse.jetty-jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) [org.eclipse.jetty-jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) [org.eclipse.jetty-jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:106) [org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) [org.eclipse.jetty-jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) [org.eclipse.jetty-jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110) [org.eclipse.jetty-jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:106) [org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.server.handler.StatisticsHandler.doStart(StatisticsHandler.java:255) [org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) [org.eclipse.jetty-jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) [org.eclipse.jetty-jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.server.Server.start(Server.java:407) [org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110) [org.eclipse.jetty-jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:106) [org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.server.Server.doStart(Server.java:371) [org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) [org.eclipse.jetty-jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813] at org.apache.pulsar.proxy.server.WebServer.start(WebServer.java:202) [org.apache.pulsar-pulsar-proxy-2.5.0.jar:2.5.0] at org.apache.pulsar.proxy.server.ProxyServiceStarter.<init>(ProxyServiceStarter.java:168) [org.apache.pulsar-pulsar-proxy-2.5.0.jar:2.5.0] at org.apache.pulsar.proxy.server.ProxyServiceStarter.main(ProxyServiceStarter.java:172) [org.apache.pulsar-pulsar-proxy-2.5.0.jar:2.5.0] Caused by: java.lang.IllegalStateException: Insufficient configured threads: required=9 < max=8 for WebExecutorThreadPool[etp425015667]@19553973{STARTED,8<=8<=8,i=8,q=0,ReservedThreadExecutor@409986fe{s=0/1,p=0}} at org.eclipse.jetty.util.thread.ThreadPoolBudget.check(ThreadPoolBudget.java:156) ~[org.eclipse.jetty-jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.util.thread.ThreadPoolBudget.leaseTo(ThreadPoolBudget.java:130) ~[org.eclipse.jetty-jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.util.thread.ThreadPoolBudget.leaseFrom(ThreadPoolBudget.java:182) ~[org.eclipse.jetty-jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.io.SelectorManager.doStart(SelectorManager.java:255) ~[org.eclipse.jetty-jetty-io-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) ~[org.eclipse.jetty-jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) ~[org.eclipse.jetty-jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) ~[org.eclipse.jetty-jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.client.AbstractConnectorHttpClientTransport.doStart(AbstractConnectorHttpClientTransport.java:64) ~[org.eclipse.jetty-jetty-client-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) ~[org.eclipse.jetty-jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) ~[org.eclipse.jetty-jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) ~[org.eclipse.jetty-jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.client.HttpClient.doStart(HttpClient.java:244) ~[org.eclipse.jetty-jetty-client-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) ~[org.eclipse.jetty-jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813] at org.apache.pulsar.proxy.server.AdminProxyHandler.createHttpClient(AdminProxyHandler.java:126) ~[org.apache.pulsar-pulsar-proxy-2.5.0.jar:2.5.0] ... 39 more 03:23:27.685 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@3dd69f5a{/,null,UNAVAILABLE} 03:23:27.685 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@1ac85b0c{/metrics,null,UNAVAILABLE} 03:23:27.685 [main] INFO org.eclipse.jetty.server.session - node0 Stopped scavenging 2020-05-28 03:23:27,686 [sun.misc.Launcher$AppClassLoader@18769467] error Uncaught exception in thread main: Failed to start HTTP server on ports [80] ``` **To Reproduce** Steps to reproduce the behavior: 1. Had a nodes running Pulsar 2. Created another group of nodes 3. Terminated nodes 4. Proxy never recovered even when I tried stateful scale of 0 and the back up to 3 **Expected behavior** Proxy recovers. Not sure why proxy is trying to get 9 threads when it says numThreads 8. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
