Hello Devs
One of our applications in maintenance mode started throwing exceptions
after OS upgrade (Ubuntu was upgraded from v14.x to 16.04)
Lib Versions
Spring Boot v1.4.7.RELEASE
Spring v4.3.9.RELEASE
Apache Tomcat 8.5.15
Axis2 1.7.9
OS Ubuntu 16.04
JVM 8 u241
REST application exposed to the customers and under the hood, it's talking
to SOAP services
Since the OS upgrade, we experience intermittent failures where we see the
sea of exceptions in the logs. Always I see exception in the red prior to
other exceptions.
Any help is appreciated. I am trying to understand what could be the reason
here...
2020-01-18 01:11:50.897 ERROR 115794 [http-nio-8081-ClientPoller-1]
--- org.apache.tomcat.util.net.NioEndpoint : Failed to register
socket with selector from poller
java.lang.NullPointerException: null
at
sun.nio.ch.EPollArrayWrapper.isEventsHighKilled(EPollArrayWrapper.java:174)
at
sun.nio.ch.EPollArrayWrapper.setUpdateEvents(EPollArrayWrapper.java:190)
at sun.nio.ch.EPollArrayWrapper.add(EPollArrayWrapper.java:239)
at sun.nio.ch.EPollSelectorImpl.implRegister(EPollSelectorImpl.java:178)
at sun.nio.ch.SelectorImpl.register(SelectorImpl.java:132)
at
java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:212)
at
org.apache.tomcat.util.net.NioEndpoint$PollerEvent.run(NioEndpoint.java:565)
at
org.apache.tomcat.util.net.NioEndpoint$Poller.events(NioEndpoint.java:680)
at
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:805)
at java.lang.Thread.run(Thread.java:748)
2020-01-18 01:11:50.956 INFO 115794 [http-nio-8081-exec-163] ---
o.a.c.httpclient.HttpMethodDirector : I/O exception
(org.apache.axis2.AxisFault) caught when processing request:
Connection has been shutdown: javax.net.ssl.SSLException:
java.net.SocketException: Socket closed
2020-01-18 01:11:50.956 INFO 115794 [http-nio-8081-exec-163] ---
o.a.c.httpclient.HttpMethodDirector : Retrying request
2020-01-18 01:11:50.962 INFO 115794 [http-nio-8081-exec-177] ---
o.a.c.httpclient.HttpMethodDirector : I/O exception
(org.apache.axis2.AxisFault) caught when processing request:
Connection has been shutdown: javax.net.ssl.SSLException:
java.net.SocketException: Socket closed
2020-01-18 01:11:50.962 INFO 115794 [http-nio-8081-exec-177] ---
o.a.c.httpclient.HttpMethodDirector : Retrying request
2020-01-18 01:11:50.963 INFO 115794 [http-nio-8081-exec-177] ---
o.a.c.httpclient.HttpMethodDirector : I/O exception
(org.apache.axis2.AxisFault) caught when processing request:
Connection has been shutdown: javax.net.ssl.SSLException:
java.net.SocketException: Socket closed
2020-01-18 01:11:50.963 INFO 115794 [http-nio-8081-exec-177] ---
o.a.c.httpclient.HttpMethodDirector : Retrying request
2020-01-18 01:11:50.964 INFO 115794 [http-nio-8081-exec-177] ---
o.a.c.httpclient.HttpMethodDirector : I/O exception
(org.apache.axis2.AxisFault) caught when processing request:
Connection has been shutdown: javax.net.ssl.SSLException:
java.net.SocketException: Socket closed
2020-01-18 01:11:50.964 INFO 115794 [http-nio-8081-exec-177] ---
o.a.c.httpclient.HttpMethodDirector : Retrying request
2020-01-18 01:11:50.966 INFO 115794 [http-nio-8081-exec-177] ---
o.a.a.t.h.i.httpclient3.HTTPSenderImpl : Unable to sendViaPost to
url[https://uwspt.mocklab.io/DeviceService.svc]
org.apache.axis2.AxisFault: Connection has been shutdown:
javax.net.ssl.SSLException: java.net.SocketException: Socket closed
at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
at
org.apache.axis2.transport.http.SOAPMessageFormatter.writeTo(SOAPMessageFormatter.java:78)
at
org.apache.axis2.transport.http.AxisRequestEntity.writeRequest(AxisRequestEntity.java:85)
at
org.apache.commons.httpclient.methods.EntityEnclosingMethod.writeRequestBody(EntityEnclosingMethod.java:499)
at
org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2114)
at
org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
at
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at
org.apache.axis2.transport.http.impl.httpclient3.HTTPSenderImpl.executeMethod(HTTPSenderImpl.java:872)
at
org.apache.axis2.transport.http.impl.httpclient3.HTTPSenderImpl.sendViaPost(HTTPSenderImpl.java:212)
at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:121)
at
org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:403)
at
org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:234)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:431)
at
org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:399)
at
org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225)
at
org.apache.axis2.client.OperationClient.execute(OperationClient.java:150)
at
com.verizon.ids.proxy.device.stubs.DeviceServiceStub.getDeviceUsageHistory(DeviceServiceStub.java:5363)
at
com.verizon.ids.proxy.device.DeviceServiceProxy.lambda$getDeviceUsageHistory$8(DeviceServiceProxy.java:168)
at com.verizon.ids.proxy.AbstractProxy.run(AbstractProxy.java:60)
at
com.verizon.ids.proxy.device.DeviceServiceProxy.getDeviceUsageHistory(DeviceServiceProxy.java:163)
at
com.verizon.ids.rs.device.service.DeviceServiceDelegator.usageList(DeviceServiceDelegator.java:269)
at
com.verizon.ids.rs.device.DeviceServiceController.usageList(DeviceServiceController.java:217)
at sun.reflect.GeneratedMethodAccessor149.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
at
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at
org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at
org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:105)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at
org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at
org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at
org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at
org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478)
at
ch.qos.logback.access.tomcat.LogbackValve.invoke(LogbackValve.java:256)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:799)
at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:861)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1455)
at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.ctc.wstx.exc.WstxIOException: Connection has been
shutdown: javax.net.ssl.SSLException: java.net.SocketException: Socket
closed
at com.ctc.wstx.sw.BaseStreamWriter.flush(BaseStreamWriter.java:261)
at
org.apache.axiom.util.stax.wrapper.XMLStreamWriterWrapper.flush(XMLStreamWriterWrapper.java:50)
at
org.apache.axiom.om.impl.MTOMXMLStreamWriter.flush(MTOMXMLStreamWriter.java:231)
at
org.apache.axiom.om.impl.MTOMXMLStreamWriter.close(MTOMXMLStreamWriter.java:223)
at
org.apache.axiom.om.impl.common.AxiomContainerSupport.ajc$interMethod$org_apache_axiom_om_impl_common_AxiomContainerSupport$org_apache_axiom_om_impl_intf_AxiomContainer$serializeAndConsume(AxiomContainerSupport.aj:324)
at
org.apache.axiom.om.impl.llom.OMElementImpl.serializeAndConsume(OMElementImpl.java:1)
at
org.apache.axis2.transport.http.SOAPMessageFormatter.writeTo(SOAPMessageFormatter.java:74)
... 88 common frames omitted
Caused by: javax.net.ssl.SSLException: Connection has been shutdown:
javax.net.ssl.SSLException: java.net.SocketException: Socket closed
at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1533)
at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1545)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at
org.apache.commons.httpclient.ChunkedOutputStream.flush(ChunkedOutputStream.java:191)
at com.ctc.wstx.io.UTF8Writer.flush(UTF8Writer.java:103)
at com.ctc.wstx.sw.BufferingXmlWriter.flush(BufferingXmlWriter.java:225)
at com.ctc.wstx.sw.BaseStreamWriter.flush(BaseStreamWriter.java:259)
... 94 common frames omitted
Caused by: javax.net.ssl.SSLException: java.net.SocketException: Socket closed
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1903)
at
sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1867)
at
sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1812)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:128)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at
org.apache.commons.httpclient.ChunkedOutputStream.flush(ChunkedOutputStream.java:191)
at com.ctc.wstx.io.UTF8Writer.flush(UTF8Writer.java:103)
at com.ctc.wstx.sw.BufferingXmlWriter.flush(BufferingXmlWriter.java:225)
at com.ctc.wstx.sw.BaseStreamWriter.flush(BaseStreamWriter.java:259)
at
org.apache.axiom.util.stax.wrapper.XMLStreamWriterWrapper.flush(XMLStreamWriterWrapper.java:50)
at
org.apache.axiom.om.impl.MTOMXMLStreamWriter.flush(MTOMXMLStreamWriter.java:231)
at
org.apache.axis2.databinding.ADBDataSource.serialize(ADBDataSource.java:50)
at
org.apache.axiom.om.impl.common.serializer.push.stax.StAXSerializer.serializePushOMDataSource(StAXSerializer.java:53)
at
org.apache.axiom.om.impl.common.serializer.push.Serializer.serialize(Serializer.java:293)
at
org.apache.axiom.om.impl.common.AxiomSourcedElementSupport.ajc$interMethod$org_apache_axiom_om_impl_common_AxiomSourcedElementSupport$org_apache_axiom_om_impl_intf_AxiomSourcedElement$internalSerialize(AxiomSourcedElementSupport.aj:434)
at
org.apache.axiom.om.impl.llom.OMSourcedElementImpl.internalSerialize(OMSourcedElementImpl.java:1)
at
org.apache.axiom.om.impl.common.AxiomContainerSupport.ajc$interMethod$org_apache_axiom_om_impl_common_AxiomContainerSupport$org_apache_axiom_om_impl_intf_AxiomContainer$serializeChildren(AxiomContainerSupport.aj:362)
at
org.apache.axiom.om.impl.llom.OMElementImpl.ajc$interMethodDispatch2$org_apache_axiom_om_impl_common$serializeChildren(OMElementImpl.java:1)
at
org.apache.axiom.om.impl.common.AxiomContainerSupport.ajc$interMethodDispatch1$org_apache_axiom_om_impl_common_AxiomContainerSupport$org_apache_axiom_om_impl_intf_AxiomContainer$serializeChildren(AxiomContainerSupport.aj)
at
org.apache.axiom.om.impl.common.AxiomElementSupport.ajc$interMethod$org_apache_axiom_om_impl_common_AxiomElementSupport$org_apache_axiom_om_impl_intf_AxiomElement$defaultInternalSerialize(AxiomElementSupport.aj:519)
at
org.apache.axiom.om.impl.llom.OMElementImpl.defaultInternalSerialize(OMElementImpl.java:1)
at
org.apache.axiom.om.impl.common.AxiomElementSupport.ajc$interMethodDispatch1$org_apache_axiom_om_impl_common_AxiomElementSupport$org_apache_axiom_om_impl_intf_AxiomElement$defaultInternalSerialize(AxiomElementSupport.aj)
at
org.apache.axiom.om.impl.common.AxiomElementSupport.ajc$interMethod$org_apache_axiom_om_impl_common_AxiomElementSupport$org_apache_axiom_om_impl_intf_AxiomElement$internalSerialize(AxiomElementSupport.aj:513)
at
org.apache.axiom.om.impl.llom.OMElementImpl.internalSerialize(OMElementImpl.java:1)
at
org.apache.axiom.om.impl.common.AxiomContainerSupport.ajc$interMethod$org_apache_axiom_om_impl_common_AxiomContainerSupport$org_apache_axiom_om_impl_intf_AxiomContainer$serializeChildren(AxiomContainerSupport.aj:362)
at
org.apache.axiom.om.impl.llom.OMElementImpl.ajc$interMethodDispatch2$org_apache_axiom_om_impl_common$serializeChildren(OMElementImpl.java:1)
at
org.apache.axiom.om.impl.common.AxiomContainerSupport.ajc$interMethodDispatch1$org_apache_axiom_om_impl_common_AxiomContainerSupport$org_apache_axiom_om_impl_intf_AxiomContainer$serializeChildren(AxiomContainerSupport.aj)
at
org.apache.axiom.om.impl.common.AxiomElementSupport.ajc$interMethod$org_apache_axiom_om_impl_common_AxiomElementSupport$org_apache_axiom_om_impl_intf_AxiomElement$defaultInternalSerialize(AxiomElementSupport.aj:519)
at
org.apache.axiom.om.impl.llom.OMElementImpl.defaultInternalSerialize(OMElementImpl.java:1)
at
org.apache.axiom.om.impl.common.AxiomElementSupport.ajc$interMethodDispatch1$org_apache_axiom_om_impl_common_AxiomElementSupport$org_apache_axiom_om_impl_intf_AxiomElement$defaultInternalSerialize(AxiomElementSupport.aj)
at
org.apache.axiom.om.impl.common.AxiomElementSupport.ajc$interMethod$org_apache_axiom_om_impl_common_AxiomElementSupport$org_apache_axiom_om_impl_intf_AxiomElement$internalSerialize(AxiomElementSupport.aj:513)
at
org.apache.axiom.soap.impl.llom.SOAPEnvelopeImpl.internalSerialize(SOAPEnvelopeImpl.java:159)
at
org.apache.axiom.om.impl.common.AxiomContainerSupport.ajc$interMethod$org_apache_axiom_om_impl_common_AxiomContainerSupport$org_apache_axiom_om_impl_intf_AxiomContainer$serializeAndConsume(AxiomContainerSupport.aj:319)
... 90 common frames omitted
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
at sun.security.ssl.SSLSocketImpl.waitForClose(SSLSocketImpl.java:1761)
at
sun.security.ssl.HandshakeOutStream.flush(HandshakeOutStream.java:124)
at sun.security.ssl.Handshaker.kickstart(Handshaker.java:1079)
at
sun.security.ssl.SSLSocketImpl.kickstartHandshake(SSLSocketImpl.java:1479)
at
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1346)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:750)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
... 120 common frames omitted
