[ https://issues.apache.org/jira/browse/CASSANDRA-20598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17952822#comment-17952822 ]
Chris Miller commented on CASSANDRA-20598: ------------------------------------------ We are also seeing this behaviour in 4.1.6 > unexpected exception caught while processing inbound messages; terminating > connection > ------------------------------------------------------------------------------------- > > Key: CASSANDRA-20598 > URL: https://issues.apache.org/jira/browse/CASSANDRA-20598 > Project: Apache Cassandra > Issue Type: Bug > Reporter: Chris Miller > Priority: Normal > > Hi, > We experiencing an intermittent issue on Cassandra 4.1.2 whereby an instance > becomes unreachable. > Nodetool status show all instances as UN. > Nodetool describecluster on the problem instance shows all the other > instances as unreachable. > The following errors are raised in the system.log for the problem instance: > {code:java} > ERROR [Messaging-EventLoop-3-13] 2025-04-25 09:31:54,041 > InboundMessageHandler.java:298 - > /xx.xx.xx.130:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-fe780889 unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-16] 2025-04-25 09:33:12,378 > InboundMessageHandler.java:298 - > /xx.xx.xx.173:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-b95efe9a unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-3] 2025-04-25 09:33:12,389 > InboundMessageHandler.java:298 - > /xx.xx.xx.64:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-88816473 unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-6] 2025-04-25 09:33:12,389 > InboundMessageHandler.java:298 - > /xx.xx.xx.151:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-71b1f4b7 unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-15] 2025-04-25 09:33:12,406 > InboundMessageHandler.java:298 - > /xx.xx.xx.121:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-9a180cbe unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-18] 2025-04-25 09:34:34,297 > InboundMessageHandler.java:298 - > /xx.xx.xx.130:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-2ec52119 unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-21] 2025-04-25 09:34:34,322 > InboundMessageHandler.java:298 - > /xx.xx.xx.151:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-7c4e7430 unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-19] 2025-04-25 09:34:34,323 > InboundMessageHandler.java:298 - > /xx.xx.xx.173:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-1a1a633e unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-1] 2025-04-25 09:36:04,409 > InboundMessageHandler.java:298 - > /xx.xx.xx.130:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-d47420fa unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-3] 2025-04-25 09:37:30,426 > InboundMessageHandler.java:298 - > /xx.xx.xx.151:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-8e2f3082 unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-4] 2025-04-25 09:39:21,017 > InboundMessageHandler.java:298 - > /xx.xx.xx.130:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-c9e5ded3 unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-5] 2025-04-25 09:40:51,131 > InboundMessageHandler.java:298 - > /xx.xx.xx.151:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-53b342c6 unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-6] 2025-04-25 09:42:30,459 > InboundMessageHandler.java:298 - > /xx.xx.xx.130:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-aa6e2d42 unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-8] 2025-04-25 09:45:25,561 > InboundMessageHandler.java:298 - > /xx.xx.xx.130:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-05fb4a27 unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-16] 2025-04-25 09:45:25,588 > InboundMessageHandler.java:298 - > /xx.xx.xx.137:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-cc2de96d unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-2] 2025-04-25 09:45:25,607 > InboundMessageHandler.java:298 - > /xx.xx.xx.227:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-17bf63a9 unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-9] 2025-04-25 09:48:03,769 > InboundMessageHandler.java:298 - > /xx.xx.xx.130:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-2ac60979 unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-5] 2025-04-25 09:48:03,794 > InboundMessageHandler.java:298 - > /xx.xx.xx.230:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-d267e81b unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-11] 2025-04-25 09:48:03,795 > InboundMessageHandler.java:298 - > /xx.xx.xx.227:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-66bb17f6 unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-10] 2025-04-25 09:48:03,797 > InboundMessageHandler.java:298 - > /xx.xx.xx.137:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-397c0fe0 unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-12] 2025-04-25 09:49:23,130 > InboundMessageHandler.java:298 - > /xx.xx.xx.130:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-10efb813 unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-14] 2025-04-25 09:51:10,139 > InboundMessageHandler.java:298 - > /xx.xx.xx.137:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-e03fac48 unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-15] 2025-04-25 09:51:10,158 > InboundMessageHandler.java:298 - > /xx.xx.xx.230:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-7ced7ea9 unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-16] 2025-04-25 09:52:22,329 > InboundMessageHandler.java:298 - > /xx.xx.xx.130:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-ce5933fe unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-18] 2025-04-25 09:53:41,178 > InboundMessageHandler.java:298 - > /xx.xx.xx.230:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-ef1bb900 unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-19] 2025-04-25 09:55:05,146 > InboundMessageHandler.java:298 - > /xx.xx.xx.130:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-3a141696 unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-20] 2025-04-25 09:56:44,986 > InboundMessageHandler.java:298 - > /xx.xx.xx.230:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-6777370c unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-21] 2025-04-25 09:58:36,090 > InboundMessageHandler.java:298 - > /xx.xx.xx.130:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-40e8d60d unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-22] 2025-04-25 09:59:48,281 > InboundMessageHandler.java:298 - > /xx.xx.xx.230:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-8a5cd748 unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-1] 2025-04-25 10:02:10,363 > InboundMessageHandler.java:298 - > /xx.xx.xx.130:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-f06ef0e9 unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-2] 2025-04-25 10:02:10,396 > InboundMessageHandler.java:298 - > /xx.xx.xx.230:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-0b442608 unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-4] 2025-04-25 10:04:48,827 > InboundMessageHandler.java:298 - > /xx.xx.xx.230:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-fc4a865c unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-5] 2025-04-25 10:07:40,860 > InboundMessageHandler.java:298 - > /xx.xx.xx.230:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-013bc248 unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-6] 2025-04-25 10:09:57,561 > InboundMessageHandler.java:298 - > /xx.xx.xx.230:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-e310655c unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-7] 2025-04-25 10:12:42,939 > InboundMessageHandler.java:298 - > /xx.xx.xx.230:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-45d27775 unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-21] 2025-04-25 10:14:01,082 > InboundMessageHandler.java:298 - > /xx.xx.xx.183:7000->/xx.xx.xx.15:7000-LARGE_MESSAGES-a8e56680 unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-9] 2025-04-25 10:15:22,170 > InboundMessageHandler.java:298 - > /xx.xx.xx.137:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-6681bfbf unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-2] 2025-04-25 10:15:22,175 > InboundMessageHandler.java:298 - > /xx.xx.xx.151:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-2af0ff09 unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-4] 2025-04-25 10:16:45,626 > InboundMessageHandler.java:298 - > /xx.xx.xx.130:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-e48a1c37 unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-13] 2025-04-25 10:17:51,160 > InboundMessageHandler.java:298 - > /xx.xx.xx.151:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-17c4b3cf unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-12] 2025-04-25 10:17:51,178 > InboundMessageHandler.java:298 - > /xx.xx.xx.137:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-5f8217cc unexpected > exception caught while processing inbound messages; terminating connection > ERROR [Messaging-EventLoop-3-14] 2025-04-25 10:17:51,180 > InboundMessageHandler.java:298 - > /xx.xx.xx.130:7000->/xx.xx.xx.15:7000-SMALL_MESSAGES-c65cabfe unexpected > exception caught while processing inbound messages; terminating connection > {code} > I tried stopping/restarting gossip which didn't work. > I also tried nodetool drain and the following errors were seen in the > system.log > {code:java} > ERROR [RMI TCP Connection(232012)-xx.xx.xx.xx] 2025-04-25 10:21:29,464 > StorageService.java:5479 - Messaging service timed out shutting down > java.lang.RuntimeException: java.util.concurrent.TimeoutException > at org.apache.cassandra.utils.Throwables.maybeFail(Throwables.java:73) > at > org.apache.cassandra.utils.Throwables.maybeFail(Throwables.java:105) > at > org.apache.cassandra.net.MessagingService.shutdown(MessagingService.java:544) > at > org.apache.cassandra.net.MessagingService.shutdown(MessagingService.java:519) > at > org.apache.cassandra.service.StorageService.drain(StorageService.java:5473) > at > org.apache.cassandra.service.StorageService.drain(StorageService.java:5423) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) > at jdk.internal.reflect.GeneratedMethodAccessor11.invoke(Unknown > Source) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at java.base/sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:260) > at > java.management/com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) > at > java.management/com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) > at > java.management/com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) > at > java.management/com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) > at > java.management/com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252) > at > java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:809) > at > com.zegelin.jmx.DelegatingMBeanServerInterceptor.invoke(DelegatingMBeanServerInterceptor.java:95) > at > java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801) > at > java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1466) > at > java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1307) > at > java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1399) > at > java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:827) > at > java.base/jdk.internal.reflect.GeneratedMethodAccessor14.invoke(Unknown > Source) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at > java.rmi/sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:359) > at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:200) > at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:197) > at java.base/java.security.AccessController.doPrivileged(Native > Method) > at > java.rmi/sun.rmi.transport.Transport.serviceCall(Transport.java:196) > at > java.rmi/sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:562) > at > java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:796) > at > java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:677) > at java.base/java.security.AccessController.doPrivileged(Native > Method) > at > java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:676) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > Caused by: java.util.concurrent.TimeoutException: null > at > org.apache.cassandra.net.RequestCallbacks.awaitTerminationUntil(RequestCallbacks.java:189) > at > org.apache.cassandra.net.MessagingService.lambda$shutdown$5(MessagingService.java:554) > at org.apache.cassandra.utils.Throwables.perform(Throwables.java:144) > at org.apache.cassandra.utils.Throwables.perform(Throwables.java:134) > ... 42 common frames omitted{code} > Stopping cassandra also seems to have raised an error associated with the > previous drain command: > {code:java} > ERROR [RMI TCP Connection(232012)-10.248.70.15] 2025-04-25 10:26:29,465 > StorageService.java:5572 - Caught an exception while draining > java.util.concurrent.TimeoutException: > org.apache.cassandra.concurrent.SEPExecutor@69c09be4 did not terminate on time > at > org.apache.cassandra.utils.ExecutorUtils.awaitTerminationUntil(ExecutorUtils.java:111) > at > org.apache.cassandra.utils.ExecutorUtils.awaitTermination(ExecutorUtils.java:100) > at > org.apache.cassandra.concurrent.Stage.shutdownAndAwaitMutatingExecutors(Stage.java:174) > at > org.apache.cassandra.service.StorageService.drain(StorageService.java:5484) > at > org.apache.cassandra.service.StorageService.drain(StorageService.java:5423) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) > at jdk.internal.reflect.GeneratedMethodAccessor11.invoke(Unknown > Source) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at java.base/sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:260) > at > java.management/com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) > at > java.management/com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) > at > java.management/com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) > at > java.management/com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) > at > java.management/com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252) > at > java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:809) > at > com.zegelin.jmx.DelegatingMBeanServerInterceptor.invoke(DelegatingMBeanServerInterceptor.java:95) > at > java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801) > at > java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1466) > at > java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1307) > at > java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1399) > at > java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:827) > at > java.base/jdk.internal.reflect.GeneratedMethodAccessor14.invoke(Unknown > Source) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at > java.rmi/sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:359) > at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:200) > at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:197) > at java.base/java.security.AccessController.doPrivileged(Native > Method) > at > java.rmi/sun.rmi.transport.Transport.serviceCall(Transport.java:196) > at > java.rmi/sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:562) > at > java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:796) > at > java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:677) > at java.base/java.security.AccessController.doPrivileged(Native > Method) > at > java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:676) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > {code} > So wondering are we hitting a bug here? Any ideas? > Thanks, > Chris. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org