i see these exceptions on 4 out of the 7 nodes in my cluster. in
addition those same four nodes all show AE-SERVICE-STAGE with pending
work, and been showing this for several hours now. each node in the
cluster has less than 2gb, so it should be finished by now.
when i do nodetool streams on these nodes i see streams with byte counts
that are never increasing.
2010-04-23 10:08:43,416 ERROR [MESSAGE-STREAMING-POOL:1]
[DebuggableThreadPoolExecutor.java:101] Error in ThreadPoolExecutor
java.lang.RuntimeException: java.net.ConnectException: Connection timed out
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.ConnectException: Connection timed out
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
at
org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 3 more
2010-04-23 10:08:43,417 ERROR [MESSAGE-STREAMING-POOL:1]
[CassandraDaemon.java:78] Fatal exception in thread
Thread[MESSAGE-STREAMING-POOL:1,5,main]
java.lang.RuntimeException: java.net.ConnectException: Connection timed out
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.ConnectException: Connection timed out
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
at
org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 3 more
