Hi,
We have c* 2.0.17 cluster with 2 DCs - DC1, DC2. We tried to add new data
center DC3 and ran "nodetool rebuild 'DC1'" and we faced below exception on
few nodes after some data got streamed to new nodes in new data center DC3.
*Exception in thread "main" java.lang.RuntimeException: Error while
rebuilding node: Stream failed*
*at
org.apache.cassandra.service.StorageService.rebuild(StorageService.java:936)*
*at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)*
*at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)*
*at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)*
*at java.lang.reflect.Method.invoke(Method.java:606)*
*at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)*
*at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)*
*at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)*
*at java.lang.reflect.Method.invoke(Method.java:606)*
*at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)*
*at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)*
*at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)*
*at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)*
*at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)*
*at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)*
*at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)*
*at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)*
*at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)*
*at
javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)*
*at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)*
*at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)*
*at
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)*
*at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)*
*at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)*
*at java.lang.reflect.Method.invoke(Method.java:606)*
*at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)*
*at sun.rmi.transport.Transport$2.run(Transport.java:202)*
*at sun.rmi.transport.Transport$2.run(Transport.java:199)*
*at java.security.AccessController.doPrivileged(Native Method)*
*at sun.rmi.transport.Transport.serviceCall(Transport.java:198)*
*at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:567)*
*at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:828)*
*at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.access$400(TCPTransport.java:619)*
*at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:684)*
*at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:681)*
*at java.security.AccessController.doPrivileged(Native Method)*
*at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:681)*
*at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)*
*at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)*
*at java.lang.Thread.run(Thread.java:745)*
We have 4 user keyspaces , so we altered all keyspaces as below before
issuing rebuild.
*ALTER KEYSPACE keyspace_name WITH replication = {'class':
'NetworkTopologyStrategy', 'DC1': '3' , 'DC2' : '3' , 'DC3' : '3'};*
*Output of describecluster*
./nodetool describecluster
Cluster Information:
Name: Ss Cluster
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
3b688e54-47be-39e8-ae45-e71ba98d69e2: [xxx.xxx.198.75, xxx.xxx.198.132,
xxx.xxx.198.133, xxx.xxx.12.115, xxx.xxx.198.78, xxx.xxx.12.123,
xxx.xxx.98.205, xxx.xxx.98.219, xxx.xxx.98.220, xxx.xxx.198.167,
xxx.xxx.98.172, xxx.xxx.98.173, xxx.xxx.98.170, xxx.xxx.198.168,
xxx.xxx.98.171, xxx.xxx.198.169, xxx.xxx.12.146, xxx.xxx.98.168,
xxx.xxx.12.145, xxx.xxx.98.169, xxx.xxx.12.144, xxx.xxx.12.143,
xxx.xxx.12.140, xxx.xxx.12.139, xxx.xxx.198.126, xxx.xxx.12.136,
xxx.xxx.12.135, xxx.xxx.198.191, xxx.xxx.12.133, xxx.xxx.12.131]
*nodetool status output:*
./nodetool status
Note: Ownership information does not include topology; for complete
information, specify a keyspace
Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN xxx.xxx.198.75 639.41 GB 256 3.2%
fdd32c67-3cea-4174-b59b-c1ea14e1a334 GRP1
UN xxx.xxx.198.132 581.94 GB 256 3.4%
6a465101-29e7-4792-8269-851200a70023 GRP2
UN xxx.xxx.198.133 618.22 GB 256 3.6%
751ce15a-10f1-44cf-9357-04da7e21b511 GRP2
UN xxx.xxx.198.167 566.83 GB 256 3.5%
45f6684f-d6a0-4cba-875c-9db459646545 GRP2
UN xxx.xxx.198.78 661.98 GB 256 3.3%
a8332f22-a75f-4d7c-8b71-7284f6fe208f GRP3
UN xxx.xxx.198.126 603.37 GB 256 3.6%
71be90d8-97db-4155-b4fc-da59d78331ef GRP1
UN xxx.xxx.198.191 571.8 GB 256 3.2%
a9023df8-a8b3-484b-a03d-0fdea35007bd GRP3
UN xxx.xxx.198.168 631.98 GB 256 3.5%
2302491a-a8b5-4aa6-bda7-f1544064c4e3 GRP3
UN xxx.xxx.198.169 631.12 GB 256 3.4%
d5e5bc3d-38de-4043-abca-08ac09f29a46 GRP1
Datacenter: DC2
==============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN xxx.xxx.98.172 630.36 GB 256 3.3%
4709fe54-2793-481a-95f0-a780ea55cdf1 GRP1
UN xxx.xxx.98.219 606.85 GB 256 3.2%
792f3185-d5f2-4f4f-a60e-e0c888871cbb GRP3
UN xxx.xxx.98.173 654.75 GB 256 3.5%
345cdedc-d6d4-4b46-9cf3-5dc726af9b99 GRP3
UN xxx.xxx.98.170 577.26 GB 256 3.6%
cfaf5519-4849-49f9-a1c2-6ca13eb19d3c GRP3
UN xxx.xxx.98.205 603.29 GB 256 2.9%
efafe9fd-65b1-41ec-b285-fa00279410ce GRP2
UN xxx.xxx.98.171 659.41 GB 256 3.4%
9585c52a-94ba-4b94-af22-97f80ab8c102 GRP2
UN xxx.xxx.98.220 634.14 GB 256 3.1%
20a5f0ec-12b3-4db0-9483-f5ce984b362a GRP1
UN xxx.xxx.98.168 617.16 GB 256 3.3%
3ee473fc-6415-45c5-9466-4aaad9181e5a GRP1
UN xxx.xxx.98.169 614.25 GB 256 3.4%
dc1f04d7-0c84-4f8c-95bb-97005baaa328 GRP2
Datacenter: DC3
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN xxx.xxx.12.143 4.76 GB 256 3.5%
626facfb-8346-4f17-aa06-38424d9575ce GRP2
UN xxx.xxx.12.140 4.88 GB 256 3.9%
88517bf9-efa6-4214-b503-943c4d5d91ba GRP1
UN xxx.xxx.12.115 4.48 GB 256 3.3%
3878532d-3980-40a8-9276-0983eebdc9c6 GRP1
UN xxx.xxx.12.139 86.72 GB 256 3.2%
2ad980fc-ffec-489e-850d-3b11824bb7c6 GRP3
UN xxx.xxx.12.136 4.78 GB 256 3.0%
0b25753f-e0c4-448d-8459-6248b3072b79 GRP3
UN xxx.xxx.12.135 4.92 GB 256 3.1%
29ac1861-da54-421f-b0b7-0ad1230460ac GRP2
UN xxx.xxx.12.133 4.15 GB 256 3.2%
fd048371-4994-486d-8de5-365e56ec912f GRP1
UN xxx.xxx.12.123 4.78 GB 256 3.3%
cbbb8973-4a33-4ce9-9610-8a7a1ddf975d GRP3
UN xxx.xxx.12.131 4.44 GB 256 3.1%
163961b1-96aa-4ef1-8517-ec0f24f0be39 GRP2
UN xxx.xxx.12.146 4.46 GB 256 3.6%
23cc92fc-2bd8-453c-ae3c-341cfba172a9 GRP2
UN xxx.xxx.12.145 5.18 GB 256 3.2%
7a929498-29ab-4921-9676-cce41714d8c3 GRP1
UN xxx.xxx.12.144 62.52 GB 256 3.3%
1631630d-a6ed-437e-9368-30fccce4f12c GRP3
We using GossipingPropertyFileSnitch , so we modified only
cassandra-rackdc.properties and made NO changes to
cassandra-topology.properties.
If anybody has faced this problem or know how to fix , please let us know.
Thanks in advance...!!!
