Thanks for the reference to nodetool resetlocalschema as that will come in handy in the future. Thanks also for the reference to https://issues.apache.org/jira/browse/CASSANDRA-11050 which seems related, but I am not sure.
I was doing a bootstrapping on 192.168.10.10 and it had nothing on it to start with it. It was in the process of transferring the schema definitions that the bootstrap was failing. In the process of trying to get something working, I tried adding the dropped columns on the existing node and the new node but had no luck with that either. I finally figured it out so I raised https://issues.apache.org/jira/browse/CASSANDRA-11273 with these details and the workaround that I found. From: Paulo Motta <pauloricard...@gmail.com> To: "user@cassandra.apache.org" <user@cassandra.apache.org>; Jason Kania <jason.ka...@ymail.com> Sent: Sunday, February 28, 2016 10:01 PM Subject: Re: How to complete bootstrap with exception due to stream failure? Were the columns sensor.lastEvaluation and sensordb.lastCheckTime dropped by any chance? If so, you might be hitting https://issues.apache.org/jira/browse/CASSANDRA-11050, fixed in upcoming 3.4. If that's the case, you may want to check if nodes other than 192.168.10.10 have the dropped columns in the system_schema.dropped_columns table, and if so, reset the local schema (nodetool resetlocalschema) of 192.168.10.10 to force a schema synchronization with other nodes. Another possible workaround is to manually include the dropped columns in the system_schema.dropped_columns table of 192.168.10.10. 2016-02-27 22:56 GMT-03:00 Jason Kania <jason.ka...@ymail.com>: Hi, I just reran the command and collected following. Any suggestions would be appreciated. Thanks, Jason from 192.168.10.8 ERROR [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857 StreamSession.java:635 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Remote peer 192.168.10.10 failed stream session. INFO [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857 StreamResultFuture.java:182 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Session with /192.168.10.10 is complete WARN [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,858 StreamResultFuture.java:209 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Stream failed from 192.168.10.8 debug DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,414 ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Received Received (79256340-bbbb-11e5-9f70-7d76a8de8480, #0) DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,854 ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Received Retry (f3a137e0-024b-11e5-bb31-0d2316086bf7, #0) DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,854 ConnectionHandler.java:334 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Sending File (Header (cfId: f3a137e0-024b-11e5-bb31-0d2316086bf7, #0, version: ma, format: BIG, estimated keys: 128, transfer size: 4653, compressed?: true, repairedAt: 0, level: 0), file: /home/cassandra/data/sensordb/sensor/ma-76-big-Data.db) DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,854 CompressedStreamWriter.java:63 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Start streaming file /home/cassandra/data/sensordb/sensor/ma-76-big-Data.db to /192.168.10.10, repairedAt = 0, totalSize = 4653 DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,854 CompressedStreamWriter.java:94 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Finished streaming file /home/cassandra/data/sensordb/sensor/ma-76-big-Data.db to /192.168.10.10, bytesTransferred = 4653, totalSize = 4653 DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,855 ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Received Retry (faa55490-024b-11e5-bb31-0d2316086bf7, #0) DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,855 ConnectionHandler.java:334 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Sending File (Header (cfId: faa55490-024b-11e5-bb31-0d2316086bf7, #0, version: ma, format: BIG, estimated keys: 128, transfer size: 705, compressed?: true, repairedAt: 0, level: 0), file: /home/cassandra/data/sensordb/sensorUnit/ma-79-big-Data.db) DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,856 CompressedStreamWriter.java:63 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Start streaming file /home/cassandra/data/sensordb/sensorUnit/ma-79-big-Data.db to /192.168.10.10, repairedAt = 0, totalSize = 705 DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,856 CompressedStreamWriter.java:94 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Finished streaming file /home/cassandra/data/sensordb/sensorUnit/ma-79-big-Data.db to /192.168.10.10, bytesTransferred = 705, totalSize = 705 DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857 ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Received Session Failed ERROR [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857 StreamSession.java:635 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Remote peer 192.168.10.10 failed stream session. DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857 ConnectionHandler.java:110 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Closing stream connection handler on /192.168.10.10 INFO [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857 StreamResultFuture.java:182 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Session with /192.168.10.10 is complete WARN [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,858 StreamResultFuture.java:209 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Stream failed from 192.168.10.10 [2016-02-27 20:37:53,413] received file /home/cassandra/data/sensordb/listedAttributes-79256340bbbb11e59f707d76a8de8480/ma-32-big-Data.db (progress: 365%) [2016-02-27 20:37:53,414] received file /home/cassandra/data/sensordb/liestedAttributes-79256340bbbb11e59f707d76a8de8480/ma-32-big-Data.db (progress: 369%) [2016-02-27 20:37:53,865] session with /192.168.10.8 complete (progress: 369%) [2016-02-27 20:37:53,866] Stream failed from 192.168.10.10 debug DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,201 CompressedStreamReader.java:80 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Start receiving file #0 from /192.168.10.8, repairedAt = 0, size = 166627, ks = 'sensordb', table = 'listAttributes'. DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,412 CompressedStreamReader.java:110 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Finished receiving file #0 from /192.168.10.8 readBytes = 166627, totalSize = 166627 DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,412 ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Received File (Header (cfId: 79256340-bbbb-11e5-9f70-7d76a8de8480, #0, version: ma, format: BIG, estimated keys: 128, transfer size: 166627, compressed?: true, repairedAt: 0, level: 0), file: /home/cassandra/data/sensordb/listAttributes-79256340bbbb11e59f707d76a8de8480/ma-32-big-Data.db) DEBUG [STREAM-OUT-/192.168.10.8] 2016-02-27 20:37:53,412 ConnectionHandler.java:334 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Sending Received (79256340-bbbb-11e5-9f70-7d76a8de8480, #0) DEBUG [CompactionExecutor:3] 2016-02-27 20:37:53,833 CompactionTask.java:217 - Compacted (e224bef0-ddbb-11e5-80c0-89f591237aca) 4 sstables to [/home/cassandra/data/system_distributed/parent_repair_history-deabd734b99d3b9c92e5fd92eb5abf14/ma-5-big,] to level=0. 2,743,164 bytes to 685,791 (~25% of original) in 1,096ms = 0.596735MB/s. 0 total partitions merged to 57. Partition merge counts were {4:57, } DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,850 CompressedStreamReader.java:80 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Start receiving file #0 from /192.168.10.8, repairedAt = 0, size = 4653, ks = 'sensordb', table = 'sensor'. WARN [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,851 StreamSession.java:641 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Retrying for following error java.lang.RuntimeException: Unknown column lastEvaluation during deserialization at org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331) ~[apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:87) ~[apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50) [apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39) [apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59) [apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) [apache-cassandra-3.0.3.jar:3.0.3] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74] DEBUG [STREAM-OUT-/192.168.10.8] 2016-02-27 20:37:53,852 ConnectionHandler.java:334 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Sending Retry (f3a137e0-024b-11e5-bb31-0d2316086bf7, #0) DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,852 ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Received null DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,853 CompressedStreamReader.java:80 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Start receiving file #0 from /192.168.10.8, repairedAt = 0, size = 705, ks = 'sensordb', table = 'sensorUnit'. WARN [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,854 StreamSession.java:641 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Retrying for following error java.lang.RuntimeException: Unknown column lastCheckTime during deserialization at org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331) ~[apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:87) ~[apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50) [apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39) [apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59) [apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) [apache-cassandra-3.0.3.jar:3.0.3] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74] DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,854 ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Received null From: Sebastian Estevez <sebastian.este...@datastax.com> To: user@cassandra.apache.org; Jason Kania <jason.ka...@ymail.com> Sent: Saturday, February 27, 2016 8:24 PM Subject: Re: How to complete bootstrap with exception due to stream failure? progress: 361% does not look right (probably a bug). Can you check the corresponding messages on the other side of the stream? I.E. the system log for 192.168.10.8 around 18:02:06? All the best, Sebastián EstévezSolutions Architect | 954 905 8615 | sebastian.este...@datastax.com DataStax is the fastest, most scalable distributed database technology, delivering Apache Cassandra to the world’s most innovative enterprises. Datastax is built to be agile, always-on, and predictably scalable to any size. With more than 500 customers in 45 countries, DataStax is the database technology and transactional backbone of choice for the worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay. On Sat, Feb 27, 2016 at 6:12 PM, Jason Kania <jason.ka...@ymail.com> wrote: Hello, I am trying to get a node bootstrapped in 3.0.3, but just at the point where the bootstrap process is to complete, a broken pipe exception occurs so the bootstrap process hangs. Once I kill the bootstrap process, I can execute "nodetool bootstrap resume" again and the same problem will occur just at the end of the bootstrap exercise. Here is the tail of the log: [2016-02-27 18:02:05,898] received file /home/cassandra/data/sensordb/listedAttributes-79256340bbbb11e59f707d76a8de8480/ma-30-big-Data.db (progress: 357%) [2016-02-27 18:02:06,479] received file /home/cassandra/data/sensordb/notification-f7e3eaa0024b11e5bb310d2316086bf7/ma-38-big-Data.db (progress: 361%) [2016-02-27 18:02:06,884] session with /192.168.10.8 complete (progress: 361%) [2016-02-27 18:02:06,886] Stream failed I attempted to run nodetool repair, but get the following which I have been told indicates that the replication factor is 1: root@bull:~# nodetool repair [2016-02-27 18:04:55,083] Nothing to repair for keyspace 'sensordb' Thanks, Jason
