Were the columns sensor.lastEvaluation and sensordb.lastCheckTime dropped
by any chance? If so, you might be hitting
https://issues.apache.org/jira/browse/CASSANDRA-11050, fixed in upcoming
3.4.
If that's the case, you may want to check if nodes other than 192.168.10.10
have the dropped columns in the system_schema.dropped_columns table, and if
so, reset the local schema (nodetool resetlocalschema) of 192.168.10.10 to
force a schema synchronization with other nodes. Another possible
workaround is to manually include the dropped columns in the
system_schema.dropped_columns table of 192.168.10.10.
2016-02-27 22:56 GMT-03:00 Jason Kania <jason.ka...@ymail.com>:
> Hi,
>
> I just reran the command and collected following. Any suggestions would be
> appreciated.
>
> Thanks,
>
> Jason
>
> *from 192.168.10.8*
>
> ERROR [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857
> StreamSession.java:635 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
> Remote peer 192.168.10.10 failed stream session.
> INFO [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857
> StreamResultFuture.java:182 - [Stream
> #c9868f90-ddbb-11e5-80c0-89f591237aca] Session with /192.168.10.10 is
> complete
> WARN [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,858
> StreamResultFuture.java:209 - [Stream
> #c9868f90-ddbb-11e5-80c0-89f591237aca] Stream failed
>
> *from 192.168.10.8 debug*
>
> DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,414
> ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
> Received Received (79256340-bbbb-11e5-9f70-7d76a8de8480, #0)
> DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,854
> ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
> Received Retry (f3a137e0-024b-11e5-bb31-0d2316086bf7, #0)
> DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,854
> ConnectionHandler.java:334 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
> Sending File (Header (cfId: f3a137e0-024b-11e5-bb31-0d2316086bf7, #0,
> version: ma, format: BIG, estimated keys: 128, transfer size: 4653,
> compressed?: true, repairedAt: 0, level: 0), file:
> /home/cassandra/data/sensordb/sensor/ma-76-big-Data.db)
> DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,854
> CompressedStreamWriter.java:63 - [Stream
> #c9868f90-ddbb-11e5-80c0-89f591237aca] Start streaming file
> /home/cassandra/data/sensordb/sensor/ma-76-big-Data.db to /192.168.10.10,
> repairedAt = 0, totalSize = 4653
> DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,854
> CompressedStreamWriter.java:94 - [Stream
> #c9868f90-ddbb-11e5-80c0-89f591237aca] Finished streaming file
> /home/cassandra/data/sensordb/sensor/ma-76-big-Data.db to /192.168.10.10,
> bytesTransferred = 4653, totalSize = 4653
> DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,855
> ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
> Received Retry (faa55490-024b-11e5-bb31-0d2316086bf7, #0)
> DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,855
> ConnectionHandler.java:334 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
> Sending File (Header (cfId: faa55490-024b-11e5-bb31-0d2316086bf7, #0,
> version: ma, format: BIG, estimated keys: 128, transfer size: 705,
> compressed?: true, repairedAt: 0, level: 0), file:
> /home/cassandra/data/sensordb/sensorUnit/ma-79-big-Data.db)
> DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,856
> CompressedStreamWriter.java:63 - [Stream
> #c9868f90-ddbb-11e5-80c0-89f591237aca] Start streaming file
> /home/cassandra/data/sensordb/sensorUnit/ma-79-big-Data.db to /
> 192.168.10.10, repairedAt = 0, totalSize = 705
> DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,856
> CompressedStreamWriter.java:94 - [Stream
> #c9868f90-ddbb-11e5-80c0-89f591237aca] Finished streaming file
> /home/cassandra/data/sensordb/sensorUnit/ma-79-big-Data.db to /
> 192.168.10.10, bytesTransferred = 705, totalSize = 705
> DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857
> ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
> Received Session Failed
> ERROR [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857
> StreamSession.java:635 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
> Remote peer 192.168.10.10 failed stream session.
> DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857
> ConnectionHandler.java:110 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
> Closing stream connection handler on /192.168.10.10
> INFO [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857
> StreamResultFuture.java:182 - [Stream
> #c9868f90-ddbb-11e5-80c0-89f591237aca] Session with /192.168.10.10 is
> complete
> WARN [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,858
> StreamResultFuture.java:209 - [Stream
> #c9868f90-ddbb-11e5-80c0-89f591237aca] Stream failed
>
>
> *from 192.168.10.10*
>
> [2016-02-27 20:37:53,413] received file
> /home/cassandra/data/sensordb/listedAttributes-79256340bbbb11e59f707d76a8de8480/ma-32-big-Data.db
> (progress: 365%)
> [2016-02-27 20:37:53,414] received file
> /home/cassandra/data/sensordb/liestedAttributes-79256340bbbb11e59f707d76a8de8480/ma-32-big-Data.db
> (progress: 369%)
> [2016-02-27 20:37:53,865] session with /192.168.10.8 complete (progress:
> 369%)
> [2016-02-27 20:37:53,866] Stream failed
>
> *from 192.168.10.10 debug*
>
> DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,201
> CompressedStreamReader.java:80 - [Stream
> #c9868f90-ddbb-11e5-80c0-89f591237aca] Start receiving file #0 from /
> 192.168.10.8, repairedAt = 0, size = 166627, ks = 'sensordb', table =
> 'listAttributes'.
> DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,412
> CompressedStreamReader.java:110 - [Stream
> #c9868f90-ddbb-11e5-80c0-89f591237aca] Finished receiving file #0 from /
> 192.168.10.8 readBytes = 166627, totalSize = 166627
> DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,412
> ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
> Received File (Header (cfId: 79256340-bbbb-11e5-9f70-7d76a8de8480, #0,
> version: ma, format: BIG, estimated keys: 128, transfer size: 166627,
> compressed?: true, repairedAt: 0, level: 0), file:
> /home/cassandra/data/sensordb/listAttributes-79256340bbbb11e59f707d76a8de8480/ma-32-big-Data.db)
> DEBUG [STREAM-OUT-/192.168.10.8] 2016-02-27 20:37:53,412
> ConnectionHandler.java:334 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
> Sending Received (79256340-bbbb-11e5-9f70-7d76a8de8480, #0)
> DEBUG [CompactionExecutor:3] 2016-02-27 20:37:53,833
> CompactionTask.java:217 - Compacted (e224bef0-ddbb-11e5-80c0-89f591237aca)
> 4 sstables to
> [/home/cassandra/data/system_distributed/parent_repair_history-deabd734b99d3b9c92e5fd92eb5abf14/ma-5-big,]
> to level=0. 2,743,164 bytes to 685,791 (~25% of original) in 1,096ms =
> 0.596735MB/s. 0 total partitions merged to 57. Partition merge counts
> were {4:57, }
> DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,850
> CompressedStreamReader.java:80 - [Stream
> #c9868f90-ddbb-11e5-80c0-89f591237aca] Start receiving file #0 from /
> 192.168.10.8, repairedAt = 0, size = 4653, ks = 'sensordb', table =
> 'sensor'.
> WARN [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,851
> StreamSession.java:641 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
> Retrying for following error
> java.lang.RuntimeException: Unknown column lastEvaluation during
> deserialization
> at
> org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
> ~[apache-cassandra-3.0.3.jar:3.0.3]
> at
> org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:87)
> ~[apache-cassandra-3.0.3.jar:3.0.3]
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
> [apache-cassandra-3.0.3.jar:3.0.3]
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
> [apache-cassandra-3.0.3.jar:3.0.3]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
> [apache-cassandra-3.0.3.jar:3.0.3]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
> [apache-cassandra-3.0.3.jar:3.0.3]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
> DEBUG [STREAM-OUT-/192.168.10.8] 2016-02-27 20:37:53,852
> ConnectionHandler.java:334 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
> Sending Retry (f3a137e0-024b-11e5-bb31-0d2316086bf7, #0)
> DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,852
> ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
> Received null
> DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,853
> CompressedStreamReader.java:80 - [Stream
> #c9868f90-ddbb-11e5-80c0-89f591237aca] Start receiving file #0 from /
> 192.168.10.8, repairedAt = 0, size = 705, ks = 'sensordb', table =
> 'sensorUnit'.
> WARN [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,854
> StreamSession.java:641 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
> Retrying for following error
> java.lang.RuntimeException: Unknown column lastCheckTime during
> deserialization
> at
> org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
> ~[apache-cassandra-3.0.3.jar:3.0.3]
> at
> org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:87)
> ~[apache-cassandra-3.0.3.jar:3.0.3]
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
> [apache-cassandra-3.0.3.jar:3.0.3]
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
> [apache-cassandra-3.0.3.jar:3.0.3]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
> [apache-cassandra-3.0.3.jar:3.0.3]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
> [apache-cassandra-3.0.3.jar:3.0.3]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
> DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,854
> ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
> Received null
>
>
> ------------------------------
> *From:* Sebastian Estevez <sebastian.este...@datastax.com>
> *To:* user@cassandra.apache.org; Jason Kania <jason.ka...@ymail.com>
> *Sent:* Saturday, February 27, 2016 8:24 PM
> *Subject:* Re: How to complete bootstrap with exception due to stream
> failure?
>
> progress: 361% does not look right (probably a bug).
>
> Can you check the corresponding messages on the other side of the stream?
> I.E. the system log for 192.168.10.8 around 18:02:06?
>
> All the best,
>
> [image: datastax_logo.png] <http://www.datastax.com/>
> Sebastián Estévez
> Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com
> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
> <https://twitter.com/datastax> [image: g+.png]
> <https://plus.google.com/+Datastax/about>
> <http://feeds.feedburner.com/datastax>
> <http://goog_410786983/>
>
> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>
> DataStax is the fastest, most scalable distributed database technology,
> delivering Apache Cassandra to the world’s most innovative enterprises.
> Datastax is built to be agile, always-on, and predictably scalable to any
> size. With more than 500 customers in 45 countries, DataStax is the
> database technology and transactional backbone of choice for the worlds
> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>
> On Sat, Feb 27, 2016 at 6:12 PM, Jason Kania <jason.ka...@ymail.com>
> wrote:
>
> Hello,
>
> I am trying to get a node bootstrapped in 3.0.3, but just at the point
> where the bootstrap process is to complete, a broken pipe exception occurs
> so the bootstrap process hangs. Once I kill the bootstrap process, I can
> execute "nodetool bootstrap resume" again and the same problem will occur
> just at the end of the bootstrap exercise. Here is the tail of the log:
>
> [2016-02-27 18:02:05,898] received file
> /home/cassandra/data/sensordb/listedAttributes-79256340bbbb11e59f707d76a8de8480/ma-30-big-Data.db
> (progress: 357%)
> [2016-02-27 18:02:06,479] received file
> /home/cassandra/data/sensordb/notification-f7e3eaa0024b11e5bb310d2316086bf7/ma-38-big-Data.db
> (progress: 361%)
> [2016-02-27 18:02:06,884] session with /192.168.10.8 complete (progress:
> 361%)
> [2016-02-27 18:02:06,886] Stream failed
>
> I attempted to run nodetool repair, but get the following which I have
> been told indicates that the replication factor is 1:
>
> root@bull:~# nodetool repair
> [2016-02-27 18:04:55,083] Nothing to repair for keyspace 'sensordb'
>
> Thanks,
>
> Jason
>
>
>
>
>
