I set the stream timeout to 1 hour this morning and started fresh trying to join this node. It took about an hour to stream over 230gb of data, and then into hour 2 I wound up back where I was yesterday, the node's load is slowly reducing and the netstats does not show sending or receiving anything. I'm not sure how long I should wait before I throw the towel in on this attempt. I'm also not really sure what to try next...
The only thing in the logs currently are three entries like this: ERROR 07:39:44,447 Exception in thread Thread[CompactionExecutor:31,1,main] java.lang.RuntimeException: Last written key DecoratedKey(8633837336094175369, 003076697369746f725f706167655f766965623936636232346331623661313935313634346638303838393465313132373700004930303030663264632d303030302d303033302d343030302d3030303030303030663264633a66376436366166382d383564352d313165342d383030302d30303030303035343764623600) >= current key DecoratedKey(-6568345298384940765, 003076697369746f725f706167655f766965623936636232346331623661313935313634346638303838393465313132373700004930303030376464652d303030302d303033302d343030302d3030303030303030376464653a64633930336533382d643766342d313165342d383030302d30303030303730626338386300) writing into /var/lib/cassandra/data/pi/__shardindex/pi-__shardindex-tmp-jb-644-Data.db at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166) at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:170) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ANY help is greatly appreciated. Thanks, Stan On Tue, Aug 4, 2015 at 2:23 PM, Sebastian Estevez < sebastian.este...@datastax.com> wrote: > That's the one. I set it to an hour to be safe (if a stream goes above the > timeout it will get restarted) but it can probably be lower. > > All the best, > > > [image: datastax_logo.png] <http://www.datastax.com/> > > Sebastián Estévez > > Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com > > [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image: > facebook.png] <https://www.facebook.com/datastax> [image: twitter.png] > <https://twitter.com/datastax> [image: g+.png] > <https://plus.google.com/+Datastax/about> > <http://feeds.feedburner.com/datastax> > > > <http://cassandrasummit-datastax.com/?utm_campaign=summit15&utm_medium=summiticon&utm_source=emailsignature> > > DataStax is the fastest, most scalable distributed database technology, > delivering Apache Cassandra to the world’s most innovative enterprises. > Datastax is built to be agile, always-on, and predictably scalable to any > size. With more than 500 customers in 45 countries, DataStax is the > database technology and transactional backbone of choice for the worlds > most innovative companies such as Netflix, Adobe, Intuit, and eBay. > > On Tue, Aug 4, 2015 at 2:21 PM, Stan Lemon <sle...@salesforce.com> wrote: > >> Sebastian, >> You're referring to streaming_socket_timeout_in_ms correct? What value >> do you recommend? All of my nodes are currently at the default 0. >> >> Thanks, >> Stan >> >> >> On Tue, Aug 4, 2015 at 2:16 PM, Sebastian Estevez < >> sebastian.este...@datastax.com> wrote: >> >>> It helps to set stream socket timeout in the yaml so that you don't hang >>> forever on a lost / broken stream. >>> >>> All the best, >>> >>> >>> [image: datastax_logo.png] <http://www.datastax.com/> >>> >>> Sebastián Estévez >>> >>> Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com >>> >>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image: >>> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png] >>> <https://twitter.com/datastax> [image: g+.png] >>> <https://plus.google.com/+Datastax/about> >>> <http://feeds.feedburner.com/datastax> >>> >>> >>> <http://cassandrasummit-datastax.com/?utm_campaign=summit15&utm_medium=summiticon&utm_source=emailsignature> >>> >>> DataStax is the fastest, most scalable distributed database technology, >>> delivering Apache Cassandra to the world’s most innovative enterprises. >>> Datastax is built to be agile, always-on, and predictably scalable to any >>> size. With more than 500 customers in 45 countries, DataStax is the >>> database technology and transactional backbone of choice for the worlds >>> most innovative companies such as Netflix, Adobe, Intuit, and eBay. >>> >>> On Tue, Aug 4, 2015 at 2:14 PM, Robert Coli <rc...@eventbrite.com> >>> wrote: >>> >>>> On Tue, Aug 4, 2015 at 11:02 AM, Stan Lemon <sle...@salesforce.com> >>>> wrote: >>>> >>>>> I am attempting to add a 13th node in one of the datacenters. I have >>>>> been monitoring this process from the node itself with nodetool netstats >>>>> and from one of the existing nodes using nodetool status. >>>>> >>>>> On the existing node I see the new node as UJ. I have watched the load >>>>> steadily climb up to about 203.4gb, and then over the last two hours it >>>>> has >>>>> fluctuated a bit and has been steadily dropping to about 203.1gb >>>>> >>>> >>>> It's probably hung. If I were you I'd probably wipe the node and >>>> re-bootstrap. >>>> >>>> (what version of cassandra/what network are you on (AWS?)/etc.) >>>> >>>> =Rob >>>> >>>> >>> >>> >> >
