What's your average data per node? Is 230gb close? All the best,
[image: datastax_logo.png] <http://www.datastax.com/> Sebastián Estévez Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image: facebook.png] <https://www.facebook.com/datastax> [image: twitter.png] <https://twitter.com/datastax> [image: g+.png] <https://plus.google.com/+Datastax/about> <http://feeds.feedburner.com/datastax> <http://cassandrasummit-datastax.com/?utm_campaign=summit15&utm_medium=summiticon&utm_source=emailsignature> DataStax is the fastest, most scalable distributed database technology, delivering Apache Cassandra to the world’s most innovative enterprises. Datastax is built to be agile, always-on, and predictably scalable to any size. With more than 500 customers in 45 countries, DataStax is the database technology and transactional backbone of choice for the worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay. On Wed, Aug 5, 2015 at 8:33 AM, Stan Lemon <sle...@salesforce.com> wrote: > I set the stream timeout to 1 hour this morning and started fresh trying > to join this node. It took about an hour to stream over 230gb of data, and > then into hour 2 I wound up back where I was yesterday, the node's load is > slowly reducing and the netstats does not show sending or receiving > anything. I'm not sure how long I should wait before I throw the towel in > on this attempt. I'm also not really sure what to try next... > > The only thing in the logs currently are three entries like this: > > ERROR 07:39:44,447 Exception in thread Thread[CompactionExecutor:31,1,main] > java.lang.RuntimeException: Last written key > DecoratedKey(8633837336094175369, > 003076697369746f725f706167655f766965623936636232346331623661313935313634346638303838393465313132373700004930303030663264632d303030302d303033302d343030302d3030303030303030663264633a66376436366166382d383564352d313165342d383030302d30303030303035343764623600) > >= current key DecoratedKey(-6568345298384940765, > 003076697369746f725f706167655f766965623936636232346331623661313935313634346638303838393465313132373700004930303030376464652d303030302d303033302d343030302d3030303030303030376464653a64633930336533382d643766342d313165342d383030302d30303030303730626338386300) > writing into > /var/lib/cassandra/data/pi/__shardindex/pi-__shardindex-tmp-jb-644-Data.db > at > org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143) > at > org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:170) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) > at > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > > > ANY help is greatly appreciated. > > Thanks, > Stan > > > > > > On Tue, Aug 4, 2015 at 2:23 PM, Sebastian Estevez < > sebastian.este...@datastax.com> wrote: > >> That's the one. I set it to an hour to be safe (if a stream goes above >> the timeout it will get restarted) but it can probably be lower. >> >> All the best, >> >> >> [image: datastax_logo.png] <http://www.datastax.com/> >> >> Sebastián Estévez >> >> Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com >> >> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image: >> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png] >> <https://twitter.com/datastax> [image: g+.png] >> <https://plus.google.com/+Datastax/about> >> <http://feeds.feedburner.com/datastax> >> >> >> <http://cassandrasummit-datastax.com/?utm_campaign=summit15&utm_medium=summiticon&utm_source=emailsignature> >> >> DataStax is the fastest, most scalable distributed database technology, >> delivering Apache Cassandra to the world’s most innovative enterprises. >> Datastax is built to be agile, always-on, and predictably scalable to any >> size. With more than 500 customers in 45 countries, DataStax is the >> database technology and transactional backbone of choice for the worlds >> most innovative companies such as Netflix, Adobe, Intuit, and eBay. >> >> On Tue, Aug 4, 2015 at 2:21 PM, Stan Lemon <sle...@salesforce.com> wrote: >> >>> Sebastian, >>> You're referring to streaming_socket_timeout_in_ms correct? What value >>> do you recommend? All of my nodes are currently at the default 0. >>> >>> Thanks, >>> Stan >>> >>> >>> On Tue, Aug 4, 2015 at 2:16 PM, Sebastian Estevez < >>> sebastian.este...@datastax.com> wrote: >>> >>>> It helps to set stream socket timeout in the yaml so that you don't >>>> hang forever on a lost / broken stream. >>>> >>>> All the best, >>>> >>>> >>>> [image: datastax_logo.png] <http://www.datastax.com/> >>>> >>>> Sebastián Estévez >>>> >>>> Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com >>>> >>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image: >>>> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png] >>>> <https://twitter.com/datastax> [image: g+.png] >>>> <https://plus.google.com/+Datastax/about> >>>> <http://feeds.feedburner.com/datastax> >>>> >>>> >>>> <http://cassandrasummit-datastax.com/?utm_campaign=summit15&utm_medium=summiticon&utm_source=emailsignature> >>>> >>>> DataStax is the fastest, most scalable distributed database >>>> technology, delivering Apache Cassandra to the world’s most innovative >>>> enterprises. Datastax is built to be agile, always-on, and predictably >>>> scalable to any size. With more than 500 customers in 45 countries, >>>> DataStax >>>> is the database technology and transactional backbone of choice for the >>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay. >>>> >>>> On Tue, Aug 4, 2015 at 2:14 PM, Robert Coli <rc...@eventbrite.com> >>>> wrote: >>>> >>>>> On Tue, Aug 4, 2015 at 11:02 AM, Stan Lemon <sle...@salesforce.com> >>>>> wrote: >>>>> >>>>>> I am attempting to add a 13th node in one of the datacenters. I have >>>>>> been monitoring this process from the node itself with nodetool netstats >>>>>> and from one of the existing nodes using nodetool status. >>>>>> >>>>>> On the existing node I see the new node as UJ. I have watched the >>>>>> load steadily climb up to about 203.4gb, and then over the last two hours >>>>>> it has fluctuated a bit and has been steadily dropping to about 203.1gb >>>>>> >>>>> >>>>> It's probably hung. If I were you I'd probably wipe the node and >>>>> re-bootstrap. >>>>> >>>>> (what version of cassandra/what network are you on (AWS?)/etc.) >>>>> >>>>> =Rob >>>>> >>>>> >>>> >>>> >>> >> >
