Re: streaming stuck on joining a node with TBs of data

2020-07-31 Thread onmstester onmstester
No Secondary index, No SASI, No materialized view Sent using https://www.zoho.com/mail/ On Sat, 01 Aug 2020 11:02:54 +0430 Jeff Jirsa wrote Are there secondary indices involved?  On Jul 31, 2020, at 10:51 PM, onmstester onmstester wrot

Re: streaming stuck on joining a node with TBs of data

2020-07-31 Thread Jeff Jirsa
Are there secondary indices involved? > On Jul 31, 2020, at 10:51 PM, onmstester onmstester > wrote: > >  > Hi, > > I'm going to join multiple new nodes to already existed and running cluster. > Each node should stream in >2TB of data, and it took a few days (with 500Mb > streaming) to alm

streaming stuck on joining a node with TBs of data

2020-07-31 Thread onmstester onmstester
Hi, I'm going to join multiple new nodes to already existed and running cluster. Each node should stream in >2TB of data, and it took a few days (with 500Mb streaming) to almost get finished. But it stuck on streaming-in from one final node, but i can not see any bottleneck on any side (sourc

Re: many instances of org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier$1 on the heap

2020-07-31 Thread Erick Ramirez
Oh, I just saw on ASF Slack that you were already discussing it earlier today with driftx in the #cassandra channel. Cheers! >

Re: many instances of org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier$1 on the heap

2020-07-31 Thread Erick Ramirez
I don't have specific experience relating to InstanceTidier but when I saw this, I immediately thought of repairs blowing up the heap. 40K instances indicates to me that you have thousands of SSTables -- are they tiny (like 1MB or less)? Otherwise, are they dense nodes (~1TB or more)? How do you r

many instances of org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier$1 on the heap

2020-07-31 Thread jelmer
Hi, I am looking into some issues we are having with a cassandra cluster. What happens is that a node will suddenly start spending all its time in garbage collection becoming non responsive and dropping in and out of the cluster. A restart will solve the problem. We created a heap dump this time a

Open Request for Reference Architectures / Case Studies of Apache Cassandra

2020-07-31 Thread Rahul Singh
Folks, I'm looking for articles, blogs, diagrams, or someone to answer some questions so that I can help repopulate a *case study database of uses of Cassandra*. I'll be publishing these on the *https://cassandra.link * site. I'm also collecting reference architectures so