Large size KS management

Hi all I have been given a 15 nodes C* 2.2.8 cluster to manage which has a large size KS (~800GB). Given the size of the KS most of the management tasks like repair take a long time to complete and disk space management is becoming tricky from the systems perspective. This KS size is going to

Re: Cassandra and Kubernetes and scaling

Looking forward to hearing from the community about this. Sent from my iPhone > On May 24, 2016, at 10:19 AM, Mike Wojcikiewicz wrote: > > I saw a thread from April 2016 talking about Cassandra and Kubernetes, and > have a few follow up questions. It seems that especially after v1.2 of > Kub

Bootstrapping multiple C* nodes in AWS

Hi all I am running C* 2.1.12 in AWS EC2 Classic with RF=3 and vnodes(256 tokens/node). My nodes are distributed in three different availability zones. I want to scale up the cluster size, given the data size per node it takes around 24 hours to add one node. I wanted to know if its safe to add mu

Advice in upgrade plan from 1.2.18 to 2.2.8

Hi everyone, I have 2 C* DCs with 12 nodes in each running 1.2.18. I plan to upgrade them to 2.2.latest and wanted to run by you experts my plan. 1. Install 2.0.latest on one node at a time, start and wait for it to join the ring. 2. Run upgradesstables on this node. 3. Repeat Step 1,2

Re: Advice in upgrade plan from 1.2.18 to 2.2.8

me window as short as possible. C*heers, --- Alain Rodriguez - @arodream - al...@thelastpickle.com<mailto:al...@thelastpickle.com> France The Last Pickle - Apache Cassandra Consulting http://www.thelastpickle.com<http://www.thelastpickle.com/> 2016-12-21 20:36 GMT+01:00 Aiman Parvaiz

Re: Decommissioned node still in Gossip

I was having exactly the same issue with the same version, check your seed list and make sure it contains only the live nodes, I know that seeds are only read when cassandra starts but updating the seed list to live nodes and then doing a roiling restart fixed this issue for me. I hope this hel

Re: Cassandra compaction appears to stall, node becomes partially unresponsive

ifferent racks. > > Has anyone seen this before? Alternatively, when this happens again, what > data can we collect that would help with the debugging process (in addition > to tpstats)? > > Thanks in advance, > > Bryan > -- *Aiman Parvaiz* Lead Systems Architect ai...@flipagram.com cell: 213-300-6377 http://flipagram.com/apz

Re: Cassandra compaction appears to stall, node becomes partially unresponsive

; We collect GC statistics through collectd via the garbage collector mbean, >> ParNew GC's report sub 500ms collection time on average (I believe >> accumulated per minute?) and CMS peaks at about 300ms collection time when >> it runs. >> >>> On Wed, Jul 22, 2

Need advice for multi DC C* setup

Hi all We are planning to move C* from EC2 (region A) to VPC in region B. I will enumerate our goals so that you guys can advice me keeping in mind the bigger picture. Goals: - Move to VPC is another region. - Enable Vnodes. - Bump up RF to 3. - Ability to have a spark cluster. I know this is a L

Re: Need advice for multi DC C* setup

using datastax version, it comes with spark. You >>> just need to change a config and spark starts along with cassandra. A >>> separate ring is advised for analytics stuff. >>> >>> >>> On Sat, Aug 15, 2015 at 1:10 AM, Aiman Parvaiz >>> wrot

Cassandra 2.1.12 Node size

Hi all, I am running a 9 node C* 2.1.12 cluster. I seek advice in data size per node. Each of my node has close to 1 TB of data. I am not seeing any issues as of now but wanted to run it by you guys if this data size is pushing the limits in any manner and if I should be working on reducing data si

Re: Cassandra 2.1.12 Node size

ons are absolutely necessary, keep it small. If you want to use the > entire disk space (50/80% of total disk space max), go ahead as long as other > resources are fine (CPU, memory, disk throughput, ...). > > C*heers, > > ------- > Alain Rodriguez - al...@t

Re: Cassandra 2.1.12 Node size

; - You are using compression (if you want too) >> >> Consider: >> >> - Adding TTLs to data you don't want to keep forever, shorten TTLs as >> much as allowed. >> - Migrating to C*3.0+ and take advantage of the new engine storage >> >> C*heers,

Re: Reaper v0.6.1 released

Great work!! Thanks Sent from my iPhone On Jun 14, 2017, at 11:30 PM, Shalom Sagges mailto:shal...@liveperson.com>> wrote: That's awesome!! Thanks for contributing! 👏👏👏 [https://signature.s3.amazonaws.com/2015/lp_logo.png] Shalom Sagges DBA T: +972-74-700-4035 [https://signature.s3.amazon

Re: Reaper 0.7 is released!

Thanks!! Love Reaper :) Sent from my iPhone On Sep 27, 2017, at 10:01 AM, Jon Haddad mailto:j...@jonhaddad.com>> wrote: Hey folks, We (The Last Pickle) are proud to announce the release of Reaper 0.7! In this release we've added support to run Reaper across multiple data centers as well as

Need help with incremental repair

Hi everyone, We seek your help in a issue we are facing in our 2.2.8 version. We have 24 nodes cluster spread over 3 DCs. Initially, when the cluster was in a single DC we were using The Last Pickle reaper 0.5 to repair it with incremental repair set to false. We added 2 more DCs. Now the prob

Re: Need help with incremental repair

shouldn't have to do anything. It's not going to hurt anything. Pre-4.0 > incremental repair has some issues that can cause a lot of extra streaming, > and inconsistencies in some edge cases, but as long as you're running full > repairs before gc grace expires, everything sho

Re: EC2 SSD cluster costs

I completely agree with others here. It depends on your use case. We were using Hi1.4xlarge boxes and paying huge amount to Amazon, lately our requirements changed and we are not hammering C* as much and our data size has gone down too, so given the new conditions we reserved and migrated to c3.4xl

Re: stalled nodetool repair?

If nodetool compactionstats says there are no Validation compactions running (and the compaction queue is empty) and netstats says there is nothing streaming there is a a good chance the repair is finished or dead. Source: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Is-it-saf

ERROR Compaction Interrupted

Hi everyone, I am running C* 2.0.9 without vnodes and RF=2. Recently while repairing, rebalancing the cluster I encountered one instance of this(just one on one node): ERROR CompactionExecutor: 55472 CassandraDaemon.uncaughtException - Exception in thread

Reading too many tombstones

Hi everyone, We are running a 10 node Cassandra 2.0.9 without vnode cluster. We are running in to a issue where we are reading too many tombstones and hence getting tons of WARN messages and some ERROR query aborted. cass-prod4 2015-06-04 14:38:34,307 WARN ReadStage:

Re: Reading too many tombstones

t time series, TTLed data. That means > no updates to old data. > > On Thu, Jun 4, 2015 at 10:58 AM Aiman Parvaiz wrote: > >> Hi everyone, >> We are running a 10 node Cassandra 2.0.9 without vnode cluster. We are >> running in to a issue where we are reading too many t

Re: Reading too many tombstones

kedin.com/in/carlosjuzarterolo>* > Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649 > www.pythian.com > > On Thu, Jun 4, 2015 at 8:31 PM, Aiman Parvaiz wrote: > >> yeah we don't update old data. One thing I am curious about is why are we >> running in to s

Re: auto clear data with ttl

So gc_grace zero will remove tombstones without any delay after compaction. So it's possible that tombstones containing SSTs still need to be compacted. So either you can wait for compaction to happen or do a manual compaction depending on your compaction strategy. Manual compaction does have so

C* 2.0.15 - java.lang.NegativeArraySizeException

Hi everyone I am running C* 2.0.9 and decided to do a rolling upgrade. Added a node of C* 2.0.15 in the existing cluster and saw this twice: Jun 9 02:27:20 prod-cass23.localdomain cassandra: 2015-06-09 02:27:20,658 INFO CompactionExecutor:4 CompactionTask.runMayThrow - Compacting [SSTableReader(p

Re: C* 2.0.15 - java.lang.NegativeArraySizeException

Quick update, saw the same error on another new node, again the node isn't really misbehaving uptill now. Thanks On Mon, Jun 8, 2015 at 9:48 PM, Aiman Parvaiz wrote: > Hi everyone > I am running C* 2.0.9 and decided to do a rolling upgrade. Added a node of > C* 2.0.15 in the ex

Re: C* 2.0.15 - java.lang.NegativeArraySizeException

> Sean Durity > > > > *From:* Aiman Parvaiz [mailto:ai...@flipagram.com] > *Sent:* Tuesday, June 09, 2015 1:29 PM > *To:* user@cassandra.apache.org > *Subject:* Re: C* 2.0.15 - java.lang.NegativeArraySizeException > > > > Quick update, saw the same error on another

Gossip Stage ERROR

We are running C* 2.0.15, recently 2 of our 10 nodes had to be forcefully removed. Cluster is behaving fine since then as we are not seeing any issues with production except that nodes every now and then throw out the following error: Jun 19 17:18:35 cass-prod5.localdomain cassandra: 2015-06-19 1

Re: VPC AWS

se, many > of the moved nodes required slightly different configurations for items > like the seeds. > > Its been a couple of years, so my memory on this maybe a little fuzzy :) > > -Mike > > -- > *From:* Aiman Parvaiz > *To:* user@cassan

Re: VPC AWS

ublic VPC, restored from a > backup taken right before the switch. > > -Mike > > ------ > *From:* Aiman Parvaiz > *To:* Michael Theroux > *Cc:* "user@cassandra.apache.org" > *Sent:* Thursday, June 5, 2014 2:39 PM > *Subject:* Re: VP

Re: how do i know if nodetool repair is finished

This is a old post, am not sure if something changed for new C* versions. If nodetool compactionstats says there are no Validation compactions running (and the compaction queue is empty) and netstats says there is nothing streaming there is a a good chance the repair is finished or dead. If a nei

Cassandra running High Load with no one using the cluster

Since last night I am seeing CPU load spikes on our cassandra boxes(Occasionally load goes up to 20, its a Amazon EC2 c1.xlarge with 300 iops EBS). After digging around a little I believe its related to heap memory and flushing memtables. >From logs: WARN 03:22:03,414 Heap is 0.7786981388910019 fu

Re: Cassandra running High Load with no one using the cluster

Correction, there was a typo in my original question, we are running cassandra 1.1.10 Thanks and sorry for the inconvenience. On May 6, 2013, at 10:23 AM, Robert Coli wrote: > including non-working Hinted Handoff

Re: Cassandra performance decreases drastically with increase in data size.

I believe you should roll out more nodes as a temporary fix to your problem, 400GB on all nodes means (as correctly mentioned in other mails of this thread) you are spending more time on GC. Check out the second comment in this link by Aaron Morton, he says the more than 300GB can be problematic

Populating seeds dynamically

Hi all I am using puppet to push cassandra.yaml file which has seeds node hardcoded, going forward I don't want to hard code the seed nodes and I plan to maintain a list of seed nodes. Since I have a cluster in place I would populate this list for now to start with and next time when I add a nod

Re: Populating seeds dynamically

ully that helps a bit :). > > Faraaz > > On Mon, Jun 03, 2013 at 04:59:23PM -0700, Aiman Parvaiz wrote: >> Hi all >> I am using puppet to push cassandra.yaml file which has seeds node >> hardcoded, going forward I don't want to hard code the seed nodes and I plan

Re: High performance hardware with lot of data per node - Global learning about configuration

Hi, We also recently migrated to 3 hi.4xlarge boxes(Raid0 SSD) and the disk IO performance is definitely better than the earlier non SSD servers, we are serving up to 14k reads/s with a latency of 3-3.5 ms/op. I wanted to share our config options and ask about the data back up strategy for Raid

Re: High performance hardware with lot of data per node - Global learning about configuration

re considering moving to a periodic snapshot > approach as the sst churn after going from 24 nodes -> 6 nodes is quite high. > > Mike > > > [1]: https://github.com/librato/tablesnap > > > On Thu, Jul 11, 2013 at 7:33 AM, Aiman Parvaiz wrote: > Hi, > We a