Re: Cassandra scale-out with no traffic on newly joined nodes
Thanks Erick. We are using datastax java driver ( https://docs.datastax.com/en/developer/java-driver/). Driver version number: 3.3.0 C* version: 3.11.6 Regards, Sandeep On Sat, Sep 12, 2020 at 2:41 PM Erick Ramirez wrote: > That would be my last option to add a new host as contant point but as per >> my understanding cassandra should auto discover newly joined nodes and >> server or load balance connections automatically but it's not happening. >> > > Yes, that sounds right to me. Sorry, this fell off my radar due to > competing priorities. I'm going to reach out to the DataStax Drivers team. > In the meantime, can you tell me which driver + version and C* version? > Cheers! >
Increased read latency with Cassandra >= 3.11.7
Dear Cassandra community, after upgrading my Cassandra nodes from version 3.11.6 to either 3.11.7 or 3.11.8 I experience a significant increase in read latency. With 3.11.6 average read latency is ~0.35 ms. With 3.11.7 average read latency increases to ~2.9 ms, almost 10 times worse! This behavior applies to all my environments, testing and productive. I do also observe that the "bloom-filter false positive rate" and "SSTable reads" increases significantly with version >= 3.11.7 I'm wondering if someone experienced the same problem when updating Cassandra from 3.11.6 to either 3.11.7 or 3.11.8 and can help me out. I already tried nodetool scrub, sstablescrub, manual compaction, and re pair on selected tables without impact. When downgrading back to 3.11.6 read latency recovers immediately. My update routine is very simple. I'm just stopping the service, install the new version (using apt), and start the service again. One node after the other. As coming from 3.11.6 updating to 3.11.7 (or 3.11.8) is just a patch update and I expect this routine should do well, but maybe I'm missing some important steps here. I can provide more information about my system or configuration if required. Appreciate your support! Kind regards Nico LANCOM Systems GmbH Adenauerstr. 20 / B2 52146 Würselen Deutschland Tel: +49 2405 49936-0 Fax: +49 2405 49936-99 Web: https://www.lancom-systems.de Geschäftsführer: Ralf Koenzen, Stefan Herrlich Sitz der Gesellschaft: Aachen, Amtsgericht Aachen, HRB 16976
Re: Increased read latency with Cassandra >= 3.11.7
Maybe compare the cache size see if anything different in two versions? On Mon, 14 Sep 2020, 19:21 Nicolai Lune Vest, wrote: > Dear Cassandra community, > > after upgrading my Cassandra nodes from version 3.11.6 to either 3.11.7 > or 3.11.8 I experience a significant increase in read latency. > With 3.11.6 average read latency is ~0.35 ms. With 3.11.7 average read > latency increases to ~2.9 ms, almost 10 times worse! This behavior > applies to all my environments, testing and productive. > > I do also observe that the "bloom-filter false positive rate" and "SSTabl > e reads" increases significantly with version >= 3.11.7 > > I'm wondering if someone experienced the same problem when updating Cassandra > from 3.11.6 to either 3.11.7 or 3.11.8 and can help me out. > > I already tried nodetool scrub, sstablescrub, manual compaction, and re > pair on selected tables without impact. When downgrading back to 3.11.6 > read latency recovers immediately. > > My update routine is very simple. I'm just stopping the service, install the > new version (using apt), and start the service again. One node after the > other. As coming from 3.11.6 updating to 3.11.7 (or 3.11.8) is just a patch > update and I expect this routine should do well, but maybe I'm missing > some important steps here. > > I can provide more information about my system or configuration if > required. > > Appreciate your support! > > Kind regards > Nico > > > -- > > > > LANCOM Systems GmbH > Adenauerstr. 20 / B2 > 52146 Würselen > Deutschland > > Tel: +49 2405 49936-0 > Fax: +49 2405 49936-99 > > Web: https://www.lancom-systems.de > > > > Geschäftsführer: Ralf Koenzen, Stefan Herrlich > Sitz der Gesellschaft: Aachen, Amtsgericht Aachen, HRB 16976 > -- > > > > >
Cassandra timeout during read query
Hi There, We are running into a strange issue in our Cassandra Cluster where one specific query is failing with following error: Cassandra timeout during read query at consistency QUORUM (3 responses were required but only 0 replica responded) This is not a typical query read timeout that we know for sure. This error is getting spit out within 5 seconds and the query timeout we have set is around 30 seconds Can we know what is happening here and how can we reproduce this in our local environment? Thanks, Deepak
Re: Cassandra timeout during read query
Deepak, Can you reply with: 1) The query you are trying to run. 2) The table definition (PRIMARY KEY, specifically). 3) Maybe a little description of what the table is designed to do. 4) How much data you're expecting returned (both # of rows and data size). Thanks, Aaron On Mon, Sep 14, 2020 at 10:58 AM Deepak Sharma wrote: > Hi There, > > We are running into a strange issue in our Cassandra Cluster where one > specific query is failing with following error: > > Cassandra timeout during read query at consistency QUORUM (3 responses > were required but only 0 replica responded) > > This is not a typical query read timeout that we know for sure. This error > is getting spit out within 5 seconds and the query timeout we have set is > around 30 seconds > > Can we know what is happening here and how can we reproduce this in our > local environment? > > Thanks, > Deepak > >
Re: Cassandra scale-out with no traffic on newly joined nodes
Small correction in cassandra driver usage, Driver used: Spring boot ( org.springframework.boot:spring-boot-starter-data-cassandra) Version: 2.3.2.RELEASE Are there any differences between spring boot (I think it's based on datastax cql driver) and Datastax core driver load balancing policies? Thanks in advance? On Tue, Sep 15, 2020 at 12:57 AM Sandeep Nethi wrote: > Thanks Erick. > > We are using datastax java driver ( > https://docs.datastax.com/en/developer/java-driver/). > Driver version number: 3.3.0 > C* version: 3.11.6 > > Regards, > Sandeep > > > On Sat, Sep 12, 2020 at 2:41 PM Erick Ramirez > wrote: > >> That would be my last option to add a new host as contant point but as >>> per my understanding cassandra should auto discover newly joined nodes and >>> server or load balance connections automatically but it's not happening. >>> >> >> Yes, that sounds right to me. Sorry, this fell off my radar due to >> competing priorities. I'm going to reach out to the DataStax Drivers team. >> In the meantime, can you tell me which driver + version and C* version? >> Cheers! >> >
Re: Re: Increased read latency with Cassandra >= 3.11.7
That is odd. I thought maybe lower version had a bigger custom cache or some other setting missed in the upgrade. On Mon, 14 Sep 2020, 23:00 Nicolai Lune Vest, wrote: > While cache configuration stays the same, cache hit rate for both, the > chunk cache and the key cache are worse (but still ok) after the update. > You can see the change in the attached screenshot (update was performed at > September the 7th ~4pm). > > @Sagar Any specific reason you are suspecting the cache? > > > > -"Sagar Jambhulkar" schrieb: - > An: user@cassandra.apache.org > Von: "Sagar Jambhulkar" > Datum: 14.09.2020 16:25 > Betreff: Re: Increased read latency with Cassandra >= 3.11.7 > > > Maybe compare the cache size see if anything different in two versions? > > > On Mon, 14 Sep 2020, 19:21 Nicolai Lune Vest, > wrote: > > > Dear Cassandra community, > > after upgrading my Cassandra nodes from version 3.11.6 to either 3.11.7 or > 3.11.8 I experience a significant increase in read latency. > With 3.11.6 average read latency is ~0.35 ms. With 3.11.7 average read > latency increases to ~2.9 ms, almost 10 times worse! This behavior applies > to all my environments, testing and productive. > > I do also observe that the "bloom-filter false positive rate" and "SSTable > reads" increases significantly with version >= 3.11.7 > > I'm wondering if someone experienced the same problem when updating > Cassandra from 3.11.6 to either 3.11.7 or 3.11.8 and can help me out. > > I already tried nodetool scrub, sstablescrub, manual compaction, and > repair on selected tables without impact. When downgrading back to 3.11.6 > read latency recovers immediately. > > My update routine is very simple. I'm just stopping the service, install > the new version (using apt), and start the service again. One node after > the other. As coming from 3.11.6 updating to 3.11.7 (or 3.11.8) is just a > patch update and I expect this routine should do well, but maybe I'm > missing some important steps here. > > I can provide more information about my system or configuration if > required. > > Appreciate your support! > > Kind regards > Nico > > > > > > > > LANCOM Systems GmbH > Adenauerstr. 20 / B2 > 52146 Würselen > Deutschland > > Tel: +49 2405 49936-0 > Fax: +49 2405 49936-99 > > Web: https://www.lancom-systems.de > > > > Geschäftsführer: Ralf Koenzen, Stefan Herrlich > Sitz der Gesellschaft: Aachen, Amtsgericht Aachen, HRB 16976 > > > > > > > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org
