topic.
BR
MK
From: Michail Kotsiouros via user
Sent: Thursday, May 11, 2023 14:08
To: user@cassandra.apache.org
Subject: RE: Questions about high read latency and related metrics
Hello Erick,
No Max/Min/Mean vs Histogram difference is clear.
What confuses me is the description of those metrics
if those questions sound trivial.
BR
MK
From: Erick Ramirez
Sent: Thursday, May 11, 2023 13:16
To: user@cassandra.apache.org; Michail Kotsiouros
Subject: Re: Questions about high read latency and related metrics
Is it the concept of histograms that's not clear? Something else?
Is it the concept of histograms that's not clear? Something else?
>
Hello Erick,
Thanks a lot for the immediate reply but still the difference between those 2
metrics is not clear to me.
BR
MK
From: Erick Ramirez
Sent: Thursday, May 11, 2023 13:04
To: user@cassandra.apache.org
Subject: Re: Questions about high read latency and related metrics
The min/max/mean
The min/max/mean partition sizes are the sizes in bytes which are the same
statistics reported by nodetool tablestats.
EstimatedPartitionSizeHistogram is the distribution of partition sizes
within specified ranges (percentiles) and is the same histogram reported by
nodetool tablehistograms (in the
Hello Cassandra community,
I see the following metrics in JMX
Metric Name
org.apache.cassandra.metrics.Table...
MinPartitionSize
Gauge
Size of the smallest compacted partition (in bytes).
MaxPartitionSize
Gauge
Size of the largest compacted partition (in bytes).
MeanPartitionSize
Gauge
Size o
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair = 'BLOCKING'
AND speculative_retry = '99p';
-Joe
On 11/29/2021 11:22 AM, Joe Obernberger wrote:
I have an 11 node cluster and am experiencing hig
I have an 11 node cluster and am experiencing high read latency on one
table. This table has ~112 million rows:
nodetool tablehistograms doc.origdoc
doc/origdoc histograms
Percentile Read Latency Write Latency SSTables Partition
Size Cell Count
(micros
>> f int static,
>>>>>>>
>>>>>>> g int static,
>>>>>>>
>>>>>>> h int,
>>>>>>>
>>>>>>> i text,
>>>>>>>
>>>>>>>
>>>
>>>>>> j text,
>>>>>>
>>>>>> k text,
>>>>>>
>>>>>> l text,
>>>>>>
>>>>>> m set
>>>>>>
>>>>>> n bigint
>>>>>>
>
>> m set
>>>>>>
>>>>>> n bigint
>>>>>>
>>>>>> o bigint
>>>>>>
>>>>>> p bigint
>>>>>>
>>>>>> q bigint
>>>>>>
>>&
gt;>>>>
>>>>> o bigint
>>>>>
>>>>> p bigint
>>>>>
>>>>> q bigint
>>>>>
>>>>> r int
>>>>>
>>>>> s text
>>>>>
>>>>>
text
>>>>
>>>> w text
>>>>
>>>> x bigint
>>>>
>>>> y bigint
>>>>
>>>> z bigint,
>>>>
>>>> primary key ((a, b), c)
>>>>
>>>> };
>>>>
>>>> -
ttings
>>
>> - Execution time of the GC
>>
>> Avg. 400ms. I do not see long pauses of GC anywhere in the log file.
>>
>>
>> On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric > <mailto:eric.le...@worldline.com>> wrote:
>> Hi,
>>
&g
p
>>>
>>> Default settings
>>>
>>> - Execution time of the GC
>>>
>>> Avg. 400ms. I do not see long pauses of GC anywhere in the log file.
>>>
>>> On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric
>>> wrote:
>>
p
>
> - Execution time of the GC
>
>
>
> 43ms for a read latency may be acceptable according to the number of request
> per second.
>
>
>
>
>
> Eric
>
>
>
> De : Jaydeep Chovatia [mailto:chovatia.jayd...@gmail.com
> <mailto:ch
ion time of the GC
>>
>> Avg. 400ms. I do not see long pauses of GC anywhere in the log file.
>>
>> On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric
>> wrote:
>>
>>> Hi,
>>>
>>>
>>>
>>>
>>>
>>> Before sp
]
Envoyé : mardi 22 septembre 2015 19:50
À : user@cassandra.apache.org
Objet : Re: High read latency
select * from test where a = ? and b = ?
On Tue, Sep 22, 2015 at 10:27 AM, sai krishnam raju potturi
mailto:pskraj...@gmail.com>> wrote:
thanks for the information. Posting the query too wo
uses of GC anywhere in the log file.
>>
>> On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric
>> wrote:
>>
>>> Hi,
>>>
>>>
>>>
>>>
>>>
>>> Before speaking about tuning, can you provide some additional
>>> informatio
;>
>> Before speaking about tuning, can you provide some additional information
>> ?
>>
>>
>>
>> - Number of req/s
>>
>> - Schema details
>>
>> - JVM settings about the heap
>>
>> - Exe
Execution time of the GC
>
>
>
> 43ms for a read latency may be acceptable according to the number of
> request per second.
>
>
>
>
>
> Eric
>
>
>
> *De :* Jaydeep Chovatia [mailto:chovatia.jayd...@gmail.com]
> *Envoyé :* mardi 22 septembr
second.
Eric
De : Jaydeep Chovatia [mailto:chovatia.jayd...@gmail.com]
Envoyé : mardi 22 septembre 2015 00:07
À : user@cassandra.apache.org
Objet : High read latency
Hi,
My application issues more read requests than write, I do see that under load
cfstats for one of the table is quite high
Hi,
My application issues more read requests than write, I do see that under
load cfstats for one of the table is quite high around 43ms
Local read count: 114479357
Local read latency: 43.442 ms
Local write count: 22288868
Local writ
nodetool cfhistograms is also very helpful in diagnosing these kinds of
data modelling issues.
On 23 March 2015 at 14:43, Chris Lohfink wrote:
>
> >> Compacted partition maximum bytes: 36904729268
>
> thats huge... 36gb rows are gonna cause a lot of problems, even when you
> specify a precise c
>> Compacted partition maximum bytes: 36904729268
thats huge... 36gb rows are gonna cause a lot of problems, even when you
specify a precise cell under this it still is going to have an enormous
column index to deserialize on every read for the partition. As mentioned
above, you should include y
I haven't deleted anything. Here's output from a traced cqlsh query (I
tried to make the spaces line up, hope it's legible):
Execute CQL3
query
| 2015-03-23 21:04:37.422000 | 172.31.32.211 | 0
Parsing select * from default.metrics where row_time = 16511 and attrs =
'[redacted]' limit
Enable tracing in cqlsh and see how many sstables are being lifted to
satisfy the query (are you repeatedly writing to the same partition
[row_time]) over time?).
Also watch for whether you're hitting a lot of tombstones (are you deleting
lots of values in the same partition over time?).
On Mon,
Duncan: I'm thinking it might be something like that. I'm also seeing just
a ton of garbage collection on the box, could it be pulling rows for all
100k attrs for a given row_time into memory since only row_time is the
partition key?
Jens: I'm not using EBS (although I used to until I read up on h
Also, two control questions:
- Are you using EBS for data storage? It might introduce additional
latencies.
- Are you doing proper paging when querying the keyspace?
Cheers,
Jens
On Mon, Mar 23, 2015 at 5:56 AM, Dave Galbraith
wrote:
> Hi! So I've got a table like this:
>
> CREATE TAB
Hi Dave,
On 23/03/15 05:56, Dave Galbraith wrote:
Hi! So I've got a table like this:
CREATE TABLE "default".metrics (row_time int,attrs varchar,offset int,value
double, PRIMARY KEY(row_time, attrs, offset)) WITH COMPACT STORAGE AND
bloom_filter_fp_chance=0.01 AND caching='KEYS_ONLY' AND comment
Hi! So I've got a table like this:
CREATE TABLE "default".metrics (row_time int,attrs varchar,offset int,value
double, PRIMARY KEY(row_time, attrs, offset)) WITH COMPACT STORAGE AND
bloom_filter_fp_chance=0.01 AND caching='KEYS_ONLY' AND comment='' AND
dclocal_read_repair_chance=0 AND gc_grace_sec
There's likely 2 things occurring
1) the cfhistograms error is due to
https://issues.apache.org/jira/browse/CASSANDRA-8028
Which is resolved in 2.1.3. Looks like voting is under way for 2.1.3. As
rcoli mentioned, you are running the latest open source of C* which should
be treated as beta until a
Hi there,
The compaction remains running with our workload.
We are using SATA HDDs RAIDs.
When trying to run cfhistograms on our user_data table, we are getting
this message:
nodetool: Unable to compute when histogram overflowed
Please see what happens when running some queries on this cf:
http:
Hello
You may not be experiencing versioning issues. Do you know if compaction is
keeping up with your workload? The behavior described in the subject is
typically associated with compaction falling behind or having a suboptimal
compaction strategy configured. What does the output of nod
an Tarbox [mailto:briantar...@gmail.com]
Sent: Friday, January 9, 2015 8:56 AM
To: user@cassandra.apache.org
Subject: Re: High read latency after data volume increased
C* seems to have more than its share of "version x doesn't work, use version y
" type issues
On Thu, Jan 8, 2015
C* seems to have more than its share of "version x doesn't work, use
version y " type issues
On Thu, Jan 8, 2015 at 2:23 PM, Robert Coli wrote:
> On Thu, Jan 8, 2015 at 11:14 AM, Roni Balthazar
> wrote:
>
>> We are using C* 2.1.2 with 2 DCs. 30 nodes DC1 and 10 nodes DC2.
>>
>
> https://eng
On Thu, Jan 8, 2015 at 6:38 PM, Roni Balthazar
wrote:
> We downgraded to 2.1.1, but got the very same result. The read latency is
> still high, but we figured out that it happens only using a specific
> keyspace.
>
Note that downgrading is officially unsupported, but is probably safe
between tho
Hi Robert,
We downgraded to 2.1.1, but got the very same result. The read latency is
still high, but we figured out that it happens only using a specific
keyspace.
Please see the graphs below...
Trying another keyspace with 600+ reads/sec, we are getting the acceptable
~30ms read latency.
Let
On Thu, Jan 8, 2015 at 11:14 AM, Roni Balthazar
wrote:
> We are using C* 2.1.2 with 2 DCs. 30 nodes DC1 and 10 nodes DC2.
>
https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/
2.1.2 in particular is known to have significant issues. You'd be better
off running 2.1.1 ...
Hi there,
We are using C* 2.1.2 with 2 DCs. 30 nodes DC1 and 10 nodes DC2.
While our data volume is increasing (34 TB now), we are running into
some problems:
1) Read latency is around 1000 ms when running 600 reads/sec (DC1
CL.LOCAL_ONE). At the same time the load average is about 20-30 on all
Disregard... heh. Was reading the latency as SECONDS. Sorry, it's been one
of those weeks.
On Wed, Mar 5, 2014 at 1:44 AM, Eric Plowe wrote:
> Background info:
>
> 6 node cluster.
> 24 gigs of ram per machine
> 8 gigs of ram dedicated to c*
> 4 4 core cpu's
> 2 250 gig SSD's raid 0
> Running c*
Background info:
6 node cluster.
24 gigs of ram per machine
8 gigs of ram dedicated to c*
4 4 core cpu's
2 250 gig SSD's raid 0
Running c* 1.2.6
The CF is configured as followed
CREATE TABLE behaviors (
uid text,
buid int,
name text,
expires text,
value text,
PRIMARY KEY (uid, buid,
> We are using Leveled Compaction with Cassandra 1.2.5. Our sstable size is
> 100M. On each node,
> we have anywhere from 700+ to 800+ sstables (for all levels). The
> bloom_filter_fp_chance is set at 0.000744.
The current default bloom_filter_fp_chance is 0.1 for levelled compaction.
Reducing t
Try doing request tracing.
http://www.datastax.com/dev/blog/tracing-in-cassandra-1-2
On Thu, Jun 27, 2013 at 2:40 PM, Bao Le wrote:
> Hi,
>
> We are using Leveled Compaction with Cassandra 1.2.5. Our sstable size
> is 100M. On each node,
> we have anywhere from 700+ to 800+ sstables (for al
Hi,
We are using Leveled Compaction with Cassandra 1.2.5. Our sstable size is
100M. On each node,
we have anywhere from 700+ to 800+ sstables (for all levels). The
bloom_filter_fp_chance is set at 0.000744.
For read requests that ask for existing row records, the latency is great,
mostly
> FlushWriter 0 0 8252 0
>299
If you are not suffering from gc pressure/pauses (possibly not, because you
don't seem to have a lot of read failures in tpstats or outlier latency on the
histograms), then the flush writer errors are s
300 GB is a lot of data for cloud machines (especially with their
weaker performance in general). If you are unhappy with performance
why not scale the cluster out to more servers, with that much data you
are usually contending with the physics of spinning disks. Three nodes
+ replication factor 3
It's really a pain to modify the data model, the problem is how to
handle "one-to-many" relation in cassandra? The limitation of the row
size will lead to impossible to store them with columns.
On Fri, Jun 4, 2010 at 4:13 PM, Sylvain Lebresne wrote:
> As written in the third point of
> http://w
As written in the third point of
http://wiki.apache.org/cassandra/CassandraLimitations,
right now, super columns are not indexed and deserialized fully when you access
them. Another way to put it is, you'll want to user super columns with
only a relatively
small number of columns in them.
Because i
we have a SupperCF which may have up to 1000 supper columns and 5
clumns for each supper column, the read latency may go up to 50ms
(even higher), I think it's a long time to response, how to tune the
storage config to optimize the performace? I read the wiki,
may help to do this, supose that
50 matches
Mail list logo
