Maybe compaction not keeping up - since you are hitting so many sstables?
Read heavy... are you using LCS?
Plenty of resources... tune to increase memtable size?
On Sat, Sep 26, 2015 at 9:19 AM, Eric Stevens <migh...@gmail.com> wrote:
> Since you have most of your reads hitting 5-8 SSTables, it's probably
> related to that increasing your latency. That makes this look like your
> write workload is either overwrite-heavy or append-heavy. Data for a
> single partition key is being written to repeatedly over long time periods,
> and this will definitely impact read performance.
>
> You can enable tracing in cqlsh and run your select to see where the time
> is going.
>
> On Fri, Sep 25, 2015 at 3:07 PM, Jaydeep Chovatia <
> chovatia.jayd...@gmail.com> wrote:
>
>> Please find histogram attached.
>>
>> On Fri, Sep 25, 2015 at 12:20 PM, Ryan Svihla <r...@foundev.pro> wrote:
>>
>>> if everything is in ram there could be a number of issues unrelated to
>>> Cassandra and there could be hardware limitations or contention problems.
>>> Otherwise cell count can really deeply impact reads, all ram or not, and
>>> some of this is because of the nature of GC and some of it is the age of
>>> the sstable format (which is due to be revamped in 3.0). Also partition
>>> size can matter just because of physics, if one of those is a 1gb
>>> partition, the network interface can only move that back across the wire so
>>> quickly not to mention the GC issues you’d run into.
>>>
>>> Anyway this is why I asked for the histograms, I wanted to get cell
>>> count and partition size. I’ve seen otherwise very stout hardware get slow
>>> on reads of large results because either a bottleneck was hit somewhere, or
>>> the CPU got slammed with GC, or other processes running on the machine were
>>> contending with Cassandra.
>>>
>>>
>>> On Sep 25, 2015, at 12:45 PM, Jaydeep Chovatia <
>>> chovatia.jayd...@gmail.com> wrote:
>>>
>>> I understand that but everything is in RAM (my data dir is tmpfs) and my
>>> row is not that wide approx. less than 5MB in size. So my question is if
>>> everything is in RAM then why does it take 43ms latency?
>>>
>>> On Fri, Sep 25, 2015 at 7:54 AM, Ryan Svihla <r...@foundev.pro> wrote:
>>>
>>>> if you run:
>>>>
>>>> nodetool cfhistograms <keyspace> <table name>
>>>>
>>>> On the given table and that will tell you how wide your rows are
>>>> getting. At some point you can get wide enough rows that just the physics
>>>> of retrieving them all take some time.
>>>>
>>>>
>>>> On Sep 25, 2015, at 9:21 AM, sai krishnam raju potturi <
>>>> pskraj...@gmail.com> wrote:
>>>>
>>>> Jaydeep; since your primary key involves a clustering column, you may
>>>> be having pretty wide rows. The read would be sequential. The latency could
>>>> be acceptable, if the read were to involve really wide rows.
>>>>
>>>> If your primary key was like ((a,b)) without the clustering column,
>>>> it's like reading a key value pair, and 40ms latency may have been a
>>>> concern.
>>>>
>>>> Bottom Line : The latency depends on how wide the row is.
>>>>
>>>> On Tue, Sep 22, 2015 at 1:27 PM, sai krishnam raju potturi <
>>>> pskraj...@gmail.com> wrote:
>>>>
>>>>> thanks for the information. Posting the query too would be of help.
>>>>>
>>>>> On Tue, Sep 22, 2015 at 11:56 AM, Jaydeep Chovatia <
>>>>> chovatia.jayd...@gmail.com> wrote:
>>>>>
>>>>>> Please find required details here:
>>>>>>
>>>>>> - Number of req/s
>>>>>>
>>>>>> 2k reads/s
>>>>>>
>>>>>> - Schema details
>>>>>>
>>>>>> create table test {
>>>>>>
>>>>>> a timeuuid,
>>>>>>
>>>>>> b bigint,
>>>>>>
>>>>>> c int,
>>>>>>
>>>>>> d int static,
>>>>>>
>>>>>> e int static,
>>>>>>
>>>>>> f int static,
>>>>>>
>>>>>> g int static,
>>>>>>
>>>>>> h int,
>>>>>>
>>>>>> i text,
>>>>>>
>>>>>> j text,
>>>>>>
>>>>>> k text,
>>>>>>
>>>>>> l text,
>>>>>>
>>>>>> m set<text>
>>>>>>
>>>>>> n bigint
>>>>>>
>>>>>> o bigint
>>>>>>
>>>>>> p bigint
>>>>>>
>>>>>> q bigint
>>>>>>
>>>>>> r int
>>>>>>
>>>>>> s text
>>>>>>
>>>>>> t bigint
>>>>>>
>>>>>> u text
>>>>>>
>>>>>> v text
>>>>>>
>>>>>> w text
>>>>>>
>>>>>> x bigint
>>>>>>
>>>>>> y bigint
>>>>>>
>>>>>> z bigint,
>>>>>>
>>>>>> primary key ((a, b), c)
>>>>>>
>>>>>> };
>>>>>>
>>>>>> - JVM settings about the heap
>>>>>>
>>>>>> Default settings
>>>>>>
>>>>>> - Execution time of the GC
>>>>>>
>>>>>> Avg. 400ms. I do not see long pauses of GC anywhere in the log file.
>>>>>>
>>>>>> On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric <eric.le...@worldline.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Before speaking about tuning, can you provide some additional
>>>>>>> information ?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> - Number of req/s
>>>>>>>
>>>>>>> - Schema details
>>>>>>>
>>>>>>> - JVM settings about the heap
>>>>>>>
>>>>>>> - Execution time of the GC
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 43ms for a read latency may be acceptable according to the number of
>>>>>>> request per second.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Eric
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *De :* Jaydeep Chovatia [mailto:chovatia.jayd...@gmail.com]
>>>>>>> *Envoyé :* mardi 22 septembre 2015 00:07
>>>>>>> *À :* user@cassandra.apache.org
>>>>>>> *Objet :* High read latency
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> My application issues more read requests than write, I do see that
>>>>>>> under load cfstats for one of the table is quite high around 43ms
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Local read count: 114479357
>>>>>>>
>>>>>>> Local read latency: 43.442 ms
>>>>>>>
>>>>>>> Local write count: 22288868
>>>>>>>
>>>>>>> Local write latency: 0.609 ms
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Here is my node configuration:
>>>>>>>
>>>>>>> RF=3, Read/Write with QUORUM, 64GB RAM, 48 CPU core. I have only 5
>>>>>>> GB of data on each node (and for experiment purpose I stored data in
>>>>>>> tmpfs)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I've tried increasing concurrent_read count upto 512 but no help in
>>>>>>> read latency. CPU/Memory/IO looks fine on system.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Any idea what should I tune?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Jaydeep
>>>>>>>
>>>>>>> ------------------------------
>>>>>>>
>>>>>>> Ce message et les pièces jointes sont confidentiels et réservés à
>>>>>>> l'usage exclusif de ses destinataires. Il peut également être protégé
>>>>>>> par
>>>>>>> le secret professionnel. Si vous recevez ce message par erreur, merci
>>>>>>> d'en
>>>>>>> avertir immédiatement l'expéditeur et de le détruire. L'intégrité du
>>>>>>> message ne pouvant être assurée sur Internet, la responsabilité de
>>>>>>> Worldline ne pourra être recherchée quant au contenu de ce message. Bien
>>>>>>> que les meilleurs efforts soient faits pour maintenir cette transmission
>>>>>>> exempte de tout virus, l'expéditeur ne donne aucune garantie à cet
>>>>>>> égard et
>>>>>>> sa responsabilité ne saurait être recherchée pour tout dommage résultant
>>>>>>> d'un virus transmis.
>>>>>>>
>>>>>>> This e-mail and the documents attached are confidential and intended
>>>>>>> solely for the addressee; it may also be privileged. If you receive this
>>>>>>> e-mail in error, please notify the sender immediately and destroy it. As
>>>>>>> its integrity cannot be secured on the Internet, the Worldline liability
>>>>>>> cannot be triggered for the message content. Although the sender
>>>>>>> endeavours
>>>>>>> to maintain a computer virus-free network, the sender does not warrant
>>>>>>> that
>>>>>>> this transmission is virus-free and will not be liable for any damages
>>>>>>> resulting from any virus transmitted.
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> Regards,
>>>>
>>>> Ryan Svihla
>>>>
>>>>
>>>
>>>
>>
>
