Re: any way to get the #writes/second, reads per second

[Tomàs Núnez]

Yes, there isn't any place in the JMX with reads/second or writes/second,
just CompletedTasks. I send this info to Graphite (
http://graphite.wikidot.com/) and use the "derivative" function to get
reads/minute. Munin also does the trick (https://github.com/tcurdt/jmx2munin).
But you can't do that with cassandra itself, you need somewhere to make the
calculations (cacti is also a good match).

Hope that helps. There is some more information about monitoring this kind
of things here:
http://www.tomas.cat/blog/en/monitoring-cassandra-relevant-data-should-be-watched-and-how-send-it-graphite



2013/5/13 Hiller, Dean 

> We running a pretty consistent load on our cluster and added a new node to
> a 6 node cluster Friday(QA worked great, but production not so much).  One
> mistake that was made was starting up the new node, then disabling the
> firewall :( which allowed nodes to discover it BEFORE the node bootstrapped
> itself.  We shutdown the node and booted him up and he bootstrapped himself
> streaming all the data in.
>
> After that though, all the ndoes have really really high load numbers now.
>  We are trying to figure out what is going on still.
>
> Is there any way to get the number of reads/second and writes/second
> through JMX or something?  The only way I can see of on doing this is
> manually calculating it by timing the read count and dividing by my manual
> stop watches start/stop times(timerange).
>
> Also, while my load is load average: 20.31, 19.10, 19.72 , what does a
> normal iostat look like?  My iostat await time is 13.66 ms which I think is
> kind of bad, but not that bad to cause a load of 20.31?
>
> Device: rrqm/s   wrqm/s r/s w/s   rsec/s   wsec/s avgrq-sz
> avgqu-sz   await  svctm  %util
> sda   0.02 0.07   11.701.96  1353.67   702.88   150.58
> 0.19   13.66   3.61   4.93
> sdb   0.00 0.020.110.4620.7297.54   206.70
> 0.001.33   0.67   0.04
>
> Thanks,
> Dean
>



-- 
[image: Groupalia] 
www.groupalia.com Tomàs NúñezIT-SysprodTel. + 34
93 159 31 00 Fax. + 34 93 396 18 52Llull, 95-97, 2º planta, 08005
BarcelonaSkype:
tomas.nunez.groupaliatomas.nu...@groupalia.com[image:
Twitter] Twitter [image: Twitter]
 Facebook [image: Twitter]
 Linkedin 
<><><><>

Logging Cassandra queries

[Tomàs Núnez]

Hi!

For quite time I've been having some unexpected loadavg in the cassandra
servers. I suspect there are lots of uncontrolled queries to the cassandra
servers causing this load, but the developers say that there are none, and
the load is due to cassandra internal processes.

Trying to get to the bottom, I've been looking into completed ReadStage and
MutationStage through JMX, and the numbers seem to confirm my theory, but
I'd like to go one step forward and, if possible, list all the queries from
the webservers to the cassandra cluster (just one node would be enough).

I've been playing with cassandra loglevels, and I can see when a Read or a
Write is done, but it would be better if I could knew the CF of the query.
For my tests I've put the in the log4j.server
" log4j.rootLogger=DEBUG,stdout,R", writing and reading a test CF, and I
can't see the name of it anywhere.

For the test I'm using Cassandra 0.8.4 (yes, still), as my production
servers, and also 1.0.11. Maybe this changes in 1.1? Maybe I'm doing
something wrong? Any hint?

And... could I be more precise when enabling logging? Because right now,
with *log4j.rootLogger=DEBUG,stdout,R* I'm getting a lot of information I
won't use ever, and I'd like to enable just what I need to see gets and
seds

Thanks in advance,
Tomàs

Re: Logging Cassandra queries

[Tomàs Núnez]

Yes, I read how to do that here, as well:
http://www.datastax.com/docs/1.1/configuration/logging_options

But I didn't know what classes to enable logging for the queries... Is
there any document with the list of classes with a bit explanation for each
of them? I can't find any, and I don't understand java enough to dive
through the code

Thanks!


2013/5/17 aaron morton 

> And... could I be more precise when enabling logging? Because right now,
> with *log4j.rootLogger=DEBUG,stdout,R* I'm getting a lot of information I
> won't use ever, and I'd like to enable just what I need to see gets and
> seds….
>
>
> see the example at the bottom of this file about setting the log level for
> a single class
> https://github.com/apache/cassandra/blob/trunk/conf/log4j-server.properties
>
> You probably want to set it for the
> org.apache.cassandra.thrift.CassandraServer class. But I cannot remember
> what the logging is like in 0.8.
>
> Cassandra gets faster in the later versions, which normally means doing
> less work. Upgrading to 1.1 would be the first step I would take in
> improving performance.
>
> Cheers
>
> -
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 18/05/2013, at 4:00 AM, Tomàs Núnez  wrote:
>
> Hi!
>
> For quite time I've been having some unexpected loadavg in the cassandra
> servers. I suspect there are lots of uncontrolled queries to the cassandra
> servers causing this load, but the developers say that there are none, and
> the load is due to cassandra internal processes.
>
> Trying to get to the bottom, I've been looking into completed ReadStage
> and MutationStage through JMX, and the numbers seem to confirm my theory,
> but I'd like to go one step forward and, if possible, list all the queries
> from the webservers to the cassandra cluster (just one node would be
> enough).
>
> I've been playing with cassandra loglevels, and I can see when a Read or a
> Write is done, but it would be better if I could knew the CF of the query.
> For my tests I've put the in the log4j.server
> " log4j.rootLogger=DEBUG,stdout,R", writing and reading a test CF, and I
> can't see the name of it anywhere.
>
> For the test I'm using Cassandra 0.8.4 (yes, still), as my production
> servers, and also 1.0.11. Maybe this changes in 1.1? Maybe I'm doing
> something wrong? Any hint?
>
> And... could I be more precise when enabling logging? Because right now,
> with *log4j.rootLogger=DEBUG,stdout,R* I'm getting a lot of information I
> won't use ever, and I'd like to enable just what I need to see gets and
> seds
>
> Thanks in advance,
> Tomàs
>
>
>

Re: Logging Cassandra queries

[Tomàs Núnez]

>
> If you're looking for logging like "get keyX with CL quorum and slice Y
> took n millis"


That would be even better! Maybe should I file a ticket in Cassandra Jira
for this feature? Do you think it would be helpful?

BTW, just "get keyX" or "set keyX" would work for me. I'll
check org.apache.cassandra.thrift.CassandraServer as Aaron suggested (but
still the list of classes would be helpful :) )

Thanks!

2013/5/19 Ilya Kirnos 

> If you're looking for logging like "get keyX with CL quorum and slice Y
> took n millis" there's nothing like that from what I could find.  We had to
> modify c* source (CassandraServer.java) to add this query logging to the
> thrift codepath.
> On May 18, 2013 3:20 PM, "Tomàs Núnez"  wrote:
>
>> Yes, I read how to do that here, as well:
>> http://www.datastax.com/docs/1.1/configuration/logging_options
>>
>> But I didn't know what classes to enable logging for the queries... Is
>> there any document with the list of classes with a bit explanation for each
>> of them? I can't find any, and I don't understand java enough to dive
>> through the code
>>
>> Thanks!
>>
>>
>> 2013/5/17 aaron morton 
>>
>>> And... could I be more precise when enabling logging? Because right now,
>>> with *log4j.rootLogger=DEBUG,stdout,R* I'm getting a lot of information
>>> I won't use ever, and I'd like to enable just what I need to see gets and
>>> seds….
>>>
>>>
>>> see the example at the bottom of this file about setting the log level
>>> for a single class
>>> https://github.com/apache/cassandra/blob/trunk/conf/log4j-server.properties
>>>
>>> You probably want to set it for the
>>> org.apache.cassandra.thrift.CassandraServer class. But I cannot remember
>>> what the logging is like in 0.8.
>>>
>>> Cassandra gets faster in the later versions, which normally means doing
>>> less work. Upgrading to 1.1 would be the first step I would take in
>>> improving performance.
>>>
>>> Cheers
>>>
>>>-
>>> Aaron Morton
>>> Freelance Cassandra Consultant
>>> New Zealand
>>>
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>>
>>> On 18/05/2013, at 4:00 AM, Tomàs Núnez 
>>> wrote:
>>>
>>> Hi!
>>>
>>> For quite time I've been having some unexpected loadavg in the cassandra
>>> servers. I suspect there are lots of uncontrolled queries to the cassandra
>>> servers causing this load, but the developers say that there are none, and
>>> the load is due to cassandra internal processes.
>>>
>>> Trying to get to the bottom, I've been looking into completed ReadStage
>>> and MutationStage through JMX, and the numbers seem to confirm my theory,
>>> but I'd like to go one step forward and, if possible, list all the queries
>>> from the webservers to the cassandra cluster (just one node would be
>>> enough).
>>>
>>> I've been playing with cassandra loglevels, and I can see when a Read or
>>> a Write is done, but it would be better if I could knew the CF of the
>>> query. For my tests I've put the in the log4j.server
>>> " log4j.rootLogger=DEBUG,stdout,R", writing and reading a test CF, and I
>>> can't see the name of it anywhere.
>>>
>>> For the test I'm using Cassandra 0.8.4 (yes, still), as my production
>>> servers, and also 1.0.11. Maybe this changes in 1.1? Maybe I'm doing
>>> something wrong? Any hint?
>>>
>>> And... could I be more precise when enabling logging? Because right now,
>>> with *log4j.rootLogger=DEBUG,stdout,R* I'm getting a lot of information
>>> I won't use ever, and I'd like to enable just what I need to see gets and
>>> seds
>>>
>>> Thanks in advance,
>>> Tomàs
>>>
>>>
>>

Alternate "major compaction"

[Tomàs Núnez]

Hi

About a year ago, we did a major compaction in our cassandra cluster (a
n00b mistake, I know), and since then we've had huge sstables that never
get compacted, and we were condemned to repeat the major compaction process
every once in a while (we are using SizeTieredCompaction strategy, and
we've not avaluated yet LeveledCompaction, because it has its downsides,
and we've had no time to test all of them in our environment).

I was trying to find a way to solve this situation (that is, do something
like a major compaction that writes small sstables, not huge as major
compaction does), and I couldn't find it in the documentation. I tried
cleanup and scrub/upgradesstables, but they don't do that (as documentation
states). Then I tried deleting all data in a node and then bootstrapping it
(or "nodetool rebuild"-ing it), hoping that this way the sstables would get
cleaned from deleted records and updates. But the deleted node just copied
the sstables from another node as they were, cleaning nothing.

So I tried a new approach: I switched the sstable compaction strategy
(SizeTiered to Leveled), forcing the sstables to be rewritten from scratch,
and then switching it back (Leveled to SizeTiered). It took a while (but so
do the major compaction process) and it worked, I have smaller sstables,
and I've regained a lot of disk space.

I'm happy with the results, but it doesn't seem a orthodox way of
"cleaning" the sstables. What do you think, is it something wrong or crazy?
Is there a different way to achieve the same thing?

Let's put an example:
Suppose you have a write-only columnfamily (no updates and no deletes, so
no need for LeveledCompaction, because SizeTiered works perfectly and
requires less I/O) and you mistakenly run a major compaction on it. After a
few months you need more space and you delete half the data, and you find
out that you're not freeing half the disk space, because most of those
records were in the "major compacted" sstables. How can you free the disk
space? Waiting will do you no good, because the huge sstable won't get
compacted anytime soon. You can run another major compaction, but that
would just postpone the real problem. Then you can switch compaction
strategy and switch it back, as I just did. Is there any other way?

-- 
[image: Groupalia] 
www.groupalia.com Tomàs NúñezIT-SysprodTel. + 34
93 159 31 00 Fax. + 34 93 396 18 52Llull, 95-97, 2º planta, 08005
BarcelonaSkype:
tomas.nunez.groupaliatomas.nu...@groupalia.com[image:
Twitter] Twitter [image: Twitter]
 Facebook [image: Twitter]
 Linkedin 
<><><><>