Re: READ messages dropped

[Tamar Fraenkel]

Hi!
Thanks for the response. My cluster is in a bad state those recent days.

I have 29 CFs, and my disk is 5% full... So I guess the VMs still have more
space to go, and I am not sure this is considered many CFs.

But maybe I have memory issues. I enlarge cassandra memory from about ~2G
to ~4G (out of ~8G). This was done because at that stage I had lots of key
caches. I then reduced them to almost 0 on all CF. I guess now I can reduce
the memory back to ~2 or ~3 G. Will that help?
Thanks
*Tamar Fraenkel *
Senior Software Engineer, TOK Media

[image: Inline image 1]

ta...@tok-media.com
Tel:   +972 2 6409736
Mob:  +972 54 8356490
Fax:   +972 2 5612956





On Thu, Oct 11, 2012 at 10:46 PM, Tyler Hobbs  wrote:

> On Wed, Oct 10, 2012 at 3:10 PM, Tamar Fraenkel wrote:
>
>>
>> What I did noticed while looking at the logs (which are also running
>> OpsCenter), is that there is some correlation between the dropped reads and
>> flushes of OpsCenter column families to disk and or compactions. What are
>> the rollups CFs? why is there so much traffic in them?
>
>
> The rollups CFs hold the performance metric data that OpsCenter stores
> about your cluster.  Typically these aren't actually very high traffic
> column families, but that depends on how many column families you have
> (more CFs require more metrics to be stored).  If you have a lot of column
> families, you have a couple of options for reducing the amount of metric
> data that's stored:
> http://www.datastax.com/docs/opscenter/trouble_shooting_opsc#limiting-the-metrics-collected-by-opscenter
>
> Assuming you don't have a large number of CFs, your nodes may legitimately
> be nearing capacity.
>
> --
> Tyler Hobbs
> DataStax 
>
>
<>

Re: unnecessary tombstone's transmission during repair process

[Alexey Zotov]

Sylvain,

I've seen to the code. Yes, you right about local deletion time. But it
contradicts to the tests results.

Do you have any thoughts how to explain result of the second test after
patch applying?


Our patch:

diff --git a/src/java/org/apache/cassandra/db/DeletedColumn.java
b/src/java/org/apache/cassandra/db/DeletedColumn.java
index 18faeef..31744f6 100644
--- a/src/java/org/apache/cassandra/db/DeletedColumn.java
+++ b/src/java/org/apache/cassandra/db/DeletedColumn.java
@@ -17,10 +17,13 @@
  */
 package org.apache.cassandra.db;

+import java.io.IOException;
 import java.nio.ByteBuffer;
+import java.security.MessageDigest;

 import org.apache.cassandra.config.CFMetaData;
 import org.apache.cassandra.db.marshal.MarshalException;
+import org.apache.cassandra.io.util.DataOutputBuffer;
 import org.apache.cassandra.utils.Allocator;
 import org.apache.cassandra.utils.ByteBufferUtil;
 import org.apache.cassandra.utils.HeapAllocator;
@@ -46,6 +49,25 @@ public class DeletedColumn extends Column
 }

 @Override
+public void updateDigest(MessageDigest digest) {
+digest.update(name.duplicate());
+// it's commented to prevent consideration of the
localDeletionTime in Merkle Tree construction
+//digest.update(value.duplicate());
+
+DataOutputBuffer buffer = new DataOutputBuffer();
+try
+{
+buffer.writeLong(timestamp);
+buffer.writeByte(serializationFlags());
+}
+catch (IOException e)
+{
+throw new RuntimeException(e);
+}
+digest.update(buffer.getData(), 0, buffer.getLength());
+}
+
+@Override
 public long getMarkedForDeleteAt()
 {
 return timestamp;




-- 

Best regards**

Zotov Alexey
Grid Dynamics
Skype: azotcsit

Re: Cassandra nodes loaded unequally

[Alexey Zotov]

Hi Ben,

I suggest you to compare amount of queries for each node. May be the
problem is on the client side.
Yoy can do that using JMX:
"org.apache.cassandra.db:type=ColumnFamilies,keyspace=,columnfamily=","ReadCount"
"org.apache.cassandra.db:type=ColumnFamilies,keyspace=,columnfamily=","WriteCount"

Also I suggest to check output of "nodetool compactionstats".

-- 
Alexey

Re: cassandra 1.0.8 memory usage

[Daniel Woo]

Hi Rob,

>>What version of Cassandra? What JVM? Are JNA and Jamm working?
cassandra 1.0.8. Sun JDK 1.7.0_05-b06, JNA memlock enabled, jamm works.

>>It sounds like the two nodes that are pathological right now have
exhausted the perm gen with actual non-garbage, probably mostly the  Bloom
filters and the JMX MBeans.
JMAP shows that the per gen is only 40% used.

>>Do you have a "large" number of ColumnFamilies? How large is the data
stored per node?
I have very few column families, maybe 30-50. The nodetool shows each node
has 5 GB load.

>> Disable swap for cassandra node
I am gonna change swappiness to 20%

Thanks,
Daniel


On Fri, Oct 12, 2012 at 2:02 AM, Rob Coli  wrote:

> On Wed, Oct 10, 2012 at 11:04 PM, Daniel Woo 
> wrote:
> > I am running a mini cluster with 6 nodes, recently we see very frequent
> > ParNewGC on two nodes. It takes 200 - 800 ms on average, sometimes it
> takes
> > 5 seconds. You know, hte ParNewGC is stop-of-wolrd GC and our client
> throws
> > SocketTimeoutException every 3 minutes.
>
> What version of Cassandra? What JVM? Are JNA and Jamm working?
>
> > I checked the load, it seems well balanced, and the two nodes are
> running on
> > the same hardware: 2 * 4 cores xeon with 16G RAM, we give cassandrda 4G
> > heap, including 800MB young generation. We did not see any swap usage
> during
> > the GC, any idea about this?
>
> It sounds like the two nodes that are pathological right now have
> exhausted the perm gen with actual non-garbage, probably mostly the
> Bloom filters and the JMX MBeans.
>
> > Then I took a heap dump, it shows that 5 instances of JmxMBeanServer
> holds
> > 500MB memory and most of the referenced objects are JMX mbean related,
> it's
> > kind of wired to me and looks like a memory leak.
>
> Do you have a "large" number of ColumnFamilies? How large is the data
> stored per node?
>
> =Rob
>
> --
> =Robert Coli
> AIM>ALK - rc...@palominodb.com
> YAHOO - rcoli.palominob
> SKYPE - rcoli_palominodb
>



-- 
Thanks & Regards,
Daniel

what is more important (RAM vs Cores)

[Hagos, A.S.]

Hi All,
For of my projects I want to buy a machine to host Casssandra database.
The options I am offered are machines with 16GB RAM with Quad-Core processor 
and 6GB RAM with Hexa-Core processor.
Which one do you recommend, big RAM or  high number of cores?

greetings 
Ambes

Re: what is more important (RAM vs Cores)

[wang liang]

Hi, Hagos,

I think it depends on your business case. Big RAM reduce latency and
improve responsibility, High number of cores increase concurrency of your
app. thanks.

On Fri, Oct 12, 2012 at 4:23 PM, Hagos, A.S.  wrote:

> Hi All,
> For of my projects I want to buy a machine to host Casssandra database.
> The options I am offered are machines with 16GB RAM with Quad-Core
> processor and 6GB RAM with Hexa-Core processor.
> Which one do you recommend, big RAM or  high number of cores?
>
> greetings
> Ambes
>



-- 
Best wishes,
Helping others is to help myself.

Re: what is more important (RAM vs Cores)

[Romain HARDOUIN]

Hi,

Sure it depends... but IMHO 6 GB is suboptimal for big data because it 
means 1,5 GB or  2 GB for Cassandra.
Maybe you could elaborate your use case. You really want a one node 
cluster ?

cheers,
Romain

wang liang  a écrit sur 12/10/2012 10:36:15 :

> Hi, Hagos,
> 
> I think it depends on your business case. Big RAM reduce latency and
> improve responsibility, High number of cores increase concurrency of
> your app. thanks.

> On Fri, Oct 12, 2012 at 4:23 PM, Hagos, A.S.  wrote:
> Hi All,
> For of my projects I want to buy a machine to host Casssandra database.
> The options I am offered are machines with 16GB RAM with Quad-Core 
> processor and 6GB RAM with Hexa-Core processor.
> Which one do you recommend, big RAM or  high number of cores?
> 
> greetings
> Ambes
> 

> 
> -- 
> Best wishes,
> Helping others is to help myself.

RE: what is more important (RAM vs Cores)

[Hagos, A.S.]

Hi there,
My application is uses Cassandra to store abstracted sensor data from a sensor 
network in large building (up to 3000 sensors).
For now I am starting one node in one floor of the building, for the future it 
will definitely be a cluster. Some of the sensors have up 16HZ sampling rate.

And now I want to make a decision if I have to focus on big RAM or large number 
of cores.
greetings 
Ambes

From: Romain HARDOUIN [romain.hardo...@urssaf.fr]
Sent: Friday, October 12, 2012 10:57 AM
To: user@cassandra.apache.org
Subject: Re: what is more important (RAM vs Cores)

Hi,

Sure it depends... but IMHO 6 GB is suboptimal for big data because it means 
1,5 GB or  2 GB for Cassandra.
Maybe you could elaborate your use case. You really want a one node cluster ?

cheers,
Romain

wang liang  a écrit sur 12/10/2012 10:36:15 :

> Hi, Hagos,
>
> I think it depends on your business case. Big RAM reduce latency and
> improve responsibility, High number of cores increase concurrency of
> your app. thanks.

> On Fri, Oct 12, 2012 at 4:23 PM, Hagos, A.S.  wrote:
> Hi All,
> For of my projects I want to buy a machine to host Casssandra database.
> The options I am offered are machines with 16GB RAM with Quad-Core
> processor and 6GB RAM with Hexa-Core processor.
> Which one do you recommend, big RAM or  high number of cores?
>
> greetings
> Ambes
>

>
> --
> Best wishes,
> Helping others is to help myself.

RE: what is more important (RAM vs Cores)

[Viktor Jevdokimov]

IMO, in most cases you'll be limited by the RAM first.
Take into account size of sstables, you will need to keep bloom filters and 
indexes in RAM and if it will not fit, 4 cores, or 24 cores doesn't matter, 
except you're on SSD.

You need to design first, stress test second, conclude last.


Best regards / Pagarbiai

Viktor Jevdokimov
Senior Developer

Email: viktor.jevdoki...@adform.com
Phone: +370 5 212 3063
Fax: +370 5 261 0453

J. Jasinskio 16C,
LT-01112 Vilnius,
Lithuania



Disclaimer: The information contained in this message and attachments is 
intended solely for the attention and use of the named addressee and may be 
confidential. If you are not the intended recipient, you are reminded that the 
information remains the property of the sender. You must not use, disclose, 
distribute, copy, print or rely on this e-mail. If you have received this 
message in error, please contact the sender immediately and irrevocably delete 
this message and any copies.> -Original Message-
> From: Hagos, A.S. [mailto:a.s.ha...@tue.nl]
> Sent: Friday, October 12, 2012 12:17
> To: user@cassandra.apache.org
> Subject: RE: what is more important (RAM vs Cores)
>
> Hi there,
> My application is uses Cassandra to store abstracted sensor data from a
> sensor network in large building (up to 3000 sensors).
> For now I am starting one node in one floor of the building, for the future it
> will definitely be a cluster. Some of the sensors have up 16HZ sampling rate.
>
> And now I want to make a decision if I have to focus on big RAM or large
> number of cores.
> greetings
> Ambes
> 
> From: Romain HARDOUIN [romain.hardo...@urssaf.fr]
> Sent: Friday, October 12, 2012 10:57 AM
> To: user@cassandra.apache.org
> Subject: Re: what is more important (RAM vs Cores)
>
> Hi,
>
> Sure it depends... but IMHO 6 GB is suboptimal for big data because it means
> 1,5 GB or  2 GB for Cassandra.
> Maybe you could elaborate your use case. You really want a one node cluster
> ?
>
> cheers,
> Romain
>
> wang liang  a écrit sur 12/10/2012 10:36:15 :
>
> > Hi, Hagos,
> >
> > I think it depends on your business case. Big RAM reduce latency and
> > improve responsibility, High number of cores increase concurrency of
> > your app. thanks.
>
> > On Fri, Oct 12, 2012 at 4:23 PM, Hagos, A.S.  wrote:
> > Hi All,
> > For of my projects I want to buy a machine to host Casssandra database.
> > The options I am offered are machines with 16GB RAM with Quad-Core
> > processor and 6GB RAM with Hexa-Core processor.
> > Which one do you recommend, big RAM or  high number of cores?
> >
> > greetings
> > Ambes
> >
>
> >
> > --
> > Best wishes,
> > Helping others is to help myself.

Super columns and arrays

[Thierry Templier]

Hello,

I wonder if it's possible to specify an array of values as a value of a 
super column... If it's not possible, is there another way to do that?

Thanks very much for your help.

Thierry

RE: what is more important (RAM vs Cores)

[Tim Wintle]

On Fri, 2012-10-12 at 10:20 +, Viktor Jevdokimov wrote:
> IMO, in most cases you'll be limited by the RAM first.

+1 - I've seen our 8-core boxes limited by RAM and inter-rack
networking, but not by CPU (yet).

Tim

RE: what is more important (RAM vs Cores)

[Romain HARDOUIN]

Also, take into account i/o since they are often a limiting factor.

RE: Super columns and arrays

[Viktor Jevdokimov]

struct SuperColumn {
   1: required binary name,
   2: required list columns,
}


Best regards / Pagarbiai

Viktor Jevdokimov
Senior Developer

Email: viktor.jevdoki...@adform.com
Phone: +370 5 212 3063
Fax: +370 5 261 0453

J. Jasinskio 16C,
LT-01112 Vilnius,
Lithuania



Disclaimer: The information contained in this message and attachments is 
intended solely for the attention and use of the named addressee and may be 
confidential. If you are not the intended recipient, you are reminded that the 
information remains the property of the sender. You must not use, disclose, 
distribute, copy, print or rely on this e-mail. If you have received this 
message in error, please contact the sender immediately and irrevocably delete 
this message and any copies.> -Original Message-
> From: Thierry Templier [mailto:thierry.templ...@restlet.com]
> Sent: Friday, October 12, 2012 13:44
> To: user@cassandra.apache.org
> Subject: Super columns and arrays
>
> Hello,
>
> I wonder if it's possible to specify an array of values as a value of a super
> column... If it's not possible, is there another way to do that?
> Thanks very much for your help.
>
> Thierry

Re: what is more important (RAM vs Cores)

[Serge Fonville]

It seems you RAM/core ratio is different between your options.

So if those two are your only options I'd suggest the 16GB option.

Still, the other things
* Concurrency
* Fit in ram
* IO

Are very relevant.

HTH

Kind regards/met vriendelijke groet,

Serge Fonville

http://www.sergefonville.nl

Convince Microsoft!
They need to add TRUNCATE PARTITION in SQL Server
https://connect.microsoft.com/SQLServer/feedback/details/417926/truncate-partition-of-partitioned-table



2012/10/12 Romain HARDOUIN 

>
> Also, take into account i/o since they are often a limiting factor.

Re: Option for ordering columns by timestamp in CF

[Hiller, Dean]

Good, I wasn't the only one confused…I was not sure at all how "make column 
timestamps optional" was related to your question either…..he must have been a 
hurry and misread the question.

From: Ertio Lew mailto:ertio...@gmail.com>>
Reply-To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Date: Friday, October 12, 2012 12:13 AM
To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Subject: Re: Option for ordering columns by timestamp in CF

"Make column timestamps optional"- kidding me, right ?:)  I do understand that 
this wont be possible as then cassandra wont be able to distinguish the latest 
among several copies of same column. I dont mean that. I just want the while 
ordering the columns, Cassandra(in an optional mode per CF) should not look at 
column names(they will exist though but for retrieval purposes not for 
ordering) but instead Cassandra would order the columns by looking at the 
timestamp values(timestamps would exist!). So the change would be just to 
provide a mode in which cassandra, while ordering, uses timestamps instead of 
column names.

On Fri, Oct 12, 2012 at 2:26 AM, Tyler Hobbs 
mailto:ty...@datastax.com>> wrote:
Without thinking too deeply about it, this is basically equivalent to disabling 
timestamps for a column family and using timestamps for column names, though in 
a very indirect (and potentially confusing) manner.  So, if you want to open a 
ticket, I would suggest framing it as "make column timestamps optional".


On Wed, Oct 10, 2012 at 4:44 AM, Ertio Lew 
mailto:ertio...@gmail.com>> wrote:
I think Cassandra should provide an configurable option on per column family 
basis to do columns sorting by time-stamp rather than column names. This would 
be really helpful to maintain time-sorted columns without using up the column 
name as time-stamps which might otherwise be used to store most relevant column 
names useful for retrievals. Very frequently we need to store data sorted in 
time order. Therefore I think this may be a very general requirement & not 
specific to just my use-case alone.

Does it makes sense to create an issue for this ?




On Fri, Mar 25, 2011 at 2:38 AM, aaron morton 
mailto:aa...@thelastpickle.com>> wrote:
If you mean order by the column timestamp (as passed by the client) that it not 
possible.

Can you use your own timestamps as the column name and store them as long 
values ?

Aaron

On 25 Mar 2011, at 09:30, Narendra Sharma wrote:

> Cassandra 0.7.4
> Column names in my CF are of type byte[] but I want to order columns by 
> timestamp. What is the best way to achieve this? Does it make sense for 
> Cassandra to support ordering of columns by timestamp as option for a column 
> family irrespective of the column name type?
>
> Thanks,
> Naren





--
Tyler Hobbs
DataStax

Re: unnecessary tombstone's transmission during repair process

[Hiller, Dean]

+1

I want to see how this plays out as well.  Anyone know the answer?

Dean

From: Alexey Zotov mailto:azo...@griddynamics.com>>
Reply-To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Date: Friday, October 12, 2012 1:33 AM
To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Subject: Re: unnecessary tombstone's transmission during repair process

iff --git a/src/java/org/apache/cassandra/db/DeletedColumn.java 
b/src/java/org/apache/cassandra/db/DeletedColumn.java
index 18faeef..31744f6 100644
--- a/src/java/org/apache/cassandra/db/DeletedColumn.java
+++ b/src/java/org/apache/cassandra/db/DeletedColumn.java
@@ -17,10 +17,13 @@
  */
 package org.apache.cassandra.db;

+import java.io.IOException;
 import java.nio.ByteBuffer;
+import java.security.MessageDigest;

 import org.apache.cassandra.config.CFMetaData;
 import org.apache.cassandra.db.marshal.MarshalException;
+import org.apache.cassandra.io.util.DataOutputBuffer;
 import org.apache.cassandra.utils.Allocator;

read performance plumetted

[Brian Tarbox]

I have a two node cluster hosting a 45 gig dataset.  I periodically have to
read a high fraction (20% or so) of my 'rows', grabbing a few thousand at a
time and then processing them.

This used to result in about 300-500 reads a second which seemed quite
good.  Recently that number has plummeted to 20-50 reads a second.  The
obvious question is what did I change?  I certainly added more
databringing my total load from 38 or so gig to 45 or so gig but its
hard to imagine that causing this problem.  The shape of my data has not
changed and I haven't changed any cassandra configuration.

Running nodetool tpstats I'm for the first time ever seeing entries under
ReadStage Active and Pending  which correlates with slow reads.
Running iostat I'm seeing a significant (10-50%) of iowait where I
previously never saw higher than 1-2%

I ran a full compaction on the relevant CF (which took 3.5 hours) to no
avail.

Any suggestions on where I can look next?

Thanks.

Re: Option for ordering columns by timestamp in CF

[Derek Williams]

You probably already know this but I'm pretty sure it wouldn't be a trivial
change, since to efficiently lookup a column by name requires the columns
to be ordered by name. A separate index would be needed in order to provide
lookup by column name if the row was sorted by timestamp (which is the way
Redis implements it's sorted set).

On Fri, Oct 12, 2012 at 12:13 AM, Ertio Lew  wrote:

> "Make column timestamps optional"- kidding me, right ?:)  I do understand
> that this wont be possible as then cassandra wont be able to distinguish
> the latest among several copies of same column. I dont mean that. I just
> want the while ordering the columns, Cassandra(in an optional mode per CF)
> should not look at column names(they will exist though but for retrieval
> purposes not for ordering) but instead Cassandra would order the columns by
> looking at the timestamp values(timestamps would exist!). So the change
> would be just to provide a mode in which cassandra, while ordering, uses
> timestamps instead of column names.
>
>
> On Fri, Oct 12, 2012 at 2:26 AM, Tyler Hobbs  wrote:
>
>> Without thinking too deeply about it, this is basically equivalent to
>> disabling timestamps for a column family and using timestamps for column
>> names, though in a very indirect (and potentially confusing) manner.  So,
>> if you want to open a ticket, I would suggest framing it as "make column
>> timestamps optional".
>>
>>
>> On Wed, Oct 10, 2012 at 4:44 AM, Ertio Lew  wrote:
>>
>>> I think Cassandra should provide an configurable option on per column
>>> family basis to do columns sorting by time-stamp rather than column names.
>>> This would be really helpful to maintain time-sorted columns without using
>>> up the column name as time-stamps which might otherwise be used to store
>>> most relevant column names useful for retrievals. Very frequently we need
>>> to store data sorted in time order. Therefore I think this may be a very
>>> general requirement & not specific to just my use-case alone.
>>>
>>> Does it makes sense to create an issue for this ?
>>>
>>>
>>>
>>>
>>> On Fri, Mar 25, 2011 at 2:38 AM, aaron morton 
>>> wrote:
>>>
 If you mean order by the column timestamp (as passed by the client)
 that it not possible.

 Can you use your own timestamps as the column name and store them as
 long values ?

 Aaron

 On 25 Mar 2011, at 09:30, Narendra Sharma wrote:

 > Cassandra 0.7.4
 > Column names in my CF are of type byte[] but I want to order columns
 by timestamp. What is the best way to achieve this? Does it make sense for
 Cassandra to support ordering of columns by timestamp as option for a
 column family irrespective of the column name type?
 >
 > Thanks,
 > Naren


>>>
>>
>>
>> --
>> Tyler Hobbs
>> DataStax 
>>
>>
>


-- 
Derek Williams

Re: Repair Failing due to bad network

[David Koblas]

Jim,

Great idea - though it doesn't look like it's in 1.1.3 (which is what 
I'm running).


My lame idea of the morning is that I'm going to just read the whole 
keyspace with QUORUM reads to force read repairs - the unfortunate truth 
is that this is about 2B reads...


--david

On 10/11/12 4:51 PM, Jim Cistaro wrote:

I am not aware of any built-in mechanism for retrying repairs.  I believe
you will have to build that into your process.

As for reducing the time of each repair command to fit in your windows:

If you have multiple reasonable size column families, and are not already
doing this, one approach might be to do repairs on a per cf basis.  This
will break your repairs up into smaller chunks that might fit in the
window.

If you are not doing -pr (primary range), using that on each node causes
the repair command to only repair the primary range on the node (not the
ones it is replicating).

Depending on your version, there is also
https://issues.apache.org/jira/browse/CASSANDRA-3912 which might help you
- but I have no experience using this feature.

jc

On 10/11/12 4:09 PM, "David Koblas"  wrote:


I'm trying to bring up a new Datacenter - while I probably could have
brought things up in another way I've now got a DC that has a ready
Cassandra with keys allocated.  The problem is that I cannot get a
repair to complete due since it appears that some part of my network
decides to restart all connections twice a day (6am and 2pm - ok 5
minutes before).

So when I start a repair job, it usually get's a ways into things before
one of the nodes goes DOWN, then back up.  What I don't see is the
repair restarting, it just stops.

Is there a workaround for this case, or is there something else I could
be doing?

--david

Re: Cassandra nodes loaded unequally

[B. Todd Burruss]

are you connecting to the same node every time?  if so, spread out
your connections across the ring

On Fri, Oct 12, 2012 at 1:22 AM, Alexey Zotov  wrote:
> Hi Ben,
>
> I suggest you to compare amount of queries for each node. May be the problem
> is on the client side.
> Yoy can do that using JMX:
> "org.apache.cassandra.db:type=ColumnFamilies,keyspace= KEYSPACE>,columnfamily=","ReadCount"
> "org.apache.cassandra.db:type=ColumnFamilies,keyspace= KEYSPACE>,columnfamily=","WriteCount"
>
> Also I suggest to check output of "nodetool compactionstats".
>
> --
> Alexey
>
>

Re: what is more important (RAM vs Cores)

[B. Todd Burruss]

i would not worry as much about the single machine specs.  find the
sweet spot on price for CPU and RAM and by that, then scale horizontal
to meet your demand.

but .. if i was pressed for a general statement -choose RAM over CPU

On Fri, Oct 12, 2012 at 4:34 AM, Serge Fonville
 wrote:
> It seems you RAM/core ratio is different between your options.
>
> So if those two are your only options I'd suggest the 16GB option.
>
> Still, the other things
> * Concurrency
> * Fit in ram
> * IO
>
> Are very relevant.
>
> HTH
>
> Kind regards/met vriendelijke groet,
>
> Serge Fonville
>
> http://www.sergefonville.nl
>
> Convince Microsoft!
> They need to add TRUNCATE PARTITION in SQL Server
> https://connect.microsoft.com/SQLServer/feedback/details/417926/truncate-partition-of-partitioned-table
>
>
>
> 2012/10/12 Romain HARDOUIN 
>>
>>
>> Also, take into account i/o since they are often a limiting factor.
>
>

Re: read performance plumetted

[B. Todd Burruss]

did the amount of data finally exceed your per machine RAM capacity?
is it the same 20% each time you read?  or do your periodic reads
eventually work through the entire dataset?

if you are essentially table scanning your data set, and the size
exceeds available RAM, then a degradation like that isn't crazy.  and
this is indicated with your iowait %%


On Fri, Oct 12, 2012 at 6:33 AM, Brian Tarbox  wrote:
> I have a two node cluster hosting a 45 gig dataset.  I periodically have to
> read a high fraction (20% or so) of my 'rows', grabbing a few thousand at a
> time and then processing them.
>
> This used to result in about 300-500 reads a second which seemed quite good.
> Recently that number has plummeted to 20-50 reads a second.  The obvious
> question is what did I change?  I certainly added more databringing my
> total load from 38 or so gig to 45 or so gig but its hard to imagine that
> causing this problem.  The shape of my data has not changed and I haven't
> changed any cassandra configuration.
>
> Running nodetool tpstats I'm for the first time ever seeing entries under
> ReadStage Active and Pending  which correlates with slow reads.
> Running iostat I'm seeing a significant (10-50%) of iowait where I
> previously never saw higher than 1-2%
>
> I ran a full compaction on the relevant CF (which took 3.5 hours) to no
> avail.
>
> Any suggestions on where I can look next?
>
> Thanks.

CQL DELETE does not work

[Sridharan Kuppa]

Hi,   I have created table schema using CQL, and I am able to insert and select 
from that table.  Everything works great but DELETE is not working.  When I 
execute the DELETE statement it throws "Bad Request: Failed parsing statement: 
[COL1, COL2, from TABLE1;] reason NullPointerException: null"  I tried without 
column names and with column names but always throws the same exception.
Thanks,-Sridharan 

Re: Repair Failing due to bad network

[Rob Coli]

https://issues.apache.org/jira/browse/CASSANDRA-3483

Is directly on point for the use case in question, and introduces
"rebuild" concept..

https://issues.apache.org/jira/browse/CASSANDRA-3487
https://issues.apache.org/jira/browse/CASSANDRA-3112

Are for improvements in repair sessions..

https://issues.apache.org/jira/browse/CASSANDRA-4767

Is for unambiguous indication of repair session status.

=Rob

-- 
=Robert Coli
AIM>ALK - rc...@palominodb.com
YAHOO - rcoli.palominob
SKYPE - rcoli_palominodb

Re: READ messages dropped

[Tyler Hobbs]

On Fri, Oct 12, 2012 at 2:24 AM, Tamar Fraenkel  wrote:

>
> Thanks for the response. My cluster is in a bad state those recent days.
>
> I have 29 CFs, and my disk is 5% full... So I guess the VMs still have
> more space to go, and I am not sure this is considered many CFs.
>

That's not too many CFs.  I don't know how much 5% of your disk space is in
absolute numbers, which is more important.  The most important measure for
whether you are approaching limits is really disk utilization (as in how
busy the disk is, not how much data it's holding).  OpsCenter exposes
metrics for this that you should check.


>
> But maybe I have memory issues. I enlarge cassandra memory from about ~2G
> to ~4G (out of ~8G). This was done because at that stage I had lots of key
> caches. I then reduced them to almost 0 on all CF. I guess now I can reduce
> the memory back to ~2 or ~3 G. Will that help?


I would leave your heap at 4G.  You really do want key caching enabled in
almost all circumstances; it can save you a lot of disk activity on reads.
If you need to bump your heap up to 4.5G to accommodate key caches, it's
worth it.

-- 
Tyler Hobbs
DataStax 

Re: cassandra 1.0.8 memory usage

[Tyler Hobbs]

On Fri, Oct 12, 2012 at 3:26 AM, Daniel Woo  wrote:

>
> >> Disable swap for cassandra node
> I am gonna change swappiness to 20%


Dead nodes are better than crippled nodes.  I'll echo Rob's suggestion that
you disable swap entirely.

-- 
Tyler Hobbs
DataStax 

Re: Option for ordering columns by timestamp in CF

[B. Todd Burruss]

trying to think of a use case where you would want to order by
timestamp, and also have unique column names for direct access.

not really trying to challenge the use case, but you can get ordering
by timestamp and still maintain a "name" for the column using
composites. if the first component of the composite is a timestamp,
then you can order on it.  when retrieved you will could have a "name"
in the second component .. and have dupes as long as the timestamp is
unique (use TimeUUID)


On Fri, Oct 12, 2012 at 7:20 AM, Derek Williams  wrote:
> You probably already know this but I'm pretty sure it wouldn't be a trivial
> change, since to efficiently lookup a column by name requires the columns to
> be ordered by name. A separate index would be needed in order to provide
> lookup by column name if the row was sorted by timestamp (which is the way
> Redis implements it's sorted set).
>
>
> On Fri, Oct 12, 2012 at 12:13 AM, Ertio Lew  wrote:
>>
>> "Make column timestamps optional"- kidding me, right ?:)  I do understand
>> that this wont be possible as then cassandra wont be able to distinguish the
>> latest among several copies of same column. I dont mean that. I just want
>> the while ordering the columns, Cassandra(in an optional mode per CF) should
>> not look at column names(they will exist though but for retrieval purposes
>> not for ordering) but instead Cassandra would order the columns by looking
>> at the timestamp values(timestamps would exist!). So the change would be
>> just to provide a mode in which cassandra, while ordering, uses timestamps
>> instead of column names.
>>
>>
>> On Fri, Oct 12, 2012 at 2:26 AM, Tyler Hobbs  wrote:
>>>
>>> Without thinking too deeply about it, this is basically equivalent to
>>> disabling timestamps for a column family and using timestamps for column
>>> names, though in a very indirect (and potentially confusing) manner.  So, if
>>> you want to open a ticket, I would suggest framing it as "make column
>>> timestamps optional".
>>>
>>>
>>> On Wed, Oct 10, 2012 at 4:44 AM, Ertio Lew  wrote:

 I think Cassandra should provide an configurable option on per column
 family basis to do columns sorting by time-stamp rather than column names.
 This would be really helpful to maintain time-sorted columns without using
 up the column name as time-stamps which might otherwise be used to store
 most relevant column names useful for retrievals. Very frequently we need 
 to
 store data sorted in time order. Therefore I think this may be a very
 general requirement & not specific to just my use-case alone.

 Does it makes sense to create an issue for this ?




 On Fri, Mar 25, 2011 at 2:38 AM, aaron morton 
 wrote:
>
> If you mean order by the column timestamp (as passed by the client)
> that it not possible.
>
> Can you use your own timestamps as the column name and store them as
> long values ?
>
> Aaron
>
> On 25 Mar 2011, at 09:30, Narendra Sharma wrote:
>
> > Cassandra 0.7.4
> > Column names in my CF are of type byte[] but I want to order columns
> > by timestamp. What is the best way to achieve this? Does it make sense 
> > for
> > Cassandra to support ordering of columns by timestamp as option for a 
> > column
> > family irrespective of the column name type?
> >
> > Thanks,
> > Naren
>

>>>
>>>
>>>
>>> --
>>> Tyler Hobbs
>>> DataStax
>>>
>>
>
>
>
> --
> Derek Williams
>

Re: cassandra 1.0.8 memory usage

[Rob Coli]

On Fri, Oct 12, 2012 at 1:26 AM, Daniel Woo  wrote:
>>>What version of Cassandra? What JVM? Are JNA and Jamm working?
> cassandra 1.0.8. Sun JDK 1.7.0_05-b06, JNA memlock enabled, jamm works.

The unusual aspect here is Sun JDK 1.7. Can you use 1.6 on an affected
node and see if the problem disappears?

https://issues.apache.org/jira/browse/CASSANDRA-4571

Exists in 1.1.x (not your case) and is for leaking descriptors and not
memory, but affects both 1.6 and 1.7.

> JMAP shows that the per gen is only 40% used.

What is the usage of the other gens?

> I have very few column families, maybe 30-50. The nodetool shows each node
> has 5 GB load.

Most of your heap being consumed by 30-50 columnfamilies MBeans seems excessive.

>>> Disable swap for cassandra node
> I am gonna change swappiness to 20%

Even setting swappiness to 0% does not prevent the kernel from
swapping if swap is defined/enabled. I re-iterate my suggestion that
you de-define/disable swap on any node running Cassandra. :)

=Rob

-- 
=Robert Coli
AIM>ALK - rc...@palominodb.com
YAHOO - rcoli.palominob
SKYPE - rcoli_palominodb

RE: Read latency issue

[Arindam Barua]

We instrumented the Cassandra and Hector code adding more logs to check where 
the time was being spent.
We found the Cassandra read times to be very low, eg. 
CassandraServer.getSlice() is only 3ms.

However, on Hector's side, creating a ColumnFamilyTemplate, 
and doing queryColumns() on it takes 90ms. 
Looking at the breakup on the Hector side, it appears ExecutionResult.execute 
takes ~30ms, and ColumnFamilyResultWrapper takes ~47ms.
(we are reading around 800 composite columns of 1000 bytes each)

Any idea if this is the expected time to process stuff on Hector/other clients?
Btw, using Hector's SliceQuery() and reading into a List, or Astynax seem to 
result in similar times too.

Thanks,
Arindam

-Original Message-
From: Arindam Barua [mailto:aba...@247-inc.com] 
Sent: Wednesday, October 03, 2012 10:54 AM
To: user@cassandra.apache.org
Subject: RE: Read latency issue


Thanks for your responses.

Just to be clear our table declaration looks something like this:
CREATE TABLE sessionevents (
  atag text,
  col2 uuid,
  col3 text,
  col4 uuid,
  col5 text,
  col6 text,
  col7 blob,
  col8 text,
  col9 timestamp,
  col10 uuid,
  col11 int,
  col12 uuid,
  PRIMARY KEY (atag, col2, col3, col4)
)

My understanding was that the (full) row key in this case would be the 'atag' 
values. The column names would then be composites like 
(:::col5), (: : 
:col6), (:::col7) ... 
(: : :col12). The columns would be sorted 
first by col2_values, then by col3 values, etc.

Hence a query like "select * from sessionevents where atag=", we are 
specifying the entire row key, and Cassandra would return all the columns for 
that row.

>> Using read consistency of ONE reduces the read latency by ~20ms, compared to 
>> using QUORUM.
>It would only have read from the local node. (I think, may be confusing 
>secondary index reads here).
For read consistency ONE, reading only from one node is my expectation as well, 
and hence I'm seeing the reduced read latency compared to read consistency 
QUORUM. Does that not sound right?
Btw, with read consistency ONE, we found the reading only happens from one 
node, but not necessarily the local node, even if the data is present in the 
local node. To check this, we turned on DEBUG logs on all the Cassandra hosts 
in the ring. We are using replication factor=3 on a 4 node ring, hence mostly 
the data is present locally. However, we noticed that the coordinator host on 
receiving the same request multiple times (i.e with the same row key) , would 
sometimes return the data locally, but sometimes would contact another host in 
the ring to fetch the data.

Thanks,
Arindam

-Original Message-
From: aaron morton [mailto:aa...@thelastpickle.com]
Sent: Wednesday, October 03, 2012 12:32 AM
To: user@cassandra.apache.org
Subject: Re: Read latency issue

> Running a query to like "select * from  where atag=", where 
> 'atag' is the first column of the composite key, from either JDBC or Hector 
> (equivalent code), results in read times of 200-300ms from a remote host on 
> the same network. 

If you send a query to select columns from a row and do not fully specify the 
row key cassandra has to do a row scan. 

If you want fast performance specify the full row key. 

> Using read consistency of ONE reduces the read latency by ~20ms, compared to 
> using QUORUM.
It would only have read from the local node. (I think, may be confusing 
secondary index reads here).
 
Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 3/10/2012, at 2:17 AM, Roshni Rajagopal  wrote:

> Arindam,
> 
>   Did you also try the cassandra stress tool & compare results?
> 
> I havent done a performance test as yet, the only ones published on 
> the internet are of YCSB on an older version of apache cassandra, and it 
> doesn't seem to be actively supported or updated 
> http://www.brianfrankcooper.net/pubs/ycsb-v4.pdf.
> 
> The numbers you have sound very low, for a read of a row by key which should 
> have been the fastest.  I hope someone can help investigate or share numbers 
> from their tests.
> 
>  
> 
> Regards,
> Roshni
>  
> 
> > From: dean.hil...@nrel.gov
> > To: user@cassandra.apache.org
> > Date: Tue, 2 Oct 2012 06:41:09 -0600
> > Subject: Re: Read latency issue
> > 
> > Interesting results. With PlayOrm, we did a 6 node test of reading 100 rows 
> > from 1,000,000 using PlayOrm Scalable SQL. It only took 60ms. Maybe we have 
> > better hardware though??? We are using 7200 RPM drives so nothing fancy on 
> > the disk side of things. More nodes puts at a higher throughput though as 
> > reading from more disks will be faster. Anyways, you may want to play with 
> > more nodes and re-run. If you run a test with PlayOrm, I would love to know 
> > the results there as well.
> > 
> > Later,
> > Dean
> > 
> > From: Arindam Barua mailto:aba...@247-inc.com>>
> > Reply-To: 
> > "user@cassandra.apache.org" 
> > mailto:user@cass