Reduce Cassandra GC

[Joel Samuelsson]

Hi,

We have a small production cluster with two nodes. The load on the nodes is
very small, around 20 reads / sec and about the same for writes. There are
around 2.5 million keys in the cluster and a RF of 2.

About 2.4 million of the rows are skinny (6 columns) and around 3kb in size
(each). Currently, scripts are running, accessing all of the keys in
timeorder to do some calculations.

While running the scripts, the nodes go down and then come back up 6-7
minutes later. This seems to be due to GC. I get lines like this in the log:
INFO [ScheduledTasks:1] 2013-04-15 14:00:02,749 GCInspector.java (line 122)
GC for ParNew: 338798 ms for 1 collections, 592212416 used; max is
1046937600

However, the heap is not full. The heap usage has a jagged pattern going
from 60% up to 70% during 5 minutes and then back down to 60% the next 5
minutes and so on. I get no "Heap is X full..." messages. Every once in a
while at one of these peaks, I get these stop-the-world GC for 6-7
minutes. Why does GC take up so much time even though the heap isn't full?

I am aware that my access patterns make key caching very unlikely to be
high. And indeed, my average key cache hit ratio during the run of the
scripts is around 0.5%. I tried disabling key caching on the accessed
column family (UPDATE COLUMN FAMILY cf WITH caching=none;) through the
cassandra-cli but I get the same behaviour. Is the turning key cache off
effective immediately?

Stop-the-world GC is fine if it happens for a few seconds but having them
for several minutes doesn't work. Any other suggestions to remove them?

Best regards,
Joel Samuelsson

C* consumes all RAM

[Mikhail Mazursky]

Hello.

C* have been running without any problem for some weeks but now it started
to consume all available ram.

The cluster have very little data in it. There are no errors in logs, CPU
is not loaded at all, jstack shows no deadlocks, there are 83 threads.
Read/write latency is 1-4ms.

The question is: how to find out where is the RAM used?

# nodetool ring keyspace name
Address DC  RackStatus State   Load
Effective-Ownership Token

114967934986393518082067045023985443411
10.0.1.220  datacenter1 rack1   Up Normal  889.71 KB
100.00% 0
10.0.1.222  datacenter1 rack1   Up Normal  918.92 KB
100.00% 56662868904138943096229641828625627278
10.0.1.221  datacenter1 rack1   Up Normal  1.04 MB
100.00% 114967934986393518082067045023985443411

We use AWS m1.medium - 3.75 RAM, 1 Core

java version "1.7.0_10"
Java(TM) SE Runtime Environment (build 1.7.0_10-b18)
Java HotSpot(TM) 64-Bit Server VM (build 23.6-b04, mixed mode)

C* is started with the following command
/usr/java/latest/bin/java -ea
-javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+UseThreadPriorities
-XX:ThreadPriorityPolicy=42 -Xms1G -Xmx1G -Xmn512M
-XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:+UseParNewGC

Thanks in advance.

Mikhail.

Re: C* consumes all RAM

[Mikhail Mazursky]

More details:

USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
219   3801  0.7 92.7 6561116 3567016 ? SLl  Mar11 372:44
/usr/java/latest/bin/java

Linux XXX@YYY 3.2.30-49.59.amzn1.x86_64 #1 SMP Wed Oct 3 19:54:33 UTC 2012
x86_64 x86_64 x86_64 GNU/Linux

total   used   free sharedbuffers cached
Mem:  3754   3695 59  0 37 75
-/+ buffers/cache:   3582171
Swap: 2055165   1890



2013/4/16 Mikhail Mazursky 

> Hello.
>
> C* have been running without any problem for some weeks but now it started
> to consume all available ram.
>
> The cluster have very little data in it. There are no errors in logs, CPU
> is not loaded at all, jstack shows no deadlocks, there are 83 threads.
> Read/write latency is 1-4ms.
>
> The question is: how to find out where is the RAM used?
>
> # nodetool ring keyspace name
> Address DC  RackStatus State   Load
> Effective-Ownership Token
>
> 114967934986393518082067045023985443411
> 10.0.1.220  datacenter1 rack1   Up Normal  889.71 KB
> 100.00% 0
> 10.0.1.222  datacenter1 rack1   Up Normal  918.92 KB
> 100.00% 56662868904138943096229641828625627278
> 10.0.1.221  datacenter1 rack1   Up Normal  1.04 MB
> 100.00% 114967934986393518082067045023985443411
>
> We use AWS m1.medium - 3.75 RAM, 1 Core
>
> java version "1.7.0_10"
> Java(TM) SE Runtime Environment (build 1.7.0_10-b18)
> Java HotSpot(TM) 64-Bit Server VM (build 23.6-b04, mixed mode)
>
> C* is started with the following command
> /usr/java/latest/bin/java -ea
> -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+UseThreadPriorities
> -XX:ThreadPriorityPolicy=42 -Xms1G -Xmx1G -Xmn512M
> -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:+UseParNewGC
>
> Thanks in advance.
>
> Mikhail.
>

RE: Reduce Cassandra GC

[Viktor Jevdokimov]

For a >40GB of data 1GB of heap is too low.

Best regards / Pagarbiai
Viktor Jevdokimov
Senior Developer

Email: viktor.jevdoki...@adform.com
Phone: +370 5 212 3063, Fax +370 5 261 0453
J. Jasinskio 16C, LT-01112 Vilnius, Lithuania
Follow us on Twitter: @adforminsider
Take a ride with Adform's Rich Media Suite

[Adform News] 
[Adform awarded the Best Employer 2012] 



Disclaimer: The information contained in this message and attachments is 
intended solely for the attention and use of the named addressee and may be 
confidential. If you are not the intended recipient, you are reminded that the 
information remains the property of the sender. You must not use, disclose, 
distribute, copy, print or rely on this e-mail. If you have received this 
message in error, please contact the sender immediately and irrevocably delete 
this message and any copies.

From: Joel Samuelsson [mailto:samuelsson.j...@gmail.com]
Sent: Tuesday, April 16, 2013 10:47
To: user@cassandra.apache.org
Subject: Reduce Cassandra GC

Hi,

We have a small production cluster with two nodes. The load on the nodes is 
very small, around 20 reads / sec and about the same for writes. There are 
around 2.5 million keys in the cluster and a RF of 2.

About 2.4 million of the rows are skinny (6 columns) and around 3kb in size 
(each). Currently, scripts are running, accessing all of the keys in timeorder 
to do some calculations.

While running the scripts, the nodes go down and then come back up 6-7 minutes 
later. This seems to be due to GC. I get lines like this in the log:
INFO [ScheduledTasks:1] 2013-04-15 14:00:02,749 GCInspector.java (line 122) GC 
for ParNew: 338798 ms for 1 collections, 592212416 used; max is 1046937600

However, the heap is not full. The heap usage has a jagged pattern going from 
60% up to 70% during 5 minutes and then back down to 60% the next 5 minutes and 
so on. I get no "Heap is X full..." messages. Every once in a while at one of 
these peaks, I get these stop-the-world GC for 6-7 minutes. Why does GC take up 
so much time even though the heap isn't full?

I am aware that my access patterns make key caching very unlikely to be high. 
And indeed, my average key cache hit ratio during the run of the scripts is 
around 0.5%. I tried disabling key caching on the accessed column family 
(UPDATE COLUMN FAMILY cf WITH caching=none;) through the cassandra-cli but I 
get the same behaviour. Is the turning key cache off effective immediately?

Stop-the-world GC is fine if it happens for a few seconds but having them for 
several minutes doesn't work. Any other suggestions to remove them?

Best regards,
Joel Samuelsson
<><>

Configuring 2 node Cassandra cluster

[Sai Kumar Ganji]

Hello Guys,

I am trying to setup a 2 node Cassandra clustesr.

My parameters are:

*Node1(ip1):*

initial_token: 0
rpc_address: ip1
listen_address: ip1
seeds: “ip1"

*Node2(ip2):*

initial_token: 85070591730234615865843651857942052864
rpc_address: ip1
listen_address: ip1
seeds: “ip1"

and in Node2 *auto_bootstrap: true;*
*
*
Then when i try to run:* **/usr/lib/apache-cassandra/bin/nodetool -host
localhost ring*
*
*
I get this error:

*Failed to connect to 'localhost:7199': Connection refused ; * on both
nodes.

Can you help here please.

Also when I create a key space and a column family in Node1, how can I
access this data from Node2 ?



-- 
Thanks & Regards

Venkata Sai Ganji
Graduate Student
Dept of Computer Science
Montana State University - Bzn

Re: Configuring 2 node Cassandra cluster

[Alain RODRIGUEZ]

Could we have the logs after starting each node ?


2013/4/16 Sai Kumar Ganji 

> Hello Guys,
>
> I am trying to setup a 2 node Cassandra clustesr.
>
> My parameters are:
>
> *Node1(ip1):*
>
> initial_token: 0
> rpc_address: ip1
> listen_address: ip1
> seeds: “ip1"
>
> *Node2(ip2):*
>
> initial_token: 85070591730234615865843651857942052864
> rpc_address: ip1
> listen_address: ip1
> seeds: “ip1"
>
> and in Node2 *auto_bootstrap: true;*
> *
> *
> Then when i try to run:* **/usr/lib/apache-cassandra/bin/nodetool -host
> localhost ring*
> *
> *
> I get this error:
>
> *Failed to connect to 'localhost:7199': Connection refused ; * on both
> nodes.
>
> Can you help here please.
>
> Also when I create a key space and a column family in Node1, how can I
> access this data from Node2 ?
>
>
>
> --
> Thanks & Regards
>
> Venkata Sai Ganji
> Graduate Student
> Dept of Computer Science
> Montana State University - Bzn
>

Re: Reduce Cassandra GC

[Joel Samuelsson]

How do you calculate the heap / data size ratio? Is this a linear ratio?

Each node has slightly more than 12 GB right now though.


2013/4/16 Viktor Jevdokimov 

>  For a >40GB of data 1GB of heap is too low.
>
> ** **
>Best regards / Pagarbiai
> *Viktor Jevdokimov*
> Senior Developer
>
> Email: viktor.jevdoki...@adform.com
> Phone: +370 5 212 3063, Fax +370 5 261 0453
> J. Jasinskio 16C, LT-01112 Vilnius, Lithuania
> Follow us on Twitter: @adforminsider 
> Take a ride with Adform's Rich Media Suite
>  [image: Adform News] 
> [image: Adform awarded the Best Employer 2012]
> 
>
> Disclaimer: The information contained in this message and attachments is
> intended solely for the attention and use of the named addressee and may be
> confidential. If you are not the intended recipient, you are reminded that
> the information remains the property of the sender. You must not use,
> disclose, distribute, copy, print or rely on this e-mail. If you have
> received this message in error, please contact the sender immediately and
> irrevocably delete this message and any copies.
>
>   *From:* Joel Samuelsson [mailto:samuelsson.j...@gmail.com]
> *Sent:* Tuesday, April 16, 2013 10:47
> *To:* user@cassandra.apache.org
> *Subject:* Reduce Cassandra GC
>
> ** **
>
> Hi,
>
> ** **
>
> We have a small production cluster with two nodes. The load on the nodes
> is very small, around 20 reads / sec and about the same for writes. There
> are around 2.5 million keys in the cluster and a RF of 2.
>
> ** **
>
> About 2.4 million of the rows are skinny (6 columns) and around 3kb in
> size (each). Currently, scripts are running, accessing all of the keys in
> timeorder to do some calculations.
>
> ** **
>
> While running the scripts, the nodes go down and then come back up 6-7
> minutes later. This seems to be due to GC. I get lines like this in the log:
> 
>
> INFO [ScheduledTasks:1] 2013-04-15 14:00:02,749 GCInspector.java (line
> 122) GC for ParNew: 338798 ms for 1 collections, 592212416 used; max is
> 1046937600
>
> ** **
>
> However, the heap is not full. The heap usage has a jagged pattern going
> from 60% up to 70% during 5 minutes and then back down to 60% the next 5
> minutes and so on. I get no "Heap is X full..." messages. Every once in a
> while at one of these peaks, I get these stop-the-world GC for 6-7
> minutes. Why does GC take up so much time even though the heap isn't full?
> 
>
> ** **
>
> I am aware that my access patterns make key caching very unlikely to be
> high. And indeed, my average key cache hit ratio during the run of the
> scripts is around 0.5%. I tried disabling key caching on the accessed
> column family (UPDATE COLUMN FAMILY cf WITH caching=none;) through the
> cassandra-cli but I get the same behaviour. Is the turning key cache off
> effective immediately?
>
> ** **
>
> Stop-the-world GC is fine if it happens for a few seconds but having them
> for several minutes doesn't work. Any other suggestions to remove them?***
> *
>
> ** **
>
> Best regards,
>
> Joel Samuelsson
>
<><>

MySQL Cluster performing faster than Cassandra cluster on single table

[jrdn hannah]

Hi,

I was wondering if anybody here had any insight into this.

I was running some tests on cassandra and mysql performance, with a two node 
and three node cassandra cluster, and a five node mysql cluster (mgmt, 2 x api, 
2 x data).

On the cassandra 2 node cluster vs mysql cluster, I was getting a couple of 
strange results. For example, on updating a single table in MySQL, with the 
equivalent super column in Cassandra, I was getting results of 0.231 ms for 
MySQL and 1.248ms for Cassandra to perform the update 1000 times.

Could anybody help explain why this is the case?

Thanks,
Hannah

Re: MySQL Cluster performing faster than Cassandra cluster on single table

[horschi]

Hi Hannah,

mysql-cluster is a in-memory database.

In-memory is fast. But I dont think you ever be able to store hundreds of
Gigabytes of data on a node, which is something you can do with Cassandra.

If your dataset is small, then maybe NDB is the better choice for you. I
myself will not even touch it with stick any more, I hate it with a
passion. But this might depend on the use-case :-)

regards,
Christian

On Tue, Apr 16, 2013 at 12:56 PM, jrdn hannah  wrote:

> Hi,
>
> I was wondering if anybody here had any insight into this.
>
> I was running some tests on cassandra and mysql performance, with a two
> node and three node cassandra cluster, and a five node mysql cluster (mgmt,
> 2 x api, 2 x data).
>
> On the cassandra 2 node cluster vs mysql cluster, I was getting a couple
> of strange results. For example, on updating a single table in MySQL, with
> the equivalent super column in Cassandra, I was getting results of 0.231 ms
> for MySQL and 1.248ms for Cassandra to perform the update 1000 times.
>
> Could anybody help explain why this is the case?
>
> Thanks,
> Hannah

Re: Lost data after expanding cluster c* 1.2.3-1

[Kais Ahmed]

Thanks aaron,

I feel that rebuilding indexes went well, but the result of my query
(SELECT * FROM userdata WHERE login='kais';) is still emty.

INFO [Creating index: userdata.userdata_login_idx] 2013-03-30 01:16:33,110
SecondaryIndex.java (line 175) Submitting index build of
userdata.userdata_login_idx
INFO [Creating index: userdata.userdata_login_idx] 2013-03-30 01:34:11,667
SecondaryIndex.java (line 202) Index build of userdata.userdata_login_idx
complete

Thanks,


2013/4/9 aaron morton 

> Look in the logs for messages from the SecondaryIndexManager
>
> starts with "Submitting index build of"
> end with "Index build of"
>
> Cheers
>
> -
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 7/04/2013, at 12:55 AM, Kais Ahmed  wrote:
>
> hi aaron,
>
> nodetool compactionstats on all nodes return 1 pending task :
>
> ubuntu@app:~$ nodetool compactionstats host
> pending tasks: 1
> Active compaction remaining time :n/a
>
> The command nodetool rebuild_index was launched several days ago.
>
> 2013/4/5 aaron morton 
>
>> but nothing's happening, how can i monitor the progress? and how can i
>> know when it's finished?
>>
>>
>> check nodetool compacitonstats
>>
>> Cheers
>>
>>-
>> Aaron Morton
>> Freelance Cassandra Consultant
>> New Zealand
>>
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 4/04/2013, at 2:51 PM, Kais Ahmed  wrote:
>>
>> Hi aaron,
>>
>> I ran the command "nodetool rebuild_index host keyspace cf" on all the
>> nodes, in the log i see :
>>
>> INFO [RMI TCP Connection(5422)-10.34.139.xxx] 2013-04-04 08:31:53,641
>> ColumnFamilyStore.java (line 558) User Requested secondary index re-build
>> for ...
>>
>> but nothing's happening, how can i monitor the progress? and how can i
>> know when it's finished?
>>
>> Thanks,
>>
>>
>> 2013/4/2 aaron morton 
>>
>>> The problem come from that i don't put  auto_boostrap to true for the
>>> new nodes, not in this documentation (
>>> http://www.datastax.com/docs/1.2/install/expand_ami)
>>>
>>> auto_bootstrap defaults to True if not specified in the yaml.
>>>
>>> can i do that at any time, or when the cluster are not loaded
>>>
>>> Not sure what the question is.
>>> Both those operations are online operations you can do while the node is
>>> processing requests.
>>>
>>> Cheers
>>>
>>>-
>>> Aaron Morton
>>> Freelance Cassandra Consultant
>>> New Zealand
>>>
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>>
>>> On 1/04/2013, at 9:26 PM, Kais Ahmed  wrote:
>>>
>>> > At this moment the errors started, we see that members and other data
>>> are gone, at this moment the nodetool status return (in red color the 3 new
>>> nodes)
>>> > What errors?
>>> The errors was in my side in the application, not cassandra errors
>>>
>>> > I put for each of them seeds = A ip, and start each with two minutes
>>> intervals.
>>> > When I'm making changes I tend to change a single node first, confirm
>>> everything is OK and then do a bulk change.
>>> Thank you for that advice.
>>>
>>> >I'm not sure what or why it went wrong, but that should get you to a
>>> stable place. If you have any problems keep an eye on the logs for errors
>>> or warnings.
>>> The problem come from that i don't put  auto_boostrap to true for the
>>> new nodes, not in this documentation (
>>> http://www.datastax.com/docs/1.2/install/expand_ami)
>>>
>>> >if you are using secondary indexes use nodetool rebuild_index to
>>> rebuild those.
>>> can i do that at any time, or when the cluster are not loaded
>>>
>>> Thanks aaron,
>>>
>>> 2013/4/1 aaron morton 
>>>
 Please do not rely on colour in your emails, the best way to get your
 emails accepted by the Apache mail servers is to use plain text.

 > At this moment the errors started, we see that members and other data
 are gone, at this moment the nodetool status return (in red color the 3 new
 nodes)
 What errors?

 > I put for each of them seeds = A ip, and start each with two minutes
 intervals.
 When I'm making changes I tend to change a single node first, confirm
 everything is OK and then do a bulk change.

 > Now the cluster seem to work normally, but i can use the secondary
 for the moment, the queryanswer are random
 run nodetool repair -pr on each node, let it finish before starting the
 next one.
 if you are using secondary indexes use nodetool rebuild_index to
 rebuild those.
 Add one node new node to the cluster and confirm everything is ok, then
 add the remaining ones.

 >I'm not sure what or why it went wrong, but that should get you to a
 stable place. If you have any problems keep an eye on the logs for errors
 or warnings.

 Cheers

 -
 Aaron Morton
 Freelance Cassandra Consultant
 New Zealand

 @aaronmorton
 http://www.thelastpickle.com

 On 31/03/

Re: Does Memtable resides in Heap?

[Jay Svc]

Thanks Edward.!



On Fri, Apr 12, 2013 at 9:46 AM, Edward Capriolo wrote:

> This issue describes the design of the arena allocation of memtabes.
> https://issues.apache.org/jira/browse/CASSANDRA-2252
>
>
> On Fri, Apr 12, 2013 at 1:35 AM, Viktor Jevdokimov <
> viktor.jevdoki...@adform.com> wrote:
>
>> Memtables resides in heap, write rate impacts GC, more writes - more
>> frequent and longer ParNew GC pauses.
>>
>>
>> From: Jay Svc [mailto:jaytechg...@gmail.com]
>> Sent: Friday, April 12, 2013 01:03
>> To: user@cassandra.apache.org
>> Subject: Does Memtable resides in Heap?
>>
>> Hi Team,
>>
>> I have got this 8GB of RAM out of that 4GB allocated to Java Heap. My
>> question is the size of Memtable does it contribute to heap size? or they
>> are part of off-heap?
>>
>> Does bigger Memtable would have impact on GC and overall memory
>> management?
>>
>> I am using DSE 3.0 / Cassandra 1.1.9.
>>
>> Thanks,
>> Jay
>>
>> Best regards / Pagarbiai
>>
>> Viktor Jevdokimov
>> Senior Developer
>>
>> Email: viktor.jevdoki...@adform.com
>> Phone: +370 5 212 3063
>> Fax: +370 5 261 0453
>>
>> J. Jasinskio 16C,
>> LT-01112 Vilnius,
>> Lithuania
>>
>>
>>
>> Disclaimer: The information contained in this message and attachments is
>> intended solely for the attention and use of the named addressee and may be
>> confidential. If you are not the intended recipient, you are reminded that
>> the information remains the property of the sender. You must not use,
>> disclose, distribute, copy, print or rely on this e-mail. If you have
>> received this message in error, please contact the sender immediately and
>> irrevocably delete this message and any copies.
>>
>
>

Re: Configuring 2 node Cassandra cluster

[Alicia Leong]

*Node2(ip2):*

initial_token: 85070591730234615865843651857942052864
rpc_address: ip*1*   < Should be  *ip2*
listen_address: ip*1*< Should be  *ip2*
seeds: “ip1"


On Tue, Apr 16, 2013 at 5:43 PM, Alain RODRIGUEZ  wrote:

> Could we have the logs after starting each node ?
>
>
> 2013/4/16 Sai Kumar Ganji 
>
>> Hello Guys,
>>
>> I am trying to setup a 2 node Cassandra clustesr.
>>
>> My parameters are:
>>
>> *Node1(ip1):*
>>
>> initial_token: 0
>> rpc_address: ip1
>> listen_address: ip1
>> seeds: “ip1"
>>
>> *Node2(ip2):*
>>
>> initial_token: 85070591730234615865843651857942052864
>> rpc_address: ip1
>> listen_address: ip1
>> seeds: “ip1"
>>
>> and in Node2 *auto_bootstrap: true;*
>> *
>> *
>> Then when i try to run:* **/usr/lib/apache-cassandra/bin/nodetool -host
>> localhost ring*
>> *
>> *
>> I get this error:
>>
>> *Failed to connect to 'localhost:7199': Connection refused ; * on both
>> nodes.
>>
>> Can you help here please.
>>
>> Also when I create a key space and a column family in Node1, how can I
>> access this data from Node2 ?
>>
>>
>>
>> --
>> Thanks & Regards
>>
>> Venkata Sai Ganji
>> Graduate Student
>> Dept of Computer Science
>> Montana State University - Bzn
>>
>
>

Re: Configuring 2 node Cassandra cluster

[Edward Capriolo]

If you are using a two node cassandra cluster locally use ccm, it builds
all the configuration files for you.

https://github.com/pcmanus/ccm


On Tue, Apr 16, 2013 at 11:06 AM, Alicia Leong  wrote:

>
>
> *Node2(ip2):*
>
> initial_token: 85070591730234615865843651857942052864
> rpc_address: ip*1*   < Should be  *ip2*
> listen_address: ip*1*< Should be  *ip2*
> seeds: “ip1"
>
>
> On Tue, Apr 16, 2013 at 5:43 PM, Alain RODRIGUEZ wrote:
>
>> Could we have the logs after starting each node ?
>>
>>
>> 2013/4/16 Sai Kumar Ganji 
>>
>>> Hello Guys,
>>>
>>> I am trying to setup a 2 node Cassandra clustesr.
>>>
>>> My parameters are:
>>>
>>> *Node1(ip1):*
>>>
>>> initial_token: 0
>>> rpc_address: ip1
>>> listen_address: ip1
>>> seeds: “ip1"
>>>
>>> *Node2(ip2):*
>>>
>>> initial_token: 85070591730234615865843651857942052864
>>> rpc_address: ip1
>>> listen_address: ip1
>>> seeds: “ip1"
>>>
>>> and in Node2 *auto_bootstrap: true;*
>>> *
>>> *
>>> Then when i try to run:* **/usr/lib/apache-cassandra/bin/nodetool -host
>>> localhost ring*
>>> *
>>> *
>>> I get this error:
>>>
>>> *Failed to connect to 'localhost:7199': Connection refused ; * on both
>>> nodes.
>>>
>>> Can you help here please.
>>>
>>> Also when I create a key space and a column family in Node1, how can I
>>> access this data from Node2 ?
>>>
>>>
>>>
>>> --
>>> Thanks & Regards
>>>
>>> Venkata Sai Ganji
>>> Graduate Student
>>> Dept of Computer Science
>>> Montana State University - Bzn
>>>
>>
>>
>

Re: MySQL Cluster performing faster than Cassandra cluster on single table

[jrdn hannah]

Ah, I see, that makes sense. Have you got a source for the storing of hundreds 
of gigabytes? And does Cassandra not store anything in memory?

Yeah, my dataset is small at the moment - perhaps I should have chosen 
something larger for the work I'm doing (University dissertation), however, it 
is far too late to change now!

Thanks for the help,
Hannah

On 16 Apr 2013, at 12:16, horschi  wrote:

> Hi Hannah,
> 
> mysql-cluster is a in-memory database.
> 
> In-memory is fast. But I dont think you ever be able to store hundreds of 
> Gigabytes of data on a node, which is something you can do with Cassandra.
> 
> If your dataset is small, then maybe NDB is the better choice for you. I 
> myself will not even touch it with stick any more, I hate it with a passion. 
> But this might depend on the use-case :-)
> 
> regards,
> Christian
> 
> On Tue, Apr 16, 2013 at 12:56 PM, jrdn hannah  wrote:
> Hi,
> 
> I was wondering if anybody here had any insight into this.
> 
> I was running some tests on cassandra and mysql performance, with a two node 
> and three node cassandra cluster, and a five node mysql cluster (mgmt, 2 x 
> api, 2 x data).
> 
> On the cassandra 2 node cluster vs mysql cluster, I was getting a couple of 
> strange results. For example, on updating a single table in MySQL, with the 
> equivalent super column in Cassandra, I was getting results of 0.231 ms for 
> MySQL and 1.248ms for Cassandra to perform the update 1000 times.
> 
> Could anybody help explain why this is the case?
> 
> Thanks,
> Hannah
> 

Re: MySQL Cluster performing faster than Cassandra cluster on single table

[jrdn hannah]

Yeah, I remember reading about that, but the schema had already been set and 
submitted. I will have to take that into consideration when discussing the 
results.

Thanks,
Hannah
On 16 Apr 2013, at 17:42, Robert Coli  wrote:

> On Tue, Apr 16, 2013 at 3:56 AM, jrdn hannah  wrote:
> For example, on updating a single table in MySQL, with the equivalent super 
> column in Cassandra, I was getting results of 0.231 ms for MySQL and 1.248ms 
> for Cassandra to perform the update 1000 times.
> 
> You probably do not want to use a Supercolumn. They have been in a state of 
> quasi-supported semi-deprecation for years now...
> 
> =Rob
>  

Re: MySQL Cluster performing faster than Cassandra cluster on single table

[Robert Coli]

On Tue, Apr 16, 2013 at 3:56 AM, jrdn hannah  wrote:

> For example, on updating a single table in MySQL, with the equivalent
> super column in Cassandra, I was getting results of 0.231 ms for MySQL and
> 1.248ms for Cassandra to perform the update 1000 times.
>

You probably do not want to use a Supercolumn. They have been in a state of
quasi-supported semi-deprecation for years now...

=Rob

Re: MySQL Cluster performing faster than Cassandra cluster on single table

[horschi]

Ah, I see, that makes sense. Have you got a source for the storing of
> hundreds of gigabytes? And does Cassandra not store anything in memory?
>
It stores bloom filters and index-samples in memory. But they are much
smaller than the actual data and they can be configured.


>
> Yeah, my dataset is small at the moment - perhaps I should have chosen
> something larger for the work I'm doing (University dissertation), however,
> it is far too late to change now!
>
On paper mysql-cluster looks great. But in daily use its not as nice as
Cassandra (where you have machines dying, networks splitting, etc.).

cheers,
Christian

RE: Reduce Cassandra GC

[Viktor Jevdokimov]

How one could provide any help without any knowledge about your cluster, node 
and environment settings?

40GB was calculated from 2 nodes with RF=2 (each has 100% data range), 2.4-2.5M 
rows * 6 cols * 3kB as a minimum without compression and any overhead (sstable, 
bloom filters and indexes).

With ParNew GC time such as yours even if it is a swapping issue I could say 
only that heap size is too small.

Check Heap, New Heap sizes, memtable and cache sizes. Are you on Linux? Is JNA 
installed and used? What is total amount of RAM?

Just for a DEV environment we use 3 virtual machines with 4GB RAM and use 2GB 
heap without any GC issue with amount of data from 0 to 16GB compressed on each 
node. Memtable space sized to 100MB, New Heap 400MB.

Best regards / Pagarbiai
Viktor Jevdokimov
Senior Developer

Email: viktor.jevdoki...@adform.com
Phone: +370 5 212 3063, Fax +370 5 261 0453
J. Jasinskio 16C, LT-01112 Vilnius, Lithuania
Follow us on Twitter: @adforminsider
Take a ride with Adform's Rich Media Suite

[Adform News] 
[Adform awarded the Best Employer 2012] 



Disclaimer: The information contained in this message and attachments is 
intended solely for the attention and use of the named addressee and may be 
confidential. If you are not the intended recipient, you are reminded that the 
information remains the property of the sender. You must not use, disclose, 
distribute, copy, print or rely on this e-mail. If you have received this 
message in error, please contact the sender immediately and irrevocably delete 
this message and any copies.

From: Joel Samuelsson [mailto:samuelsson.j...@gmail.com]
Sent: Tuesday, April 16, 2013 12:52
To: user@cassandra.apache.org
Subject: Re: Reduce Cassandra GC

How do you calculate the heap / data size ratio? Is this a linear ratio?

Each node has slightly more than 12 GB right now though.

2013/4/16 Viktor Jevdokimov 
mailto:viktor.jevdoki...@adform.com>>
For a >40GB of data 1GB of heap is too low.

Best regards / Pagarbiai
Viktor Jevdokimov
Senior Developer

Email: viktor.jevdoki...@adform.com
Phone: +370 5 212 3063, Fax +370 5 261 
0453
J. Jasinskio 16C, LT-01112 Vilnius, Lithuania
Follow us on Twitter: @adforminsider
Take a ride with Adform's Rich Media Suite

[Adform News]
[Adform awarded the Best Employer 
2012]


Disclaimer: The information contained in this message and attachments is 
intended solely for the attention and use of the named addressee and may be 
confidential. If you are not the intended recipient, you are reminded that the 
information remains the property of the sender. You must not use, disclose, 
distribute, copy, print or rely on this e-mail. If you have received this 
message in error, please contact the sender immediately and irrevocably delete 
this message and any copies.

From: Joel Samuelsson 
[mailto:samuelsson.j...@gmail.com]
Sent: Tuesday, April 16, 2013 10:47
To: user@cassandra.apache.org
Subject: Reduce Cassandra GC

Hi,

We have a small production cluster with two nodes. The load on the nodes is 
very small, around 20 reads / sec and about the same for writes. There are 
around 2.5 million keys in the cluster and a RF of 2.

About 2.4 million of the rows are skinny (6 columns) and around 3kb in size 
(each). Currently, scripts are running, accessing all of the keys in timeorder 
to do some calculations.

While running the scripts, the nodes go down and then come back up 6-7 minutes 
later. This seems to be due to GC. I get lines like this in the log:
INFO [ScheduledTasks:1] 2013-04-15 14:00:02,749 GCInspector.java (line 122) GC 
for ParNew: 338798 ms for 1 collections, 592212416 used; max is 1046937600

However, the heap is not full. The heap usage has a jagged pattern going from 
60% up to 70% during 5 minutes and then back down to 60% the next 5 minutes and 
so on. I get no "Heap is X full..." messages. Every once in a while at one of 
these peaks, I get these stop-the-world GC for 6-7 minutes. Why does GC take up 
so much time even though the heap isn't full?

I am aware that my access patterns make key caching very unlikely to be high. 
And indeed, my average key cache hit ratio during the run of the scripts is 
around 0.5%. I tried disabling key caching on the accessed column family 
(UPDATE COLUMN FAMILY cf WITH caching=none;) through the cassandra-cli but I 
get the same behaviour. Is the turning key cache off effective immediately?

Stop-the-world GC is fine if it happens for a few seconds but having them for 
several minut

Re: 1.1.9 to 1.2.3 upgrade issue

[aaron morton]

> Is this a known issue? Or rolling upgrade form 1.1.x to 1.2.x not possible?
Definitely supported. 

The error is from the coordinator processing the response from a replica. The 
size of the digest in the response has been miss-reported.

Was this only reported on the 1.1.9 nodes ? 
Did you complete the upgrade ? 

I see you created https://issues.apache.org/jira/browse/CASSANDRA-5476 thanks. 
I cannot see any resolved issues in the 1.2 branch that would cover this. 

Cheers



-
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 15/04/2013, at 9:20 AM, John Watson  wrote:

> Started doing a rolling upgrade of nodes from 1.1.9 to 1.2.3 and nodes on 
> 1.1.9 started flooding this error:
> 
> Exception in thread Thread[RequestResponseStage:19496,5,main]
> java.io.IOError: java.io.EOFException
> at 
> org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
> at 
> org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:155)
> at 
> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at 
> org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:100)
> at 
> org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:81)
> at 
> org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64)
> ... 6 more
> 
> As I understand the Hints CF changed from 1.1.x to 1.2.x so I assume that's 
> the cause of the 1.2.3 nodes flooding (for various IPs still being 1.1.9):
> 
> Unable to store hint for host with missing ID, /10.37.62.71 (old node?)
> 
> Is this a known issue? Or rolling upgrade form 1.1.x to 1.2.x not possible?
> 
> Thanks,
> 
> John

Re: re-execution of failed queries with rpc_timeout

[aaron morton]

If you are using Counters you need to do everything you can to avoid timeouts. 
In the worse case we do not know where it has been applied. The increment is 
applied on a lead and then replicated to the others, if the coordinator is not  
the lead it may not know if the increments was applied at all. 

Start by reducing the size of the updates. Larger batches do not always mean 
better performance. 

>  In all other cases, the rpc_timeout might be thrown from a remote node (not 
> the one I'm connected to), and hence some parts of the update will be 
> performed and others parts will not.
TimedOutException is always thrown from the coordinator you are connected to. 

Cheers

-
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 15/04/2013, at 1:38 PM, Moty Kosharovsky  wrote:

> Sorry, not LOCAL QUORUM, I meant "ANY" quorum.
> 
> 
> On Mon, Apr 15, 2013 at 4:12 AM, Moty Kosharovsky  wrote:
> Hello,
> 
> I'm running a 12 node cluser with cassandra 1.1.5 and oracle jdk 1.6.0_35. 
> Our application constantly writes large updates with cql. Once in a while, an 
> rpc_time will occur.
> 
> Since a lot of the information is counters, its impossible for me to 
> understand if the updates complete partially on rpc_timeout, or cassandra 
> somehow rolls back the change completely, and hence I can't tell if I should 
> re-execute the query on rpc_timeout (with double processing being a bigger 
> concern than missing updates).
> 
> I am thinking, but unsure of this, that if I'll switch to LOCAL_QUORUM, 
> rpc_timeout will always mean that the update was not processes as a whole. In 
> all other cases, the rpc_timeout might be thrown from a remote node (not the 
> one I'm connected to), and hence some parts of the update will be performed 
> and others parts will not.
> 
> Anyone solved this issue before?
> 
> Kind Regards,
> Kosha
> 

Re: re-execution of failed queries with rpc_timeout

[Edward Capriolo]

Q: The newer versions of Cassandra include extra information in the
exception, I **think** you can use that information to determine how many
machines the operation succeeded on. However I do not think that
information means you can make counters that timed out "bulletproof"


On Tue, Apr 16, 2013 at 5:08 PM, aaron morton wrote:

> If you are using Counters you need to do everything you can to avoid
> timeouts. In the worse case we do not know where it has been applied. The
> increment is applied on a lead and then replicated to the others, if the
> coordinator is not  the lead it may not know if the increments was applied
> at all.
>
> Start by reducing the size of the updates. Larger batches do not always
> mean better performance.
>
>  In all other cases, the rpc_timeout might be thrown from a remote node
>> (not the one I'm connected to), and hence some parts of the update will be
>> performed and others parts will not.
>>
> TimedOutException is always thrown from the coordinator you are connected
> to.
>
> Cheers
>
> -
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 15/04/2013, at 1:38 PM, Moty Kosharovsky  wrote:
>
> Sorry, not LOCAL QUORUM, I meant "ANY" quorum.
>
>
> On Mon, Apr 15, 2013 at 4:12 AM, Moty Kosharovsky wrote:
>
>> Hello,
>>
>> I'm running a 12 node cluser with cassandra 1.1.5 and oracle jdk
>> 1.6.0_35. Our application constantly writes large updates with cql. Once in
>> a while, an rpc_time will occur.
>>
>> Since a lot of the information is counters, its impossible for me to
>> understand if the updates complete partially on rpc_timeout, or cassandra
>> somehow rolls back the change completely, and hence I can't tell if I
>> should re-execute the query on rpc_timeout (with double processing being a
>> bigger concern than missing updates).
>>
>> I am thinking, but unsure of this, that if I'll switch to LOCAL_QUORUM,
>> rpc_timeout will always mean that the update was not processes as a whole.
>> In all other cases, the rpc_timeout might be thrown from a remote node (not
>> the one I'm connected to), and hence some parts of the update will be
>> performed and others parts will not.
>>
>> Anyone solved this issue before?
>>
>> Kind Regards,
>> Kosha
>>
>
>
>

Re: Rename failed while cassandra is starting up

[aaron morton]

I forgot to add I created a ticket for it 
https://issues.apache.org/jira/browse/CASSANDRA-5469

See that ticket for recent changes to the MeteredFlusher. 

IMHO this is not related to the metered flusher. Index rebuilds force a flush.

Cheers
 
-
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 15/04/2013, at 1:57 PM, Boris Yen  wrote:

> Hi Aaron,
> 
> "startup is single threaded and the scrub runs before the tables are opened".
> 
> This is what I was thinking too. However, after using the debugger to trace 
> the code, I realized that MeteredFlusher (see the "countFlushBytes" method) 
> might open the sstables before the scrub is completed. I suppose this is the 
> cause of the exceptions I saw.
> 
> My plan is to add a boolean flag named "scrubCompleted" at 
> AbstractCassandraDaemon or StorageService. By default, it is false, after the 
> scrub is completed the AbstractCassandraDaemon needs to set it to true. The 
> MeterdFluster needs to make sure the scrub is completed by checking this 
> boolean value and starts to do all the calculation.
> 
> Is this a good plan? or it might have side effects?
> 
> Thanks and Regards,
> Boris
> 
> 
> On Mon, Apr 15, 2013 at 4:26 AM, aaron morton  wrote:
>> From the log messages, it looked like the table/keyspace was
>> opened before the scrubDataDirectories was executed. This created a race
>> condition between two threads.
> Seems odd. 
> AFAIK that startup is single threaded and the scrub runs before the tables 
> are opened. See AbstractCassandraDaemon.setup()
> 
>> INFO [OptionalTasks:1] 2013-04-09 02:49:39,900 SecondaryIndexManager.java
>> (line 184) Creating new index :
>> ColumnDefinition{name=6d6f62696c6974795a6f6e654944,
>> validator=org.apache.cassandra.db.marshal.UTF8Type, index_type=KEYS,
>> index_name='fmzd_ap_mobilityZoneUUID'}
>> ERROR [FlushWriter:1] 2013-04-09 02:49:39,916 AbstractCassandraDaemon.java
>> (line 139) Fatal exception in thread Thread[FlushWriter:1,5,main]
>> java.io.IOError: java.io.IOException: rename failed of
>> /test/db/data/fmzd/alarm.fmzd_alarm_alarmCode-hd-21-Data.db
> 
> Looks like a secondary index is being created at startup and there is an 
> error renaming the file. 
> OR
> The node was shut down before the index was built and it's been rebuilt at 
> startup.
> 
> Both of these are async operations and cause a race with scrubDirectories(). 
> 
> Probably not the log replaying because it looks like the sstables have not 
> been opened. 
> 
> I *think* the way around this is to um…. 
> * move all existing data and commit log out of the way 
> * start with node with -Dcassandra.join_ring=false JVM option in 
> cassandra-env.sh
> * check that all indexes are built using nodetool cfstats
> * shut it down
> * put the commit log and data dirs back in place. 
> 
> All we want to do is get the system KS updated, but in 1.0 that's a 
> serialised object and not easy to poke. 
> 
> Hope that helps. 
>  
> -
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
> 
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 14/04/2013, at 3:50 PM, Boris Yen  wrote:
> 
>> Hi All,
>> 
>> Recently, we encountered an error on 1.0.12 that prevented cassandra from
>> starting up. From the log messages, it looked like the table/keyspace was
>> opened before the scrubDataDirectories was executed. This created a race
>> condition between two threads. One was trying to rename files while the
>> other was trying to remove tmp files. I was wondering if anyone could
>> provide us some information or workaround for this.
>> 
>> INFO [MemoryMeter:1] 2013-04-09 02:49:39,868 Memtable.java (line 186)
>> CFS(Keyspace='fmzd', ColumnFamily='alarm.fmzd_alarm_category') liveRatio is
>> 3.7553409423470883 (just-counted was 3.1413828689370487).  calculation took
>> 2ms for 265 columns
>> INFO [SSTableBatchOpen:1] 2013-04-09 02:49:39,868 SSTableReader.java (line
>> 153) Opening /test/db/data/fmzd/ap.fmzd_ap_meshRole-hd-2 (83 bytes)
>> INFO [SSTableBatchOpen:2] 2013-04-09 02:49:39,868 SSTableReader.java (line
>> 153) Opening /test/db/data/fmzd/ap.fmzd_ap_meshRole-hd-1 (123 bytes)
>> INFO [Creating index: alarm.fmzd_alarm_category] 2013-04-09 02:49:39,874
>> ColumnFamilyStore.java (line 705) Enqueuing flush of
>> Memtable-alarm.fmzd_alarm_category@413535513(14025/65835 serialized/live
>> bytes, 275 ops)
>> INFO [OptionalTasks:1] 2013-04-09 02:49:39,877 SecondaryIndexManager.java
>> (line 184) Creating new index : ColumnDefinition{name=6d65736853534944,
>> validator=org.apache.cassandra.db.marshal.UTF8Type, index_type=KEYS,
>> index_name='fmzd_ap_meshSSID'}
>> INFO [SSTableBatchOpen:1] 2013-04-09 02:49:39,895 SSTableReader.java (line
>> 153) Opening /test/db/data/fmzd/ap.fmzd_ap_meshSSID-hd-1 (122 bytes)
>> INFO [SSTableBatchOpen:2] 2013-04-09 02:49:39,896 SSTableReader.java (line
>> 153) Opening /test/db/data/fmzd/ap.fmzd_ap_meshSSID-hd-2 (82 bytes)
>> INFO [OptionalTask

Re: Added extra column as composite key while creation counter column family

[aaron morton]

What version are you using ? 

WIth 1.2.4 …

cqlsh:dev>  CREATE TABLE counters (
   ...   key text,
   ...   value counter,
   ...   PRIMARY KEY (key)
   ...  ) WITH COMPACT STORAGE;
cqlsh:dev> describe table counters;

CREATE TABLE counters (
  key text PRIMARY KEY,
  value counter
) WITH COMPACT STORAGE AND
  bloom_filter_fp_chance=0.01 AND
  caching='KEYS_ONLY' AND
  comment='' AND
  dclocal_read_repair_chance=0.00 AND
  gc_grace_seconds=864000 AND
  read_repair_chance=0.10 AND
  replicate_on_write='true' AND
  populate_io_cache_on_flush='false' AND
  compaction={'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'SnappyCompressor'};

Cheers

-
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 15/04/2013, at 3:58 PM, Kuldeep Mishra  wrote:

> Hi,
>While I creating counter column family a extra column is being added what 
> I do ?
> Table creation script
>  CREATE TABLE counters (
>   key text,
>   value counter,
>   PRIMARY KEY (key)
>  ) WITH COMPACT STORAGE
> 
> after describing column family I am getting following 
> CREATE TABLE counters (
>   key text,
>   column1 text,
>   value counter,
>   PRIMARY KEY (key, column1)
> ) WITH COMPACT STORAGE AND
>   bloom_filter_fp_chance=0.01 AND
>   caching='KEYS_ONLY' AND
>   comment='' AND
>   dclocal_read_repair_chance=0.00 AND
>   gc_grace_seconds=864000 AND
>   read_repair_chance=0.10 AND
>   replicate_on_write='true' AND
>   compaction={'class': 'SizeTieredCompactionStrategy'} AND
>   compression={'sstable_compression': 'SnappyCompressor'};
> 
> extra column column1 is added 
> 
> Please help
> 
> -- 
> Thanks and Regards
> Kuldeep Kumar Mishra
> +919540965199

Re: Any experience of 20 node mini-itx cassandra cluster

[aaron morton]

> Can't we use LCS? 
Do some reading and some tests…

http://www.datastax.com/dev/blog/leveled-compaction-in-apache-cassandra
http://www.datastax.com/dev/blog/when-to-use-leveled-compaction

Cheers

-
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 15/04/2013, at 10:44 PM, Jabbar Azam  wrote:

> I know the SSD's are a bit small but they should be enough for our 
> application. Out test data is 1.6 TB(including replication of rf=3). Can't we 
> use LCS? This will give us more space at the expensive of more I/O but SSD's 
> have loads of I/Os.
> 
> 
> 
> 
> 
> Thanks
> 
> Jabbar Azam
> 
> 
> On 14 April 2013 20:20, Jabbar Azam  wrote:
> Thanks Aaron.
> 
> Thanks
> 
> Jabbar Azam
> 
> 
> On 14 April 2013 19:39, aaron morton  wrote:
> That's better. 
> 
> The SSD size is a bit small, and be warned that you will want to leave 50Gb 
> to 100GB free to allow room for compaction (using the default size tiered). 
> 
> On the ram side you will want to run about 4GB (assuming cass 1.2) for the 
> JVM the rest can be off heap Cassandra structures. This may not leave too 
> much free space for the os page cache, but SSD may help there.
> 
> Cheers
>   
> -
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
> 
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 13/04/2013, at 4:47 PM, Jabbar Azam  wrote:
> 
>> What about using quad core athlon x4 740 3.2 GHz with 8gb of ram and 256gb 
>> ssds?
>> 
>> I know it will depend on our workload but will be better than a dual core 
>> CPU. I think
>> 
>> Jabbar Azam
>> 
>> On 13 Apr 2013 01:05, "Edward Capriolo"  wrote:
>> Duel core not the greatest you might run into GC issues before you run out 
>> of IO from your ssd devices. Also cassandra has other concurrency settings 
>> that are tuned roughly around the number of processors/cores. It is not 
>> uncommon to see 4-6 cores of cpu (600 % in top dealing with young gen 
>> garbage managing lots of sockets whatever.
>> 
>> 
>> On Fri, Apr 12, 2013 at 12:02 PM, Jabbar Azam  wrote:
>> That's my guess. My colleague is still looking at CPU's so I'm hoping he can 
>> get quad core CPU's for the servers.
>> 
>> Thanks
>> 
>> Jabbar Azam
>> 
>> 
>> On 12 April 2013 16:48, Colin Blower  wrote:
>> If you have not seen it already, checkout the Netflix blog post on their 
>> performance testing of AWS SSD instances.
>> 
>> http://techblog.netflix.com/2012/07/benchmarking-high-performance-io-with.html
>> 
>> My guess, based on very little experience, is that you will be CPU bound.
>> 
>> 
>> On 04/12/2013 03:05 AM, Jabbar Azam wrote:
>>> Hello,
>>> 
>>> I'm going to be building a 20 node cassandra cluster in one datacentre. The 
>>> spec of the servers will roughly be dual core Celeron CPU, 256 GB SSD, 16GB 
>>> RAM and two nics.
>>> 
>>> 
>>> Has anybody done any performance testing with this setup or have any 
>>> gotcha's I should be aware of wrt to the hardware?
>>> 
>>>  I do realise the CPU is fairly low computational power but I'm going to 
>>> assume the system is going to be IO bound hence the RAM and SSD's.
>>> 
>>> 
>>> Thanks
>>> 
>>> Jabbar Azam
>> 
>> 
>> -- 
>> Colin Blower
>> Software Engineer
>> Barracuda Networks Inc.
>> +1 408-342-5576 (o)
>> 
>> 
> 
> 
> 

Re: Any experience of 20 node mini-itx cassandra cluster

[Jabbar Azam]

I already have thanks. I'll do the tests with the hardware arrives.

Thanks

Jabbar Azam


On 16 April 2013 22:27, aaron morton  wrote:

> Can't we use LCS?
>
> Do some reading and some tests…
>
> http://www.datastax.com/dev/blog/leveled-compaction-in-apache-cassandra
> http://www.datastax.com/dev/blog/when-to-use-leveled-compaction
>
> Cheers
>
> -
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 15/04/2013, at 10:44 PM, Jabbar Azam  wrote:
>
> I know the SSD's are a bit small but they should be enough for our
> application. Out test data is 1.6 TB(including replication of rf=3). Can't
> we use LCS? This will give us more space at the expensive of more I/O but
> SSD's have loads of I/Os.
>
>
>
>
>
> Thanks
>
> Jabbar Azam
>
>
> On 14 April 2013 20:20, Jabbar Azam  wrote:
>
>> Thanks Aaron.
>>
>> Thanks
>>
>> Jabbar Azam
>>
>>
>> On 14 April 2013 19:39, aaron morton  wrote:
>>
>>> That's better.
>>>
>>> The SSD size is a bit small, and be warned that you will want to leave
>>> 50Gb to 100GB free to allow room for compaction (using the default size
>>> tiered).
>>>
>>> On the ram side you will want to run about 4GB (assuming cass 1.2) for
>>> the JVM the rest can be off heap Cassandra structures. This may not leave
>>> too much free space for the os page cache, but SSD may help there.
>>>
>>> Cheers
>>>
>>>-
>>> Aaron Morton
>>> Freelance Cassandra Consultant
>>> New Zealand
>>>
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>>
>>> On 13/04/2013, at 4:47 PM, Jabbar Azam  wrote:
>>>
>>> What about using quad core athlon x4 740 3.2 GHz with 8gb of ram and
>>> 256gb ssds?
>>>
>>> I know it will depend on our workload but will be better than a dual
>>> core CPU. I think
>>>
>>> Jabbar Azam
>>> On 13 Apr 2013 01:05, "Edward Capriolo"  wrote:
>>>
 Duel core not the greatest you might run into GC issues before you run
 out of IO from your ssd devices. Also cassandra has other concurrency
 settings that are tuned roughly around the number of processors/cores. It
 is not uncommon to see 4-6 cores of cpu (600 % in top dealing with young
 gen garbage managing lots of sockets whatever.


 On Fri, Apr 12, 2013 at 12:02 PM, Jabbar Azam  wrote:

> That's my guess. My colleague is still looking at CPU's so I'm hoping
> he can get quad core CPU's for the servers.
>
> Thanks
>
> Jabbar Azam
>
>
> On 12 April 2013 16:48, Colin Blower  wrote:
>
>>  If you have not seen it already, checkout the Netflix blog post on
>> their performance testing of AWS SSD instances.
>>
>>
>> http://techblog.netflix.com/2012/07/benchmarking-high-performance-io-with.html
>>
>> My guess, based on very little experience, is that you will be CPU
>> bound.
>>
>>
>> On 04/12/2013 03:05 AM, Jabbar Azam wrote:
>>
>>   Hello,
>>
>>  I'm going to be building a 20 node cassandra cluster in one
>> datacentre. The spec of the servers will roughly be dual core Celeron 
>> CPU,
>> 256 GB SSD, 16GB RAM and two nics.
>>
>>
>>  Has anybody done any performance testing with this setup or have any
>> gotcha's I should be aware of wrt to the hardware?
>>
>>  I do realise the CPU is fairly low computational power but I'm going
>> to assume the system is going to be IO bound hence the RAM and SSD's.
>>
>>
>>  Thanks
>>
>> Jabbar Azam
>>
>>
>>
>> --
>>  *Colin Blower*
>> *Software Engineer*
>> Barracuda Networks Inc.
>> +1 408-342-5576 (o)
>>
>
>

>>>
>>
>
>

Re: Extracting data from SSTable files with MapReduce

[aaron morton]

> I did try to upgrade to 1.2 but it did not work out. Maybe to many versions 
> in between.
Newer versions should be able to read older file formats. What was the error?

> Why would later formats make this easier you think?
it will be easier to write against the current code base and you find it easier 
to get help. 

Cheers


-
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 16/04/2013, at 5:37 AM, Jasper K.  wrote:

> Hi Aaron,
> 
> I did try to upgrade to 1.2 but it did not work out. Maybe to many versions 
> in between.
> 
> Why would later formats make this easier you think?
> 
> Jasper
> 
> 
> 
> 2013/4/14 aaron morton 
>> The SSTable files are in the -f- format from 0.8.10.
> 
> If you can upgrade to the latest version it will make things easier. 
> Start a node and use nodetool upgradesstables. 
> 
> The org.apache.cassandra.tools.SSTableExport class provides a blue print for 
> reading rows from disk.
> 
> hope that helps. 
> 
> -
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
> 
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 13/04/2013, at 7:58 PM, Jasper K.  wrote:
> 
>> Hi,
>> 
>> Does anyone have any experience with running a MapReduce directly against a 
>> CF's SSTable files?
>> 
>> I have a use case where this seems to be an option. I want to export all 
>> data from a CF to a flat file format for statistical analysis.
>> 
>> Some factors that make it (more) doable in my case:
>> -The Cassandra instance is not 'on-line' (no writes- no reads)
>> -The .db files were exported from another instance. I got them all in one 
>> place now
>> 
>> The SSTable files are in the -f- format from 0.8.10.
>> 
>> Looking at this : http://wiki.apache.org/cassandra/ArchitectureSSTable it 
>> should be possible to write a Hadoop RecordReader for Cassandra rowkeys.
>> 
>> But maybe I am not fully aware of what I am up to.
>> 
>> -- 
>> 
>> Jasper 
> 
> 
> 
> 
> -- 
> 

Re: StatusLogger format?

[aaron morton]

> 99% sure it's in bytes.
+1 to your confidence level. 

Cheers

-
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 16/04/2013, at 6:14 AM, William Oberman  wrote:

> 99% sure it's in bytes.
> 
> 
> On Mon, Apr 15, 2013 at 11:25 AM, William Oberman  
> wrote:
> Mainly the:
> "ColumnFamilyMemtable ops,data" 
> section.  
> 
> Is data in bytes/kb/mb/etc?
> 
> Example line:
> StatusLogger.java (line 116) civicscience.sessions4963,1799916
> 
> Thanks!
> 
> 
> 
> 
> 

Re: Thrift message length exceeded

[aaron morton]

Can you confirm the you are using the same thrift version that ships 1.2.3 ? 

Cheers

-
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 16/04/2013, at 10:17 AM, Lanny Ripple  wrote:

> A bump to say I found this
> 
>  
> http://stackoverflow.com/questions/15487540/pig-cassandra-message-length-exceeded
> 
> so others are seeing similar behavior.
> 
> From what I can see of org.apache.cassandra.hadoop nothing has changed since 
> 1.1.5 when we didn't see such things but sure looks like there's a bug that's 
> slipped in (or been uncovered) somewhere.  I'll try to narrow down to a 
> dataset and code that can reproduce.
> 
> On Apr 10, 2013, at 6:29 PM, Lanny Ripple  wrote:
> 
>> We are using Astyanax in production but I cut back to just Hadoop and 
>> Cassandra to confirm it's a Cassandra (or our use of Cassandra) problem.
>> 
>> We do have some extremely large rows but we went from everything working 
>> with 1.1.5 to almost everything carping with 1.2.3.  Something has changed.  
>> Perhaps we were doing something wrong earlier that 1.2.3 exposed but 
>> surprises are never welcome in production.
>> 
>> On Apr 10, 2013, at 8:10 AM,  wrote:
>> 
>>> I also saw this when upgrading from C* 1.0 to 1.2.2, and from hector 0.6 to 
>>> 0.8
>>> Turns out the Thrift message really was too long.
>>> The mystery to me: Why no complaints in previous versions? Were some checks 
>>> added in Thrift or Hector?
>>> 
>>> -Original Message-
>>> From: Lanny Ripple [mailto:la...@spotright.com] 
>>> Sent: Tuesday, April 09, 2013 6:17 PM
>>> To: user@cassandra.apache.org
>>> Subject: Thrift message length exceeded
>>> 
>>> Hello,
>>> 
>>> We have recently upgraded to Cass 1.2.3 from Cass 1.1.5.  We ran 
>>> sstableupgrades and got the ring on its feet and we are now seeing a new 
>>> issue.
>>> 
>>> When we run MapReduce jobs against practically any table we find the 
>>> following errors:
>>> 
>>> 2013-04-09 09:58:47,746 INFO org.apache.hadoop.util.NativeCodeLoader: 
>>> Loaded the native-hadoop library
>>> 2013-04-09 09:58:47,899 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: 
>>> Initializing JVM Metrics with processName=MAP, sessionId=
>>> 2013-04-09 09:58:48,021 INFO org.apache.hadoop.util.ProcessTree: setsid 
>>> exited with exit code 0
>>> 2013-04-09 09:58:48,024 INFO org.apache.hadoop.mapred.Task:  Using 
>>> ResourceCalculatorPlugin : 
>>> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4a48edb5
>>> 2013-04-09 09:58:50,475 INFO org.apache.hadoop.mapred.TaskLogsTruncater: 
>>> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
>>> 2013-04-09 09:58:50,477 WARN org.apache.hadoop.mapred.Child: Error running 
>>> child
>>> java.lang.RuntimeException: org.apache.thrift.TException: Message length 
>>> exceeded: 106
>>> at 
>>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.maybeInit(ColumnFamilyRecordReader.java:384)
>>> at 
>>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:390)
>>> at 
>>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:313)
>>> at 
>>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>>> at 
>>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>>> at 
>>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader.getProgress(ColumnFamilyRecordReader.java:103)
>>> at 
>>> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.getProgress(MapTask.java:444)
>>> at 
>>> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:460)
>>> at 
>>> org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
>>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
>>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>> at 
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
>>> at org.apache.hadoop.mapred.Child.main(Child.java:260)
>>> Caused by: org.apache.thrift.TException: Message length exceeded: 106
>>> at 
>>> org.apache.thrift.protocol.TBinaryProtocol.checkReadLength(TBinaryProtocol.java:393)
>>> at 
>>> org.apache.thrift.protocol.TBinaryProtocol.readBinary(TBinaryProtocol.java:363)
>>> at org.apache.cassandra.thrift.Column.read(Column.java:528)
>>> at 
>>> org.apache.cassandra.thrift.ColumnOrSuperColumn.read(ColumnOrSuperColumn.java:507)
>>> at org.apache.cassandra.thrift.KeySlice.read(KeySlice.java:408)
>>> at 
>>> org.apache.cassandra.thrift.Cassandra$get_ran

Re: C* consumes all RAM

[aaron morton]

You are probably seeing this http://wiki.apache.org/cassandra/FAQ#mmap

Cheers

-
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 16/04/2013, at 8:43 PM, Mikhail Mazursky  wrote:

> More details:
> 
> USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND 
> 219   3801  0.7 92.7 6561116 3567016 ? SLl  Mar11 372:44 
> /usr/java/latest/bin/java
> 
> Linux XXX@YYY 3.2.30-49.59.amzn1.x86_64 #1 SMP Wed Oct 3 19:54:33 UTC 2012 
> x86_64 x86_64 x86_64 GNU/Linux
> 
> total   used   free sharedbuffers cached 
> Mem:  3754   3695 59  0 37 75 
> -/+ buffers/cache:   3582171 
> Swap: 2055165   1890
> 
> 
> 
> 2013/4/16 Mikhail Mazursky 
> Hello.
> 
> C* have been running without any problem for some weeks but now it started to 
> consume all available ram.
> 
> The cluster have very little data in it. There are no errors in logs, CPU is 
> not loaded at all, jstack shows no deadlocks, there are 83 threads. 
> Read/write latency is 1-4ms.
> 
> The question is: how to find out where is the RAM used?
> 
> # nodetool ring keyspace name
> Address DC  RackStatus State   Load
> Effective-Ownership Token
>   
>  114967934986393518082067045023985443411  
> 10.0.1.220  datacenter1 rack1   Up Normal  889.71 KB   
> 100.00% 0
> 10.0.1.222  datacenter1 rack1   Up Normal  918.92 KB   
> 100.00% 56662868904138943096229641828625627278   
> 10.0.1.221  datacenter1 rack1   Up Normal  1.04 MB 
> 100.00% 114967934986393518082067045023985443411
> 
> We use AWS m1.medium - 3.75 RAM, 1 Core 
> 
> java version "1.7.0_10" 
> Java(TM) SE Runtime Environment (build 1.7.0_10-b18) 
> Java HotSpot(TM) 64-Bit Server VM (build 23.6-b04, mixed mode)
> 
> C* is started with the following command
> /usr/java/latest/bin/java -ea 
> -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+UseThreadPriorities 
> -XX:ThreadPriorityPolicy=42 -Xms1G -Xmx1G -Xmn512M 
> -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:+UseParNewGC
> 
> Thanks in advance.
> 
> Mikhail.
> 

Re: Lost data after expanding cluster c* 1.2.3-1

[aaron morton]

Sorry can you repost the details of that issue including the CL you are using. 

Aaron

-
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 17/04/2013, at 12:57 AM, Kais Ahmed  wrote:

> Thanks aaron,
> 
> I feel that rebuilding indexes went well, but the result of my query (SELECT 
> * FROM userdata WHERE login='kais';) is still emty.
> 
> INFO [Creating index: userdata.userdata_login_idx] 2013-03-30 01:16:33,110 
> SecondaryIndex.java (line 175) Submitting index build of 
> userdata.userdata_login_idx
> INFO [Creating index: userdata.userdata_login_idx] 2013-03-30 01:34:11,667 
> SecondaryIndex.java (line 202) Index build of userdata.userdata_login_idx 
> complete
> 
> Thanks,
> 
> 
> 2013/4/9 aaron morton 
> Look in the logs for messages from the SecondaryIndexManager 
> 
> starts with "Submitting index build of"
> end with "Index build of"
> 
> Cheers
> 
> -
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
> 
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 7/04/2013, at 12:55 AM, Kais Ahmed  wrote:
> 
>> hi aaron,
>> 
>> nodetool compactionstats on all nodes return 1 pending task :
>> 
>> ubuntu@app:~$ nodetool compactionstats host
>> pending tasks: 1
>> Active compaction remaining time :n/a
>> 
>> The command nodetool rebuild_index was launched several days ago.
>> 
>> 2013/4/5 aaron morton 
>>> but nothing's happening, how can i monitor the progress? and how can i know 
>>> when it's finished?
>> 
>> check nodetool compacitonstats
>> 
>> Cheers
>> 
>> -
>> Aaron Morton
>> Freelance Cassandra Consultant
>> New Zealand
>> 
>> @aaronmorton
>> http://www.thelastpickle.com
>> 
>> On 4/04/2013, at 2:51 PM, Kais Ahmed  wrote:
>> 
>>> Hi aaron,
>>> 
>>> I ran the command "nodetool rebuild_index host keyspace cf" on all the 
>>> nodes, in the log i see :
>>> 
>>> INFO [RMI TCP Connection(5422)-10.34.139.xxx] 2013-04-04 08:31:53,641 
>>> ColumnFamilyStore.java (line 558) User Requested secondary index re-build 
>>> for ...
>>> 
>>> but nothing's happening, how can i monitor the progress? and how can i know 
>>> when it's finished?
>>> 
>>> Thanks,
>>>  
>>> 
>>> 2013/4/2 aaron morton 
 The problem come from that i don't put  auto_boostrap to true for the new 
 nodes, not in this documentation 
 (http://www.datastax.com/docs/1.2/install/expand_ami)
>>> auto_bootstrap defaults to True if not specified in the yaml. 
>>> 
 can i do that at any time, or when the cluster are not loaded
>>> Not sure what the question is. 
>>> Both those operations are online operations you can do while the node is 
>>> processing requests. 
>>>  
>>> Cheers
>>> 
>>> -
>>> Aaron Morton
>>> Freelance Cassandra Consultant
>>> New Zealand
>>> 
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>> 
>>> On 1/04/2013, at 9:26 PM, Kais Ahmed  wrote:
>>> 
 > At this moment the errors started, we see that members and other data 
 > are gone, at this moment the nodetool status return (in red color the 3 
 > new nodes)
 > What errors?
 The errors was in my side in the application, not cassandra errors
 
 > I put for each of them seeds = A ip, and start each with two minutes 
 > intervals.
 > When I'm making changes I tend to change a single node first, confirm 
 > everything is OK and then do a bulk change.
 Thank you for that advice.
 
 >I'm not sure what or why it went wrong, but that should get you to a 
 >stable place. If you have any problems keep an eye on the logs for errors 
 >or warnings.
 The problem come from that i don't put  auto_boostrap to true for the new 
 nodes, not in this documentation 
 (http://www.datastax.com/docs/1.2/install/expand_ami)
 
 >if you are using secondary indexes use nodetool rebuild_index to rebuild 
 >those.
 can i do that at any time, or when the cluster are not loaded
 
 Thanks aaron,
 
 2013/4/1 aaron morton 
 Please do not rely on colour in your emails, the best way to get your 
 emails accepted by the Apache mail servers is to use plain text.
 
 > At this moment the errors started, we see that members and other data 
 > are gone, at this moment the nodetool status return (in red color the 3 
 > new nodes)
 What errors?
 
 > I put for each of them seeds = A ip, and start each with two minutes 
 > intervals.
 When I'm making changes I tend to change a single node first, confirm 
 everything is OK and then do a bulk change.
 
 > Now the cluster seem to work normally, but i can use the secondary for 
 > the moment, the queryanswer are random
 run nodetool repair -pr on each node, let it finish before starting the 
 next one.
 if you are using secondary indexes use nodetool rebuild_index to rebuild 
 those.
 Add one node new node to the cluster and confirm everything is 

Re: Does Memtable resides in Heap?

[aaron morton]

Compression Meta data is also off heap 
http://www.datastax.com/dev/blog/performance-improvements-in-cassandra-1-2

Cheers

-
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 17/04/2013, at 3:02 AM, Jay Svc  wrote:

> Thanks Edward.!
> 
> 
> 
> On Fri, Apr 12, 2013 at 9:46 AM, Edward Capriolo  
> wrote:
> This issue describes the design of the arena allocation of memtabes.
> https://issues.apache.org/jira/browse/CASSANDRA-2252
> 
> 
> On Fri, Apr 12, 2013 at 1:35 AM, Viktor Jevdokimov 
>  wrote:
> Memtables resides in heap, write rate impacts GC, more writes - more frequent 
> and longer ParNew GC pauses.
> 
> 
> From: Jay Svc [mailto:jaytechg...@gmail.com]
> Sent: Friday, April 12, 2013 01:03
> To: user@cassandra.apache.org
> Subject: Does Memtable resides in Heap?
> 
> Hi Team,
> 
> I have got this 8GB of RAM out of that 4GB allocated to Java Heap. My 
> question is the size of Memtable does it contribute to heap size? or they are 
> part of off-heap?
> 
> Does bigger Memtable would have impact on GC and overall memory management?
> 
> I am using DSE 3.0 / Cassandra 1.1.9.
> 
> Thanks,
> Jay
> 
> Best regards / Pagarbiai
> 
> Viktor Jevdokimov
> Senior Developer
> 
> Email: viktor.jevdoki...@adform.com
> Phone: +370 5 212 3063
> Fax: +370 5 261 0453
> 
> J. Jasinskio 16C,
> LT-01112 Vilnius,
> Lithuania
> 
> 
> 
> Disclaimer: The information contained in this message and attachments is 
> intended solely for the attention and use of the named addressee and may be 
> confidential. If you are not the intended recipient, you are reminded that 
> the information remains the property of the sender. You must not use, 
> disclose, distribute, copy, print or rely on this e-mail. If you have 
> received this message in error, please contact the sender immediately and 
> irrevocably delete this message and any copies.
> 
> 

Repair Freeze / Gossip Invisibility / EC2 Public IP configuration

[Arya Goudarzi]

TL;DR; An EC2 Multi-Region Setup's Repair/Gossip Works with 1.1.10 but with
1.2.4, gossip does not see the nodes after restarting all nodes at once,
and repair gets stuck.

This is a working configuration:
Cassandra 1.1.10 Cluster with 12 nodes in us-east-1 and 12 nodes in
us-west-2
Using Ec2MultiRegionSnitch and SSL enabled for DC_ONLY and
NetworkTopologyStrategy with strategy_options: us-east-1:3;us-west-2:3;
C* instances have a security group called 'cluster1'
security group 'cluster1' in each region is configured as such
Allow TCP:
7199 from cluster1 (JMX)
1024 - 65535 from cluster1 (JMX Random Ports - This supersedes all specific
ports, but I have the specific ports just for clarity )
7100 from cluster1 (Configured Normal Storage)
7103 from cluster1 (Configured SSL Storage)
9160 from cluster1 (Configured Thrift RPC Port)
9160 from 
foreach node's public IP we also have this rule set to enable cross region
comminication:
7103 from public_ip (Open SSL storage)

The above is a functioning and happy setup. You run repair, and it finishes
successfully.

Broken Setup:

Upgrade to 1.2.4 without changing any of the above security group settings:

Run repair. The repair will get stuck. Thus hanging.

Now for each public_ip add a security group rule as such to cluster1
security group:

Allow TCP: 7100 from public_ip

Run repair. Things will work now. Also after restarting all nodes at the
same time, gossip will see everyone again.

I was told on https://issues.apache.org/jira/browse/CASSANDRA-5432 that
nothing in terms of networking was changed. If nothing in terms of port and
networking was changed in 1.2, then why the above is happening? I can
constantly reproduce it.

Please advice.

-Arya

Re: Repair Freeze / Gossip Invisibility / EC2 Public IP configuration

[Edward Capriolo]

So cassandra does inter node compression. I have not checked but this might
be accidentally getting turned on by default. Because the storage port is
typically 7000. Not sure why you are allowing 7100. In any case try
allowing 7000 or with internode compression off.


On Tue, Apr 16, 2013 at 6:42 PM, Arya Goudarzi  wrote:

> TL;DR; An EC2 Multi-Region Setup's Repair/Gossip Works with 1.1.10 but
> with 1.2.4, gossip does not see the nodes after restarting all nodes at
> once, and repair gets stuck.
>
> This is a working configuration:
> Cassandra 1.1.10 Cluster with 12 nodes in us-east-1 and 12 nodes in
> us-west-2
> Using Ec2MultiRegionSnitch and SSL enabled for DC_ONLY and
> NetworkTopologyStrategy with strategy_options: us-east-1:3;us-west-2:3;
> C* instances have a security group called 'cluster1'
> security group 'cluster1' in each region is configured as such
> Allow TCP:
> 7199 from cluster1 (JMX)
> 1024 - 65535 from cluster1 (JMX Random Ports - This supersedes all
> specific ports, but I have the specific ports just for clarity )
> 7100 from cluster1 (Configured Normal Storage)
> 7103 from cluster1 (Configured SSL Storage)
> 9160 from cluster1 (Configured Thrift RPC Port)
> 9160 from 
> foreach node's public IP we also have this rule set to enable cross region
> comminication:
> 7103 from public_ip (Open SSL storage)
>
> The above is a functioning and happy setup. You run repair, and it
> finishes successfully.
>
> Broken Setup:
>
> Upgrade to 1.2.4 without changing any of the above security group settings:
>
> Run repair. The repair will get stuck. Thus hanging.
>
> Now for each public_ip add a security group rule as such to cluster1
> security group:
>
> Allow TCP: 7100 from public_ip
>
> Run repair. Things will work now. Also after restarting all nodes at the
> same time, gossip will see everyone again.
>
> I was told on https://issues.apache.org/jira/browse/CASSANDRA-5432 that
> nothing in terms of networking was changed. If nothing in terms of port and
> networking was changed in 1.2, then why the above is happening? I can
> constantly reproduce it.
>
> Please advice.
>
> -Arya
>
>

Re: unexplained hinted handoff

[Dane Miller]

On Sun, Apr 14, 2013 at 11:28 AM, aaron morton  wrote:
>>  If hints are being stored, doesn't that imply DOWN nodes, and why don't I
>> see that in the logs?
>
> Hints are stored for two reasons. First if the node is down when the write
> request starts, second if the node does not reply to the coordinator before
> rpc_timeout. If you are not seeing dropped write messages it may indicate
> network issues between the nodes.

Very helpful! I increased the timeouts for *_request_timeout settings
in cassandra.yaml and no longer see HH in the logs.  Still not sure
why I saw logs about dropped reads, and not dropped mutations.

Anyhow, big timeouts helps in this case.  Thanks :)

Dane

How to stop Cassandra and then restart it in windows?

[Raihan Jamal]

Hello,

I installed single node cluster in my local dev box which is running
Windows 7 and it was working fine. Due to some reason, I need to restart my
desktop and then after that whenever I am doing like this on the command
prompt, it always gives me the below exception-

S:\Apache Cassandra\apache-cassandra-1.2.3\bin>cassandra -f
Starting Cassandra Server
Error: Exception thrown by the agent : java.rmi.server.ExportException:
Port already in use: 7199; nested exception is:
java.net.BindException: Address already in use: JVM_Bind


Meaning port being used somewhere. I have made some changes in *cassandra.yaml
*file so I need to shutdown the Cassandra server and then restart it again.

Can anybody help me with this?

Thanks for the help.

Cassandra Client Recommendation

[Techy Teck]

Hello,
I have recently started working with Cassandra Database. Now I am in the
process of evaluating which Cassandra client I should go forward with.

I am mainly interested in these three-

--1)  Astyanax client

2--)  New Datastax client that uses Binary protocol.

--3)  Pelops client


Can anyone provide some thoughts on this? Some advantages and disadvantages
for these three will be great start for me.


Keeping in mind, we are running Cassandra 1.2.2 in production environment.



Thanks for the help.

Re: MySQL Cluster performing faster than Cassandra cluster on single table

[Jabbar Azam]

MySQL cluster also has the index in ram.  So with lots of rows the ram
becomes a limiting factor.

That's what my colleague found and hence why were sticking with Cassandra.
On 16 Apr 2013 21:05, "horschi"  wrote:

>
>
> Ah, I see, that makes sense. Have you got a source for the storing of
>> hundreds of gigabytes? And does Cassandra not store anything in memory?
>>
> It stores bloom filters and index-samples in memory. But they are much
> smaller than the actual data and they can be configured.
>
>
>>
>> Yeah, my dataset is small at the moment - perhaps I should have chosen
>> something larger for the work I'm doing (University dissertation), however,
>> it is far too late to change now!
>>
> On paper mysql-cluster looks great. But in daily use its not as nice as
> Cassandra (where you have machines dying, networks splitting, etc.).
>
> cheers,
> Christian
>

Re: C* consumes all RAM

[Mikhail Mazursky]

Thank you, Aaron.

p.s. we're on 1.1.9 - i forgot to mention that.


2013/4/17 aaron morton 

> You are probably seeing this http://wiki.apache.org/cassandra/FAQ#mmap
>
> Cheers
>
>-
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 16/04/2013, at 8:43 PM, Mikhail Mazursky  wrote:
>
> More details:
>
> USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
> 219   3801  0.7 92.7 6561116 3567016 ? SLl  Mar11 372:44
> /usr/java/latest/bin/java
>
> Linux XXX@YYY 3.2.30-49.59.amzn1.x86_64 #1 SMP Wed Oct 3 19:54:33 UTC
> 2012 x86_64 x86_64 x86_64 GNU/Linux
>
> total   used   free sharedbuffers cached
> Mem:  3754   3695 59  0 37 75
> -/+ buffers/cache:   3582171
> Swap: 2055165   1890
>
>
>
> 2013/4/16 Mikhail Mazursky 
>
>> Hello.
>>
>> C* have been running without any problem for some weeks but now it
>> started to consume all available ram.
>>
>> The cluster have very little data in it. There are no errors in logs, CPU
>> is not loaded at all, jstack shows no deadlocks, there are 83 threads.
>> Read/write latency is 1-4ms.
>>
>> The question is: how to find out where is the RAM used?
>>
>> # nodetool ring keyspace name
>> Address DC  RackStatus State   Load
>> Effective-Ownership Token
>>
>> 114967934986393518082067045023985443411
>> 10.0.1.220  datacenter1 rack1   Up Normal  889.71 KB
>> 100.00% 0
>> 10.0.1.222  datacenter1 rack1   Up Normal  918.92 KB
>> 100.00% 56662868904138943096229641828625627278
>> 10.0.1.221  datacenter1 rack1   Up Normal  1.04 MB
>> 100.00% 114967934986393518082067045023985443411
>>
>> We use AWS m1.medium - 3.75 RAM, 1 Core
>>
>> java version "1.7.0_10"
>> Java(TM) SE Runtime Environment (build 1.7.0_10-b18)
>> Java HotSpot(TM) 64-Bit Server VM (build 23.6-b04, mixed mode)
>>
>> C* is started with the following command
>> /usr/java/latest/bin/java -ea
>> -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+UseThreadPriorities
>> -XX:ThreadPriorityPolicy=42 -Xms1G -Xmx1G -Xmn512M
>> -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:+UseParNewGC
>>
>> Thanks in advance.
>>
>> Mikhail.
>>
>
>
>

Commit Log question

[aaron morton]

I'm looking into a case where it appears that recycling a commit log segment 
and flushing the dirty CF's results in 46 CF's being flushed. Out of 47 in the 
keyspace. All this flush activity blocks writes. 

Before I dig further I wanted to confirm my understanding.

At 10:46 the MeteredFlusher kicks and flushed the CF…. 

2013-03-18T10:46:59-07:00 INFO [OptionalTasks:1] 2013-03-18 10:46:59,009 
MeteredFlusher.java (line 62) flushing high-traffic column family 
CFS(Keyspace='KSNAME', ColumnFamily='CFNAME') (estimated 198959043 bytes)
2013-03-18T10:46:59-07:00 INFO [OptionalTasks:1] 2013-03-18 10:46:59,009 
ColumnFamilyStore.java (line 659) Enqueuing flush of 
Memtable-CFNAME@172178999(158553073/198959043 serialized/live bytes, 470065 ops)
2013-03-18T10:46:59-07:00 INFO [FlushWriter:482] 2013-03-18 10:46:59,010 
Memtable.java (line 264) Writing Memtable-CFNAME@172178999(158553073/198959043 
serialized/live bytes, 470065 ops)
2013-03-18T10:47:01-07:00 INFO [FlushWriter:482] 2013-03-18 10:47:01,503 
Memtable.java (line 305) Completed flushing 
/mnt/cassandra/data/KSNAME/CFNAME/KSNAME-CFNAME-hf-590-Data.db (29212692 bytes) 
for commitlog position ReplayPosition(segmentId=1363374084494, 
position=982002808)

I expect that the CF is marked as clean in all CL segments other than the 
active one (1363374084494), and in the active one if the last write location 
for the CF is before the current context position (982002808).

There is nothing else in the log (other than liveRatio measuring) then a storm 
of flushing without the MeteredFlusher being involved. I attribute this to the 
CL growing over capacity and the CF's marked as dirty in the oldest segment 
being flushed.

As part of that the CF from above is flushed. 

2013-03-18T10:50:05-07:00 INFO [OptionalTasks:1] 2013-03-18 10:50:05,777 
ColumnFamilyStore.java (line 659) Enqueuing flush of 
Memtable-CFNAME@1841064520(93240219/117001736 serialized/live bytes, 71513 ops)
2013-03-18T10:50:05-07:00 INFO [FlushWriter:483] 2013-03-18 10:50:05,942 
Memtable.java (line 264) Writing Memtable-CFNAME@1841064520(93240219/117001736 
serialized/live bytes, 71513 ops)
2013-03-18T10:50:07-07:00 INFO [FlushWriter:483] 2013-03-18 10:50:07,277 
Memtable.java (line 305) Completed flushing 
/mnt/cassandra/data/KSNAME/CFNAME/KSNAME-CFNAME-hf-591-Data.db (12050522 bytes) 
for commitlog position ReplayPosition(segmentId=1363374084495, position=8542666)

Not that the segmentId has only increased by 1, so we have not put a huge 
amount of traffic through the commit log. 

My understanding is that after the first flush the CF would be clean in the 
oldest CL segment and I should not see the CF being flushed a second time. Is 
that correct? 

running on 1.1.6
  
Cheers

-
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

Re: Added extra column as composite key while creation counter column family

[Kuldeep Mishra]

cassandra 1.2.0

Is it a bug in  1.2.0 ?


Thanks
KK


On Wed, Apr 17, 2013 at 2:56 AM, aaron morton wrote:

> What version are you using ?
>
> WIth 1.2.4 …
>
> cqlsh:dev>  CREATE TABLE counters (
>...   key text,
>...   value counter,
>...   PRIMARY KEY (key)
>...  ) WITH COMPACT STORAGE;
> cqlsh:dev> describe table counters;
>
> CREATE TABLE counters (
>   key text PRIMARY KEY,
>   value counter
> ) WITH COMPACT STORAGE AND
>   bloom_filter_fp_chance=0.01 AND
>   caching='KEYS_ONLY' AND
>   comment='' AND
>   dclocal_read_repair_chance=0.00 AND
>   gc_grace_seconds=864000 AND
>   read_repair_chance=0.10 AND
>   replicate_on_write='true' AND
>   populate_io_cache_on_flush='false' AND
>   compaction={'class': 'SizeTieredCompactionStrategy'} AND
>   compression={'sstable_compression': 'SnappyCompressor'};
>
> Cheers
>
> -
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 15/04/2013, at 3:58 PM, Kuldeep Mishra 
> wrote:
>
> > Hi,
> >While I creating counter column family a extra column is being added
> what I do ?
> > Table creation script
> >  CREATE TABLE counters (
> >   key text,
> >   value counter,
> >   PRIMARY KEY (key)
> >  ) WITH COMPACT STORAGE
> >
> > after describing column family I am getting following
> > CREATE TABLE counters (
> >   key text,
> >   column1 text,
> >   value counter,
> >   PRIMARY KEY (key, column1)
> > ) WITH COMPACT STORAGE AND
> >   bloom_filter_fp_chance=0.01 AND
> >   caching='KEYS_ONLY' AND
> >   comment='' AND
> >   dclocal_read_repair_chance=0.00 AND
> >   gc_grace_seconds=864000 AND
> >   read_repair_chance=0.10 AND
> >   replicate_on_write='true' AND
> >   compaction={'class': 'SizeTieredCompactionStrategy'} AND
> >   compression={'sstable_compression': 'SnappyCompressor'};
> >
> > extra column column1 is added
> >
> > Please help
> >
> > --
> > Thanks and Regards
> > Kuldeep Kumar Mishra
> > +919540965199
>
>


-- 
Thanks and Regards
Kuldeep Kumar Mishra
+919540965199

Re: Cassandra Client Recommendation

[Everton Lima]

Hi Techy,

We are using Astyanax with cassandra 1.2.4.

beneficits:
 * It is so easy to configure and use.
 * Good wiki
 * Mantained by Netflix
 * Solution to manage the store of big files (more than 15mb)
 * Solution to read all rows efficiently

problems:
 * It consume more memory


2013/4/16 Techy Teck 

> Hello,
> I have recently started working with Cassandra Database. Now I am in the
> process of evaluating which Cassandra client I should go forward with.
>
> I am mainly interested in these three-
>
> --1)  Astyanax client
>
> 2--)  New Datastax client that uses Binary protocol.
>
> --3)  Pelops client
>
>
> Can anyone provide some thoughts on this? Some advantages and
> disadvantages for these three will be great start for me.
>
>
> Keeping in mind, we are running Cassandra 1.2.2 in production environment.
>
>
>
> Thanks for the help.
>



-- 
Everton Lima Aleixo
Bacharel em Ciência da Computação pela UFG
Mestrando em Ciência da Computação pela UFG
Programador no LUPA

Re: Cassandra Client Recommendation

[Techy Teck]

Thanks Everton for the suggestion. Couple of questions-

1) Does Astyanax client have any problem with previous version of Cassandra?
2) You said one problem, that it will consume more memory? Can you
elaborate that slightly? What do you mean by that?
3) Does Astyanax supports asynch capabilities?


On Tue, Apr 16, 2013 at 11:05 PM, Everton Lima wrote:

> Hi Techy,
>
> We are using Astyanax with cassandra 1.2.4.
>
> beneficits:
>  * It is so easy to configure and use.
>  * Good wiki
>  * Mantained by Netflix
>  * Solution to manage the store of big files (more than 15mb)
>  * Solution to read all rows efficiently
>
> problems:
>  * It consume more memory
>
>
> 2013/4/16 Techy Teck 
>
>>  Hello,
>> I have recently started working with Cassandra Database. Now I am in the
>> process of evaluating which Cassandra client I should go forward with.
>>
>> I am mainly interested in these three-
>>
>> --1)  Astyanax client
>>
>> 2--)  New Datastax client that uses Binary protocol.
>>
>> --3)  Pelops client
>>
>>
>> Can anyone provide some thoughts on this? Some advantages and
>> disadvantages for these three will be great start for me.
>>
>>
>> Keeping in mind, we are running Cassandra 1.2.2 in production environment.
>>
>>
>>
>> Thanks for the help.
>>
>
>
>
> --
> Everton Lima Aleixo
> Bacharel em Ciência da Computação pela UFG
> Mestrando em Ciência da Computação pela UFG
> Programador no LUPA
>
>

Re: Cassandra Client Recommendation

[Everton Lima]

1) Does Astyanax client have any problem with previous version of Cassandra?
We have used with 1.1.8, but for this version we do not use the last
version of Astyanax. But I think that to Cassandra 1.2.* the last version
of astyanax will work.

2) You said one problem, that it will consume more memory? Can you
elaborate that slightly? What do you mean by that?
In our tests, when we use Astyanax the process memory elavate comparing
with with using direct TBinaryProtocol (cassandra-all.jar). So it is
necessary that you have more memory to your process.

3) Does Astyanax supports asynch capabilities?
What is an asynch capabilites example?



2013/4/17 Techy Teck 

> Thanks Everton for the suggestion. Couple of questions-
>
> 1) Does Astyanax client have any problem with previous version of
> Cassandra?
> 2) You said one problem, that it will consume more memory? Can you
> elaborate that slightly? What do you mean by that?
> 3) Does Astyanax supports asynch capabilities?
>
>
> On Tue, Apr 16, 2013 at 11:05 PM, Everton Lima wrote:
>
>> Hi Techy,
>>
>> We are using Astyanax with cassandra 1.2.4.
>>
>> beneficits:
>>  * It is so easy to configure and use.
>>  * Good wiki
>>  * Mantained by Netflix
>>  * Solution to manage the store of big files (more than 15mb)
>>  * Solution to read all rows efficiently
>>
>> problems:
>>  * It consume more memory
>>
>>
>> 2013/4/16 Techy Teck 
>>
>>>  Hello,
>>> I have recently started working with Cassandra Database. Now I am in the
>>> process of evaluating which Cassandra client I should go forward with.
>>>
>>> I am mainly interested in these three-
>>>
>>> --1)  Astyanax client
>>>
>>> 2--)  New Datastax client that uses Binary protocol.
>>>
>>> --3)  Pelops client
>>>
>>>
>>> Can anyone provide some thoughts on this? Some advantages and
>>> disadvantages for these three will be great start for me.
>>>
>>>
>>> Keeping in mind, we are running Cassandra 1.2.2 in production
>>> environment.
>>>
>>>
>>>
>>> Thanks for the help.
>>>
>>
>>
>>
>> --
>> Everton Lima Aleixo
>> Bacharel em Ciência da Computação pela UFG
>> Mestrando em Ciência da Computação pela UFG
>> Programador no LUPA
>>
>>
>


-- 
Everton Lima Aleixo
Bacharel em Ciência da Computação pela UFG
Mestrando em Ciência da Computação pela UFG
Programador no LUPA

RE: Cassandra Client Recommendation

[Francisco Trujillo]

Hi

We are using Cassandra 1.6 at this moment. We start to work with Hector, 
because it is the first recommendation that you can find in a simple google 
search for java clients Cassandra.

We start using Hector but when we start to have non dynamically column 
families, that can be managed using cql, we start to use astyanax because:

-  It is easy to understand the code even for people who has never 
worked with Cassandra.

-  The cql implementation offer more capabilities

-  Astyanax is prepared to use Cql 3 and with hector we experienced 
some problems (probably our fault, but with Astyanax everything works from the 
beginning).

-  Astyanax allow to use compound primary keys.

In next months we are going to substitute Hector by Astyanax totally but at 
this moment we are using both:


-  Astyanax for cql.

-  Hector for dynamic column families.


From: Techy Teck [mailto:comptechge...@gmail.com]
Sent: woensdag 17 april 2013 8:14
To: user
Subject: Re: Cassandra Client Recommendation

Thanks Everton for the suggestion. Couple of questions-

1) Does Astyanax client have any problem with previous version of Cassandra?
2) You said one problem, that it will consume more memory? Can you elaborate 
that slightly? What do you mean by that?
3) Does Astyanax supports asynch capabilities?

On Tue, Apr 16, 2013 at 11:05 PM, Everton Lima 
mailto:peitin.inu...@gmail.com>> wrote:
Hi Techy,
We are using Astyanax with cassandra 1.2.4.

beneficits:
 * It is so easy to configure and use.
 * Good wiki
 * Mantained by Netflix
 * Solution to manage the store of big files (more than 15mb)
 * Solution to read all rows efficiently
problems:
 * It consume more memory

2013/4/16 Techy Teck mailto:comptechge...@gmail.com>>

Hello,
I have recently started working with Cassandra Database. Now I am in the 
process of evaluating which Cassandra client I should go forward with.

I am mainly interested in these three-

--1)  Astyanax client

2--)  New Datastax client that uses Binary protocol.

--3)  Pelops client



Can anyone provide some thoughts on this? Some advantages and disadvantages for 
these three will be great start for me.


Keeping in mind, we are running Cassandra 1.2.2 in production environment.



Thanks for the help.


--
Everton Lima Aleixo
Bacharel em Ciência da Computação pela UFG
Mestrando em Ciência da Computação pela UFG
Programador no LUPA