Re: memtable mem usage off by 10?

[Idrén , Johan]

I’m using the datastax rpms, using the bundled launch scripts.

grep -i jamm *
cassandra-env.sh:# add the jamm javaagent
cassandra-env.sh:JVM_OPTS="$JVM_OPTS 
-javaagent:$CASSANDRA_HOME/lib/jamm-0.2.5.jar”

And it’s part of the commandline used to start cassandra:

/usr/java/latest/bin/java -ea 
-javaagent:/usr/share/cassandra//lib/jamm-0.2.5.jar -XX:+UseThreadPriorities 
-XX:ThreadPriorityPolicy=42 -Xms10G -Xmx10G -Xmn2400M 
-XX:+HeapDumpOnOutOfMemoryError -Xss240k -XX:+UseParNewGC 
-XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 
-XX:MaxTenuringThreshold=4 -XX:CMSInitiatingOccupancyFraction=75 
-XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSIncrementalMode -XX:+UseCondCardMark 
-Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 
-Dcom.sun.management.jmxremote.ssl=false 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true 
-Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp 
/etc/cassandra/conf:/usr/share/java/jna.jar:/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/apache-cassandra-2.0.7.jar:/usr/share/cassandra/lib/apache-cassandra-clientutil-2.0.7.jar:/usr/share/cassandra/lib/apache-cassandra-thrift-2.0.7.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-15.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/lib/stress.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.3.jar
 org.apache.cassandra.service.CassandraDaemon



From: Benedict Elliott Smith 
mailto:belliottsm...@datastax.com>>
Reply-To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Date: Wednesday 4 June 2014 17:18
To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Subject: Re: memtable mem usage off by 10?

In that case I would assume the problem is that for some reason JAMM is failing 
to load, and so the liveRatio it would ordinarily calculate is defaulting to 10 
- are you using the bundled cassandra launch scripts?


On 4 June 2014 15:51, Idrén, Johan 
mailto:johan.id...@dice.se>> wrote:
I wasn’t supplying it, I was assuming it was using the default. It does not 
exist in my config file. Sorry for the confusion.



From: Benedict Elliott Smith 
mailto:belliottsm...@datastax.com>>
Reply-To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Date: Wednesday 4 June 2014 16:36
To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>

Subject: Re: memtable mem usage off by 10?

Oh, well ok that explains why I'm not seeing a flush at 750MB. Sorry, I was 
going by the documentation. It claims that the property is around in 2.0.

But something else is wrong, as Cassandra will crash if you supply an invalid 
property, implying it's not sourcing the config file you're using.

I'm afraid I don't have the context for why it was removed, but it happened as 
part of the 2.0 release.


On 4 June 2014 13:59, Jack Krupansky 
mailto:j...@basetechnology.com>> wrote:
Yeah, it is in the doc:
http://www.datastax.com/documentation/cassandra/2.0/cassandra/configuration/configCassandra_yaml_r.html

And I don’t find a Jira issue mentioning it being removed, so... what’s the 
full story there?!

-- Jack Krupansky

From: Idrén, Johan
Sent: Wednesday, June 4, 2014 8:26 AM
To: user@cassandra.apache.org
Subject: RE: memtable mem usage off by 10?


Oh, well ok that explains why I'm not seeing a flush at 750MB. Sorry, I was 
going by the documentation. It claims that the property is around in 2.0.



If we skip that, part of my reply still makes sense:



Having memtable_total_size_in_mb set to 20480, memtables are flushed at a

CQLSSTableWriter memory leak

[Xu Zhongxing]

I am using Cassandra's CQLSSTableWriter to import a large amount of data into 
Cassandra. When I use CQLSSTableWriter to write to a table with compound 
primary key, the memory consumption keeps growing. The GC of JVM cannot collect 
any used memory. When writing to tables with no compound primary key, the JVM 
GC works fine.

My Cassandra version is 2.0.5. The OS is Ubuntu 14.04 x86-64. JVM parameters 
are -Xms1g -Xmx2g. This is sufficient for all other non-compound primary key 
cases.

The problem can be reproduced by the following test case:

import org.apache.cassandra.io.sstable.CQLSSTableWriter;
import org.apache.cassandra.exceptions.InvalidRequestException;

import java.io.IOException;
import java.util.UUID;

class SS {
public static void main(String[] args) {
String schema = "create table test.t (x uuid, y uuid, primary key (x, 
y))";


String insert = "insert into test.t (x, y) values (?, ?)";
CQLSSTableWriter writer = CQLSSTableWriter.builder()
.inDirectory("/tmp/test/t")
.forTable(schema).withBufferSizeInMB(32)
.using(insert).build();

UUID id = UUID.randomUUID();
try {
for (int i = 0; i < 5000; i++) {
UUID id2 = UUID.randomUUID();
writer.addRow(id, id2);
}

writer.close();
} catch (Exception e) {
System.err.println("hell");
}
}
}

Re: Consolidating records and TTL

[Aaron Morton]

As Tyler says, with atomic batches which are enabled by default the cluster 
will keep trying to replay the insert / deletes. 

Nodes check their local batch log for failed batches, ones where the 
coordinator did not acknowledge it had successfully completed, every 60 
seconds. So there is a window where it’s possible for not all mutations in the 
batch to be completed. This could happen when a write timeout occurs when 
processing a batch of 2 rows; the request CL will not have been achieved on one 
or more of the rows. The coordinator will leave it up to the batch log to 
replay the request, and the client driver will (by default config) not retry. 

You can use a model like this. 

create table ledger (
account int, 
tx_id   timeuuid, 
sub_total   int,
primary key (account, tx_id)
);

create table account (
account int, 
total   int, 
last_tx_id  timeuuid, 
primary key (account)
);

To get the total:

select * from account where account = X;

Then get the ledger entries you need

select * from ledger where account = X and tx_id > last_tx_id;

This query will degrade when the partition size in the ledger table gets 
bigger, as it will need to read the column index (see column_index_size_in_kb 
in yaml). It will use that to find the first page that contains the rows we are 
interested in and then read forwards to the end of the row. It’s not the most 
efficient type of read but if you are going to delete ledger entries this 
*should* be able to skip over the tombstones without reading them. 

When you want to update the total in the account write to the account table and 
update both the total and the last_tx_id. You can then delete ledger entries if 
needed. Don’t forget to ensure that only one client thread is doing this at a 
time. 

Hope that helps. 
Aaron


-
Aaron Morton
New Zealand
@aaronmorton

Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

On 5/06/2014, at 10:37 am, Tyler Hobbs  wrote:

> Just use an atomic batch that holds both the insert and deletes: 
> http://www.datastax.com/dev/blog/atomic-batches-in-cassandra-1-2
> 
> 
> On Tue, Jun 3, 2014 at 2:13 PM, Charlie Mason  wrote:
> Hi All.
> 
> I have a system thats going to make possibly several concurrent changes to a 
> running total. I know I could use a counter for this. However I have extra 
> meta data I can store with the changes which would allow me to reply the 
> changes. If I use a counter and it looses some writes I can't recover it as I 
> will only have its current total not the extra meta data to know where to 
> replay from.
> 
> What I was planning to do was write each change of the value to a CQL table 
> with a Time UUID as a row level primary key as well as a partition key. Then 
> when I need to read the running total back I will do a query for all the 
> changes and add them up to get the total.
> 
> As there could be tens of thousands of these I want to have a period after 
> which these are consolidated. Most won't be any where near that but a few 
> will which I need to be able to support. So I was also going to have a 
> consolidated total table which holds the UUID of the values consolidated up 
> to. Since I can bound the query for the recent updates by the UUID I should 
> be able to avoid all the tombstones. So if the read encounters any changes 
> that can be consolidated it inserts a new consolidated value and deletes the 
> newly consolidated changes.
> 
> What I am slightly worried about is what happens if the consolidated value 
> insert fails but the deletes to the change records succeed. I would be left 
> with an inconsistent total indefinitely. I have come up with a couple of 
> ideas:
> 
> 
> 1, I could make it require all nodes to acknowledge it before deleting the 
> difference records.
> 
> 2, May be I could have another period after its consolidated but before its 
> deleted?
> 
> 3, Is there anyway I could use the TTL to allow to it to be deleted after a 
> period of time? Chances are another read would come in and fix the value.
> 
> 
> Anyone got any other suggestions on how I could implement this?
> 
> 
> Thanks,
> 
> Charlie M
> 
> 
> 
> -- 
> Tyler Hobbs
> DataStax

VPC AWS

[Alain RODRIGUEZ]

Hi guys,

We are going to move from a cluster made of simple Amazon EC2 servers to a
VPC cluster. We are using Cassandra 1.2.11 and I have some questions
regarding this switch and the Cassandra configuration inside a VPC.

Actually I found no documentation on this topic, but I am quite sure that
some people are already using VPC. If you can point me to any documentation
regarding VPC / Cassandra, it would be very nice of you. We have only one
DC for now, but we need to remain multi DC compatible, since we will add DC
very soon.

Else, I would like to know if I should keep using EC2MultiRegionSnitch or
change the snitch to anything else.

What about broadcast/listen ip, seeds...?

We currently use public ip as for broadcast address and for seeds. We use
private ones for listen address. Machines inside the VPC will only have
private IP AFAIK. Should I keep using a broadcast address ?

Is there any other incidence when switching to a VPC ?

Sorry if the topic was already discussed, I was unable to find any useful
information...

Re: VPC AWS

[Idrén , Johan]

Using 2.0.x, we hit https://issues.apache.org/jira/browse/CASSANDRA-2380 when 
deploying in a VPC.
We worked around this by adding a hostname in /etc/hosts.

Apart from that, nothing specific. We use only internal ip's, no public ip's 
and no broadcast etc. It works very well.

/Johan

From: Alain RODRIGUEZ mailto:arodr...@gmail.com>>
Reply-To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Date: Thursday 5 June 2014 14:12
To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Subject: VPC AWS

Hi guys,

We are going to move from a cluster made of simple Amazon EC2 servers to a VPC 
cluster. We are using Cassandra 1.2.11 and I have some questions regarding this 
switch and the Cassandra configuration inside a VPC.

Actually I found no documentation on this topic, but I am quite sure that some 
people are already using VPC. If you can point me to any documentation 
regarding VPC / Cassandra, it would be very nice of you. We have only one DC 
for now, but we need to remain multi DC compatible, since we will add DC very 
soon.

Else, I would like to know if I should keep using EC2MultiRegionSnitch or 
change the snitch to anything else.

What about broadcast/listen ip, seeds...?

We currently use public ip as for broadcast address and for seeds. We use 
private ones for listen address. Machines inside the VPC will only have private 
IP AFAIK. Should I keep using a broadcast address ?

Is there any other incidence when switching to a VPC ?

Sorry if the topic was already discussed, I was unable to find any useful 
information...

Re: VPC AWS

[Michael Theroux]

Hello Alain,

We switched from EC2 to VPC a couple of years ago.  The process for us was 
long, slow, and multi step for our (at the time) 6 node cluster.

In our case, we don't need to consider multi-DC.  However, in our 
infrastructure we were rapidly running out of IP addresses, and wished to move 
to VPC to give us a nearly inexhaustible supply.  In addition, AWS VPC gives us 
an additional layer of security for our Cassandra cluster. 

To do this, we setup our VPC to have both private and public subnets.  Public 
subnets were accessible to the Internet (when instances were assigned a public 
IP), while private subnets could not (although instances on the subnet could 
access the Internet via a NAT instance).  We wished for to be Cassandra on the 
private subnet.  However, this introduced a complication.  EC2 instances would 
not be able to communicate directly to our VPC instances on a private subnet. 

So, to achieve this, while still having an operating Cassandra DB without 
downtime, we essentially had to stage Cassandra instances on our public subnet, 
assigning IPs and reconfiguring nodes until we had a mixed EC2/VPC Public 
subnet cluster, then start moving systems to the private subnet, continuing the 
process until all instances were on a private subnet.  During the process we 
carefully orchestrated configuration like broadcast and seeds to make sure the 
cluster continued to function properly and all nodes could communicate with 
each other.  We also had to carefully orchestrate the assigning of AWS security 
groups to make sure everyone could talk to each other during this process.

Also keep in mind that the use of public IPs for communications will add to 
your AWS costs.  During our transition we had to do this for a short time while 
EC2 instances were communicating with VPC instances, but we were able to switch 
to 100% internal IPs when we completed (you will still get inter availability 
zone charges regardless)

This process was complex enough that I wrote detailed series of steps, for each 
node in our cluster.

-Mike
 


 From: Alain RODRIGUEZ 
To: user@cassandra.apache.org 
Sent: Thursday, June 5, 2014 8:12 AM
Subject: VPC AWS
 


Hi guys,

We are going to move from a cluster made of simple Amazon EC2 servers to a VPC 
cluster. We are using Cassandra 1.2.11 and I have some questions regarding this 
switch and the Cassandra configuration inside a VPC.

Actually I found no documentation on this topic, but I am quite sure that some 
people are already using VPC. If you can point me to any documentation 
regarding VPC / Cassandra, it would be very nice of you. We have only one DC 
for now, but we need to remain multi DC compatible, since we will add DC very 
soon.

Else, I would like to know if I should keep using EC2MultiRegionSnitch or 
change the snitch to anything else.

What about broadcast/listen ip, seeds...?

We currently use public ip as for broadcast address and for seeds. We use 
private ones for listen address. Machines inside the VPC will only have private 
IP AFAIK. Should I keep using a broadcast address ?

Is there any other incidence when switching to a VPC ?

Sorry if the topic was already discussed, I was unable to find any useful 
information...

Re: VPC AWS

[Michael Theroux]

Hello Alain,

We switched from EC2 to VPC a couple of years ago.  The process for us was 
long, slow and multi step.

In our case, we don't need to consider multi-DC.  However, in our 
infrastructure we were rapidly running out of IP addresses, and wished to move 
to VPC to give us a nearly inexhaustible supply.  In addition, AWS VPC gives us 
an additional layer of security for our Cassandra cluster. 

To do this, we setup our VPC to have both private and public subnets.  Public 
subnets were accessible to the Internet (when instances were assigned a public 
IP, while private subnets could not (although instances on the subnet could 
access the Internet via a NAT instance).  We wished for Cassandra on the 
private subnet. 

So, to achieve this, while still having an operating Cassandra DB without 
downtime, we essentially had to stage Cassandra instances on our public subnet, 
assigning IPs and reconfiguring nodes until we had a mixed EC2/VPC Public 
cluster, then start moving systems to the private subnet, continuing the 
process until all instances were on a private subnet.  During the process we 
carefully orchestrated configuration like broadcast and seeds to make sure the 
cluster continued to function properly.  We also had to orchestrate the 
assigning of AWS security groups to make sure everyone could talk to each other 
during this process.

Also keep in mind that the use of public IPs for communications will add to 
your AWS costs.  During our transition we had to do this for a short time while 
EC2 instances were communicating with VPC instances, but we were able to switch 
to 100% internal IPs when we completed (you will still get inter availability 
zone charges regardless)

In order to make this successful, I created a script out


 From: Alain RODRIGUEZ 
To: user@cassandra.apache.org 
Sent: Thursday, June 5, 2014 8:12 AM
Subject: VPC AWS
 


Hi guys,

We are going to move from a cluster made of simple Amazon EC2 servers to a VPC 
cluster. We are using Cassandra 1.2.11 and I have some questions regarding this 
switch and the Cassandra configuration inside a VPC.

Actually I found no documentation on this topic, but I am quite sure that some 
people are already using VPC. If you can point me to any documentation 
regarding VPC / Cassandra, it would be very nice of you. We have only one DC 
for now, but we need to remain multi DC compatible, since we will add DC very 
soon.

Else, I would like to know if I should keep using EC2MultiRegionSnitch or 
change the snitch to anything else.

What about broadcast/listen ip, seeds...?

We currently use public ip as for broadcast address and for seeds. We use 
private ones for listen address. Machines inside the VPC will only have private 
IP AFAIK. Should I keep using a broadcast address ?

Is there any other incidence when switching to a VPC ?

Sorry if the topic was already discussed, I was unable to find any useful 
information...

Re: VPC AWS

[William Oberman]

I don't think traffic will flow between "classic" ec2 and vpc directly.
There is some kind of gateway bridge instance that sits between, acting as
a NAT.   I would think that would cause new challenges for:
-transitions
-clients

Sorry this response isn't heavy on content!  I'm curious how this thread
goes...

Will

On Thursday, June 5, 2014, Alain RODRIGUEZ  wrote:

> Hi guys,
>
> We are going to move from a cluster made of simple Amazon EC2 servers to a
> VPC cluster. We are using Cassandra 1.2.11 and I have some questions
> regarding this switch and the Cassandra configuration inside a VPC.
>
> Actually I found no documentation on this topic, but I am quite sure that
> some people are already using VPC. If you can point me to any documentation
> regarding VPC / Cassandra, it would be very nice of you. We have only one
> DC for now, but we need to remain multi DC compatible, since we will add DC
> very soon.
>
> Else, I would like to know if I should keep using EC2MultiRegionSnitch or
> change the snitch to anything else.
>
> What about broadcast/listen ip, seeds...?
>
> We currently use public ip as for broadcast address and for seeds. We use
> private ones for listen address. Machines inside the VPC will only have
> private IP AFAIK. Should I keep using a broadcast address ?
>
>  Is there any other incidence when switching to a VPC ?
>
> Sorry if the topic was already discussed, I was unable to find any useful
> information...
>


-- 
Will Oberman
Civic Science, Inc.
6101 Penn Avenue, Fifth Floor
Pittsburgh, PA 15206
(M) 412-480-7835
(E) ober...@civicscience.com

Re: VPC AWS

[Alain RODRIGUEZ]

I think you can define VPC subnet to be public (to have public + private
IPs) or private only.

Any insight regarding snitches ? What snitch do you guys use ?


2014-06-05 15:06 GMT+02:00 William Oberman :

> I don't think traffic will flow between "classic" ec2 and vpc directly.
> There is some kind of gateway bridge instance that sits between, acting as
> a NAT.   I would think that would cause new challenges for:
> -transitions
> -clients
>
> Sorry this response isn't heavy on content!  I'm curious how this thread
> goes...
>
> Will
>
> On Thursday, June 5, 2014, Alain RODRIGUEZ  wrote:
>
>> Hi guys,
>>
>> We are going to move from a cluster made of simple Amazon EC2 servers to
>> a VPC cluster. We are using Cassandra 1.2.11 and I have some questions
>> regarding this switch and the Cassandra configuration inside a VPC.
>>
>> Actually I found no documentation on this topic, but I am quite sure that
>> some people are already using VPC. If you can point me to any documentation
>> regarding VPC / Cassandra, it would be very nice of you. We have only one
>> DC for now, but we need to remain multi DC compatible, since we will add DC
>> very soon.
>>
>> Else, I would like to know if I should keep using EC2MultiRegionSnitch or
>> change the snitch to anything else.
>>
>> What about broadcast/listen ip, seeds...?
>>
>> We currently use public ip as for broadcast address and for seeds. We use
>> private ones for listen address. Machines inside the VPC will only have
>> private IP AFAIK. Should I keep using a broadcast address ?
>>
>>  Is there any other incidence when switching to a VPC ?
>>
>> Sorry if the topic was already discussed, I was unable to find any useful
>> information...
>>
>
>
> --
> Will Oberman
> Civic Science, Inc.
> 6101 Penn Avenue, Fifth Floor
> Pittsburgh, PA 15206
> (M) 412-480-7835
> (E) ober...@civicscience.com
>

Re: VPC AWS

[William Oberman]

I was making assumptions (sorry!), namely the goal to switching to vpc
was getting "backend resources" (like cassandra) off the public internet.

Will

On Thursday, June 5, 2014, Alain RODRIGUEZ  wrote:

> I think you can define VPC subnet to be public (to have public + private
> IPs) or private only.
>
> Any insight regarding snitches ? What snitch do you guys use ?
>
>
> 2014-06-05 15:06 GMT+02:00 William Oberman  >:
>
>> I don't think traffic will flow between "classic" ec2 and vpc directly.
>> There is some kind of gateway bridge instance that sits between, acting as
>> a NAT.   I would think that would cause new challenges for:
>> -transitions
>> -clients
>>
>> Sorry this response isn't heavy on content!  I'm curious how this thread
>> goes...
>>
>> Will
>>
>> On Thursday, June 5, 2014, Alain RODRIGUEZ > > wrote:
>>
>>> Hi guys,
>>>
>>> We are going to move from a cluster made of simple Amazon EC2 servers to
>>> a VPC cluster. We are using Cassandra 1.2.11 and I have some questions
>>> regarding this switch and the Cassandra configuration inside a VPC.
>>>
>>> Actually I found no documentation on this topic, but I am quite sure
>>> that some people are already using VPC. If you can point me to any
>>> documentation regarding VPC / Cassandra, it would be very nice of you. We
>>> have only one DC for now, but we need to remain multi DC compatible, since
>>> we will add DC very soon.
>>>
>>> Else, I would like to know if I should keep using EC2MultiRegionSnitch
>>> or change the snitch to anything else.
>>>
>>> What about broadcast/listen ip, seeds...?
>>>
>>> We currently use public ip as for broadcast address and for seeds. We
>>> use private ones for listen address. Machines inside the VPC will only have
>>> private IP AFAIK. Should I keep using a broadcast address ?
>>>
>>>  Is there any other incidence when switching to a VPC ?
>>>
>>> Sorry if the topic was already discussed, I was unable to find any
>>> useful information...
>>>
>>
>>
>> --
>> Will Oberman
>> Civic Science, Inc.
>> 6101 Penn Avenue, Fifth Floor
>> Pittsburgh, PA 15206
>> (M) 412-480-7835
>> (E) ober...@civicscience.com
>> 
>>
>
>

-- 
Will Oberman
Civic Science, Inc.
6101 Penn Avenue, Fifth Floor
Pittsburgh, PA 15206
(M) 412-480-7835
(E) ober...@civicscience.com

Re: VPC AWS

[David McNelis]

A general thought, if you're using AWS for this, I'd strongly recommend you
consider using OpsWorks and custom chef recipes for your node deployment if
its an option for you.  The easy of provisioning new nodes without the need
for snapshotting is certainly worth the hassle, and there are already
several Cassandra chef recipes available with a little searching.

David


On Thu, Jun 5, 2014 at 9:26 AM, William Oberman 
wrote:

> I was making assumptions (sorry!), namely the goal to switching to vpc
> was getting "backend resources" (like cassandra) off the public internet.
>
> Will
>
> On Thursday, June 5, 2014, Alain RODRIGUEZ  wrote:
>
>> I think you can define VPC subnet to be public (to have public + private
>> IPs) or private only.
>>
>> Any insight regarding snitches ? What snitch do you guys use ?
>>
>>
>> 2014-06-05 15:06 GMT+02:00 William Oberman :
>>
>>> I don't think traffic will flow between "classic" ec2 and vpc directly.
>>> There is some kind of gateway bridge instance that sits between, acting as
>>> a NAT.   I would think that would cause new challenges for:
>>> -transitions
>>> -clients
>>>
>>> Sorry this response isn't heavy on content!  I'm curious how this thread
>>> goes...
>>>
>>> Will
>>>
>>> On Thursday, June 5, 2014, Alain RODRIGUEZ  wrote:
>>>
 Hi guys,

 We are going to move from a cluster made of simple Amazon EC2 servers
 to a VPC cluster. We are using Cassandra 1.2.11 and I have some questions
 regarding this switch and the Cassandra configuration inside a VPC.

 Actually I found no documentation on this topic, but I am quite sure
 that some people are already using VPC. If you can point me to any
 documentation regarding VPC / Cassandra, it would be very nice of you. We
 have only one DC for now, but we need to remain multi DC compatible, since
 we will add DC very soon.

 Else, I would like to know if I should keep using EC2MultiRegionSnitch
 or change the snitch to anything else.

 What about broadcast/listen ip, seeds...?

 We currently use public ip as for broadcast address and for seeds. We
 use private ones for listen address. Machines inside the VPC will only have
 private IP AFAIK. Should I keep using a broadcast address ?

  Is there any other incidence when switching to a VPC ?

 Sorry if the topic was already discussed, I was unable to find any
 useful information...

>>>
>>>
>>> --
>>> Will Oberman
>>> Civic Science, Inc.
>>> 6101 Penn Avenue, Fifth Floor
>>> Pittsburgh, PA 15206
>>> (M) 412-480-7835
>>> (E) ober...@civicscience.com
>>>
>>
>>
>
> --
> Will Oberman
> Civic Science, Inc.
> 6101 Penn Avenue, Fifth Floor
> Pittsburgh, PA 15206
> (M) 412-480-7835
> (E) ober...@civicscience.com
>

Re: VPC AWS

[Alain RODRIGUEZ]

Well if you are talking about the transition I think we are going to do it
this way

- Existing cluster out of any VPC -> DC1
- New VPC cluster, with subnet private + public -> DC2
- Add / Rebuild DC2
- Switch clients from DC1 to DC2
- Remove DC1
- Switch client inside the VPC
- Remove public VPC subnet

--> Issue in this last step how to switch cassandra configuation from
public to private IPs without any downtime ?

What do you think about this approach ?


2014-06-05 15:29 GMT+02:00 David McNelis :

> A general thought, if you're using AWS for this, I'd strongly recommend
> you consider using OpsWorks and custom chef recipes for your node
> deployment if its an option for you.  The easy of provisioning new nodes
> without the need for snapshotting is certainly worth the hassle, and there
> are already several Cassandra chef recipes available with a little
> searching.
>
> David
>
>
> On Thu, Jun 5, 2014 at 9:26 AM, William Oberman 
> wrote:
>
>> I was making assumptions (sorry!), namely the goal to switching to vpc
>> was getting "backend resources" (like cassandra) off the public internet.
>>
>> Will
>>
>> On Thursday, June 5, 2014, Alain RODRIGUEZ  wrote:
>>
>>> I think you can define VPC subnet to be public (to have public + private
>>> IPs) or private only.
>>>
>>>  Any insight regarding snitches ? What snitch do you guys use ?
>>>
>>>
>>> 2014-06-05 15:06 GMT+02:00 William Oberman :
>>>
 I don't think traffic will flow between "classic" ec2 and vpc directly.
 There is some kind of gateway bridge instance that sits between, acting as
 a NAT.   I would think that would cause new challenges for:
 -transitions
 -clients

 Sorry this response isn't heavy on content!  I'm curious how this
 thread goes...

 Will

 On Thursday, June 5, 2014, Alain RODRIGUEZ  wrote:

> Hi guys,
>
> We are going to move from a cluster made of simple Amazon EC2 servers
> to a VPC cluster. We are using Cassandra 1.2.11 and I have some questions
> regarding this switch and the Cassandra configuration inside a VPC.
>
> Actually I found no documentation on this topic, but I am quite sure
> that some people are already using VPC. If you can point me to any
> documentation regarding VPC / Cassandra, it would be very nice of you. We
> have only one DC for now, but we need to remain multi DC compatible, since
> we will add DC very soon.
>
> Else, I would like to know if I should keep using EC2MultiRegionSnitch
> or change the snitch to anything else.
>
> What about broadcast/listen ip, seeds...?
>
> We currently use public ip as for broadcast address and for seeds. We
> use private ones for listen address. Machines inside the VPC will only 
> have
> private IP AFAIK. Should I keep using a broadcast address ?
>
>  Is there any other incidence when switching to a VPC ?
>
> Sorry if the topic was already discussed, I was unable to find any
> useful information...
>


 --
 Will Oberman
 Civic Science, Inc.
 6101 Penn Avenue, Fifth Floor
 Pittsburgh, PA 15206
 (M) 412-480-7835
 (E) ober...@civicscience.com

>>>
>>>
>>
>> --
>> Will Oberman
>> Civic Science, Inc.
>> 6101 Penn Avenue, Fifth Floor
>> Pittsburgh, PA 15206
>> (M) 412-480-7835
>> (E) ober...@civicscience.com
>>
>
>

Re: memtable mem usage off by 10?

[Benedict Elliott Smith]

What does

/usr/java/latest/bin/java -version

print?


On 5 June 2014 08:15, Idrén, Johan  wrote:

>  I’m using the datastax rpms, using the bundled launch scripts.
>
>  grep -i jamm *
> cassandra-env.sh:# add the jamm javaagent
> cassandra-env.sh:JVM_OPTS="$JVM_OPTS
> -javaagent:$CASSANDRA_HOME/lib/jamm-0.2.5.jar”
>
>  And it’s part of the commandline used to start cassandra:
>
>  /usr/java/latest/bin/java -ea
> -javaagent:/usr/share/cassandra//lib/jamm-0.2.5.jar
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms10G -Xmx10G
> -Xmn2400M -XX:+HeapDumpOnOutOfMemoryError -Xss240k -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
> -XX:MaxTenuringThreshold=4 -XX:CMSInitiatingOccupancyFraction=75
> -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSIncrementalMode
> -XX:+UseCondCardMark -Djava.net.preferIPv4Stack=true
> -Dcom.sun.management.jmxremote.port=7199
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dlog4j.configuration=log4j-server.properties
> -Dlog4j.defaultInitOverride=true
> -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp
> /etc/cassandra/conf:/usr/share/java/jna.jar:/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/apache-cassandra-2.0.7.jar:/usr/share/cassandra/lib/apache-cassandra-clientutil-2.0.7.jar:/usr/share/cassandra/lib/apache-cassandra-thrift-2.0.7.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-15.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/lib/stress.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.3.jar
> org.apache.cassandra.service.CassandraDaemon
>
>
>
>   From: Benedict Elliott Smith 
> Reply-To: "user@cassandra.apache.org" 
> Date: Wednesday 4 June 2014 17:18
>
> To: "user@cassandra.apache.org" 
> Subject: Re: memtable mem usage off by 10?
>
>   In that case I would assume the problem is that for some reason JAMM is
> failing to load, and so the liveRatio it would ordinarily calculate is
> defaulting to 10 - are you using the bundled cassandra launch scripts?
>
>
> On 4 June 2014 15:51, Idrén, Johan  wrote:
>
>>  I wasn’t supplying it, I was assuming it was using the default. It does
>> not exist in my config file. Sorry for the confusion.
>>
>>
>>
>>   From: Benedict Elliott Smith 
>>  Reply-To: "user@cassandra.apache.org" 
>> Date: Wednesday 4 June 2014 16:36
>> To: "user@cassandra.apache.org" 
>>
>> Subject: Re: memtable mem usage off by 10?
>>
>>Oh, well ok that explains why I'm not seeing a flush at 750MB. Sorry,
>>> I was going by the documentation. It claims that the property is around in
>>> 2.0.
>>
>> But something else is wrong, as Cassandra will crash if you supply an
>> invalid property, implying it's not sourcing the config file you're using.
>>  I'm afraid I don't have the context for why it was removed, but it
>> happened as part of the 2.0 release.
>>
>>>
>>
>> On 4 June 2014 13:59, Jack Krupansky  wrote:
>>
>>>   Yeah, it is in the doc:
>>>
>>> http://www.datastax.com/documentation/cassandra/2.0/cassandra/configuration/configCassandra_yaml_r.html
>>>
>>> And I don’t find a Jira issue mentioning it being removed, so... what’s
>>> the full story there?!
>>>
>>> -- Jack Krupansky
>>>
>>>  *From:* Idrén, Johan 
>>> *Sent:* Wednesday, June 4, 2014 8:26 AM
>>> *To:* user@cassandra.apache.org
>>> *Subject:* RE: memtable mem usage off by 10?
>>>
>>>
>>> Oh, well ok that explains why I'm not seeing a flush at 750MB. Sorry, I
>>> was going by the documentation. It claims that the property is around in
>>> 2.0.
>>>
>>>
>>>
>>> If we skip that, part of my reply still makes sense:
>>>
>>>
>>>
>>> Having memtable_total_size_in_mb set to 20480, memtables are flushed at
>>> a reported value of ~2GB.
>>>
>>>
>>>
>>> With a constant overhead of ~10x, as suggested, this would mean that it
>>> used 20G

Re: VPC AWS

[Michael Theroux]

We personally use the EC2Snitch, however, we don't have the multi-region 
requirements you do,

-Mike



 From: Alain RODRIGUEZ 
To: user@cassandra.apache.org 
Sent: Thursday, June 5, 2014 9:14 AM
Subject: Re: VPC AWS
 


I think you can define VPC subnet to be public (to have public + private IPs) 
or private only.

Any insight regarding snitches ? What snitch do you guys use ?



2014-06-05 15:06 GMT+02:00 William Oberman :

I don't think traffic will flow between "classic" ec2 and vpc directly. There 
is some kind of gateway bridge instance that sits between, acting as a NAT.   I 
would think that would cause new challenges for:
>
>-transitions 
>-clients
>
>Sorry this response isn't heavy on content!  I'm curious how this thread 
>goes...
>
>
>Will
>
>
>On Thursday, June 5, 2014, Alain RODRIGUEZ  wrote:
>
>Hi guys,
>>
>>
>>We are going to move from a cluster made of simple Amazon EC2 servers to a 
>>VPC cluster. We are using Cassandra 1.2.11 and I have some questions 
>>regarding this switch and the Cassandra configuration inside a VPC.
>>
>>
>>Actually I found no documentation on this topic, but I am quite sure that 
>>some people are already using VPC. If you can point me to any documentation 
>>regarding VPC / Cassandra, it would be very nice of you. We have only one DC 
>>for now, but we need to remain multi DC compatible, since we will add DC very 
>>soon.
>>
>>
>>Else, I would like to know if I should keep using EC2MultiRegionSnitch or 
>>change the snitch to anything else.
>>
>>
>>What about broadcast/listen ip, seeds...?
>>
>>
>>We currently use public ip as for broadcast address and for seeds. We use 
>>private ones for listen address. Machines inside the VPC will only have 
>>private IP AFAIK. Should I keep using a broadcast address ?
>>
>>
>>Is there any other incidence when switching to a VPC ?
>>
>>
>>Sorry if the topic was already discussed, I was unable to find any useful 
>>information...
>
>-- 
>Will Oberman
>Civic Science, Inc.
>6101 Penn Avenue, Fifth Floor
>Pittsburgh, PA 15206
>(M) 412-480-7835
>(E) ober...@civicscience.com
>

Migration 1.2.14 to 2.0.8 causes "Tried to create duplicate hard link" at startup

[Tom van den Berge]

Hi,

I'm trying to migrate a development cluster from 1.2.14 to 2.0.8. When
starting up 2.0.8, I'm seeing the following error in the logs:


 INFO 17:40:25,405 Snapshotting drillster, Account to
pre-sstablemetamigration
ERROR 17:40:25,407 Exception encountered during startup
java.lang.RuntimeException: Tried to create duplicate hard link to
/Users/tom/cassandra-data/data/drillster/Account/snapshots/pre-sstablemetamigration/drillster-Account-ic-65-Filter.db
at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:75)
at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:129)
at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91)
at
org.apache.cassandra.db.compaction.LeveledManifest.maybeMigrateManifests(LeveledManifest.java:617)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:274)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)


Does anyone have an idea how to solve this?


Thanks,
Tom

Re: CQLSSTableWriter memory leak

[Jack Krupansky]

How many rows (primary key values) are you writing for each partition of the 
primary key? I mean, are there relatively few, or are these very wide 
partitions?

Oh, I see! You’re writing 50,000,000 rows to a single partition! My, that IS 
ambitious.

-- Jack Krupansky

From: Xu Zhongxing 
Sent: Thursday, June 5, 2014 3:34 AM
To: user@cassandra.apache.org 
Subject: CQLSSTableWriter memory leak

I am using Cassandra's CQLSSTableWriter to import a large amount of data into 
Cassandra. When I use CQLSSTableWriter to write to a table with compound 
primary key, the memory consumption keeps growing. The GC of JVM cannot collect 
any used memory. When writing to tables with no compound primary key, the JVM 
GC works fine.

My Cassandra version is 2.0.5. The OS is Ubuntu 14.04 x86-64. JVM parameters 
are -Xms1g -Xmx2g. This is sufficient for all other non-compound primary key 
cases.

The problem can be reproduced by the following test case:

import org.apache.cassandra.io.sstable.CQLSSTableWriter;
import org.apache.cassandra.exceptions.InvalidRequestException;

import java.io.IOException;
import java.util.UUID;

class SS {
public static void main(String[] args) {
String schema = "create table test.t (x uuid, y uuid, primary key (x, 
y))";


String insert = "insert into test.t (x, y) values (?, ?)";
CQLSSTableWriter writer = CQLSSTableWriter.builder()
.inDirectory("/tmp/test/t")
.forTable(schema).withBufferSizeInMB(32)
.using(insert).build();

UUID id = UUID.randomUUID();
try {
for (int i = 0; i < 5000; i++) {
UUID id2 = UUID.randomUUID();
writer.addRow(id, id2);
}

writer.close();
} catch (Exception e) {
System.err.println("hell");
}
}
}

RE: Consolidating records and TTL

[James Campbell]

Thanks for creating and opening the discussion on this use case.  I have been 
evaluating Cassandra for a very similar problem, but with the small twist that 
I'd like to roll up the ledger entries into the aggregated 'account' 
information on a regular basis to ensure that even rarely-read (but often 
written) data will be periodically rolled up and ready for quick reads.  There 
is also a larger twist that for me 'account' is a compound key and the typical 
read pattern will include reading from multiple rows within the same partition 
(each of which would require separate merging from the ledger).


This response prompts three questions for me about that:


1. Just to clarify the requirement that only one client thread does this at a 
time, I assume you mean only one thread should handle any given 'account' at a 
time, right?  There wouldn't be problems having multiple clients do this data 
maintenance on different accounts at the same time, I hope.


2. For my use case, hadoop integration seems the natural fit, because I'd like 
to batch these updates to run at night during query down-time, but the current 
hadoop tools don't really support reading from or writing to multiple tables, 
as is required for this implementation.  I've seen a few posts from people who 
have written homegrown input/output formats for hadoop, but I haven't tried to 
use them or evaluate their stability.  Is there another mechanism I should be 
thinking about for that sort of batch updating?​


James Campbell


From: Aaron Morton 
Sent: Thursday, June 5, 2014 5:26 AM
To: Cassandra User
Cc: charlie@gmail.com
Subject: Re: Consolidating records and TTL

As Tyler says, with atomic batches which are enabled by default the cluster 
will keep trying to replay the insert / deletes.

Nodes check their local batch log for failed batches, ones where the 
coordinator did not acknowledge it had successfully completed, every 60 
seconds. So there is a window where it’s possible for not all mutations in the 
batch to be completed. This could happen when a write timeout occurs when 
processing a batch of 2 rows; the request CL will not have been achieved on one 
or more of the rows. The coordinator will leave it up to the batch log to 
replay the request, and the client driver will (by default config) not retry.

You can use a model like this.

create table ledger (
account int,
tx_id  timeuuid,
sub_total  int,
primary key (account, tx_id)
);

create table account (
account  int,
total int,
last_tx_id timeuuid,
primary key (account)
);

To get the total:

select * from account where account = X;

Then get the ledger entries you need

select * from ledger where account = X and tx_id > last_tx_id;

This query will degrade when the partition size in the ledger table gets 
bigger, as it will need to read the column index (see column_index_size_in_kb 
in yaml). It will use that to find the first page that contains the rows we are 
interested in and then read forwards to the end of the row. It’s not the most 
efficient type of read but if you are going to delete ledger entries this 
*should* be able to skip over the tombstones without reading them.

When you want to update the total in the account write to the account table and 
update both the total and the last_tx_id. You can then delete ledger entries if 
needed. Don’t forget to ensure that only one client thread is doing this at a 
time.

Hope that helps.
Aaron


-
Aaron Morton
New Zealand
@aaronmorton

Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

On 5/06/2014, at 10:37 am, Tyler Hobbs 
mailto:ty...@datastax.com>> wrote:

Just use an atomic batch that holds both the insert and deletes: 
http://www.datastax.com/dev/blog/atomic-batches-in-cassandra-1-2


On Tue, Jun 3, 2014 at 2:13 PM, Charlie Mason 
mailto:charlie@gmail.com>> wrote:
Hi All.

I have a system thats going to make possibly several concurrent changes to a 
running total. I know I could use a counter for this. However I have extra meta 
data I can store with the changes which would allow me to reply the changes. If 
I use a counter and it looses some writes I can't recover it as I will only 
have its current total not the extra meta data to know where to replay from.

What I was planning to do was write each change of the value to a CQL table 
with a Time UUID as a row level primary key as well as a partition key. Then 
when I need to read the running total back I will do a query for all the 
changes and add them up to get the total.

As there could be tens of thousands of these I want to have a period after 
which these are consolidated. Most won't be any where near that but a few will 
which I need to be able to support. So I was also going to have a consolidated 
total table which holds the UUID of the values consolidated up to. Since I can 
bound the query for the recent updates by the UUID I should be able to avoid 
all the tombs

Re: migration to a new model

[Marcelo Elias Del Valle]

Michael,

I will try to test it up to tomorrow and I will let you know all the
results.

Thanks a lot!

Best regards,
Marcelo.


2014-06-04 22:28 GMT-03:00 Laing, Michael :

> BTW you might want to put a LIMIT clause on your SELECT for testing. -ml
>
>
> On Wed, Jun 4, 2014 at 6:04 PM, Laing, Michael 
> wrote:
>
>> Marcelo,
>>
>> Here is a link to the preview of the python fast copy program:
>>
>> https://gist.github.com/michaelplaing/37d89c8f5f09ae779e47
>>
>> It will copy a table from one cluster to another with some
>> transformation- they can be the same cluster.
>>
>> It has 3 main throttles to experiment with:
>>
>>1. fetch_size: size of source pages in rows
>>2. worker_count: number of worker subprocesses
>>3. concurrency: number of async callback chains per worker subprocess
>>
>> It is easy to overrun Cassandra and the python driver, so I recommend
>> starting with the defaults: fetch_size: 1000; worker_count: 2; concurrency:
>> 10.
>>
>> Additionally there are switches to set 'policies' by source and
>> destination: retry (downgrade consistency), dc_aware, and token_aware.
>> retry is useful if you are getting timeouts. For the others YMMV.
>>
>> To use it you need to define the SELECT and UPDATE cql statements as well
>> as the 'map_fields' method.
>>
>> The worker subprocesses divide up the token range among themselves and
>> proceed quasi-independently. Each worker opens a connection to each cluster
>> and the driver sets up connection pools to the nodes in the cluster. Anyway
>> there are a lot of processes, threads, callbacks going at once so it is fun
>> to watch.
>>
>> On my regional cluster of small nodes in AWS I got about 3000 rows per
>> second transferred after things warmed up a bit - each row about 6kb.
>>
>> ml
>>
>>
>> On Wed, Jun 4, 2014 at 11:49 AM, Laing, Michael <
>> michael.la...@nytimes.com> wrote:
>>
>>> OK Marcelo, I'll work on it today. -ml
>>>
>>>
>>> On Tue, Jun 3, 2014 at 8:24 PM, Marcelo Elias Del Valle <
>>> marc...@s1mbi0se.com.br> wrote:
>>>
 Hi Michael,

 For sure I would be interested in this program!

 I am new both to python and for cql. I started creating this copier,
 but was having problems with timeouts. Alex solved my problem here on the
 list, but I think I will still have a lot of trouble making the copy to
 work fine.

 I open sourced my version here:
 https://github.com/s1mbi0se/cql_record_processor

 Just in case it's useful for anything.

 However, I saw CQL has support for concurrency itself and having
 something made by someone who knows Python CQL Driver better would be very
 helpful.

 My two servers today are at OVH (ovh.com), we have servers at AWS but
 but several cases we prefer other hosts. Both servers have SDD and 64 Gb
 RAM, I could use the script as a benchmark for you if you want. Besides, we
 have some bigger clusters, I could run on the just to test the speed if
 this is going to help.

 Regards
 Marcelo.


 2014-06-03 11:40 GMT-03:00 Laing, Michael :

 Hi Marcelo,
>
> I could create a fast copy program by repurposing some python apps
> that I am using for benchmarking the python driver - do you still need 
> this?
>
> With high levels of concurrency and multiple subprocess workers, based
> on my current actual benchmarks, I think I can get well over 1,000
> rows/second on my mac and significantly more in AWS. I'm using variable
> size rows averaging 5kb.
>
> This would be the initial version of a piece of the benchmark suite we
> will release as part of our nyt⨍aбrik project on 21 June for my
> Cassandra Day NYC talk re the python driver.
>
> ml
>
>
> On Mon, Jun 2, 2014 at 2:15 PM, Marcelo Elias Del Valle <
> marc...@s1mbi0se.com.br> wrote:
>
>> Hi Jens,
>>
>> Thanks for trying to help.
>>
>> Indeed, I know I can't do it using just CQL. But what would you use
>> to migrate data manually? I tried to create a python program using auto
>> paging, but I am getting timeouts. I also tried Hive, but no success.
>> I only have two nodes and less than 200Gb in this cluster, any simple
>> way to extract the data quickly would be good enough for me.
>>
>> Best regards,
>> Marcelo.
>>
>>
>>
>> 2014-06-02 15:08 GMT-03:00 Jens Rantil :
>>
>> Hi Marcelo,
>>>
>>> Looks like you can't do this without migrating your data manually:
>>> https://stackoverflow.com/questions/18421668/alter-cassandra-column-family-primary-key-using-cassandra-cli-or-cql
>>>
>>> Cheers,
>>> Jens
>>>
>>>
>>> On Mon, Jun 2, 2014 at 7:48 PM, Marcelo Elias Del Valle <
>>> marc...@s1mbi0se.com.br> wrote:
>>>
 Hi,

 I have some cql CFs in a 2 node Cassandra 2.0.8 cluster.

 I realized I created my column family with

Re: nodetool move seems slow

[Jason Tyler]

Hi Rob,

THX for you response and link to the issue.

The move did complete after a restart!


Cheers,

~Jason
***
From: Robert Coli mailto:rc...@eventbrite.com>>
Reply-To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Date: Wednesday, June 4, 2014 at 5:01 PM
To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Cc: Francois Richard mailto:frich...@yahoo-inc.com>>
Subject: Re: nodetool move seems slow

On Wed, Jun 4, 2014 at 2:34 PM, Jason Tyler 
mailto:jaty...@yahoo-inc.com>> wrote:
I wrote 'apparent progress' because it reports “MOVING” and the Pending 
Commands/Responses are changing over time.  However, I haven’t seen the 
individual .db files progress go above 0%.

Your move is hung. Restart the affected nodes [1] and then restart the move.

=Rob
[1] https://issues.apache.org/jira/browse/CASSANDRA-3486

Re: Cassandra 2.0 unbalanced ring with vnodes after adding new node

[Marcelo Elias Del Valle]

Actually, I have the same doubt. The same happens to me, but I guess it's
because of lack of knowledge in Cassandra vnodes, somehow...

I just added 3 nodes to my old 2 nodes cluster, now I have a 5 nodes
cluster.

As rows should be in a node calculated by HASH / number of nodes, adding a
new node should move data from all other nodes to the new ones, right?
Considering I have an enough number of different row keys.

I noticed that:


   1. Even reading data with read consistency = ALL, I get the wrong
   results while the repair is not complete. Should this happen?
   2. I have run nodetool repair in each new node and nodetool cleanup in
   the 2 old nodes. There is some streaming happening, but it's really slow,
   considering my bandwith and use of SSDs.

What should I do make the data stream from the old nodes to the new ones
faster?

And everytime I add new nodes to the cluster I will have to stop my
processes that reads data from cassandra until the move is complete? Isn't
there any other way?

Best regards,
Marcelo.



2014-06-04 13:52 GMT-03:00 Владимир Рудев :

> Hello to everyone!
>
> Please, can someone explain where we made a mistake?
>
> We have cluster with 4 nodes which uses vnodes(256 per node, default
> settings), snitch is default on every node: SimpleSnitch.
> These four nodes was from beginning of a cluster.
> In this cluster we have keyspace with this options:
> Keyspace: K:
>   Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
>   Durable Writes: true
> Options: [replication_factor:3]
>
> All was normal and nodetool status K shows that each node owns 75% of all
> key range. All 4 nodes are located in same datacenter and have same first
> two bytes in IP address(others are different).
>
> Then we buy new server on different datacenter and add it to the cluster
> with same settings as in previous four nodes(difference only in
> listen_address), assuming that the effective own of each node for this
> keyspace will be 300/5=60% or near. But after 3-5 minutes after start nodetool
> status K show this:
> nodetool status K;
> Datacenter: datacenter1
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  AddressLoad   Tokens  Owns (effective)  Host ID
> Rack
> UN  N1   6,06 GB256 50.0%
> 62f295b3-0da6-4854-a53a-f03d6b424b03  rack1
> UN  N2   5,89 GB256 50.0%
> af4e4a23-2610-44dd-9061-09c7a6512a54  rack1
> UN  N3   6,02 GB256 50.0%
> 0f0e4e78-6fb2-479f-ad76-477006f76795  rack1
> UN  N4   5,8 GB 256 50.0%
> 670344c0-9856-48cf-9ec9-1a98f9a89460  rack1
> UN  N5   7,51 GB256 100.0%
>  82473d14-9e36-4ae7-86d2-a3e526efb53f  rack1
>
> N5 is newly added node
>
> nodetool repair -pr on N5 doesn't change anything
>
> nodetool describering K shows that new node N5 participate in EACH range.
> This is not we want at all.
>
> It looks like cassandra add new node to each range because it located in
> different datacenter, but all settings and output are exactly prevent this.
>
> Also interesting point is that while in all config files snitch is defined
> as SimpleSnitch the output of the command nodetool describecluster is:
> Cluster Information:
> Name: Some Cluster Name
> Snitch: org.apache.cassandra.locator.*DynamicEndpointSnitch*
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Schema versions:
> 26b8fa37-e666-31ed-aa3b-85be75f2aa1a: [N1, N2, N3, N4, N5]
>
> We use Cassandra 2.0.6
>
> Questions we have at this moment:
> 1. How to rebalance ring so all nodes will own 60% of range?
>1a. Removing node from cluster and adding it again is a solution?
> 2. Where we possibly make a mistake when adding new node?
> 3. If we add new 6th node to ring it will take 50% from N5 or some portion
> from each node?
>
> Thanks in advance!
>
> --
> С уважением,
> Владимир Рудев
> (With regards, Vladimir Rudev)
> vladimir.ru...@gmail.com
>
>
>

Re: VPC AWS

[Michael Theroux]

The implementation of moving from EC2 to a VPC was a bit of a juggling act.  
Our motivation was two fold:


1) We were running out of static IP addresses, and it was becoming increasingly 
difficult in EC2 to design around limiting the number of static IP addresses to 
the number of public IP addresses EC2 allowed
2) VPC affords us an additional level of security that was desirable.

However, we needed to consider the following limitations:

1) By default, you have a limited number of available public IPs for both EC2 
and VPC.  
2) AWS security groups need to be configured to allow traffic for Cassandra 
to/from instances in EC2 and the VPC.

You are correct at the high level that the migration goes from EC2->Public VPC 
(VPC with an Internet Gateway)->Private VPC (VPC with a NAT).  The first phase 
was moving instances to the public VPC, setting broadcast and seeds to the 
public IPs we had available.  Basically:

1) Take down a node, taking a snapshot for a backup

2) Restore the node on the public VPC, assigning it to the correct security 
group, manually setting the seeds to other available nodes
3) Verify the cluster can communicate
4) Repeat

Realize the NAT instance on the private subnet will also require a public IP.  
What got really interesting is that near the end of the process we ran out of 
available IPs, requiring us to switch the final node that was on EC2 directly 
to the private VPC (and taking down two nodes at once, which our setup allowed 
given we had 6 nodes with an RF of 3).  

What we did, and highly suggest for the switch, is to write down every step 
that has to happen on every node during the switch.  In our case, many of the 
moved nodes required slightly different configurations for items like the seeds.

Its been a couple of years, so my memory on this maybe a little fuzzy :)

-Mike



 From: Aiman Parvaiz 
To: user@cassandra.apache.org; Michael Theroux  
Sent: Thursday, June 5, 2014 12:55 PM
Subject: Re: VPC AWS
 


Michael, 
Thanks for the response, I am about to head in to something very similar if not 
exactly same. I envision things happening on the same lines as you mentioned. 
I would be grateful if you could please throw some more light on how you went 
about switching cassandra nodes from public subnet to private with out any 
downtime.
I have not started on this project yet, still in my research phase. I plan to 
have a ec2+public VPC cluster and then decomission ec2 nodes to have everything 
in public subnet, next would be to move it to private subnet.

Thanks



On Thu, Jun 5, 2014 at 8:14 AM, Michael Theroux  wrote:

We personally use the EC2Snitch, however, we don't have the multi-region 
requirements you do,
>
>
>-Mike
>
>
>
>
> 
>From: Alain RODRIGUEZ 
>To: user@cassandra.apache.org 
>Sent: Thursday, June 5, 2014 9:14 AM
>Subject: Re: VPC AWS
>
>
>
>I think you can define VPC subnet to be public (to have public + private IPs) 
>or private only.
>
>
>Any insight regarding snitches ? What snitch do you guys use ?
>
>
>
>2014-06-05 15:06 GMT+02:00 William Oberman :
>
>I don't think traffic will flow between "classic" ec2 and vpc directly. There 
>is some kind of gateway bridge instance that sits between, acting as a NAT.   
>I would think that would cause new challenges for:
>>
>>-transitions 
>>-clients
>>
>>Sorry this response isn't heavy on content!  I'm curious how this thread 
>>goes...
>>
>>
>>Will
>>
>>
>>On Thursday, June 5, 2014, Alain RODRIGUEZ  wrote:
>>
>>Hi guys,
>>>
>>>
>>>We are going to move from a cluster made of simple Amazon EC2 servers to a 
>>>VPC cluster. We are using Cassandra 1.2.11 and I have some questions 
>>>regarding this switch and the Cassandra configuration inside a VPC.
>>>
>>>
>>>Actually I found no documentation on this topic, but I am quite sure that 
>>>some people are already using VPC. If you can point me to any documentation 
>>>regarding VPC / Cassandra, it would be very nice of you. We have only one DC 
>>>for now, but we need to remain multi DC compatible, since we will add DC 
>>>very soon.
>>>
>>>
>>>Else, I would like to know if I should keep using EC2MultiRegionSnitch or 
>>>change the snitch to anything else.
>>>
>>>
>>>What about broadcast/listen ip, seeds...?
>>>
>>>
>>>We currently use public ip as for broadcast address and for seeds. We use 
>>>private ones for listen address. Machines inside the VPC will only have 
>>>private IP AFAIK. Should I keep using a broadcast address ?
>>>
>>>
>>>Is there any other incidence when switching to a VPC ?
>>>
>>>
>>>Sorry if the topic was already discussed, I was unable to find any useful 
>>>information...
>>
>>-- 
>>Will Oberman
>>Civic Science, Inc.
>>6101 Penn Avenue, Fifth Floor
>>Pittsburgh, PA 15206
>>(M) 412-480-7835
>>(E) ober...@civicscience.com
>>
>
>
>

Re: VPC AWS

[Aiman Parvaiz]

Thanks for this info Michael. As far as restoring node in public VPC is
concerned I was thinking ( and I might be wrong here) if we can have a ring
spread across EC2 and public subnet of a VPC, this way I can simply
decommission nodes in Ec2 as I gradually introduce new nodes in public
subnet of VPC and I will end up with a ring in public subnet and then
migrate them from public to private in a similar way may be.

If anyone has any experience/ suggestions with this please share, would
really appreciate it.

Aiman


On Thu, Jun 5, 2014 at 10:37 AM, Michael Theroux 
wrote:

> The implementation of moving from EC2 to a VPC was a bit of a juggling
> act.  Our motivation was two fold:
>
> 1) We were running out of static IP addresses, and it was becoming
> increasingly difficult in EC2 to design around limiting the number of
> static IP addresses to the number of public IP addresses EC2 allowed
> 2) VPC affords us an additional level of security that was desirable.
>
> However, we needed to consider the following limitations:
>
> 1) By default, you have a limited number of available public IPs for both
> EC2 and VPC.
> 2) AWS security groups need to be configured to allow traffic for
> Cassandra to/from instances in EC2 and the VPC.
>
> You are correct at the high level that the migration goes from EC2->Public
> VPC (VPC with an Internet Gateway)->Private VPC (VPC with a NAT).  The
> first phase was moving instances to the public VPC, setting broadcast and
> seeds to the public IPs we had available.  Basically:
>
> 1) Take down a node, taking a snapshot for a backup
> 2) Restore the node on the public VPC, assigning it to the correct
> security group, manually setting the seeds to other available nodes
> 3) Verify the cluster can communicate
> 4) Repeat
>
> Realize the NAT instance on the private subnet will also require a public
> IP.  What got really interesting is that near the end of the process we
> ran out of available IPs, requiring us to switch the final node that was on
> EC2 directly to the private VPC (and taking down two nodes at once, which
> our setup allowed given we had 6 nodes with an RF of 3).
>
> What we did, and highly suggest for the switch, is to write down every
> step that has to happen on every node during the switch.  In our case, many
> of the moved nodes required slightly different configurations for items
> like the seeds.
>
> Its been a couple of years, so my memory on this maybe a little fuzzy :)
>
> -Mike
>
>   --
>  *From:* Aiman Parvaiz 
> *To:* user@cassandra.apache.org; Michael Theroux 
> *Sent:* Thursday, June 5, 2014 12:55 PM
> *Subject:* Re: VPC AWS
>
> Michael,
> Thanks for the response, I am about to head in to something very similar
> if not exactly same. I envision things happening on the same lines as you
> mentioned.
> I would be grateful if you could please throw some more light on how you
> went about switching cassandra nodes from public subnet to private with out
> any downtime.
> I have not started on this project yet, still in my research phase. I plan
> to have a ec2+public VPC cluster and then decomission ec2 nodes to have
> everything in public subnet, next would be to move it to private subnet.
>
> Thanks
>
>
> On Thu, Jun 5, 2014 at 8:14 AM, Michael Theroux 
> wrote:
>
> We personally use the EC2Snitch, however, we don't have the multi-region
> requirements you do,
>
> -Mike
>
>   --
>  *From:* Alain RODRIGUEZ 
> *To:* user@cassandra.apache.org
> *Sent:* Thursday, June 5, 2014 9:14 AM
> *Subject:* Re: VPC AWS
>
> I think you can define VPC subnet to be public (to have public + private
> IPs) or private only.
>
> Any insight regarding snitches ? What snitch do you guys use ?
>
>
> 2014-06-05 15:06 GMT+02:00 William Oberman :
>
> I don't think traffic will flow between "classic" ec2 and vpc directly.
> There is some kind of gateway bridge instance that sits between, acting as
> a NAT.   I would think that would cause new challenges for:
> -transitions
> -clients
>
> Sorry this response isn't heavy on content!  I'm curious how this thread
> goes...
>
> Will
>
> On Thursday, June 5, 2014, Alain RODRIGUEZ  wrote:
>
> Hi guys,
>
> We are going to move from a cluster made of simple Amazon EC2 servers to a
> VPC cluster. We are using Cassandra 1.2.11 and I have some questions
> regarding this switch and the Cassandra configuration inside a VPC.
>
> Actually I found no documentation on this topic, but I am quite sure that
> some people are already using VPC. If you can point me to any documentation
> regarding VPC / Cassandra, it would be very nice of you. We have only one
> DC for now, but we need to remain multi DC compatible, since we will add DC
> very soon.
>
> Else, I would like to know if I should keep using EC2MultiRegionSnitch or
> change the snitch to anything else.
>
> What about broadcast/listen ip, seeds...?
>
> We currently use public ip as for broadcast address and for seeds. We use
> private o

Re: VPC AWS

[Michael Theroux]

You can have a ring spread across EC2 and the public subnet of a VPC.  That is 
how we did our migration.  In our case, we simply replaced the existing EC2 
node with a new instance in the public VPC, restored from a backup taken right 
before the switch.

-Mike



 From: Aiman Parvaiz 
To: Michael Theroux  
Cc: "user@cassandra.apache.org"  
Sent: Thursday, June 5, 2014 2:39 PM
Subject: Re: VPC AWS
 


Thanks for this info Michael. As far as restoring node in public VPC is 
concerned I was thinking ( and I might be wrong here) if we can have a ring 
spread across EC2 and public subnet of a VPC, this way I can simply 
decommission nodes in Ec2 as I gradually introduce new nodes in public subnet 
of VPC and I will end up with a ring in public subnet and then migrate them 
from public to private in a similar way may be.

If anyone has any experience/ suggestions with this please share, would really 
appreciate it.

Aiman



On Thu, Jun 5, 2014 at 10:37 AM, Michael Theroux  wrote:

The implementation of moving from EC2 to a VPC was a bit of a juggling act.  
Our motivation was two fold:
>
>
>
>1) We were running out of static IP addresses, and it was becoming 
>increasingly difficult in EC2 to design around limiting the number of static 
>IP addresses to the number of public IP addresses EC2 allowed
>2) VPC affords us an additional level of security that was desirable.
>
>
>However, we needed to consider the following limitations:
>
>
>1) By default, you have a limited number of available public IPs for both EC2 
>and VPC.  
>2) AWS security groups need to be configured to allow traffic for Cassandra 
>to/from instances in EC2 and the VPC.
>
>
>You are correct at the high level that the migration goes from EC2->Public VPC 
>(VPC with an Internet Gateway)->Private VPC (VPC with a NAT).  The first phase 
>was moving instances to the public VPC, setting broadcast and seeds to the 
>public IPs we had available.  Basically:
>
>
>1) Take down a node, taking a snapshot for a backup
>
>2) Restore the node on the public VPC, assigning it to the correct security 
>group, manually setting the seeds to other available nodes
>3) Verify the cluster can communicate
>4) Repeat
>
>
>Realize the NAT instance on the private subnet will also require a public IP.  
>What got really interesting is that near the end of the process we ran out of 
>available IPs, requiring us to switch the final node that was on EC2 directly 
>to the private VPC (and taking down two nodes at once, which our setup allowed 
>given we had 6 nodes with an RF of 3).  
>
>
>What we did, and highly suggest for the switch, is to write down every step 
>that has to happen on every node during the switch.  In our case, many of the 
>moved nodes required slightly different configurations for items like the 
>seeds.
>
>
>Its been a couple of years, so my memory on this maybe a little fuzzy :)
>
>
>-Mike
>
>
>
>
> From: Aiman Parvaiz 
>To: user@cassandra.apache.org; Michael Theroux  
>Sent: Thursday, June 5, 2014 12:55 PM
>Subject: Re: VPC AWS
> 
>
>
>Michael, 
>Thanks for the response, I am about to head in to something very similar if 
>not exactly same. I envision things happening on the same lines as you 
>mentioned. 
>I would be grateful if you could please throw some more light on how you went 
>about switching cassandra nodes from public subnet to private with out any 
>downtime.
>I have not started on this project yet, still in my research phase. I plan to 
>have a ec2+public VPC cluster and then decomission ec2 nodes to have 
>everything in public subnet, next would be to move it to private subnet.
>
>
>Thanks
>
>
>
>On Thu, Jun 5, 2014 at 8:14 AM, Michael Theroux  wrote:
>
>We personally use the EC2Snitch, however, we don't have the multi-region 
>requirements you do,
>>
>>
>>-Mike
>>
>>
>>
>>
>> 
>>From: Alain RODRIGUEZ 
>>To: user@cassandra.apache.org 
>>Sent: Thursday, June 5, 2014 9:14 AM
>>Subject: Re: VPC AWS
>>
>>
>>
>>I think you can define VPC subnet to be public (to have public + private IPs) 
>>or private only.
>>
>>
>>Any insight regarding snitches ? What snitch do you guys use ?
>>
>>
>>
>>2014-06-05 15:06 GMT+02:00 William Oberman :
>>
>>I don't think traffic will flow between "classic" ec2 and vpc directly. There 
>>is some kind of gateway bridge instance that sits between, acting as a NAT.   
>>I would think that would cause new challenges for:
>>>
>>>-transitions 
>>>-clients
>>>
>>>Sorry this response isn't heavy on content!  I'm curious how this thread 
>>>goes...
>>>
>>>
>>>Will
>>>
>>>
>>>On Thursday, June 5, 2014, Alain RODRIGUEZ  wrote:
>>>
>>>Hi guys,


We are going to move from a cluster made of simple Amazon EC2 servers to a 
VPC cluster. We are using Cassandra 1.2.11 and I have some questions 
regarding this switch and the Cassandra configuration inside a VPC.


Actually I found no documentation on thi

Re: VPC AWS

[Aiman Parvaiz]

Cool, thanks again for this.


On Thu, Jun 5, 2014 at 11:51 AM, Michael Theroux 
wrote:

> You can have a ring spread across EC2 and the public subnet of a VPC.
>  That is how we did our migration.  In our case, we simply replaced the
> existing EC2 node with a new instance in the public VPC, restored from a
> backup taken right before the switch.
>
> -Mike
>
>   --
>  *From:* Aiman Parvaiz 
> *To:* Michael Theroux 
> *Cc:* "user@cassandra.apache.org" 
> *Sent:* Thursday, June 5, 2014 2:39 PM
> *Subject:* Re: VPC AWS
>
> Thanks for this info Michael. As far as restoring node in public VPC is
> concerned I was thinking ( and I might be wrong here) if we can have a ring
> spread across EC2 and public subnet of a VPC, this way I can simply
> decommission nodes in Ec2 as I gradually introduce new nodes in public
> subnet of VPC and I will end up with a ring in public subnet and then
> migrate them from public to private in a similar way may be.
>
> If anyone has any experience/ suggestions with this please share, would
> really appreciate it.
>
> Aiman
>
>
> On Thu, Jun 5, 2014 at 10:37 AM, Michael Theroux 
> wrote:
>
> The implementation of moving from EC2 to a VPC was a bit of a juggling
> act.  Our motivation was two fold:
>
> 1) We were running out of static IP addresses, and it was becoming
> increasingly difficult in EC2 to design around limiting the number of
> static IP addresses to the number of public IP addresses EC2 allowed
> 2) VPC affords us an additional level of security that was desirable.
>
> However, we needed to consider the following limitations:
>
> 1) By default, you have a limited number of available public IPs for both
> EC2 and VPC.
> 2) AWS security groups need to be configured to allow traffic for
> Cassandra to/from instances in EC2 and the VPC.
>
> You are correct at the high level that the migration goes from EC2->Public
> VPC (VPC with an Internet Gateway)->Private VPC (VPC with a NAT).  The
> first phase was moving instances to the public VPC, setting broadcast and
> seeds to the public IPs we had available.  Basically:
>
> 1) Take down a node, taking a snapshot for a backup
> 2) Restore the node on the public VPC, assigning it to the correct
> security group, manually setting the seeds to other available nodes
> 3) Verify the cluster can communicate
> 4) Repeat
>
> Realize the NAT instance on the private subnet will also require a public
> IP.  What got really interesting is that near the end of the process we
> ran out of available IPs, requiring us to switch the final node that was on
> EC2 directly to the private VPC (and taking down two nodes at once, which
> our setup allowed given we had 6 nodes with an RF of 3).
>
> What we did, and highly suggest for the switch, is to write down every
> step that has to happen on every node during the switch.  In our case, many
> of the moved nodes required slightly different configurations for items
> like the seeds.
>
> Its been a couple of years, so my memory on this maybe a little fuzzy :)
>
> -Mike
>
>   --
>  *From:* Aiman Parvaiz 
> *To:* user@cassandra.apache.org; Michael Theroux 
> *Sent:* Thursday, June 5, 2014 12:55 PM
> *Subject:* Re: VPC AWS
>
> Michael,
> Thanks for the response, I am about to head in to something very similar
> if not exactly same. I envision things happening on the same lines as you
> mentioned.
> I would be grateful if you could please throw some more light on how you
> went about switching cassandra nodes from public subnet to private with out
> any downtime.
> I have not started on this project yet, still in my research phase. I plan
> to have a ec2+public VPC cluster and then decomission ec2 nodes to have
> everything in public subnet, next would be to move it to private subnet.
>
> Thanks
>
>
> On Thu, Jun 5, 2014 at 8:14 AM, Michael Theroux 
> wrote:
>
> We personally use the EC2Snitch, however, we don't have the multi-region
> requirements you do,
>
> -Mike
>
>   --
>  *From:* Alain RODRIGUEZ 
> *To:* user@cassandra.apache.org
> *Sent:* Thursday, June 5, 2014 9:14 AM
> *Subject:* Re: VPC AWS
>
> I think you can define VPC subnet to be public (to have public + private
> IPs) or private only.
>
> Any insight regarding snitches ? What snitch do you guys use ?
>
>
> 2014-06-05 15:06 GMT+02:00 William Oberman :
>
> I don't think traffic will flow between "classic" ec2 and vpc directly.
> There is some kind of gateway bridge instance that sits between, acting as
> a NAT.   I would think that would cause new challenges for:
> -transitions
> -clients
>
> Sorry this response isn't heavy on content!  I'm curious how this thread
> goes...
>
> Will
>
> On Thursday, June 5, 2014, Alain RODRIGUEZ  wrote:
>
> Hi guys,
>
> We are going to move from a cluster made of simple Amazon EC2 servers to a
> VPC cluster. We are using Cassandra 1.2.11 and I have some questions
> regarding this switch and the Cassandra configuration inside a

Re: High latency on 5 node Cassandra Cluster

[Arup Chakrabarti]

Yes, I meant RF: 2,2,1. We have the following in our network topo config:

# cass08: us-west-1c

private.ip=DC-A:1

public.ip=DC-A:1

# cass09: us-west-1b

private.ip=DC-A:2

public.ip=DC-A:2

# cass10: Linode-Fremont-CA

public.ip=DC-C:1

# cass11: us-west-2c

private.ip=DC-B:1

public.ip=DC-B:1

# cass12: us-west-2b

private.ip=DC-B:2

public.ip=DC-B:2

We're figuring out the upgrade right now. In general, what is the expected
upgrade time per node for ~50GB (on each node)?

Thanks

arup


On Wed, Jun 4, 2014 at 11:08 AM, Robert Coli  wrote:

> On Wed, Jun 4, 2014 at 12:12 AM, Arup Chakrabarti 
> wrote:
>
>> Size: 5 nodes (2 in AWS US-West-1, 2 in AWS US-West-2, 1 in Linode
>> Fremont)
>> Replication Factor: 5
>>
>
> You're operating with a single-DC strategy across multiple data centers?
> If so, I'm surprised you get sane latency ever.
>
> (Or do you mean RF : 2,2,1?)
>
> I agree with others that problems which can cause cluster wide outages
> exist in Gossip in the version of Cassandra you are running.
>
> As a general piece of feedback, I suggest an upgrade, first to 1.1 HEAD,
> then 1.2.16.
>
> =Rob
>
>

Re: Consolidating records and TTL

[Charlie Mason]

Hi All,

Thanks for the replies. These detailed explanations are extremely useful.
The are much appreciated!

One other question that came to mind with this. Is there any way I can use
a TTLs to keep the old account entries around for a fixed length of time
rather than deleting them immediately? That way if I had an issue I could
replay the process form one of these earlier points. For the regular reads
of the account table I can use a limit of 1, with the reverse comparator
set I should always get the newest record without hitting an older record
first.

I can't do the initial account insert with a TTL as I can't guarantee when
a new value would come along and so replace this account record. However
when I insert the new account record, instead of deleting the old one could
I reinsert it with a TTL of say 1 month.

How would compaction handle this. Would the original record get compacted
away after 1 month + the GC Grace period or would it hang around still?

Thanks,

Charlie M


On Thu, Jun 5, 2014 at 5:32 PM, James Campbell  wrote:

>  Thanks for creating and opening the discussion on this use case.  I have
> been evaluating Cassandra for a very similar problem, but with the
> small twist that I'd like to roll up the ledger entries into the
> aggregated 'account' information on a regular basis to ensure that even
> rarely-read (but often written) data will be periodically rolled up and
> ready for quick reads.  There is also a larger twist that for me 'account'
> is a compound key and the typical read pattern will include reading from 
> multiple
> rows within the same partition (each of which would require separate
> merging from the ledger).
>
>
>  This response prompts three questions for me about that:
>
>
>  1. Just to clarify the requirement that only one client thread does this
> at a time, I assume you mean only one thread should handle any given
> 'account' at a time, right?  There wouldn't be problems having multiple
> clients do this data maintenance on different accounts at the same time, I
> hope.
>
>
>  2. For my use case, hadoop integration seems the natural fit, because
> I'd like to batch these updates to run at night during query down-time, but
> the current hadoop tools don't really support reading from or writing to
> multiple tables, as is required for this implementation.  I've seen a few
> posts from people who have written homegrown input/output formats for
> hadoop, but I haven't tried to use them or evaluate their stability.  Is
> there another mechanism I should be thinking about for that sort of batch
> updating?​
>
>
>  James Campbell
>  --
> *From:* Aaron Morton 
> *Sent:* Thursday, June 5, 2014 5:26 AM
> *To:* Cassandra User
> *Cc:* charlie@gmail.com
> *Subject:* Re: Consolidating records and TTL
>
>  As Tyler says, with atomic batches which are enabled by default the
> cluster will keep trying to replay the insert / deletes.
>
>  Nodes check their local batch log for failed batches, ones where the
> coordinator did not acknowledge it had successfully completed, every 60
> seconds. So there is a window where it’s possible for not all mutations in
> the batch to be completed. This could happen when a write timeout occurs
> when processing a batch of 2 rows; the request CL will not have been
> achieved on one or more of the rows. The coordinator will leave it up to
> the batch log to replay the request, and the client driver will (by default
> config) not retry.
>
>  You can use a model like this.
>
>  create table ledger (
> account int,
> tx_id  timeuuid,
> sub_total  int,
> primary key (account, tx_id)
> );
>
>  create table account (
> account  int,
> total int,
> last_tx_id timeuuid,
> primary key (account)
> );
>
>  To get the total:
>
>  select * from account where account = X;
>
>  Then get the ledger entries you need
>
>  select * from ledger where account = X and tx_id > last_tx_id;
>
>  This query will degrade when the partition size in the ledger table gets
> bigger, as it will need to read the column index
> (see column_index_size_in_kb in yaml). It will use that to find the first
> page that contains the rows we are interested in and then read forwards to
> the end of the row. It’s not the most efficient type of read but if you are
> going to delete ledger entries this *should* be able to skip over the
> tombstones without reading them.
>
>  When you want to update the total in the account write to the account
> table and update both the total and the last_tx_id. You can then delete
> ledger entries if needed. Don’t forget to ensure that only one client
> thread is doing this at a time.
>
>  Hope that helps.
> Aaron
>
>
>   -
> Aaron Morton
> New Zealand
> @aaronmorton
>
>  Co-Founder & Principal Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>
>  On 5/06/2014, at 10:37 am, Tyler Hobbs  wrote:
>
>  Just use an atomic batch that holds both the insert and deletes:
> http://www.datastax.c

Sizing a new Cassandra cluster

[Koppel, Jeremy]

I have been able to find lots of general information about sizing each node in 
a new Cassandra cluster, but have not come across any specific recommendations 
about the total size and configuration of the cluster (the number of nodes 
required per data center, the number of data centers, throughput requirements 
between data centers, etc.).   I am currently in the process of sizing a new 
Cassandra cluster to support the following:

  *   Probably more write intensive than read, or at least 65% / 35%.
  *   Writes per day:  200,000,000 (~2315 per second).
  *   Data retention = 30 days.
  *   Replication Factor = 3.  (I anticipate reads and writes of CL = Quorum or 
Quorum Local.)
  *   My developers estimate a payload of ~300 bytes per record.
 *   Throughput per second (MiB):  (Records per second * Replication Facor 
* Event Payload) / 1024 / 1024 = 1.99 MiB/Sec.
 *   Storage required (TiB):  (Events per day * Event Payload * Replication 
Factor * Data Retention * 2) / 1024 / 1024 / 1024 / 1024 = 9.82 TiB.
*   Size doubled to provide room for Compaction.

I’m wondering if I’m on the right track with my math, and if the following 
configuration would perform well, and leave a little overhead:

  *   2 Data Centers (they could co-exist with the application clusters).
  *   12 nodes (6 per data center) with:
 *   1 TiB storage capacity each.
*   I’ve seen varying information for RAID usage / configuration.  Is a 
RAID 1 mirrored over 2x 1 TiB SSD drives performant?  That might be a good 
configuration for us, and provide some high availability so that we can lose a 
drive without having to repair a node.  Or is it better to buy an additional 
node for extra capacity, save the data to single SSDs and let it fail?  (Or 
stripe 2x 500 GiB SSD drives…)
*   Do we need to store the CommitLog on a separate drive if we’re 
using SSD?  How much space do we leave for it?  Do we really need separate 
controllers?
 *   8 CPU cores.
 *   32GB RAM.

Thoughts?  Is this enough?  Overkill?

—Jeremy

Re: CQLSSTableWriter memory leak

[Xu Zhongxing]

Is writing too many rows to a single partition the cause of memory consumption?


What I want to achieve is this: say I have 5 partition ID. Each corresponds to 
50 million IDs.  Given a partition ID, I need to get its corresponding 50 
million IDs. Is there another way to design the schema to avoid such a compound 
primary key?


I could use the clustering IDs as the primary key, and create index on the 
partition ID. But is that equivalent to create another table with compound keys?


At 2014-06-06 00:16:13, "Jack Krupansky"  wrote:
How many rows (primary key values) are you writing for each partition of the 
primary key? I mean, are there relatively few, or are these very wide 
partitions?
 
Oh, I see! You’re writing 50,000,000 rows to a single partition! My, that IS 
ambitious.
 
-- Jack Krupansky
 
From:Xu Zhongxing
Sent: Thursday, June 5, 2014 3:34 AM
To:user@cassandra.apache.org
Subject: CQLSSTableWriter memory leak
 

I am using Cassandra's CQLSSTableWriter to import a large amount of data into 
Cassandra. When I use CQLSSTableWriter to write to a table with compound 
primary key, the memory consumption keeps growing. The GC of JVM cannot collect 
any used memory. When writing to tables with no compound primary key, the JVM 
GC works fine.

My Cassandra version is 2.0.5. The OS is Ubuntu 14.04 x86-64. JVM parameters 
are -Xms1g -Xmx2g. This is sufficient for all other non-compound primary key 
cases.

The problem can be reproduced by the following test case:

import org.apache.cassandra.io.sstable.CQLSSTableWriter;
import org.apache.cassandra.exceptions.InvalidRequestException;

import java.io.IOException;
import java.util.UUID;

class SS {
public static void main(String[] args) {
String schema = "create table test.t (x uuid, y uuid, primary key (x, 
y))";


String insert = "insert into test.t (x, y) values (?, ?)";
CQLSSTableWriter writer = CQLSSTableWriter.builder()
.inDirectory("/tmp/test/t")
.forTable(schema).withBufferSizeInMB(32)
.using(insert).build();

UUID id = UUID.randomUUID();
try {
for (int i = 0; i < 5000; i++) {
UUID id2 = UUID.randomUUID();
writer.addRow(id, id2);
}

writer.close();
} catch (Exception e) {
System.err.println("hell");
}
}
}

Bad Request: Type error: cannot assign result of function token (type bigint) to id (type int)

[Kevin Burton]

I'm building a new schema which I need to read externally by paging through
the result set.

My understanding from reading the documentation , and this list, is that I
can do that but I need to use the token() function.

Only it doesn't work.

Here's a reduction:


create table test_paging (
id int,
primary key(id)
);

insert into test_paging (id) values (1);
insert into test_paging (id) values (2);
insert into test_paging (id) values (3);
insert into test_paging (id) values (4);
insert into test_paging (id) values (5);

select * from test_paging where id > token(0);

… but it gives me:

Bad Request: Type error: cannot assign result of function token (type
bigint) to id (type int)

…

What's that about?  I can't find any documentation for this and there
aren't any concise examples.


-- 

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
Skype: *burtonator*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile


War is peace. Freedom is slavery. Ignorance is strength. Corporations are
people.

Re: Bad Request: Type error: cannot assign result of function token (type bigint) to id (type int)

[Jonathan Haddad]

You should read through the token docs, it has examples and specifications:
http://cassandra.apache.org/doc/cql3/CQL.html#tokenFun


On Thu, Jun 5, 2014 at 10:22 PM, Kevin Burton  wrote:

> I'm building a new schema which I need to read externally by paging
> through the result set.
>
> My understanding from reading the documentation , and this list, is that I
> can do that but I need to use the token() function.
>
> Only it doesn't work.
>
> Here's a reduction:
>
>
> create table test_paging (
> id int,
> primary key(id)
> );
>
> insert into test_paging (id) values (1);
> insert into test_paging (id) values (2);
> insert into test_paging (id) values (3);
> insert into test_paging (id) values (4);
> insert into test_paging (id) values (5);
>
> select * from test_paging where id > token(0);
>
> … but it gives me:
>
> Bad Request: Type error: cannot assign result of function token (type
> bigint) to id (type int)
>
> …
>
> What's that about?  I can't find any documentation for this and there
> aren't any concise examples.
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> Skype: *burtonator*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> 
> 
> War is peace. Freedom is slavery. Ignorance is strength. Corporations are
> people.
>
>


-- 
Jon Haddad
http://www.rustyrazorblade.com
skype: rustyrazorblade

Re: Bad Request: Type error: cannot assign result of function token (type bigint) to id (type int)

[Jonathan Haddad]

Sorry, the datastax docs are actually a bit better:
http://www.datastax.com/documentation/cql/3.0/cql/cql_using/paging_c.html

Jon


On Thu, Jun 5, 2014 at 10:46 PM, Jonathan Haddad  wrote:

> You should read through the token docs, it has examples and
> specifications: http://cassandra.apache.org/doc/cql3/CQL.html#tokenFun
>
>
> On Thu, Jun 5, 2014 at 10:22 PM, Kevin Burton  wrote:
>
>> I'm building a new schema which I need to read externally by paging
>> through the result set.
>>
>> My understanding from reading the documentation , and this list, is that
>> I can do that but I need to use the token() function.
>>
>> Only it doesn't work.
>>
>> Here's a reduction:
>>
>>
>> create table test_paging (
>> id int,
>> primary key(id)
>> );
>>
>> insert into test_paging (id) values (1);
>> insert into test_paging (id) values (2);
>> insert into test_paging (id) values (3);
>> insert into test_paging (id) values (4);
>> insert into test_paging (id) values (5);
>>
>> select * from test_paging where id > token(0);
>>
>> … but it gives me:
>>
>> Bad Request: Type error: cannot assign result of function token (type
>> bigint) to id (type int)
>>
>> …
>>
>> What's that about?  I can't find any documentation for this and there
>> aren't any concise examples.
>>
>>
>> --
>>
>> Founder/CEO Spinn3r.com
>> Location: *San Francisco, CA*
>> Skype: *burtonator*
>> blog: http://burtonator.wordpress.com
>> … or check out my Google+ profile
>> 
>> 
>> War is peace. Freedom is slavery. Ignorance is strength. Corporations are
>> people.
>>
>>
>
>
> --
> Jon Haddad
> http://www.rustyrazorblade.com
> skype: rustyrazorblade
>



-- 
Jon Haddad
http://www.rustyrazorblade.com
skype: rustyrazorblade