Re: Strange trouble with JNA on CentOS6

[Nate McCall]

>
>
> In the new setup /tmp was mounted as noexec. So it looks like JNA was
> putting the native library here and then it was unable to execute it.
>

If you wanted to keep /tmp noexec (not a bad idea depending on environment)
you could also append the following to cassandra-env.sh:

JVM_OPTS="$JVM_OPTS -Djava.io.tmpdir=/some/other/dir"




-- 
-
Nate McCall
Austin, TX
@zznate

Co-Founder & Sr. Technical Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: Strange trouble with JNA on CentOS6

[Holger Hoffstätte]

On Tue, 27 May 2014 09:26:41 -0500, Nate McCall wrote:

>> In the new setup /tmp was mounted as noexec. So it looks like JNA was
>> putting the native library here and then it was unable to execute it.
>>
>>
> If you wanted to keep /tmp noexec (not a bad idea depending on
> environment) you could also append the following to cassandra-env.sh:
> 
> JVM_OPTS="$JVM_OPTS -Djava.io.tmpdir=/some/other/dir"

Recent versions of JNA have a dedicated property "jna.tmpdir" for this, 
so that files that are supposed to go into /tmp (i.e. regular temp files) 
are unaffected. It has precedence over java.io.tmpdir.

-h

Cassandra CSV & JSON uploader

[Joyabrata Das]

Hi,

Could someone please help on how to import data from Apache Cassandra to
Oracle Coherence.

As per my understanding it could be done by sstable to JSON-->JSON upload
to oracle coherence, however is there any code/tool/script to upload JSON
to Oracle Coherence.

Thanks,
Joy

Re: decommissioning a node

[Steven A Robenalt]

Hi Tim,

In case the above doesn't work, another thing to be aware of is that JMX
uses 2 different ports. The initial connection to 7199 causes a second port
to be opened, which is normally assigned randomly to an available and
otherwise unused port above 1024. If your server has a
firewall/ACL/Security Group configured, it probably doesn't expose the port.

Assuming you are on Java 7 (u4 or later), you can add the following to your
environment:
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=7200"

to force a fixed port (7200 in this case) for the second one, which is more
firewall friendly. You'll still need to be sure access to port 7200 is
allowed, but at least it's only a single port.

Hope that helps,

Steve



On Sun, May 25, 2014 at 7:48 AM, Tim Dunphy  wrote:

> ok I copied the cassandra.env from the host that had cassandra listening
> on port 7199 to the node that wasn't.
>
> That got it listening on the JMX port:
>
> [root@beta:~] #lsof -i :7199
> COMMAND  PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
> java9197 root   45u  IPv4 6411278  0t0  TCP *:7199 (LISTEN)
>
> But even tho I can telnet to that port from the seed node:
>
> [root@beta-new:~] #telnet  10.10.1.18 7199
> Trying 166.78.27.18...
> Connected to 166.78.27.18.
> Escape character is '^]'.
>
>
> I still get connection refused when trying to decommission the node:
>
> [root@beta-new:~] #nodetool -host 10.10.1.18 decommission
> Failed to connect to '166.78.27.18:7199': Connection refused
>
> To Colin, thanks for the information!
>
> nodetool decomission 
>
> UN means UP, NORMAL
>
> Oh and thanks I tried that and it seems to be working!
>
> [root@beta-new:~] #nodetool decommission
> f2a48fc7-a362-43f5-9061-4bb3739fdeaf
> Decommission will decommission the node you are connected to and does not
> take arguments!
>
> Sorry guys, wrote this email in a hurry as we're checking out of a hotel
> room currently. :) I'll let you know if this does work.
>
>
>
> On Sun, May 25, 2014 at 10:19 AM, Colin Clark  wrote:
>
>> Try this:
>>
>> nodetool decomission 
>>
>> UN means UP, NORMAL
>>
>> --
>> Colin
>> +1 320 221 9531
>>
>>
>>
>> On Sun, May 25, 2014 at 9:09 AM, Tim Dunphy  wrote:
>>
>>> Also for information that may help diagnose this issue I am running
>>> cassandra 2.0.7
>>>
>>> I am also using these java options:
>>>
>>> [root@beta:/etc/alternatives/cassandrahome] #grep -i jvm_opts
>>> conf/cassandra-env.sh  | grep -v '#'
>>> JVM_OPTS="$JVM_OPTS -ea"
>>> JVM_OPTS="$JVM_OPTS -javaagent:$CASSANDRA_HOME/lib/jamm-0.2.5.jar"
>>> JVM_OPTS="$JVM_OPTS -XX:+CMSClassUnloadingEnabled"
>>> JVM_OPTS="$JVM_OPTS -XX:+UseThreadPriorities"
>>> JVM_OPTS="$JVM_OPTS -XX:ThreadPriorityPolicy=42"
>>> JVM_OPTS="$JVM_OPTS -Xms${MAX_HEAP_SIZE}"
>>> JVM_OPTS="$JVM_OPTS -Xmx${MAX_HEAP_SIZE}"
>>> JVM_OPTS="$JVM_OPTS -Xmn${HEAP_NEWSIZE}"
>>> JVM_OPTS="$JVM_OPTS -XX:+HeapDumpOnOutOfMemoryError"
>>> JVM_OPTS="$JVM_OPTS
>>> -XX:HeapDumpPath=$CASSANDRA_HEAPDUMP_DIR/cassandra-`date +%s`-pid$$.hprof"
>>> JVM_OPTS="$JVM_OPTS -Xss256k"
>>> JVM_OPTS="$JVM_OPTS -XX:StringTableSize=103"
>>> JVM_OPTS="$JVM_OPTS -XX:+UseParNewGC"
>>> JVM_OPTS="$JVM_OPTS -XX:+UseConcMarkSweepGC"
>>> JVM_OPTS="$JVM_OPTS -XX:+CMSParallelRemarkEnabled"
>>> JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=8"
>>> JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=1"
>>> JVM_OPTS="$JVM_OPTS -XX:CMSInitiatingOccupancyFraction=75"
>>> JVM_OPTS="$JVM_OPTS -XX:+UseCMSInitiatingOccupancyOnly"
>>> JVM_OPTS="$JVM_OPTS -XX:+UseTLAB"
>>> JVM_OPTS="$JVM_OPTS -XX:+UseCondCardMark"
>>> JVM_OPTS="$JVM_OPTS -Djava.net.preferIPv4Stack=true"
>>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT"
>>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=false"
>>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
>>> JVM_OPTS="$JVM_OPTS $JVM_EXTRA_OPTS"
>>>
>>>
>>> Still need to figure out why the node I want to decommission isn't
>>> listening on port 7199 and how I can actually decommission it.
>>>
>>> Thanks
>>> Tim
>>>
>>>
>>> On Sun, May 25, 2014 at 9:20 AM, Tim Dunphy wrote:
>>>

 Hey all,

 I'm attempting to decommission a node I want to remove.

 First I get a status of the ring

 [root@beta-new:~] #nodetool status

 Datacenter: datacenter1

 ===

 Status=Up/Down

 |/ State=Normal/Leaving/Joining/Moving

 --  Address Load   Tokens  Owns   Host ID
 Rack

 UN  10.10.1.94  197.37 KB  256 49.4%
 fd2f76ae-8dcf-4e93-a37f-bf1e9088696e  rack1

 UN  10.10.1.18216.95 KB  256 50.6%
 f2a48fc7-a362-43f5-9061-4bb3739fdeaf  rack


 I see that the node I want to remove is UP. Tho I believe UN means up I
 don't know what it stands for.


 [root@beta-new:~] #nodetool -host  10.10.1.18 decommission

 Failed to connect to ' 10.10.1.18 : Connection timed out

 The connection to the n

Re: Strange trouble with JNA on CentOS6

[Nate McCall]

I missed that - thanks Holger!


On Tue, May 27, 2014 at 10:14 AM, Holger Hoffstätte <
holger.hoffstae...@googlemail.com> wrote:

> On Tue, 27 May 2014 09:26:41 -0500, Nate McCall wrote:
>
> >> In the new setup /tmp was mounted as noexec. So it looks like JNA was
> >> putting the native library here and then it was unable to execute it.
> >>
> >>
> > If you wanted to keep /tmp noexec (not a bad idea depending on
> > environment) you could also append the following to cassandra-env.sh:
> >
> > JVM_OPTS="$JVM_OPTS -Djava.io.tmpdir=/some/other/dir"
>
> Recent versions of JNA have a dedicated property "jna.tmpdir" for this,
> so that files that are supposed to go into /tmp (i.e. regular temp files)
> are unaffected. It has precedence over java.io.tmpdir.
>
> -h
>
>


-- 
-
Nate McCall
Austin, TX
@zznate

Co-Founder & Sr. Technical Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Suggestions for upgrading cassandra

[Eric Plowe]

i have a cluster that is running 1.2.6. I'd like to upgrade that cluster to
2.0.7

Any suggestions/tips that would make the upgrade process smooth?

Re: Suggestions for upgrading cassandra

[Paulo Ricardo Motta Gomes]

I've written a bit about upgrading from 1.1 to 1.2, non-vnodes:
http://monkeys.chaordic.com.br/operation/zero-downtime-cassandra-upgrade/

Some tips may be valid for a more recent upgrade, but I'm sure the
community has more specific tips regarding the upgrade from 1.2 to 2.0.

On Tue, May 27, 2014 at 2:57 PM, Eric Plowe  wrote:

> i have a cluster that is running 1.2.6. I'd like to upgrade that cluster
> to 2.0.7
>
> Any suggestions/tips that would make the upgrade process smooth?
>



-- 
*Paulo Motta*

Chaordic | *Platform*
*www.chaordic.com.br *
+55 48 3232.3200

Re: Strange trouble with JNA on CentOS6

[david.daeschler]

Great! Thanks for the feedback gents.


Have a nice week.








From: Nate McCall
Sent: ‎Tuesday‎, ‎May‎ ‎27‎, ‎2014 ‎12‎:‎39‎ ‎PM
To: Cassandra Users





I missed that - thanks Holger!




On Tue, May 27, 2014 at 10:14 AM, Holger Hoffstätte 
 wrote:

On Tue, 27 May 2014 09:26:41 -0500, Nate McCall wrote:

>> In the new setup /tmp was mounted as noexec. So it looks like JNA was
>> putting the native library here and then it was unable to execute it.
>>
>>
> If you wanted to keep /tmp noexec (not a bad idea depending on
> environment) you could also append the following to cassandra-env.sh:
>
> JVM_OPTS="$JVM_OPTS -Djava.io.tmpdir=/some/other/dir"

Recent versions of JNA have a dedicated property "jna.tmpdir" for this,
so that files that are supposed to go into /tmp (i.e. regular temp files)
are unaffected. It has precedence over java.io.tmpdir.

-h







-- 

-
Nate McCall
Austin, TX
@zznate

Co-Founder & Sr. Technical Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: Suggestions for upgrading cassandra

[Robert Coli]

On Tue, May 27, 2014 at 10:57 AM, Eric Plowe  wrote:

> i have a cluster that is running 1.2.6. I'd like to upgrade that cluster
> to 2.0.7
>
> Any suggestions/tips that would make the upgrade process smooth?
>

As indicated in NOTES.txt, upgrades to 2.0.x must pass through at least
(iirc?) 1.2.9.

=Rob

Re: Avoiding High Cell Tombstone Count

[Robert Coli]

On Sun, May 25, 2014 at 12:01 PM, Charlie Mason wrote:

> I have a table which has one column per user. It revives at lot of updates
> to these columns through out the life time. They are always updates on a
> few specific columns Firstly is Cassandra storing a Tombstone for each of
> these old column values.
> ...
> As you can see that's awful lot of tombstoned cells. That's after a full
> compaction as well. Just so you are aware this table is updated using a
> Paxos IF statement.
>

If you do a lot of UPDATEs, perhaps a log structured database with
immutable datafiles from which row fragments are reconciled on read is not
for you. Especially if you have to use lightweight "transactions" to make
your application semantics work.


> Would I better off adding a time based key to the primary key. Then doing
> a sepperate insert and then deleting the original. If I did the query with
> a limit of one it should always find the first rows before hitting a
> tombstone. Is that correct?
>

I have no idea what you're asking regarding a LIMIT of 1... in general
anything that scans over multiple partitions is bad. I'm pretty sure you
almost always want to use a design which allows you to use FIRST instead of
LIMIT for this reason.

The overall form of your questions suggests you might be better off using
the right tool for the job, which may not be Cassandra.

=Rob

Re: Avoiding High Cell Tombstone Count

[Colin]

Charlie,

I would be willing to help you out with your issues tomorrow afternoon, feel 
free to give me a call after 4m ET.  There are lots of people who store *and* 
update data with cassandra (at scale).

--
Colin Clark   | Solutions Architect
DataStax  |  www.datastax.com 
m | +1-320-221-9531
e  | colin.cl...@datastax.com


We power the big data applications that transform business.

More than 400 customers, including startups and twenty-five percent of the 
Fortune 100 rely on DataStax's massively scalable, flexible, fast and 
continuously available big data platform built on Apache Cassandra™. DataStax 
integrates in one cluster (thus requiring no ETL)  enterprise-ready Cassandra, 
Apache Hadoop™ for analytics and Apache Solr™ for search, across multiple data 
centers and in the cloud all while providing advanced enterprise security 
features that keep data safe.
 

> On May 27, 2014, at 4:16 PM, Robert Coli  wrote:
> 
>> On Sun, May 25, 2014 at 12:01 PM, Charlie Mason  
>> wrote:
>> I have a table which has one column per user. It revives at lot of updates 
>> to these columns through out the life time. They are always updates on a few 
>> specific columns Firstly is Cassandra storing a Tombstone for each of these 
>> old column values. 
>> ...
>> As you can see that's awful lot of tombstoned cells. That's after a full 
>> compaction as well. Just so you are aware this table is updated using a 
>> Paxos IF statement.
> 
> If you do a lot of UPDATEs, perhaps a log structured database with immutable 
> datafiles from which row fragments are reconciled on read is not for you. 
> Especially if you have to use lightweight "transactions" to make your 
> application semantics work.
>  
>> Would I better off adding a time based key to the primary key. Then doing a 
>> sepperate insert and then deleting the original. If I did the query with a 
>> limit of one it should always find the first rows before hitting a 
>> tombstone. Is that correct? 
> 
> I have no idea what you're asking regarding a LIMIT of 1... in general 
> anything that scans over multiple partitions is bad. I'm pretty sure you 
> almost always want to use a design which allows you to use FIRST instead of 
> LIMIT for this reason.
> 
> The overall form of your questions suggests you might be better off using the 
> right tool for the job, which may not be Cassandra.
> 
> =Rob

Re: Suggestions for upgrading cassandra

[Patricia Gorla]

I'd review the CHANGES.txt to see what has changed and the NEWS.txt to see
what you need to do before upgrading (config changes, data storage changes,
etc).

Generally:

- Determine what version you are upgrading to, and why
- Snapshot the data across all nodes
- Upgrade the nodes one at a time (rolling upgrade)

For a change this major, you should set up a test cluster and see if there
are any changes after the upgrade.

-- 
Patricia Gorla
@patriciagorla

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com 

Re: Suggestions for upgrading cassandra

[Robert Coli]

On Tue, May 27, 2014 at 1:24 PM, Robert Coli  wrote:

> On Tue, May 27, 2014 at 10:57 AM, Eric Plowe  wrote:
>
>> i have a cluster that is running 1.2.6. I'd like to upgrade that cluster
>> to 2.0.7
>>
>> Any suggestions/tips that would make the upgrade process smooth?
>>
>
> As indicated in NOTES.txt, upgrades to 2.0.x must pass through at least
> (iirc?) 1.2.9.
>

s/NOTES/NEWS/

=Rob

Re: Suggestions for upgrading cassandra

[Eric Plowe]

Thanks guys!

Yea I am going to do this on a test environment that simulates our
production environment. I was just looking for any potential gotchas, etc.




On Tue, May 27, 2014 at 6:42 PM, Robert Coli  wrote:

> On Tue, May 27, 2014 at 1:24 PM, Robert Coli  wrote:
>
>> On Tue, May 27, 2014 at 10:57 AM, Eric Plowe wrote:
>>
>>> i have a cluster that is running 1.2.6. I'd like to upgrade that cluster
>>> to 2.0.7
>>>
>>> Any suggestions/tips that would make the upgrade process smooth?
>>>
>>
>> As indicated in NOTES.txt, upgrades to 2.0.x must pass through at least
>> (iirc?) 1.2.9.
>>
>
> s/NOTES/NEWS/
>
> =Rob
>
>

Re: Cassandra CSV & JSON uploader

[Samir Faci]

http://www.datastax.com/docs/1.0/references/sstable2json  might be what
you're looking for.  It's in the bin folder of your cassandra installation.

Though I really doubt you'd want to just drop what is in Oracle into
cassandra.  SQL to NoSQL is rarely ever a 1 to 1 mapping.


On Tue, May 27, 2014 at 8:23 AM, Joyabrata Das  wrote:

> Hi,
>
> Could someone please help on how to import data from Apache Cassandra to
> Oracle Coherence.
>
> As per my understanding it could be done by sstable to JSON-->JSON upload
> to oracle coherence, however is there any code/tool/script to upload JSON
> to Oracle Coherence.
>
> Thanks,
> Joy
>



-- 
Samir Faci
*insert title*
fortune | cowsay -f /usr/share/cows/tux.cow

Sent from my non-iphone laptop.

Re: Cassandra CSV & JSON uploader

[Colin]

Why wouldnt you use datastax enterprise in-memory option vs oracle coherence?


 

> On May 27, 2014, at 10:33 PM, Samir Faci  wrote:
> 
> http://www.datastax.com/docs/1.0/references/sstable2json  might be what 
> you're looking for.  It's in the bin folder of your cassandra installation.
> 
> Though I really doubt you'd want to just drop what is in Oracle into 
> cassandra.  SQL to NoSQL is rarely ever a 1 to 1 mapping.  
> 
> 
>> On Tue, May 27, 2014 at 8:23 AM, Joyabrata Das 
>>  wrote:
>> Hi,
>> 
>> Could someone please help on how to import data from Apache Cassandra to 
>> Oracle Coherence.
>> 
>> As per my understanding it could be done by sstable to JSON-->JSON upload to 
>> oracle coherence, however is there any code/tool/script to upload JSON to 
>> Oracle Coherence.
>> 
>> Thanks,
>> Joy
> 
> 
> 
> -- 
> Samir Faci
> *insert title*
> fortune | cowsay -f /usr/share/cows/tux.cow
> 
> Sent from my non-iphone laptop.

Re: decommissioning a node

[Tim Dunphy]

>
> In case the above doesn't work, another thing to be aware of is that JMX
> uses 2 different ports. The initial connection to 7199 causes a second port
> to be opened, which is normally assigned randomly to an available and
> otherwise unused port above 1024. If your server has a
> firewall/ACL/Security Group configured, it probably doesn't expose the port.
>
> Assuming you are on Java 7 (u4 or later), you can add the following to
> your environment:
> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=7200"
>
> to force a fixed port (7200 in this case) for the second one, which is
> more firewall friendly. You'll still need to be sure access to port 7200 is
> allowed, but at least it's only a single port.
>

Hope that helps,
>

Hi Steven,

 Thank you for that. I'll give this a try. Right now I am down to one
cassandra node with one node reported down in nodetool stats. I'll probably
make another attempt to revive another cassandra node in order to get rid
of the lingering dead one later on this week.

But in the meantime, as an FYI I am on java 8 currently. So this should
hopefully work.

java version "1.8.0"
Java(TM) SE Runtime Environment (build 1.8.0-b132)
Java HotSpot(TM) 64-Bit Server VM (build 25.0-b70, mixed mode)




> Thanks
>
Tim


On Tue, May 27, 2014 at 12:04 PM, Steven A Robenalt
wrote:

> Hi Tim,
>
> In case the above doesn't work, another thing to be aware of is that JMX
> uses 2 different ports. The initial connection to 7199 causes a second port
> to be opened, which is normally assigned randomly to an available and
> otherwise unused port above 1024. If your server has a
> firewall/ACL/Security Group configured, it probably doesn't expose the port.
>
> Assuming you are on Java 7 (u4 or later), you can add the following to
> your environment:
> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=7200"
>
> to force a fixed port (7200 in this case) for the second one, which is
> more firewall friendly. You'll still need to be sure access to port 7200 is
> allowed, but at least it's only a single port.
>
> Hope that helps,
>
> Steve
>
>
>
> On Sun, May 25, 2014 at 7:48 AM, Tim Dunphy  wrote:
>
>> ok I copied the cassandra.env from the host that had cassandra listening
>> on port 7199 to the node that wasn't.
>>
>> That got it listening on the JMX port:
>>
>> [root@beta:~] #lsof -i :7199
>>  COMMAND  PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
>> java9197 root   45u  IPv4 6411278  0t0  TCP *:7199 (LISTEN)
>>
>> But even tho I can telnet to that port from the seed node:
>>
>> [root@beta-new:~] #telnet  10.10.1.18 7199
>> Trying 166.78.27.18...
>> Connected to 166.78.27.18.
>> Escape character is '^]'.
>>
>>
>> I still get connection refused when trying to decommission the node:
>>
>> [root@beta-new:~] #nodetool -host 10.10.1.18 decommission
>> Failed to connect to '166.78.27.18:7199': Connection refused
>>
>> To Colin, thanks for the information!
>>
>> nodetool decomission 
>>
>> UN means UP, NORMAL
>>
>> Oh and thanks I tried that and it seems to be working!
>>
>> [root@beta-new:~] #nodetool decommission
>> f2a48fc7-a362-43f5-9061-4bb3739fdeaf
>> Decommission will decommission the node you are connected to and does not
>> take arguments!
>>
>> Sorry guys, wrote this email in a hurry as we're checking out of a hotel
>> room currently. :) I'll let you know if this does work.
>>
>>
>>
>> On Sun, May 25, 2014 at 10:19 AM, Colin Clark  wrote:
>>
>>> Try this:
>>>
>>> nodetool decomission 
>>>
>>>  UN means UP, NORMAL
>>>
>>> --
>>> Colin
>>> +1 320 221 9531
>>>
>>>
>>>
>>> On Sun, May 25, 2014 at 9:09 AM, Tim Dunphy wrote:
>>>
 Also for information that may help diagnose this issue I am running
 cassandra 2.0.7

 I am also using these java options:

 [root@beta:/etc/alternatives/cassandrahome] #grep -i jvm_opts
 conf/cassandra-env.sh  | grep -v '#'
 JVM_OPTS="$JVM_OPTS -ea"
 JVM_OPTS="$JVM_OPTS -javaagent:$CASSANDRA_HOME/lib/jamm-0.2.5.jar"
 JVM_OPTS="$JVM_OPTS -XX:+CMSClassUnloadingEnabled"
 JVM_OPTS="$JVM_OPTS -XX:+UseThreadPriorities"
 JVM_OPTS="$JVM_OPTS -XX:ThreadPriorityPolicy=42"
 JVM_OPTS="$JVM_OPTS -Xms${MAX_HEAP_SIZE}"
 JVM_OPTS="$JVM_OPTS -Xmx${MAX_HEAP_SIZE}"
 JVM_OPTS="$JVM_OPTS -Xmn${HEAP_NEWSIZE}"
 JVM_OPTS="$JVM_OPTS -XX:+HeapDumpOnOutOfMemoryError"
 JVM_OPTS="$JVM_OPTS
 -XX:HeapDumpPath=$CASSANDRA_HEAPDUMP_DIR/cassandra-`date +%s`-pid$$.hprof"
 JVM_OPTS="$JVM_OPTS -Xss256k"
 JVM_OPTS="$JVM_OPTS -XX:StringTableSize=103"
 JVM_OPTS="$JVM_OPTS -XX:+UseParNewGC"
 JVM_OPTS="$JVM_OPTS -XX:+UseConcMarkSweepGC"
 JVM_OPTS="$JVM_OPTS -XX:+CMSParallelRemarkEnabled"
 JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=8"
 JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=1"
 JVM_OPTS="$JVM_OPTS -XX:CMSInitiatingOccupancyFraction=75"
 JVM_OPTS="$JVM_OPTS -XX:+UseCMSInitiatingOccupancyOnly"
 JVM_OPTS="$JVM_OPTS -XX:+UseTLAB"
 JVM_OPTS="$JVM_OPTS -XX:+Us

Multi-DC Repairs and Token Questions

[Matthew Allen]

Hi,

Am a bit confused regarding data ownership in a multi-dc environment.

I have the following setup in a test cluster with a keyspace with
(placement_strategy = 'NetworkTopologyStrategy' and strategy_options =
{'DC_NSW':2,'DC_VIC':2};)

Datacenter: DC_NSW
==
Replicas: 2
Address RackStatus State   Load
OwnsToken

0
nsw1  rack1   Up Normal  1007.43 MB  100.00%
-9223372036854775808
nsw2  rack1   Up Normal  1008.08 MB  100.00% 0


Datacenter: DC_VIC
==
Replicas: 2
Address RackStatus State   Load
OwnsToken

100
vic1   rack1   Up Normal  1015.1 MB   100.00%
-9223372036854775708
vic2   rack1   Up Normal  1015.13 MB  100.00% 100

My understanding is that both Datacenters have a complete copy of the data,
but when I run a repair -pr on each of the nodes, the vic hosts only take a
couple of seconds, while the nsw nodes take about 5 minutes each.

Does this mean that nsw nodes "own" the majority of the data given their
key ranges and that repairs will need to cross datacenters ?

Thanks

Matt

command>nodetool -h vic1 repair -pr   (takes seconds)
Starting NodeTool
[2014-05-28 15:11:02,783] Starting repair command #1, repairing 1 ranges
for keyspace MY_KEYSPACE
[2014-05-28 15:11:03,110] Repair session
76d170f0-e626-11e3-af4e-218541ad23a1 for range
(-9223372036854775808,-9223372036854775708] finished
[2014-05-28 15:11:03,110] Repair command #1 finished
[2014-05-28 15:11:03,126] Nothing to repair for keyspace 'system'
[2014-05-28 15:11:03,126] Nothing to repair for keyspace 'system_traces'

command>nodetool -h vic2 repair -pr (takes seconds)
Starting NodeTool
[2014-05-28 15:11:28,746] Starting repair command #1, repairing 1 ranges
for keyspace MY_KEYSPACE
[2014-05-28 15:11:28,840] Repair session
864b14a0-e626-11e3-9612-07b0c029e3c7 for range (0,100] finished
[2014-05-28 15:11:28,840] Repair command #1 finished
[2014-05-28 15:11:28,866] Nothing to repair for keyspace 'system'
[2014-05-28 15:11:28,866] Nothing to repair for keyspace 'system_traces'

command>nodetool -h nsw1 repair -pr (takes minutes)
Starting NodeTool
[2014-05-28 15:11:32,579] Starting repair command #1, repairing 1 ranges
for keyspace SN_KEYSPACE
[2014-05-28 15:14:07,187] Repair session
88966430-e626-11e3-81eb-c991646ac2bf for range (100,-9223372036854775808]
finished
[2014-05-28 15:14:07,187] Repair command #1 finished
[2014-05-28 15:14:11,393] Nothing to repair for keyspace 'system'
[2014-05-28 15:14:11,440] Nothing to repair for keyspace 'system_traces'

command>nodetool -h nsw2 repair -pr (takes minutes)
Starting NodeTool
[2014-05-28 15:14:18,670] Starting repair command #1, repairing 1 ranges
for keyspace SN_KEYSPACE
[2014-05-28 15:17:27,300] Repair session
eb936ce0-e626-11e3-81e2-8790242f886e for range (-9223372036854775708,0]
finished
[2014-05-28 15:17:27,300] Repair command #1 finished
[2014-05-28 15:17:32,017] Nothing to repair for keyspace 'system'
[2014-05-28 15:17:32,064] Nothing to repair for keyspace 'system_traces'

Re: Multi-DC Repairs and Token Questions

[chovatia jaydeep]

What is your partition type? Is it org.apache.cassandra.dht.Murmur3Partitioner?
In your repair command i do see there are two different KeySpaces "MY_KEYSPACE" 
and "SN_KEYSPACE", are these two separate key spaces or typo?  

  
-jaydeep



On Tuesday, 27 May 2014 10:26 PM, Matthew Allen  
wrote:
 


Hi,

Am a bit confused regarding data ownership in a multi-dc environment.

I have the following setup in a test cluster with a keyspace with 
(placement_strategy = 'NetworkTopologyStrategy' and strategy_options = 
{'DC_NSW':2,'DC_VIC':2};)


Datacenter: DC_NSW
==
Replicas: 2
Address Rack    Status State   Load    Owns    
Token
   0
nsw1  rack1   Up Normal  1007.43 MB  100.00% 
-9223372036854775808
nsw2  rack1   Up Normal  1008.08 MB  100.00% 0


Datacenter: DC_VIC
==
Replicas: 2
Address Rack    Status State   Load    Owns    
Token
   
100
vic1   rack1   Up Normal  1015.1 MB   100.00% 
-9223372036854775708
vic2   rack1   Up Normal  1015.13 MB  100.00% 100


My understanding is that both Datacenters have a complete copy of the data, but 
when I run a repair -pr on each of the nodes, the vic hosts only take a couple 
of seconds, while the nsw nodes take about 5 minutes each.

Does this mean that nsw nodes "own" the majority of the data given their key 
ranges and that repairs will need to cross datacenters ?


Thanks

Matt


command>nodetool -h vic1 repair -pr   (takes seconds)
Starting NodeTool
[2014-05-28 15:11:02,783] Starting repair command #1, repairing 1 ranges for 
keyspace MY_KEYSPACE
[2014-05-28 15:11:03,110] Repair session 76d170f0-e626-11e3-af4e-218541ad23a1 
for range (-9223372036854775808,-9223372036854775708] finished
[2014-05-28 15:11:03,110] Repair command #1 finished
[2014-05-28 15:11:03,126] Nothing to repair for keyspace 'system'
[2014-05-28 15:11:03,126] Nothing to repair for keyspace 'system_traces'

command>nodetool -h vic2 repair -pr (takes seconds)
Starting NodeTool
[2014-05-28 15:11:28,746] Starting repair command #1, repairing 1 ranges for 
keyspace MY_KEYSPACE
[2014-05-28 15:11:28,840] Repair session 864b14a0-e626-11e3-9612-07b0c029e3c7 
for range (0,100] finished
[2014-05-28 15:11:28,840] Repair command #1 finished
[2014-05-28 15:11:28,866] Nothing to repair for keyspace 'system'
[2014-05-28 15:11:28,866] Nothing to repair for keyspace 'system_traces'

command>nodetool -h nsw1 repair -pr (takes minutes)
Starting NodeTool
[2014-05-28 15:11:32,579] Starting repair command #1, repairing 1 ranges for 
keyspace SN_KEYSPACE
[2014-05-28 15:14:07,187] Repair session 88966430-e626-11e3-81eb-c991646ac2bf 
for range (100,-9223372036854775808] finished
[2014-05-28 15:14:07,187] Repair command #1 finished
[2014-05-28 15:14:11,393] Nothing to repair for keyspace 'system'
[2014-05-28 15:14:11,440] Nothing to repair for keyspace 'system_traces'

command>nodetool -h nsw2 repair -pr (takes minutes)
Starting NodeTool
[2014-05-28 15:14:18,670] Starting repair command #1, repairing 1 ranges for 
keyspace SN_KEYSPACE
[2014-05-28 15:17:27,300] Repair session eb936ce0-e626-11e3-81e2-8790242f886e 
for range (-9223372036854775708,0] finished
[2014-05-28 15:17:27,300] Repair command #1 finished
[2014-05-28 15:17:32,017] Nothing to repair for keyspace 'system'
[2014-05-28 15:17:32,064] Nothing to repair for keyspace 'system_traces'