Re: Problem in getting data from a 2 node cluster of Cassandra

[Roshni Rajagopal]

Hi Prakrati,

 In an ideal situation, no data should be lost when a node is added. How are 
you getting the statistics below.
The output below looks like its from some code using Hector or Thrift..is the 
code to get statistics from a 1 node cluster or 2 exactly the same- with the 
only change being a node being added or removed?
Could you verify the number of rows & cols in the column family using CLI or 
CQL..

Regards,
Roshni




From: Prakrati Agrawal 
mailto:prakrati.agra...@mu-sigma.com>>
Reply-To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Date: Friday 8 June 2012 11:50 AM
To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Subject: Problem in getting data from a 2 node cluster of Cassandra

Dear all

I was originally having a 1 node cluster. Then I added one more node to it with 
initial token configured appropriately. Now when I run my queries I am not 
getting all my data ie all columns.
 Output on 2 nodes
Time taken to retrieve columns 43707 of key range is 1276
Time taken to retrieve columns 2084199 of all tickers is 54334
Time taken to count is 230776
Total number of rows in the database are 183
Total number of columns in the database are 7903753
Output on 1 node
Time taken to retrieve columns 43707 of key range is 767
Time taken to retrieve columns 382 of all tickers is 52793
Time taken to count is 268135
Total number of rows in the database are 396
Total number of columns in the database are 16316426
Please help me. Where is my data going or how should I retrieve it. I have 
consistency level specified as ONE and I did not specify any replication factor.



Prakrati Agrawal | Developer - Big Data(I&D)| 9731648376 | www.mu-sigma.com



This email message may contain proprietary, private and confidential 
information. The information transmitted is intended only for the person(s) or 
entities to which it is addressed. Any review, retransmission, dissemination or 
other use of, or taking of any action in reliance upon, this information by 
persons or entities other than the intended recipient is prohibited and may be 
illegal. If you received this in error, please contact the sender and delete 
the message from your system.

Mu Sigma takes all reasonable steps to ensure that its electronic 
communications are free from viruses. However, given Internet accessibility, 
the Company cannot accept liability for any virus introduced by this e-mail or 
any attachment and you are advised to use up-to-date virus checking software.

This email and any files transmitted with it are confidential and intended 
solely for the individual or entity to whom they are addressed. If you have 
received this email in error destroy it immediately. *** Walmart Confidential 
***

Problem in adding a new node

[Prakrati Agrawal]

Dear all,

I had a 1 node cluster of Cassandra. Then I added one more node to it and 
started Cassandra on it. I got the following error:

INFO 12:44:49,588 Loading persisted ring state
ERROR 12:44:49,613 Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
java.io.IOError: java.io.IOException: Map failed
at 
org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:127)
at 
org.apache.cassandra.db.commitlog.CommitLogAllocator$3.run(CommitLogAllocator.java:191)
at 
org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
at 
org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:119)
... 4 more
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)

Please tell me what is the reason for this error and how should I rectify it.

Thanks and Regards
Prakrati



This email message may contain proprietary, private and confidential 
information. The information transmitted is intended only for the person(s) or 
entities to which it is addressed. Any review, retransmission, dissemination or 
other use of, or taking of any action in reliance upon, this information by 
persons or entities other than the intended recipient is prohibited and may be 
illegal. If you received this in error, please contact the sender and delete 
the message from your system.

Mu Sigma takes all reasonable steps to ensure that its electronic 
communications are free from viruses. However, given Internet accessibility, 
the Company cannot accept liability for any virus introduced by this e-mail or 
any attachment and you are advised to use up-to-date virus checking software.

RE: Problem in getting data from a 2 node cluster of Cassandra

[Prakrati Agrawal]

Yes the code is the same for both 1 and 2 node cluster. It's a Hector code. How 
do I get the number of rows and columns from Cassandra CLI as the data is very 
large.

Thanks and Regards
Prakrati


-Original Message-
From: Roshni Rajagopal [mailto:roshni.rajago...@wal-mart.com]
Sent: Friday, June 08, 2012 12:43 PM
To: user@cassandra.apache.org
Subject: Re: Problem in getting data from a 2 node cluster of Cassandra

Hi Prakrati,

 In an ideal situation, no data should be lost when a node is added. How are 
you getting the statistics below.
The output below looks like its from some code using Hector or Thrift..is the 
code to get statistics from a 1 node cluster or 2 exactly the same- with the 
only change being a node being added or removed?
Could you verify the number of rows & cols in the column family using CLI or 
CQL..

Regards,
Roshni




From: Prakrati Agrawal 
mailto:prakrati.agra...@mu-sigma.com>>
Reply-To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Date: Friday 8 June 2012 11:50 AM
To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Subject: Problem in getting data from a 2 node cluster of Cassandra

Dear all

I was originally having a 1 node cluster. Then I added one more node to it with 
initial token configured appropriately. Now when I run my queries I am not 
getting all my data ie all columns.
 Output on 2 nodes
Time taken to retrieve columns 43707 of key range is 1276
Time taken to retrieve columns 2084199 of all tickers is 54334
Time taken to count is 230776
Total number of rows in the database are 183
Total number of columns in the database are 7903753
Output on 1 node
Time taken to retrieve columns 43707 of key range is 767
Time taken to retrieve columns 382 of all tickers is 52793
Time taken to count is 268135
Total number of rows in the database are 396
Total number of columns in the database are 16316426
Please help me. Where is my data going or how should I retrieve it. I have 
consistency level specified as ONE and I did not specify any replication factor.



Prakrati Agrawal | Developer - Big Data(I&D)| 9731648376 | www.mu-sigma.com



This email message may contain proprietary, private and confidential 
information. The information transmitted is intended only for the person(s) or 
entities to which it is addressed. Any review, retransmission, dissemination or 
other use of, or taking of any action in reliance upon, this information by 
persons or entities other than the intended recipient is prohibited and may be 
illegal. If you received this in error, please contact the sender and delete 
the message from your system.

Mu Sigma takes all reasonable steps to ensure that its electronic 
communications are free from viruses. However, given Internet accessibility, 
the Company cannot accept liability for any virus introduced by this e-mail or 
any attachment and you are advised to use up-to-date virus checking software.

This email and any files transmitted with it are confidential and intended 
solely for the individual or entity to whom they are addressed. If you have 
received this email in error destroy it immediately. *** Walmart Confidential 
***

 This email message may contain proprietary, private and confidential 
information. The information transmitted is intended only for the person(s) or 
entities to which it is addressed. Any review, retransmission, dissemination or 
other use of, or taking of any action in reliance upon, this information by 
persons or entities other than the intended recipient is prohibited and may be 
illegal. If you received this in error, please contact the sender and delete 
the message from your system.

Mu Sigma takes all reasonable steps to ensure that its electronic 
communications are free from viruses. However, given Internet accessibility, 
the Company cannot accept liability for any virus introduced by this e-mail or 
any attachment and you are advised to use up-to-date virus checking software.

RE: Problem in adding a new node

[MOHD ARSHAD SALEEM]

Hi,

the node which you added in that (cassandra.yaml file)give the ip address of 
1st node in seeds option.

Regards
Arshad

From: Prakrati Agrawal [prakrati.agra...@mu-sigma.com]
Sent: Friday, June 08, 2012 12:44 PM
To: user@cassandra.apache.org
Subject: Problem in adding a new node

Dear all,

I had a 1 node cluster of Cassandra. Then I added one more node to it and 
started Cassandra on it. I got the following error:

INFO 12:44:49,588 Loading persisted ring state
ERROR 12:44:49,613 Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
java.io.IOError: java.io.IOException: Map failed
at 
org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:127)
at 
org.apache.cassandra.db.commitlog.CommitLogAllocator$3.run(CommitLogAllocator.java:191)
at 
org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
at 
org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:119)
... 4 more
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)

Please tell me what is the reason for this error and how should I rectify it.

Thanks and Regards
Prakrati



This email message may contain proprietary, private and confidential 
information. The information transmitted is intended only for the person(s) or 
entities to which it is addressed. Any review, retransmission, dissemination or 
other use of, or taking of any action in reliance upon, this information by 
persons or entities other than the intended recipient is prohibited and may be 
illegal. If you received this in error, please contact the sender and delete 
the message from your system.

Mu Sigma takes all reasonable steps to ensure that its electronic 
communications are free from viruses. However, given Internet accessibility, 
the Company cannot accept liability for any virus introduced by this e-mail or 
any attachment and you are advised to use up-to-date virus checking software.

Re: Problem in getting data from a 2 node cluster of Cassandra

[rohit bhatia]

run nodetool -h localhost cfstats on the nodes... this gives node
specific column family based data...
just run this for both nodes...

On Fri, Jun 8, 2012 at 12:46 PM, Prakrati Agrawal
 wrote:
> Yes the code is the same for both 1 and 2 node cluster. It's a Hector code. 
> How do I get the number of rows and columns from Cassandra CLI as the data is 
> very large.
>
> Thanks and Regards
> Prakrati
>
>
> -Original Message-
> From: Roshni Rajagopal [mailto:roshni.rajago...@wal-mart.com]
> Sent: Friday, June 08, 2012 12:43 PM
> To: user@cassandra.apache.org
> Subject: Re: Problem in getting data from a 2 node cluster of Cassandra
>
> Hi Prakrati,
>
>  In an ideal situation, no data should be lost when a node is added. How are 
> you getting the statistics below.
> The output below looks like its from some code using Hector or Thrift..is the 
> code to get statistics from a 1 node cluster or 2 exactly the same- with the 
> only change being a node being added or removed?
> Could you verify the number of rows & cols in the column family using CLI or 
> CQL..
>
> Regards,
> Roshni
>
>
>
>
> From: Prakrati Agrawal 
> mailto:prakrati.agra...@mu-sigma.com>>
> Reply-To: "user@cassandra.apache.org" 
> mailto:user@cassandra.apache.org>>
> Date: Friday 8 June 2012 11:50 AM
> To: "user@cassandra.apache.org" 
> mailto:user@cassandra.apache.org>>
> Subject: Problem in getting data from a 2 node cluster of Cassandra
>
> Dear all
>
> I was originally having a 1 node cluster. Then I added one more node to it 
> with initial token configured appropriately. Now when I run my queries I am 
> not getting all my data ie all columns.
>  Output on 2 nodes
> Time taken to retrieve columns 43707 of key range is 1276
> Time taken to retrieve columns 2084199 of all tickers is 54334
> Time taken to count is 230776
> Total number of rows in the database are 183
> Total number of columns in the database are 7903753
> Output on 1 node
> Time taken to retrieve columns 43707 of key range is 767
> Time taken to retrieve columns 382 of all tickers is 52793
> Time taken to count is 268135
> Total number of rows in the database are 396
> Total number of columns in the database are 16316426
> Please help me. Where is my data going or how should I retrieve it. I have 
> consistency level specified as ONE and I did not specify any replication 
> factor.
>
>
>
> Prakrati Agrawal | Developer - Big Data(I&D)| 9731648376 | www.mu-sigma.com
>
>
> 
> This email message may contain proprietary, private and confidential 
> information. The information transmitted is intended only for the person(s) 
> or entities to which it is addressed. Any review, retransmission, 
> dissemination or other use of, or taking of any action in reliance upon, this 
> information by persons or entities other than the intended recipient is 
> prohibited and may be illegal. If you received this in error, please contact 
> the sender and delete the message from your system.
>
> Mu Sigma takes all reasonable steps to ensure that its electronic 
> communications are free from viruses. However, given Internet accessibility, 
> the Company cannot accept liability for any virus introduced by this e-mail 
> or any attachment and you are advised to use up-to-date virus checking 
> software.
>
> This email and any files transmitted with it are confidential and intended 
> solely for the individual or entity to whom they are addressed. If you have 
> received this email in error destroy it immediately. *** Walmart Confidential 
> ***
>
>  This email message may contain proprietary, private and confidential 
> information. The information transmitted is intended only for the person(s) 
> or entities to which it is addressed. Any review, retransmission, 
> dissemination or other use of, or taking of any action in reliance upon, this 
> information by persons or entities other than the intended recipient is 
> prohibited and may be illegal. If you received this in error, please contact 
> the sender and delete the message from your system.
>
> Mu Sigma takes all reasonable steps to ensure that its electronic 
> communications are free from viruses. However, given Internet accessibility, 
> the Company cannot accept liability for any virus introduced by this e-mail 
> or any attachment and you are advised to use up-to-date virus checking 
> software.

kswapd0 causing read timeouts

[Gurpreet Singh]

Hi,
I am testing cassandra 1.1 on a 1 node cluster.
8 core, 16 gb ram, 6 data disks raid0, no swap configured

cassandra 1.1.1
heap size: 8 gigs
key cache size in mb: 800 (used only 200mb till now)
memtable_total_space_in_mb : 2048

I am running a read workload.. about 30 reads/second. no writes at all.
The system runs fine for roughly 12 hours.

jconsole shows that my heap size has hardly touched 4 gigs.
top shows -
  SHR increasing slowly from 100 mb to 6.6 gigs in  these 12 hrs
  RES increases slowly from 6 gigs all the way to 15 gigs
  buffers are at a healthy 25 mb at some point and that goes down to 2 mb
in these 12 hrs
  VIRT stays at 85 gigs

I understand that SHR goes up because of mmap, RES goes up because it is
showing SHR value as well.

After around 10-12 hrs, the cpu utilization of the system starts
increasing, and i notice that kswapd0 process starts becoming more active.
Gradually, the system cpu becomes high almost 70%, and the client starts
getting continuous timeouts. The fact that the buffers went down from 20 mb
to 2 mb suggests that kswapd0 is probably swapping out the pagecache.

Is there a way out of this to avoid the kswapd0 starting to do things even
when there is no swap configured?
This is very easily reproducible for me, and would like a way out of this
situation. Do i need to adjust vm memory management stuff like pagecache,
vfs_cache_pressure.. things like that?

just some extra information, jna is installed, mlockall is successful.
there is no compaction running.
would appreciate any help on this.
Thanks
Gurpreet

Re: kswapd0 causing read timeouts

[ruslan usifov]

disk_access_mode: mmap??

set to disk_access_mode: mmap_index_only in cassandra yaml

2012/6/8 Gurpreet Singh :
> Hi,
> I am testing cassandra 1.1 on a 1 node cluster.
> 8 core, 16 gb ram, 6 data disks raid0, no swap configured
>
> cassandra 1.1.1
> heap size: 8 gigs
> key cache size in mb: 800 (used only 200mb till now)
> memtable_total_space_in_mb : 2048
>
> I am running a read workload.. about 30 reads/second. no writes at all.
> The system runs fine for roughly 12 hours.
>
> jconsole shows that my heap size has hardly touched 4 gigs.
> top shows -
>   SHR increasing slowly from 100 mb to 6.6 gigs in  these 12 hrs
>   RES increases slowly from 6 gigs all the way to 15 gigs
>   buffers are at a healthy 25 mb at some point and that goes down to 2 mb in
> these 12 hrs
>   VIRT stays at 85 gigs
>
> I understand that SHR goes up because of mmap, RES goes up because it is
> showing SHR value as well.
>
> After around 10-12 hrs, the cpu utilization of the system starts increasing,
> and i notice that kswapd0 process starts becoming more active. Gradually,
> the system cpu becomes high almost 70%, and the client starts getting
> continuous timeouts. The fact that the buffers went down from 20 mb to 2 mb
> suggests that kswapd0 is probably swapping out the pagecache.
>
> Is there a way out of this to avoid the kswapd0 starting to do things even
> when there is no swap configured?
> This is very easily reproducible for me, and would like a way out of this
> situation. Do i need to adjust vm memory management stuff like pagecache,
> vfs_cache_pressure.. things like that?
>
> just some extra information, jna is installed, mlockall is successful. there
> is no compaction running.
> would appreciate any help on this.
> Thanks
> Gurpreet
>
>

RE: Problem in adding a new node

[Prakrati Agrawal]

Yes I gave the ip address of the 1st node in the seeds option

Thanks and Regards
Prakrati
From: MOHD ARSHAD SALEEM [mailto:marshadsal...@tataelxsi.co.in]
Sent: Friday, June 08, 2012 12:51 PM
To: user@cassandra.apache.org
Subject: RE: Problem in adding a new node

Hi,

the node which you added in that (cassandra.yaml file)give the ip address of 
1st node in seeds option.

Regards
Arshad

From: Prakrati Agrawal [prakrati.agra...@mu-sigma.com]
Sent: Friday, June 08, 2012 12:44 PM
To: user@cassandra.apache.org
Subject: Problem in adding a new node
Dear all,

I had a 1 node cluster of Cassandra. Then I added one more node to it and 
started Cassandra on it. I got the following error:

INFO 12:44:49,588 Loading persisted ring state
ERROR 12:44:49,613 Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
java.io.IOError: java.io.IOException: Map failed
at 
org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:127)
at 
org.apache.cassandra.db.commitlog.CommitLogAllocator$3.run(CommitLogAllocator.java:191)
at 
org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
at 
org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:119)
... 4 more
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)

Please tell me what is the reason for this error and how should I rectify it.

Thanks and Regards
Prakrati



This email message may contain proprietary, private and confidential 
information. The information transmitted is intended only for the person(s) or 
entities to which it is addressed. Any review, retransmission, dissemination or 
other use of, or taking of any action in reliance upon, this information by 
persons or entities other than the intended recipient is prohibited and may be 
illegal. If you received this in error, please contact the sender and delete 
the message from your system.

Mu Sigma takes all reasonable steps to ensure that its electronic 
communications are free from viruses. However, given Internet accessibility, 
the Company cannot accept liability for any virus introduced by this e-mail or 
any attachment and you are advised to use up-to-date virus checking software.


This email message may contain proprietary, private and confidential 
information. The information transmitted is intended only for the person(s) or 
entities to which it is addressed. Any review, retransmission, dissemination or 
other use of, or taking of any action in reliance upon, this information by 
persons or entities other than the intended recipient is prohibited and may be 
illegal. If you received this in error, please contact the sender and delete 
the message from your system.

Mu Sigma takes all reasonable steps to ensure that its electronic 
communications are free from viruses. However, given Internet accessibility, 
the Company cannot accept liability for any virus introduced by this e-mail or 
any attachment and you are advised to use up-to-date virus checking software.

Re: kswapd0 causing read timeouts

[Gurpreet Singh]

Thanks Ruslan.
I will try the mmap_index_only.
Is there any guideline as to when to leave it to auto and when to use
mmap_index_only?

/G

On Fri, Jun 8, 2012 at 1:21 AM, ruslan usifov wrote:

> disk_access_mode: mmap??
>
> set to disk_access_mode: mmap_index_only in cassandra yaml
>
> 2012/6/8 Gurpreet Singh :
> > Hi,
> > I am testing cassandra 1.1 on a 1 node cluster.
> > 8 core, 16 gb ram, 6 data disks raid0, no swap configured
> >
> > cassandra 1.1.1
> > heap size: 8 gigs
> > key cache size in mb: 800 (used only 200mb till now)
> > memtable_total_space_in_mb : 2048
> >
> > I am running a read workload.. about 30 reads/second. no writes at all.
> > The system runs fine for roughly 12 hours.
> >
> > jconsole shows that my heap size has hardly touched 4 gigs.
> > top shows -
> >   SHR increasing slowly from 100 mb to 6.6 gigs in  these 12 hrs
> >   RES increases slowly from 6 gigs all the way to 15 gigs
> >   buffers are at a healthy 25 mb at some point and that goes down to 2
> mb in
> > these 12 hrs
> >   VIRT stays at 85 gigs
> >
> > I understand that SHR goes up because of mmap, RES goes up because it is
> > showing SHR value as well.
> >
> > After around 10-12 hrs, the cpu utilization of the system starts
> increasing,
> > and i notice that kswapd0 process starts becoming more active. Gradually,
> > the system cpu becomes high almost 70%, and the client starts getting
> > continuous timeouts. The fact that the buffers went down from 20 mb to 2
> mb
> > suggests that kswapd0 is probably swapping out the pagecache.
> >
> > Is there a way out of this to avoid the kswapd0 starting to do things
> even
> > when there is no swap configured?
> > This is very easily reproducible for me, and would like a way out of this
> > situation. Do i need to adjust vm memory management stuff like pagecache,
> > vfs_cache_pressure.. things like that?
> >
> > just some extra information, jna is installed, mlockall is successful.
> there
> > is no compaction running.
> > would appreciate any help on this.
> > Thanks
> > Gurpreet
> >
> >
>

Out of memory error

[Prakrati Agrawal]

Dear all,

When I try to list the entire data in my column family I get the following 
error:

Using default limit of 100
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at 
org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:140)
at 
org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at 
org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at 
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at 
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at 
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at 
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at 
org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:683)
at 
org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:667)
at 
org.apache.cassandra.cli.CliClient.executeList(CliClient.java:1373)
at 
org.apache.cassandra.cli.CliClient.executeCLIStatement(CliClient.java:264)
at 
org.apache.cassandra.cli.CliMain.processStatementInteractive(CliMain.java:219)
at org.apache.cassandra.cli.CliMain.main(CliMain.java:346)

Please help me

Thanks and Regards
Prakrati




This email message may contain proprietary, private and confidential 
information. The information transmitted is intended only for the person(s) or 
entities to which it is addressed. Any review, retransmission, dissemination or 
other use of, or taking of any action in reliance upon, this information by 
persons or entities other than the intended recipient is prohibited and may be 
illegal. If you received this in error, please contact the sender and delete 
the message from your system.

Mu Sigma takes all reasonable steps to ensure that its electronic 
communications are free from viruses. However, given Internet accessibility, 
the Company cannot accept liability for any virus introduced by this e-mail or 
any attachment and you are advised to use up-to-date virus checking software.

Re: Out of memory error

[shashwat shriparv]

Check this slide,

http://www.slideshare.net/cloudera/hadoop-troubleshooting-101-kate-ting-cloudera

Regards

∞
Shashwat Shriparv


On Fri, Jun 8, 2012 at 2:34 PM, Prakrati Agrawal <
prakrati.agra...@mu-sigma.com> wrote:

>  Dear all,
>
> ** **
>
> When I try to list the entire data in my column family I get the following
> error: 
>
> ** **
>
> Using default limit of 100
>
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>
> at
> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:140)
> 
>
> at
> org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
> 
>
> at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
> 
>
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
> 
>
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
> 
>
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
>
> at
> org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:683)
> 
>
> at
> org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:667)
> 
>
> at
> org.apache.cassandra.cli.CliClient.executeList(CliClient.java:1373)
>
> at
> org.apache.cassandra.cli.CliClient.executeCLIStatement(CliClient.java:264)
> 
>
> at
> org.apache.cassandra.cli.CliMain.processStatementInteractive(CliMain.java:219)
> 
>
> at org.apache.cassandra.cli.CliMain.main(CliMain.java:346)
> 
>
> ** **
>
> Please help me
>
> ** **
>
> Thanks and Regards
>
> Prakrati
>
> ** **
>
> ** **
>
> --
> This email message may contain proprietary, private and confidential
> information. The information transmitted is intended only for the person(s)
> or entities to which it is addressed. Any review, retransmission,
> dissemination or other use of, or taking of any action in reliance upon,
> this information by persons or entities other than the intended recipient
> is prohibited and may be illegal. If you received this in error, please
> contact the sender and delete the message from your system.
>
> Mu Sigma takes all reasonable steps to ensure that its electronic
> communications are free from viruses. However, given Internet
> accessibility, the Company cannot accept liability for any virus introduced
> by this e-mail or any attachment and you are advised to use up-to-date
> virus checking software.
>



-- 


∞
Shashwat Shriparv

Time taken to retrieve data from a 2 node cluster is more than 1 node cluster

[Prakrati Agrawal]

Dear all

Initially I had a one node cluster and I flooded my data into it. I then ran my 
Hector code to get some rows and columns. It took around 52.793 seconds.
 Then I added one more node to the cluster. I again ran the same code and it 
took around 112.065 seconds.
Cassandra should perform faster when there are more nodes was my belief.Is my 
belief wrong or am I doing something wrong? Please help me

Thanks and Regards
Prakrati



This email message may contain proprietary, private and confidential 
information. The information transmitted is intended only for the person(s) or 
entities to which it is addressed. Any review, retransmission, dissemination or 
other use of, or taking of any action in reliance upon, this information by 
persons or entities other than the intended recipient is prohibited and may be 
illegal. If you received this in error, please contact the sender and delete 
the message from your system.

Mu Sigma takes all reasonable steps to ensure that its electronic 
communications are free from viruses. However, given Internet accessibility, 
the Company cannot accept liability for any virus introduced by this e-mail or 
any attachment and you are advised to use up-to-date virus checking software.

Re: about multitenant datamodel

[Toru Inoko]

See virtual keyspaces in Hector.
Yes, at first, I tried to desigen data model like POD architecture  
(http://goo.gl/Uw1yD) with this.
But, it is problem for me that strong consistency isn't guaranteed among  
metadata schemas.


Every CF has a certain amount of overhead in memory. It's just not how  
Cassandra is designed to be used.
Thanks. I'll try to design meta schma data model again which has strong  
consistency.


Thank you for your advices!

On Wed, 06 Jun 2012 03:35:40 +0900, aaron morton   
wrote:


With an abstraction layer you can store practically anything in  
Cassandra.

See virtual keyspaces in Hector.

why do you think so? I'll let users create ristricted CFs, and limit a  
number of CFs which users create.

is it still a bad one?

Depends what your limits are, but in general still yes.

If someone creates a CF with 10 secondary indexes they will use more  
resources than someone who creates a CF with none. Same thing would  
happen in a multitenant RDBMS server.


If you have 200 CF's in a cluster it will use more memory than one with  
20 CF's. The extra memory use will result in more disk IO.


Cheers


-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 5/06/2012, at 7:52 PM, R. Verlangen wrote:

Every CF has a certain amount of overhead in memory. It's just not how  
Cassandra is designed to be used. Maybe you could think of a way to  
smash data down to indices and entities. With an abstraction layer you  
can store practically anything in Cassandra.


2012/6/5 Toru Inoko 
IMHO a model that allows external users to create CF's is a bad one.

why do you think so? I'll let users create ristricted CFs, and limit a  
number of CFs which users create.

is it still a bad one?


On Thu, 31 May 2012 06:44:05 +0900, aaron morton  
 wrote:


- Do a lot of keyspaces cause some problems? (If I have 1,000 users,  
cassandra creates 1,000 keyspaces…)

It's not keyspaces, but the number of column families.

Without storing any data each CF uses about 1MB of ram. When they start  
storing and reading data they use more.


IMHO a model that allows external users to create CF's is a bad one.

Hope that helps.
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 25/05/2012, at 12:52 PM, Toru Inoko wrote:

Hi, all.

I'm designing data api service(like cassandra.io but not using  
dedicated server for each user) on cassandra 1.1 on which users can do  
DML/DDL method like cql.

Followings are api which users can use( almost same to cassandra api).
- create/read/delete ColumnFamilies/Rows/Columns

Now I'm thinking about multitenant datamodel on that.
My data model like the following.
I'm going to prepare a keyspace for each user as a user's tenant space.

| keyspace1 | --- | column family |
|(for user1)|  |
 ...

| keyspace2 | --- | column family |
|(for user2)|  |
 ...

Followings are my question!
- Is this data model a good for multitenant?
- Do a lot of keyspaces cause some problems? (If I have 1,000 users,  
cassandra creates 1,000 keyspaces...)


please, help.
thank you in advance.

Toru Inoko.











--
With kind regards,

Robin Verlangen
Software engineer

W http://www.robinverlangen.nl
E ro...@us2.nl

Disclaimer: The information contained in this message and attachments  
is intended solely for the attention and use of the named addressee and  
may be confidential. If you are not the intended recipient, you are  
reminded that the information remains the property of the sender. You  
must not use, disclose, distribute, copy, print or rely on this e-mail.  
If you have received this message in error, please contact the sender  
immediately and irrevocably delete this message and any copies.







--
---
SCSK Corp.

Toru Inoko
tel   : 03-6438-3544
mail  : in...@ms.scsk.jp
---

Re: Cassandra 1.1.1 stack overflow on an infinite loop building IntervalTree

[Omid Aladini]

Also looks similar to this ticket:

https://issues.apache.org/jira/browse/CASSANDRA-4078


On Thu, Jun 7, 2012 at 6:48 PM, Omid Aladini  wrote:

> Hi,
>
> One of my 1.1.1 nodes doesn't restart due to stack overflow on building
> the interval tree. Bumping the stack size doesn't help. Here's the stack
> trace:
>
> https://gist.github.com/2889611
>
> It looks more like an infinite loop on IntervalNode constructor's logic
> than a deep tree since DEBUG log shows looping over the same intervals:
>
> https://gist.github.com/2889862
>
> Running it with assertions enabled shows a number of sstables which the
> first key > last key, for example:
>
> 2012-06-07_16:12:18.18781 java.lang.AssertionError: SSTable first key
> DecoratedKey(2254009252149354268486114339861094,
> 3730343137317c3438333632333932) > last key
> DecoratedKey(22166106697727078019854024428005234814,
> 313138323637397c3432373931353435)
>
> and let's the node come up without hitting IntervalNode constructor. I
> wonder how invalid sstables get create in the first place? Is there a way
> to verify if other nodes in the cluster are affected as well?
>
> Speaking of a solution to get the node back up without wiping the data off
> and let it bootstrap again, I was wondering if I remove affected sstables
> and restart the node followed by a repair, will the node end up in a
> consistent state?
>
> SStables contain counter columns and leveled compaction is used.
>
> Thanks,
> Omid
>

Re: Time taken to retrieve data from a 2 node cluster is more than 1 node cluster

[rohit bhatia]

Is ur client code calling with asyncrhynous requests?? and whats ur
replication factor and read consistency level.

In any case, 2 nodes might take as much time as one, but should not be
slow (unless u also doubled the data)...

On Fri, Jun 8, 2012 at 2:41 PM, Prakrati Agrawal
 wrote:
>
>
> Dear all
>
>
>
> Initially I had a one node cluster and I flooded my data into it. I then ran
> my Hector code to get some rows and columns. It took around 52.793 seconds.
>
>  Then I added one more node to the cluster. I again ran the same code and it
> took around 112.065 seconds.
>
> Cassandra should perform faster when there are more nodes was my belief.Is
> my belief wrong or am I doing something wrong? Please help me
>
>
>
> Thanks and Regards
>
> Prakrati
>
>
>
>
> 
> This email message may contain proprietary, private and confidential
> information. The information transmitted is intended only for the person(s)
> or entities to which it is addressed. Any review, retransmission,
> dissemination or other use of, or taking of any action in reliance upon,
> this information by persons or entities other than the intended recipient is
> prohibited and may be illegal. If you received this in error, please contact
> the sender and delete the message from your system.
>
> Mu Sigma takes all reasonable steps to ensure that its electronic
> communications are free from viruses. However, given Internet accessibility,
> the Company cannot accept liability for any virus introduced by this e-mail
> or any attachment and you are advised to use up-to-date virus checking
> software.

RE: Time taken to retrieve data from a 2 node cluster is more than 1 node cluster

[Prakrati Agrawal]

My consistency level is 1 and I did not set any replication factor. I am using 
Hector API for getting data and don't know whether they are asynchronous or not.

Thanks and Regards
Prakrati


-Original Message-
From: rohit bhatia [mailto:rohit2...@gmail.com]
Sent: Friday, June 08, 2012 3:58 PM
To: user@cassandra.apache.org
Subject: Re: Time taken to retrieve data from a 2 node cluster is more than 1 
node cluster

Is ur client code calling with asyncrhynous requests?? and whats ur
replication factor and read consistency level.

In any case, 2 nodes might take as much time as one, but should not be
slow (unless u also doubled the data)...

On Fri, Jun 8, 2012 at 2:41 PM, Prakrati Agrawal
 wrote:
>
>
> Dear all
>
>
>
> Initially I had a one node cluster and I flooded my data into it. I then ran
> my Hector code to get some rows and columns. It took around 52.793 seconds.
>
>  Then I added one more node to the cluster. I again ran the same code and it
> took around 112.065 seconds.
>
> Cassandra should perform faster when there are more nodes was my belief.Is
> my belief wrong or am I doing something wrong? Please help me
>
>
>
> Thanks and Regards
>
> Prakrati
>
>
>
>
> 
> This email message may contain proprietary, private and confidential
> information. The information transmitted is intended only for the person(s)
> or entities to which it is addressed. Any review, retransmission,
> dissemination or other use of, or taking of any action in reliance upon,
> this information by persons or entities other than the intended recipient is
> prohibited and may be illegal. If you received this in error, please contact
> the sender and delete the message from your system.
>
> Mu Sigma takes all reasonable steps to ensure that its electronic
> communications are free from viruses. However, given Internet accessibility,
> the Company cannot accept liability for any virus introduced by this e-mail
> or any attachment and you are advised to use up-to-date virus checking
> software.

 This email message may contain proprietary, private and confidential 
information. The information transmitted is intended only for the person(s) or 
entities to which it is addressed. Any review, retransmission, dissemination or 
other use of, or taking of any action in reliance upon, this information by 
persons or entities other than the intended recipient is prohibited and may be 
illegal. If you received this in error, please contact the sender and delete 
the message from your system.

Mu Sigma takes all reasonable steps to ensure that its electronic 
communications are free from viruses. However, given Internet accessibility, 
the Company cannot accept liability for any virus introduced by this e-mail or 
any attachment and you are advised to use up-to-date virus checking software.

Dead node still being pinged

[Nicolas Lalevée]

I had a configuration where I had 4 nodes, data-1,4. We then bought 3 bigger 
machines, data-5,7. And we moved all data from data-1,4 to data-5,7.
To move all the data without interruption of service, I added one new node at a 
time. And then I removed one by one the old machines via a "remove token".

Everything was working fine. Until there was an expected load on our cluster, 
the machine started to swap and become unresponsive. We fixed the unexpected 
load and the three new machines were restarted. After that the new cassandra 
machines were stating that some old token were not assigned, namely from data-2 
and data-4. To fix this I issued again some "remove token" commands.

Everything seems to be back to normal, but on the network I still see some 
packet from the new cluster to the old machines. On the port 7000.
How I can tell cassandra to completely forget about the old machines ?

Nicolas

RE Dead node still being pinged

[Samuel CARRIERE]

Hi Nicolas,

What does nodetool ring says ? (Ask every node)
Have you checked that the list of seeds in every yaml is correct ?
What version of cassandra are you using ?

Samuel




Nicolas Lalevée  
08/06/2012 14:10
Veuillez répondre à
user@cassandra.apache.org


A
user@cassandra.apache.org
cc

Objet
Dead node still being pinged






I had a configuration where I had 4 nodes, data-1,4. We then bought 3 
bigger machines, data-5,7. And we moved all data from data-1,4 to 
data-5,7.
To move all the data without interruption of service, I added one new node 
at a time. And then I removed one by one the old machines via a "remove 
token".

Everything was working fine. Until there was an expected load on our 
cluster, the machine started to swap and become unresponsive. We fixed the 
unexpected load and the three new machines were restarted. After that the 
new cassandra machines were stating that some old token were not assigned, 
namely from data-2 and data-4. To fix this I issued again some "remove 
token" commands.

Everything seems to be back to normal, but on the network I still see some 
packet from the new cluster to the old machines. On the port 7000.
How I can tell cassandra to completely forget about the old machines ?

Nicolas

Re: Nodes not picking up data on repair, disk loaded unevenly

[Luke Hospadaruk]

Follow-up:
After adding the EBS nodes, I successfully compacted, the node that had ~1.3T 
is now down to about 400/500GB (some of that is compression savings).  You're 
right about the load – lots of overwrites.

I'm going to get things back off the EBS and add a couple more nodes (I've got 
4 right now, maybe move up to 6 or 8 for the time being.

I also plan on copying all my CFs to new ones to un-do the major compaction.  
I've got some fairly minor schema changes in mind, so it's a good time to copy 
over my data anyways.

Thanks for all the help, it's been very informative

Luke

From: aaron morton mailto:aa...@thelastpickle.com>>
Reply-To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Subject: Re: Nodes not picking up data on repair, disk loaded unevenly

 I am now running major compactions on those nodes (and all is well so far).
Major compaction in this situation will make things worse. When end up with one 
big file you will need that much space again to compact / upgrade / re-write it.

back down to a normal size, can I move all the data back off the ebs volumes?
something along the lines of:
Yup.

Then add some more nodes to the cluster to keep this from happening in the 
future.
Yerp. Get everything settled and repair running it should be a simple operation.

I assume all the files stored in any of the data directories are all uniquely 
named and cassandra won't really care where they are as long as everything it 
wants is in it's data directories.
Unique on each node.

So it looks like I never got the tree from node #2 (the node which has 
particularly out of control disk usage).
If you look at the logs for 2. you will probably find an error.
Or it may still be running, check nodetool compactionstats

-Is there any way to force replay of hints to empty this out – just a full 
cluster restart when everything is working again maybe?
Normally I would say stop the nodes and delete the hints CF's. As you have 
deleted CF's from one of the nodes there is a risk of losing data though.

If you have been working at CL QUORUM and have not been getting 
TimedOutException you can still delete the hints. As the writes they contain 
should be on at least one other node and they will be repaired by repair.

 I have a high replication factor and all my writes have been at cl=ONE (so all 
the data in the hints should actually exist in a CF somewhere right?).
There is a chance that a write was only applied locally on the node that you 
delete the data from, and it recorded hints to send to the othe nodes. It's a 
remote chance but still there.

 how much working space does this need?  Problem is that node #2 is so full I'm 
not sure any major rebuild or compaction will be susccessful.  The other nodes 
seem to be handiling things ok although they are still heavily loaded.
upgradetables processes one SSTable at a time, it only needs enough space to 
re-write the SSTable.

This is why major compaction hurts in these situations. If you have 1.5T of 
small files, you may have enough free space to re-write all the files. If you 
have a single 1.5T file you don't.

This cluster has a super high write load currently since I'm still building it 
out.  I frequently update every row in my CFs
 Sounds like a lot of overwrites. When you get compaction running it may purge 
a lot of data.


Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 7/06/2012, at 2:51 AM, Luke Hospadaruk wrote:

Thanks for the tips

Some things I found looking around:

grepping the logs for a specific repair I ran yesterday:

/var/log/cassandra# grep df14e460-af48-11e1--e9014560c7bd system.log
INFO [AntiEntropySessions:13] 2012-06-05 19:58:51,303 AntiEntropyService.java 
(line 658) [repair #df14e460-af48-11e1--e9014560c7bd] new session: will 
sync /4.xx.xx.xx, /1.xx.xx.xx, /3.xx.xx.xx, /2.xx.xx.xx on range 
(85070591730234615865843651857942052864,127605887595351923798765477786913079296]
 for content.[article2]
INFO [AntiEntropySessions:13] 2012-06-05 19:58:51,304 AntiEntropyService.java 
(line 837) [repair #df14e460-af48-11e1--e9014560c7bd] requests for merkle 
tree sent for article2 (to [ /4.xx.xx.xx, /1.xx.xx.xx, /3.xx.xx.xx, 
/2.xx.xx.xx])
INFO [AntiEntropyStage:1] 2012-06-05 20:07:01,169 AntiEntropyService.java (line 
190) [repair #df14e460-af48-11e1--e9014560c7bd] Received merkle tree for 
article2 from /4.xx.xx.xx
INFO [AntiEntropyStage:1] 2012-06-06 04:12:30,633 AntiEntropyService.java (line 
190) [repair #df14e460-af48-11e1--e9014560c7bd] Received merkle tree for 
article2 from /3.xx.xx.xx
INFO [AntiEntropyStage:1] 2012-06-06 07:02:51,497 AntiEntropyService.java (line 
190) [repair #df14e460-af48-11e1--e9014560c7bd] Received merkle tree for 
article2 from /1.xx.xx.xx

So it looks like I never got the tree from nod

Re: Problem in adding a new node

[Sylvain Lebresne]

Do you use a 32 bit JVM ? If so I refer you to the following thread:
http://mail-archives.apache.org/mod_mbox/cassandra-user/201204.mbox/%3ccaldd-zgthksc2bikp3h4trjxo5vcnhkl2wpwclsf+d9sqty...@mail.gmail.com%3E

In short, avoids 32 bits, but if you really cannot, set
commitlog_total_space_in_mb to a low value (128-256MB)

--
Sylvain

On Fri, Jun 8, 2012 at 9:14 AM, Prakrati Agrawal
 wrote:
> Dear all,
>
>
>
> I had a 1 node cluster of Cassandra. Then I added one more node to it and
> started Cassandra on it. I got the following error:
>
>
>
> INFO 12:44:49,588 Loading persisted ring state
>
> ERROR 12:44:49,613 Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
>
> java.io.IOError: java.io.IOException: Map failed
>
>     at
> org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:127)
>
>     at
> org.apache.cassandra.db.commitlog.CommitLogAllocator$3.run(CommitLogAllocator.java:191)
>
>     at
> org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
>
>     at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>
>     at java.lang.Thread.run(Thread.java:662)
>
> Caused by: java.io.IOException: Map failed
>
>     at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
>
>     at
> org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:119)
>
>     ... 4 more
>
> Caused by: java.lang.OutOfMemoryError: Map failed
>
>     at sun.nio.ch.FileChannelImpl.map0(Native Method)
>
>     at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)
>
>
>
> Please tell me what is the reason for this error and how should I rectify
> it.
>
>
>
> Thanks and Regards
>
> Prakrati
>
>
>
>
> 
> This email message may contain proprietary, private and confidential
> information. The information transmitted is intended only for the person(s)
> or entities to which it is addressed. Any review, retransmission,
> dissemination or other use of, or taking of any action in reliance upon,
> this information by persons or entities other than the intended recipient is
> prohibited and may be illegal. If you received this in error, please contact
> the sender and delete the message from your system.
>
> Mu Sigma takes all reasonable steps to ensure that its electronic
> communications are free from viruses. However, given Internet accessibility,
> the Company cannot accept liability for any virus introduced by this e-mail
> or any attachment and you are advised to use up-to-date virus checking
> software.

Re: Cassandra 1.1.1 stack overflow on an infinite loop building IntervalTree

[Sylvain Lebresne]

Looks a lot like https://issues.apache.org/jira/browse/CASSANDRA-4321.
Feel free to add a comment on there if you have any additional info.

--
Sylvain

On Fri, Jun 8, 2012 at 12:06 PM, Omid Aladini  wrote:
> Also looks similar to this ticket:
>
> https://issues.apache.org/jira/browse/CASSANDRA-4078
>
>
>
> On Thu, Jun 7, 2012 at 6:48 PM, Omid Aladini  wrote:
>>
>> Hi,
>>
>> One of my 1.1.1 nodes doesn't restart due to stack overflow on building
>> the interval tree. Bumping the stack size doesn't help. Here's the stack
>> trace:
>>
>> https://gist.github.com/2889611
>>
>> It looks more like an infinite loop on IntervalNode constructor's logic
>> than a deep tree since DEBUG log shows looping over the same intervals:
>>
>> https://gist.github.com/2889862
>>
>> Running it with assertions enabled shows a number of sstables which the
>> first key > last key, for example:
>>
>> 2012-06-07_16:12:18.18781 java.lang.AssertionError: SSTable first key
>> DecoratedKey(2254009252149354268486114339861094,
>> 3730343137317c3438333632333932) > last key
>> DecoratedKey(22166106697727078019854024428005234814,
>> 313138323637397c3432373931353435)
>>
>> and let's the node come up without hitting IntervalNode constructor. I
>> wonder how invalid sstables get create in the first place? Is there a way to
>> verify if other nodes in the cluster are affected as well?
>>
>> Speaking of a solution to get the node back up without wiping the data off
>> and let it bootstrap again, I was wondering if I remove affected sstables
>> and restart the node followed by a repair, will the node end up in a
>> consistent state?
>>
>> SStables contain counter columns and leveled compaction is used.
>>
>> Thanks,
>> Omid
>
>

Re: Cassandra 1.1.1 Fails to Start

[Javier Sotelo]

Different node same hardware now gets the stack overflow error but I found
the part of the stack trace that is more interesting:


at com.google.common.collect.Iterators$5.hasNext(Iterators.java:517)
at com.google.common.collect.Iterators$3.hasNext(Iterators.java:114)
at com.google.common.collect.Iterators$5.hasNext(Iterators.java:517)
at com.google.common.collect.Iterators$3.hasNext(Iterators.java:114)
at
com.google.common.collect.Iterators$7.computeNext(Iterators.java:614)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at com.google.common.collect.Iterators.size(Iterators.java:129)
at com.google.common.collect.Sets$3.size(Sets.java:670)
at com.google.common.collect.Iterables.size(Iterables.java:80)
at
org.apache.cassandra.db.DataTracker.buildIntervalTree(DataTracker.java:557)
at
org.apache.cassandra.db.compaction.CompactionController.(CompactionController.java:79)
at
org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:105)
at
org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:50)

Is it time for a JIRA ticket?


On Thu, Jun 7, 2012 at 7:03 AM, Javier Sotelo wrote:

> nodetool ring showed 34.89GB load. Upgrading from 1.1.0. One small
> keyspace with no compression, about 250MB. The rest taken by the second
> keyspace with leveled compaction and snappy compressed.
>
> The blade is an Intel(R) Xeon(R) CPU E5620 @ 2.40GHz with 6GB of RAM.
>
>
> On Thu, Jun 7, 2012 at 2:52 AM, aaron morton wrote:
>
>> How much data do you have on the node ?
>> Was this a previously running system that was upgraded ?
>>
>> > with disk_access_mode mmap_index_only and mmap I see OOM map failed
>> error on SSTableBatchOpen thread
>> Do you have the stack trace from the log ?
>>
>> > ERROR [CompactionExecutor:6] 2012-06-06 20:24:19,772
>> AbstractCassandraDaemon.java (line 134) Exception in thread
>> Thread[CompactionExecutor:6,1,main]
>> > java.lang.StackOverflowError
>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578)
>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578)
>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578)
>> Was there more to this stack trace ?
>> What were the log messages before this error ?
>>
>>
>> >  INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java (line
>> 122) Heap size: 1525415936/1525415936
>> The JVM only has 1.5 G of ram, this is at the lower limit. If you have
>> some data to load I would not be surprised if it failed to start.
>>
>> Cheers
>>
>> -
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 7/06/2012, at 8:41 AM, Javier Sotelo wrote:
>>
>> > Hi All,
>> >
>> > On SuSe Linux blade with 6GB of RAM.
>> >
>> > with disk_access_mode mmap_index_only and mmap I see OOM map failed
>> error on SSTableBatchOpen thread. cat /proc//maps shows a peak of
>> 53521 right before it dies. vm.max_map_count = 1966080 and
>> /proc//limits shows unlimited locked memory.
>> >
>> > with disk_access_mode standard, the node does start up but I see the
>> repeated error:
>> > ERROR [CompactionExecutor:6] 2012-06-06 20:24:19,772
>> AbstractCassandraDaemon.java (line 134) Exception in thread
>> Thread[CompactionExecutor:6,1,main]
>> > java.lang.StackOverflowError
>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578)
>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578)
>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578)
>> > ...
>> >
>> > I'm not sure the second error is related to the first. I prefer to run
>> with full mmap but I have run out of ideas. Is there anything else I can do
>> to debug this?
>> >
>> > Here's startup settings from debug log:
>> >  INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java (line
>> 121) JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_31
>> >  INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java (line
>> 122) Heap size: 1525415936/1525415936
>> >  ...
>> >  INFO [main] 2012-06-06 20:17:10,946 CLibrary.java (line 111) JNA
>> mlockall successful
>> >  ...
>> >  INFO [main] 2012-06-06 20:17:11,055 DatabaseDescriptor.java (line 191)
>> DiskAccessMode is standard, indexAccessMode is standard
>> >  INFO [main] 2012-06-06 20:17:11,213 DatabaseDescriptor.java (line 247)
>> Global memtable threshold is enabled at 484MB
>> >  INFO [main] 2012-06-06 20:17:11,499 CacheService.java (line 96)
>> Initializing key cache with capacity of 72 MBs.
>> >  INFO [main] 2012-06-06 20:17:11,509 CacheService.java (line 107)
>> Scheduling key cache save to each 14400 seconds (going to save all keys).
>> >  INFO [main] 2012-06-06 20:17:11,510 CacheService.java (line 121)
>> Initializing 

Re: Cassandra 1.1.1 crash during compaction

[Sylvain Lebresne]

A JVM crash means that it's unlikely linked to Cassandra itself. We do
use a few unsafe calls here and there, but nothing as changed on that
front for 1.1.1 and we never had any report of this crashing the JVM.
So imo there is a very good chance that this is a JVM bug. In
particular, I doubt many people have tested the Diablo JDK with
Cassandra and I would suggest preferring the sun JDK (the most tested)
or openJDK. I suppose there is also the possibility that snappy has a
bug on FreeBSD.

--
Sylvain


On Thu, Jun 7, 2012 at 7:51 PM, Dustin Wenz  wrote:
> We observed a JRE crash on one node in a seven node cluster about a half hour 
> after upgrading to version 1.1.1 yesterday. Immediately after the upgrade, 
> everything seemed to be working fine. The last item in the cassandra log was 
> a info-level notification that compaction had started on a data file. Four 
> minutes later, the process crashed.
>
> The host OS is FreeBSD 8.2, built for the amd64 architecture. Most of the 
> cluster settings are left to their defaults and the replication factor is set 
> to 2 for our keyspace. We are using the RandomPartitioner and 
> RackInferringSnitch. JNA is enabled, but cannot use mlockall since the 
> process runs as a non-privileged user. It was also necessary to build our own 
> Snappy compressor jar file, since the required architecture was not built-in 
> to the public distribution.
>
> Cassandra is a fairly new software deployment for us, and I was hoping 
> someone could give me some pointers on interpreting the crash report below.
>
> Thanks,
>
>        - .Dustin
>
> #
> # An unexpected error has been detected by Java Runtime Environment:
> #
> #  SIGBUS (0xa) at pc=0x000801199140, pid=44897, tid=0x8d1fdc80
> #
> # Java VM: Diablo Java HotSpot(TM) 64-Bit Server VM (10.0-b23 mixed mode 
> bsd-amd64)
> # Problematic frame:
> # V  [libjvm.so+0x599140]
> #
> # Please submit bug reports to freebsd-j...@freebsd.org
> #
>
> ---  T H R E A D  ---
>
> Current thread (0x000aa9d56000):  JavaThread "CompactionExecutor:30" 
> daemon [_thread_in_vm, id=-1927291776, 
> stack(0x7898e000,0x78a8e000)]
>
> siginfo:si_signo=SIGBUS: si_errno=0, si_code=3 (BUS_OBJERR), 
> si_addr=0x000801199140
>
> Registers:
> RAX=0x000aa95f8fe8, RBX=0x000aa95fc2c0, RCX=0x000987a334f0, 
> RDX=0x0009274e9888
> RSP=0x78a8d630, RBP=0x78a8d640, RSI=0x0009274e9888, 
> RDI=0xc90009274e99
> R8 =0x00098c203bd8, R9 =0x0008809ff4b8, R10=0x000801488580, 
> R11=0x0001
> R12=0x000aa9d56000, R13=0x000aa95f8c00, R14=0x78a8d818, 
> R15=0x000aa95f8c10
> RIP=0x000801199140, EFL=0x003b003b0001, ERR=0x
>  TRAPNO=0x001b00130009
>
> Top of Stack: (sp=0x78a8d630)
> 0x78a8d630:   000aa95fc2c0 000aa9d56000
> 0x78a8d640:   78a8d660 00080119917e
> 0x78a8d650:   00080345b690 000aa95fc2c0
> 0x78a8d660:   78a8d6a0 000800f3a1fd
> 0x78a8d670:   000aa95f8fe8 0009274e9869
> 0x78a8d680:   000986f7acb0 000986f7e591
> 0x78a8d690:   78a8d818 000aa9d56000
> 0x78a8d6a0:   78a8d700 00080346556f
> 0x78a8d6b0:   0009274e9888 00080346553b
> 0x78a8d6c0:   78a8d6c0 000986f7e591
> 0x78a8d6d0:   78a8d818 000987a30198
> 0x78a8d6e0:   000987a334f0 000986f7e918
> 0x78a8d6f0:    78a8d810
> 0x78a8d700:   78a8d870 00080345c04e
> 0x78a8d710:    
> 0x78a8d720:    
> 0x78a8d730:    
> 0x78a8d740:   0009274aa428 00080817df80
> 0x78a8d750:   0008081714c0 
> 0x78a8d760:   00080817df30 
> 0x78a8d770:   00091b5ed0a0 
> 0x78a8d780:    
> 0x78a8d790:    
> 0x78a8d7a0:    
> 0x78a8d7b0:    
> 0x78a8d7c0:    
> 0x78a8d7d0:    0017974b
> 0x78a8d7e0:   deaddeaf 0137c3fdd558
> 0x78a8d7f0:   deaddeaf 
> 0x78a8d800:    00091b638910
> 0x78a8d810:    
> 0x78a8d820:   000882d0d2e8 0008406a3de8
>
> Instructions: (pc=0x000801199140)
> 0x000801199130:   55 48 89 e5 48 89 5d f0 4c 89 65 f8 48 83 ec 10
> 0x000801199140:   0f b7 47 10 48 89 fb 44 8d 60 01 49 63 fc e8 4d
>
> Stack: [0x7898e000,0x78a8e000],  sp=0x78a8d630,  free 
> space=1021k
> Native frames: (J=compiled Java c

Re: Cassandra 1.1.1 Fails to Start

[Javier Sotelo]

Looks like someone beat me to it,
https://issues.apache.org/jira/browse/CASSANDRA-4321

On Fri, Jun 8, 2012 at 9:06 AM, Javier Sotelo wrote:

> Different node same hardware now gets the stack overflow error but I found
> the part of the stack trace that is more interesting:
>
>
> at
> com.google.common.collect.Iterators$5.hasNext(Iterators.java:517)
> at
> com.google.common.collect.Iterators$3.hasNext(Iterators.java:114)
> at
> com.google.common.collect.Iterators$5.hasNext(Iterators.java:517)
> at
> com.google.common.collect.Iterators$3.hasNext(Iterators.java:114)
> at
> com.google.common.collect.Iterators$7.computeNext(Iterators.java:614)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> at com.google.common.collect.Iterators.size(Iterators.java:129)
> at com.google.common.collect.Sets$3.size(Sets.java:670)
> at com.google.common.collect.Iterables.size(Iterables.java:80)
> at
> org.apache.cassandra.db.DataTracker.buildIntervalTree(DataTracker.java:557)
> at
> org.apache.cassandra.db.compaction.CompactionController.(CompactionController.java:79)
> at
> org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:105)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:50)
>
> Is it time for a JIRA ticket?
>
>
> On Thu, Jun 7, 2012 at 7:03 AM, Javier Sotelo 
> wrote:
>
>> nodetool ring showed 34.89GB load. Upgrading from 1.1.0. One small
>> keyspace with no compression, about 250MB. The rest taken by the second
>> keyspace with leveled compaction and snappy compressed.
>>
>> The blade is an Intel(R) Xeon(R) CPU E5620 @ 2.40GHz with 6GB of RAM.
>>
>>
>> On Thu, Jun 7, 2012 at 2:52 AM, aaron morton wrote:
>>
>>> How much data do you have on the node ?
>>> Was this a previously running system that was upgraded ?
>>>
>>> > with disk_access_mode mmap_index_only and mmap I see OOM map failed
>>> error on SSTableBatchOpen thread
>>> Do you have the stack trace from the log ?
>>>
>>> > ERROR [CompactionExecutor:6] 2012-06-06 20:24:19,772
>>> AbstractCassandraDaemon.java (line 134) Exception in thread
>>> Thread[CompactionExecutor:6,1,main]
>>> > java.lang.StackOverflowError
>>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578)
>>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578)
>>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578)
>>> Was there more to this stack trace ?
>>> What were the log messages before this error ?
>>>
>>>
>>> >  INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java
>>> (line 122) Heap size: 1525415936/1525415936
>>> The JVM only has 1.5 G of ram, this is at the lower limit. If you have
>>> some data to load I would not be surprised if it failed to start.
>>>
>>> Cheers
>>>
>>> -
>>> Aaron Morton
>>> Freelance Developer
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>>
>>> On 7/06/2012, at 8:41 AM, Javier Sotelo wrote:
>>>
>>> > Hi All,
>>> >
>>> > On SuSe Linux blade with 6GB of RAM.
>>> >
>>> > with disk_access_mode mmap_index_only and mmap I see OOM map failed
>>> error on SSTableBatchOpen thread. cat /proc//maps shows a peak of
>>> 53521 right before it dies. vm.max_map_count = 1966080 and
>>> /proc//limits shows unlimited locked memory.
>>> >
>>> > with disk_access_mode standard, the node does start up but I see the
>>> repeated error:
>>> > ERROR [CompactionExecutor:6] 2012-06-06 20:24:19,772
>>> AbstractCassandraDaemon.java (line 134) Exception in thread
>>> Thread[CompactionExecutor:6,1,main]
>>> > java.lang.StackOverflowError
>>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578)
>>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578)
>>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578)
>>> > ...
>>> >
>>> > I'm not sure the second error is related to the first. I prefer to run
>>> with full mmap but I have run out of ideas. Is there anything else I can do
>>> to debug this?
>>> >
>>> > Here's startup settings from debug log:
>>> >  INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java
>>> (line 121) JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_31
>>> >  INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java
>>> (line 122) Heap size: 1525415936/1525415936
>>> >  ...
>>> >  INFO [main] 2012-06-06 20:17:10,946 CLibrary.java (line 111) JNA
>>> mlockall successful
>>> >  ...
>>> >  INFO [main] 2012-06-06 20:17:11,055 DatabaseDescriptor.java (line
>>> 191) DiskAccessMode is standard, indexAccessMode is standard
>>> >  INFO [main] 2012-06-06 20:17:11,213 DatabaseDescriptor.java (line
>>> 247) Global memtable threshold is enabled at 484MB
>>> >  INFO [main] 2012-06-06 20:17:11,499 CacheService.java (line 9

Re: Problem joining new node to cluster in 1.1.1

[Sylvain Lebresne]

That very much look like a bug. Would you mind opening a ticket on
https://issues.apache.org/jira/browse/CASSANDRA with those stack
traces and maybe a little bit more precision on what you were doing
when that happened?

--
Sylvain

On Fri, Jun 8, 2012 at 12:28 AM, Bryce Godfrey
 wrote:
> As the new node starts up I get this error before boostrap starts:
>
>
>
> INFO 08:20:51,584 Enqueuing flush of Memtable-schema_columns@1493418651(0/0
> serialized/live bytes, 1 ops)
>
> INFO 08:20:51,584 Writing Memtable-schema_columns@1493418651(0/0
> serialized/live bytes, 1 ops)
>
> INFO 08:20:51,589 Completed flushing
> /opt/cassandra/data/system/schema_columns/system-schema_columns-hc-1-Data.db
> (61 bytes)
>
> ERROR 08:20:51,889 Exception in thread Thread[MigrationStage:1,5,main]
>
> java.lang.IllegalArgumentException: value already present: 1015
>
>     at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:115)
>
>     at
> com.google.common.collect.AbstractBiMap.putInBothMaps(AbstractBiMap.java:111)
>
>     at
> com.google.common.collect.AbstractBiMap.put(AbstractBiMap.java:96)
>
>     at com.google.common.collect.HashBiMap.put(HashBiMap.java:84)
>
>     at org.apache.cassandra.config.Schema.load(Schema.java:385)
>
>     at
> org.apache.cassandra.db.DefsTable.addColumnFamily(DefsTable.java:426)
>
>     at
> org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:361)
>
>     at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
>
>     at
> org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:248)
>
>     at
> org.apache.cassandra.service.MigrationManager$MigrationTask.runMayThrow(MigrationManager.java:416)
>
>     at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>
>     at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> Source)
>
>     at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
>
>     at java.util.concurrent.FutureTask.run(Unknown Source)
>
>     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source)
>
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
>
>     at java.lang.Thread.run(Unknown Source)
>
> INFO 08:20:51,931 Enqueuing flush of
> Memtable-schema_keyspaces@833041663(943/1178 serialized/live bytes, 20 ops)
>
> INFO 08:20:51,932 Writing Memtable-schema_keyspaces@833041663(943/1178
> serialized/live bytes, 20 ops)
>
>
>
>
>
> Then it starts spewing these errors nonstop until I kill it.
>
>
>
> ERROR 08:21:45,959 Error in row mutation
>
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=1019
>
>     at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
>
>     at
> org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
>
>     at
> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
>
>     at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>
>     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source)
>
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
>
>     at java.lang.Thread.run(Unknown Source)
>
> ERROR 08:21:45,814 Error in row mutation
>
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=1019
>
>     at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
>
>     at
> org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
>
>     at
> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
>
>     at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>
>     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source)
>
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
>
>     at java.lang.Thread.run(Unknown Source)
>
> ERROR 08:21:45,813 Error in row mutation
>
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=1020
>
>     at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
>
>     at
> org.apache.cassandra.db

Re: Dead node still being pinged

[Nicolas Lalevée]

Le 8 juin 2012 à 15:17, Samuel CARRIERE a écrit :

> What does nodetool ring says ? (Ask every node) 

currently, each of new node see only the tokens of the new nodes.

> Have you checked that the list of seeds in every yaml is correct ? 

yes, it is correct, every of my new node point to the first of my new node

> What version of cassandra are you using ?

Sorry I should have wrote this in my first mail.
I use the 1.0.9

Nicolas

> 
> Samuel 
> 
> 
> 
> Nicolas Lalevée 
> 08/06/2012 14:10
> Veuillez répondre à
> user@cassandra.apache.org
> 
> A
> user@cassandra.apache.org
> cc
> Objet
> Dead node still being pinged
> 
> 
> 
> 
> 
> I had a configuration where I had 4 nodes, data-1,4. We then bought 3 bigger 
> machines, data-5,7. And we moved all data from data-1,4 to data-5,7.
> To move all the data without interruption of service, I added one new node at 
> a time. And then I removed one by one the old machines via a "remove token".
> 
> Everything was working fine. Until there was an expected load on our cluster, 
> the machine started to swap and become unresponsive. We fixed the unexpected 
> load and the three new machines were restarted. After that the new cassandra 
> machines were stating that some old token were not assigned, namely from 
> data-2 and data-4. To fix this I issued again some "remove token" commands.
> 
> Everything seems to be back to normal, but on the network I still see some 
> packet from the new cluster to the old machines. On the port 7000.
> How I can tell cassandra to completely forget about the old machines ?
> 
> Nicolas
> 
> 

Re: Problem joining new node to cluster in 1.1.1

[Brandon Williams]

This sounds related to https://issues.apache.org/jira/browse/CASSANDRA-4251

On Thu, Jun 7, 2012 at 5:28 PM, Bryce Godfrey  wrote:
> As the new node starts up I get this error before boostrap starts:
>
>
>
> INFO 08:20:51,584 Enqueuing flush of Memtable-schema_columns@1493418651(0/0
> serialized/live bytes, 1 ops)
>
> INFO 08:20:51,584 Writing Memtable-schema_columns@1493418651(0/0
> serialized/live bytes, 1 ops)
>
> INFO 08:20:51,589 Completed flushing
> /opt/cassandra/data/system/schema_columns/system-schema_columns-hc-1-Data.db
> (61 bytes)
>
> ERROR 08:20:51,889 Exception in thread Thread[MigrationStage:1,5,main]
>
> java.lang.IllegalArgumentException: value already present: 1015
>
>     at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:115)
>
>     at
> com.google.common.collect.AbstractBiMap.putInBothMaps(AbstractBiMap.java:111)
>
>     at
> com.google.common.collect.AbstractBiMap.put(AbstractBiMap.java:96)
>
>     at com.google.common.collect.HashBiMap.put(HashBiMap.java:84)
>
>     at org.apache.cassandra.config.Schema.load(Schema.java:385)
>
>     at
> org.apache.cassandra.db.DefsTable.addColumnFamily(DefsTable.java:426)
>
>     at
> org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:361)
>
>     at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
>
>     at
> org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:248)
>
>     at
> org.apache.cassandra.service.MigrationManager$MigrationTask.runMayThrow(MigrationManager.java:416)
>
>     at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>
>     at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> Source)
>
>     at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
>
>     at java.util.concurrent.FutureTask.run(Unknown Source)
>
>     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source)
>
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
>
>     at java.lang.Thread.run(Unknown Source)
>
> INFO 08:20:51,931 Enqueuing flush of
> Memtable-schema_keyspaces@833041663(943/1178 serialized/live bytes, 20 ops)
>
> INFO 08:20:51,932 Writing Memtable-schema_keyspaces@833041663(943/1178
> serialized/live bytes, 20 ops)
>
>
>
>
>
> Then it starts spewing these errors nonstop until I kill it.
>
>
>
> ERROR 08:21:45,959 Error in row mutation
>
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=1019
>
>     at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
>
>     at
> org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
>
>     at
> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
>
>     at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>
>     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source)
>
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
>
>     at java.lang.Thread.run(Unknown Source)
>
> ERROR 08:21:45,814 Error in row mutation
>
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=1019
>
>     at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
>
>     at
> org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
>
>     at
> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
>
>     at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>
>     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source)
>
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
>
>     at java.lang.Thread.run(Unknown Source)
>
> ERROR 08:21:45,813 Error in row mutation
>
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=1020
>
>     at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
>
>     at
> org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
>
>     at
> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
>
>  

Re: Dead node still being pinged

[Samuel CARRIERE]

I'm in the train but just a guess : maybe it's hinted handoff. A look in the 
logs of the new nodes could confirm that : look for the IP of an old node and 
maybe you'll find hinted handoff related messages.


- Message d'origine -
De : Nicolas Lalevée [nicolas.lale...@hibnet.org]
Envoyé : 08/06/2012 19:26 ZE2
À : user@cassandra.apache.org
Objet : Re: Dead node still being pinged



Le 8 juin 2012 à 15:17, Samuel CARRIERE a écrit :

> What does nodetool ring says ? (Ask every node)

currently, each of new node see only the tokens of the new nodes.

> Have you checked that the list of seeds in every yaml is correct ?

yes, it is correct, every of my new node point to the first of my new node

> What version of cassandra are you using ?

Sorry I should have wrote this in my first mail.
I use the 1.0.9

Nicolas

>
> Samuel
>
>
>
> Nicolas Lalevée 
> 08/06/2012 14:10
> Veuillez répondre à
> user@cassandra.apache.org
>
> A
> user@cassandra.apache.org
> cc
> Objet
> Dead node still being pinged
>
>
>
>
>
> I had a configuration where I had 4 nodes, data-1,4. We then bought 3 bigger 
> machines, data-5,7. And we moved all data from data-1,4 to data-5,7.
> To move all the data without interruption of service, I added one new node at 
> a time. And then I removed one by one the old machines via a "remove token".
>
> Everything was working fine. Until there was an expected load on our cluster, 
> the machine started to swap and become unresponsive. We fixed the unexpected 
> load and the three new machines were restarted. After that the new cassandra 
> machines were stating that some old token were not assigned, namely from 
> data-2 and data-4. To fix this I issued again some "remove token" commands.
>
> Everything seems to be back to normal, but on the network I still see some 
> packet from the new cluster to the old machines. On the port 7000.
> How I can tell cassandra to completely forget about the old machines ?
>
> Nicolas
>
>

Re: memtable_flush_queue_size and memtable_flush_writers

[aaron morton]

Single queue for all writers. 

Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 8/06/2012, at 12:49 AM, rohit bhatia wrote:

> Hi
> 
> I can't find this in any documentation online, so just wanted to ask
> 
> Do all flush writers share the same flush queue or do they maintain
> their separate queues..
> 
> Thanks
> Rohit

RE: Problem joining new node to cluster in 1.1.1

[Bryce Godfrey]

https://issues.apache.org/jira/browse/CASSANDRA-4323

Not sure if it's a dupe of what Brandon sent (4251), so created the bug anyway.

-Original Message-
From: Sylvain Lebresne [mailto:sylv...@datastax.com] 
Sent: Friday, June 08, 2012 9:08 AM
To: user@cassandra.apache.org
Subject: Re: Problem joining new node to cluster in 1.1.1

That very much look like a bug. Would you mind opening a ticket on 
https://issues.apache.org/jira/browse/CASSANDRA with those stack traces and 
maybe a little bit more precision on what you were doing when that happened?

--
Sylvain

On Fri, Jun 8, 2012 at 12:28 AM, Bryce Godfrey  
wrote:
> As the new node starts up I get this error before boostrap starts:
>
>
>
> INFO 08:20:51,584 Enqueuing flush of 
> Memtable-schema_columns@1493418651(0/0
> serialized/live bytes, 1 ops)
>
> INFO 08:20:51,584 Writing Memtable-schema_columns@1493418651(0/0
> serialized/live bytes, 1 ops)
>
> INFO 08:20:51,589 Completed flushing
> /opt/cassandra/data/system/schema_columns/system-schema_columns-hc-1-D
> ata.db
> (61 bytes)
>
> ERROR 08:20:51,889 Exception in thread Thread[MigrationStage:1,5,main]
>
> java.lang.IllegalArgumentException: value already present: 1015
>
>     at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:
> 115)
>
>     at
> com.google.common.collect.AbstractBiMap.putInBothMaps(AbstractBiMap.ja
> va:111)
>
>     at
> com.google.common.collect.AbstractBiMap.put(AbstractBiMap.java:96)
>
>     at com.google.common.collect.HashBiMap.put(HashBiMap.java:84)
>
>     at org.apache.cassandra.config.Schema.load(Schema.java:385)
>
>     at
> org.apache.cassandra.db.DefsTable.addColumnFamily(DefsTable.java:426)
>
>     at
> org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:3
> 61)
>
>     at 
> org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
>
>     at
> org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:248
> )
>
>     at
> org.apache.cassandra.service.MigrationManager$MigrationTask.runMayThro
> w(MigrationManager.java:416)
>
>     at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30
> )
>
>     at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> Source)
>
>     at java.util.concurrent.FutureTask$Sync.innerRun(Unknown 
> Source)
>
>     at java.util.concurrent.FutureTask.run(Unknown Source)
>
>     at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source)
>
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
>
>     at java.lang.Thread.run(Unknown Source)
>
> INFO 08:20:51,931 Enqueuing flush of
> Memtable-schema_keyspaces@833041663(943/1178 serialized/live bytes, 20 
> ops)
>
> INFO 08:20:51,932 Writing Memtable-schema_keyspaces@833041663(943/1178
> serialized/live bytes, 20 ops)
>
>
>
>
>
> Then it starts spewing these errors nonstop until I kill it.
>
>
>
> ERROR 08:21:45,959 Error in row mutation
>
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=1019
>
>     at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamil
> ySerializer.java:126)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(
> RowMutation.java:439)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(
> RowMutation.java:447)
>
>     at
> org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
>
>     at
> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbH
> andler.java:42)
>
>     at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.j
> ava:59)
>
>     at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source)
>
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
>
>     at java.lang.Thread.run(Unknown Source)
>
> ERROR 08:21:45,814 Error in row mutation
>
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=1019
>
>     at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamil
> ySerializer.java:126)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(
> RowMutation.java:439)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(
> RowMutation.java:447)
>
>     at
> org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
>
>     at
> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbH
> andler.java:42)
>
>     at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.j
> ava:59)
>
>     at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source)
>
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
>
>     at java.lang.Thread.run(Unknown Source)
>
> ERROR 08:21:45,813 Error in row mutation
>
> org.apache.cassandra.db.Unknown

Re: Failing operations & repair

[aaron morton]

> I am making some cassandra presentations in Kyiv and would like to check that 
> I am telling people truth :)
Thanks for spreading the word :)

> 1) Failed (from client-side view) operation may still be applied to cluster

Yes. 
If you fail with UnavailableException it's because from the coordinators view 
of the cluster there is less than CL nodes available. So retry. Somewhat 
similar story with TimedOutException. 

> 2) Coordinator does not try anything to "roll-back" operation that failed 
> because it was processed by less then consitency level number of nodes.

Correct.

> 3) Hinted handoff works only for successfull operations.

HH will be stored if the coordinator proceeds with the request.
In 1.X HH is stored on the coordinator if a replica is down when the request 
starts and if the node does not reply in rpc_timeout. 

> 4) Counters are not reliable because of (1)

If you get a TimedOutException when writing a counter you should not re-send 
the request. 

> 5) Read-repair may help to propagate operation that was failed it's 
> consistency level, but was persisted to some nodes.

Yes. It works in the background, by default is only enabled on 10% of requests. 
Note that RR is not the same as the Consistent Level for read. If you work as a 
CL > ONE the results from CL nodes are always compared and differences 
resolved. RR is concerned with the replicas not involved in the CL read. 

> 6) Manual repair is still needed because of (2) and (3)

Manual repair is *the* was to achieve consistency of data on disk. HH and RR 
are optimisations designed to reduce the chance of a Digest Mismatch during a 
read with CL > ONE. 
It is also essential for distributing Tombstones before they are purged by 
compaction.
> P.S. If some points apply only to some cassandra versions, I will be happy to 
> know this too.

Assume everyone for version 1.X

Thanks

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 8/06/2012, at 1:20 AM, Віталій Тимчишин wrote:

> Hello.
> 
> I am making some cassandra presentations in Kyiv and would like to check that 
> I am telling people truth :)
> Could community tell me if next points are true:
> 1) Failed (from client-side view) operation may still be applied to cluster
> 2) Coordinator does not try anything to "roll-back" operation that failed 
> because it was processed by less then consitency level number of nodes.
> 3) Hinted handoff works only for successfull operations.
> 4) Counters are not reliable because of (1)
> 5) Read-repair may help to propagate operation that was failed it's 
> consistency level, but was persisted to some nodes.
> 6) Manual repair is still needed because of (2) and (3)
> 
> P.S. If some points apply only to some cassandra versions, I will be happy to 
> know this too.
> -- 
> Best regards,
>  Vitalii Tymchyshyn

Re: Data corruption issues with 1.1

[aaron morton]

Was this old data written using an older version of cassandra ? 

In the current versions UTF data is verified. 

Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 8/06/2012, at 7:19 AM, Oleg Dulin wrote:

> I can't quite describe what happened, but essentially one day I found that my 
> column values that are supposed to be UTF-8 strings started getting bogus 
> characters.
> 
> Is there a known data corruption issue with 1.1 ?
> 
> 

Re: Setting column to null

[aaron morton]

You don't nee to set columns to null, delete the column instead.

Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 8/06/2012, at 9:34 AM, Leonid Ilyevsky wrote:

> Is it possible to explicitly set a column value to null?
>  
> I see that if insert statement does not include a specific column, that 
> column comes up as null (assuming we are creating a record with new unique 
> key).
> But if we want to update a record, how we set it to null?
>  
> Another situation is when I use prepared cql3 statement (in Java) and send 
> parameters when I execute it. If I want to leave some column unassigned, I 
> need a special statement without that column.
> What I would like is, prepare one statement including all columns, and then 
> be able to set some of them to null. I tried to set corresponding ByteBuffer 
> parameter to null, obviously got an exception.
> 
> This email, along with any attachments, is confidential and may be legally 
> privileged or otherwise protected from disclosure. Any unauthorized 
> dissemination, copying or use of the contents of this email is strictly 
> prohibited and may be in violation of law. If you are not the intended 
> recipient, any disclosure, copying, forwarding or distribution of this email 
> is strictly prohibited and this email and any attachments should be deleted 
> immediately. This email and any attachments do not constitute an offer to 
> sell or a solicitation of an offer to purchase any interest in any investment 
> vehicle sponsored by Moon Capital Management LP (“Moon Capital”). Moon 
> Capital does not provide legal, accounting or tax advice. Any statement 
> regarding legal, accounting or tax matters was not intended or written to be 
> relied upon by any person as advice. Moon Capital does not waive 
> confidentiality or privilege as a result of this email.

Re: Problem in getting data from a 2 node cluster

[aaron morton]

1

Data Stax has some good documentation here 
http://www.datastax.com/docs/1.1/index

Cheers


-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 8/06/2012, at 2:50 PM, Prakrati Agrawal wrote:

> What is the default replication factor? I did not set any replication factor.
> 
> Prakrati Agrawal | Developer - Big Data(I&D)| 9731648376 | www.mu-sigma.com
> 
> -Original Message-
> From: Tim Wintle [mailto:timwin...@gmail.com]
> Sent: Wednesday, June 06, 2012 5:42 PM
> To: user@cassandra.apache.org
> Subject: RE: Problem in getting data from a 2 node cluster
> 
> On Wed, 2012-06-06 at 06:54 -0500, Prakrati Agrawal wrote:
>> This node will not auto bootstrap because it is configured to be a
>> seed node
> 
> This means the cassandra.yaml on that node references itself as a seed
> node.
> 
> 
> After you decommission the second node, can you still access the entire
> dataset in the single node cluser, or has it been lost along the way?
> 
> What is the replication factor for your data?
> 
> 
> Tim Wintle
> 
> 
> 
> This email message may contain proprietary, private and confidential 
> information. The information transmitted is intended only for the person(s) 
> or entities to which it is addressed. Any review, retransmission, 
> dissemination or other use of, or taking of any action in reliance upon, this 
> information by persons or entities other than the intended recipient is 
> prohibited and may be illegal. If you received this in error, please contact 
> the sender and delete the message from your system.
> 
> Mu Sigma takes all reasonable steps to ensure that its electronic 
> communications are free from viruses. However, given Internet accessibility, 
> the Company cannot accept liability for any virus introduced by this e-mail 
> or any attachment and you are advised to use up-to-date virus checking 
> software.

Re: kswapd0 causing read timeouts

[aaron morton]

Ruslan, 
Why did you suggest changing the disk_access_mode ? 

Gurpreet,
I would leave the disk_access_mode with the default until you have a 
reason to change it. 

> > 8 core, 16 gb ram, 6 data disks raid0, no swap configured
is swap disabled ?

> Gradually,
> > the system cpu becomes high almost 70%, and the client starts getting
> > continuous timeouts

70% of one core or 70% of all cores ?
Check the server logs, is there GC activity ? 
check nodetool cfstats to see the read latency for the cf. 

Take a look at vmstat to see if you are swapping, and look at iostats to see if 
io is the problem 
http://spyced.blogspot.co.nz/2010/01/linux-performance-basics.html

Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 8/06/2012, at 9:00 PM, Gurpreet Singh wrote:

> Thanks Ruslan.
> I will try the mmap_index_only.
> Is there any guideline as to when to leave it to auto and when to use 
> mmap_index_only?
> 
> /G
> 
> On Fri, Jun 8, 2012 at 1:21 AM, ruslan usifov  wrote:
> disk_access_mode: mmap??
> 
> set to disk_access_mode: mmap_index_only in cassandra yaml
> 
> 2012/6/8 Gurpreet Singh :
> > Hi,
> > I am testing cassandra 1.1 on a 1 node cluster.
> > 8 core, 16 gb ram, 6 data disks raid0, no swap configured
> >
> > cassandra 1.1.1
> > heap size: 8 gigs
> > key cache size in mb: 800 (used only 200mb till now)
> > memtable_total_space_in_mb : 2048
> >
> > I am running a read workload.. about 30 reads/second. no writes at all.
> > The system runs fine for roughly 12 hours.
> >
> > jconsole shows that my heap size has hardly touched 4 gigs.
> > top shows -
> >   SHR increasing slowly from 100 mb to 6.6 gigs in  these 12 hrs
> >   RES increases slowly from 6 gigs all the way to 15 gigs
> >   buffers are at a healthy 25 mb at some point and that goes down to 2 mb in
> > these 12 hrs
> >   VIRT stays at 85 gigs
> >
> > I understand that SHR goes up because of mmap, RES goes up because it is
> > showing SHR value as well.
> >
> > After around 10-12 hrs, the cpu utilization of the system starts increasing,
> > and i notice that kswapd0 process starts becoming more active. Gradually,
> > the system cpu becomes high almost 70%, and the client starts getting
> > continuous timeouts. The fact that the buffers went down from 20 mb to 2 mb
> > suggests that kswapd0 is probably swapping out the pagecache.
> >
> > Is there a way out of this to avoid the kswapd0 starting to do things even
> > when there is no swap configured?
> > This is very easily reproducible for me, and would like a way out of this
> > situation. Do i need to adjust vm memory management stuff like pagecache,
> > vfs_cache_pressure.. things like that?
> >
> > just some extra information, jna is installed, mlockall is successful. there
> > is no compaction running.
> > would appreciate any help on this.
> > Thanks
> > Gurpreet
> >
> >
> 

Re: Out of memory error

[aaron morton]

When you ask a question please include the query or function call you have 
made. An any other information that would help someone understand what you are 
trying to do. 

Also, please list things you have already tried to work around the problem. 

Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 8/06/2012, at 9:04 PM, Prakrati Agrawal wrote:

> Dear all,
>  
> When I try to list the entire data in my column family I get the following 
> error:
>  
> Using default limit of 100
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at 
> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:140)
> at 
> org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
> at 
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at 
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
> at 
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
> at 
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
> at 
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> at 
> org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:683)
> at 
> org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:667)
> at 
> org.apache.cassandra.cli.CliClient.executeList(CliClient.java:1373)
> at 
> org.apache.cassandra.cli.CliClient.executeCLIStatement(CliClient.java:264)
> at 
> org.apache.cassandra.cli.CliMain.processStatementInteractive(CliMain.java:219)
> at org.apache.cassandra.cli.CliMain.main(CliMain.java:346)
>  
> Please help me
>  
> Thanks and Regards
> Prakrati
>  
>  
> 
> This email message may contain proprietary, private and confidential 
> information. The information transmitted is intended only for the person(s) 
> or entities to which it is addressed. Any review, retransmission, 
> dissemination or other use of, or taking of any action in reliance upon, this 
> information by persons or entities other than the intended recipient is 
> prohibited and may be illegal. If you received this in error, please contact 
> the sender and delete the message from your system.
> 
> Mu Sigma takes all reasonable steps to ensure that its electronic 
> communications are free from viruses. However, given Internet accessibility, 
> the Company cannot accept liability for any virus introduced by this e-mail 
> or any attachment and you are advised to use up-to-date virus checking 
> software.

Re: Dead node still being pinged

[aaron morton]

Are the old machines listed in the seed list on the new ones ?

Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 9/06/2012, at 12:10 AM, Nicolas Lalevée wrote:

> I had a configuration where I had 4 nodes, data-1,4. We then bought 3 bigger 
> machines, data-5,7. And we moved all data from data-1,4 to data-5,7.
> To move all the data without interruption of service, I added one new node at 
> a time. And then I removed one by one the old machines via a "remove token".
> 
> Everything was working fine. Until there was an expected load on our cluster, 
> the machine started to swap and become unresponsive. We fixed the unexpected 
> load and the three new machines were restarted. After that the new cassandra 
> machines were stating that some old token were not assigned, namely from 
> data-2 and data-4. To fix this I issued again some "remove token" commands.
> 
> Everything seems to be back to normal, but on the network I still see some 
> packet from the new cluster to the old machines. On the port 7000.
> How I can tell cassandra to completely forget about the old machines ?
> 
> Nicolas
> 

Re: Nodes not picking up data on repair, disk loaded unevenly

[aaron morton]

Nice work :)

A

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 9/06/2012, at 1:48 AM, Luke Hospadaruk wrote:

> Follow-up:
> After adding the EBS nodes, I successfully compacted, the node that had ~1.3T 
> is now down to about 400/500GB (some of that is compression savings).  You're 
> right about the load – lots of overwrites.
> 
> I'm going to get things back off the EBS and add a couple more nodes (I've 
> got 4 right now, maybe move up to 6 or 8 for the time being.
> 
> I also plan on copying all my CFs to new ones to un-do the major compaction.  
> I've got some fairly minor schema changes in mind, so it's a good time to 
> copy over my data anyways.
> 
> Thanks for all the help, it's been very informative
> 
> Luke
> 
> From: aaron morton mailto:aa...@thelastpickle.com>>
> Reply-To: "user@cassandra.apache.org" 
> mailto:user@cassandra.apache.org>>
> To: "user@cassandra.apache.org" 
> mailto:user@cassandra.apache.org>>
> Subject: Re: Nodes not picking up data on repair, disk loaded unevenly
> 
> I am now running major compactions on those nodes (and all is well so far).
> Major compaction in this situation will make things worse. When end up with 
> one big file you will need that much space again to compact / upgrade / 
> re-write it.
> 
> back down to a normal size, can I move all the data back off the ebs volumes?
> something along the lines of:
> Yup.
> 
> Then add some more nodes to the cluster to keep this from happening in the 
> future.
> Yerp. Get everything settled and repair running it should be a simple 
> operation.
> 
> I assume all the files stored in any of the data directories are all uniquely 
> named and cassandra won't really care where they are as long as everything it 
> wants is in it's data directories.
> Unique on each node.
> 
> So it looks like I never got the tree from node #2 (the node which has 
> particularly out of control disk usage).
> If you look at the logs for 2. you will probably find an error.
> Or it may still be running, check nodetool compactionstats
> 
> -Is there any way to force replay of hints to empty this out – just a full 
> cluster restart when everything is working again maybe?
> Normally I would say stop the nodes and delete the hints CF's. As you have 
> deleted CF's from one of the nodes there is a risk of losing data though.
> 
> If you have been working at CL QUORUM and have not been getting 
> TimedOutException you can still delete the hints. As the writes they contain 
> should be on at least one other node and they will be repaired by repair.
> 
> I have a high replication factor and all my writes have been at cl=ONE (so 
> all the data in the hints should actually exist in a CF somewhere right?).
> There is a chance that a write was only applied locally on the node that you 
> delete the data from, and it recorded hints to send to the othe nodes. It's a 
> remote chance but still there.
> 
> how much working space does this need?  Problem is that node #2 is so full 
> I'm not sure any major rebuild or compaction will be susccessful.  The other 
> nodes seem to be handiling things ok although they are still heavily loaded.
> upgradetables processes one SSTable at a time, it only needs enough space to 
> re-write the SSTable.
> 
> This is why major compaction hurts in these situations. If you have 1.5T of 
> small files, you may have enough free space to re-write all the files. If you 
> have a single 1.5T file you don't.
> 
> This cluster has a super high write load currently since I'm still building 
> it out.  I frequently update every row in my CFs
> Sounds like a lot of overwrites. When you get compaction running it may purge 
> a lot of data.
> 
> 
> Cheers
> 
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 7/06/2012, at 2:51 AM, Luke Hospadaruk wrote:
> 
> Thanks for the tips
> 
> Some things I found looking around:
> 
> grepping the logs for a specific repair I ran yesterday:
> 
> /var/log/cassandra# grep df14e460-af48-11e1--e9014560c7bd system.log
> INFO [AntiEntropySessions:13] 2012-06-05 19:58:51,303 AntiEntropyService.java 
> (line 658) [repair #df14e460-af48-11e1--e9014560c7bd] new session: will 
> sync /4.xx.xx.xx, /1.xx.xx.xx, /3.xx.xx.xx, /2.xx.xx.xx on range 
> (85070591730234615865843651857942052864,127605887595351923798765477786913079296]
>  for content.[article2]
> INFO [AntiEntropySessions:13] 2012-06-05 19:58:51,304 AntiEntropyService.java 
> (line 837) [repair #df14e460-af48-11e1--e9014560c7bd] requests for merkle 
> tree sent for article2 (to [ /4.xx.xx.xx, /1.xx.xx.xx, /3.xx.xx.xx, 
> /2.xx.xx.xx])
> INFO [AntiEntropyStage:1] 2012-06-05 20:07:01,169 AntiEntropyService.java 
> (line 190) [repair #df14e460-af48-11e1--e9014560c7bd] Received merkle 
> tree for article2 from /4.xx.xx.xx
> INFO [AntiEntropyStage:1] 2012-06-06 04:12

Re: kswapd0 causing read timeouts

[ruslan usifov]

2012/6/8 aaron morton :
> Ruslan,
> Why did you suggest changing the disk_access_mode ?

Because this bring problems on empty seat, in any case for me mmap
bring similar problem and i doesn't have find any solution to resolve
it, only  change disk_access_mode:-((. For me also will be interesting
hear results of author of this theme

>
> Gurpreet,
> I would leave the disk_access_mode with the default until you have a reason
> to change it.
>
>> > 8 core, 16 gb ram, 6 data disks raid0, no swap configured
>
> is swap disabled ?
>
>> Gradually,
>> > the system cpu becomes high almost 70%, and the client starts getting
>> > continuous timeouts
>
> 70% of one core or 70% of all cores ?
> Check the server logs, is there GC activity ?
> check nodetool cfstats to see the read latency for the cf.
>
> Take a look at vmstat to see if you are swapping, and look at iostats to see
> if io is the problem
> http://spyced.blogspot.co.nz/2010/01/linux-performance-basics.html
>
> Cheers
>
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 8/06/2012, at 9:00 PM, Gurpreet Singh wrote:
>
> Thanks Ruslan.
> I will try the mmap_index_only.
> Is there any guideline as to when to leave it to auto and when to use
> mmap_index_only?
>
> /G
>
> On Fri, Jun 8, 2012 at 1:21 AM, ruslan usifov 
> wrote:
>>
>> disk_access_mode: mmap??
>>
>> set to disk_access_mode: mmap_index_only in cassandra yaml
>>
>> 2012/6/8 Gurpreet Singh :
>> > Hi,
>> > I am testing cassandra 1.1 on a 1 node cluster.
>> > 8 core, 16 gb ram, 6 data disks raid0, no swap configured
>> >
>> > cassandra 1.1.1
>> > heap size: 8 gigs
>> > key cache size in mb: 800 (used only 200mb till now)
>> > memtable_total_space_in_mb : 2048
>> >
>> > I am running a read workload.. about 30 reads/second. no writes at all.
>> > The system runs fine for roughly 12 hours.
>> >
>> > jconsole shows that my heap size has hardly touched 4 gigs.
>> > top shows -
>> >   SHR increasing slowly from 100 mb to 6.6 gigs in  these 12 hrs
>> >   RES increases slowly from 6 gigs all the way to 15 gigs
>> >   buffers are at a healthy 25 mb at some point and that goes down to 2
>> > mb in
>> > these 12 hrs
>> >   VIRT stays at 85 gigs
>> >
>> > I understand that SHR goes up because of mmap, RES goes up because it is
>> > showing SHR value as well.
>> >
>> > After around 10-12 hrs, the cpu utilization of the system starts
>> > increasing,
>> > and i notice that kswapd0 process starts becoming more active.
>> > Gradually,
>> > the system cpu becomes high almost 70%, and the client starts getting
>> > continuous timeouts. The fact that the buffers went down from 20 mb to 2
>> > mb
>> > suggests that kswapd0 is probably swapping out the pagecache.
>> >
>> > Is there a way out of this to avoid the kswapd0 starting to do things
>> > even
>> > when there is no swap configured?
>> > This is very easily reproducible for me, and would like a way out of
>> > this
>> > situation. Do i need to adjust vm memory management stuff like
>> > pagecache,
>> > vfs_cache_pressure.. things like that?
>> >
>> > just some extra information, jna is installed, mlockall is successful.
>> > there
>> > is no compaction running.
>> > would appreciate any help on this.
>> > Thanks
>> > Gurpreet
>> >
>> >
>
>
>

Re: Dead node still being pinged

[Nicolas Lalevée]

Le 8 juin 2012 à 20:02, Samuel CARRIERE a écrit :

> I'm in the train but just a guess : maybe it's hinted handoff. A look in the 
> logs of the new nodes could confirm that : look for the IP of an old node and 
> maybe you'll find hinted handoff related messages.

I grepped on every node about every old node, I got nothing since the "crash".

If it can be of some help, here is some grepped log of the crash:

system.log.1: WARN [RMI TCP Connection(1037)-10.10.0.26] 2012-05-06 
00:39:30,241 StorageService.java (line 2417) Endpoint /10.10.0.24 is down and 
will not receive data for re-replication of /10.10.0.22
system.log.1: WARN [RMI TCP Connection(1037)-10.10.0.26] 2012-05-06 
00:39:30,242 StorageService.java (line 2417) Endpoint /10.10.0.24 is down and 
will not receive data for re-replication of /10.10.0.22
system.log.1: WARN [RMI TCP Connection(1037)-10.10.0.26] 2012-05-06 
00:39:30,242 StorageService.java (line 2417) Endpoint /10.10.0.24 is down and 
will not receive data for re-replication of /10.10.0.22
system.log.1: WARN [RMI TCP Connection(1037)-10.10.0.26] 2012-05-06 
00:39:30,243 StorageService.java (line 2417) Endpoint /10.10.0.24 is down and 
will not receive data for re-replication of /10.10.0.22
system.log.1: WARN [RMI TCP Connection(1037)-10.10.0.26] 2012-05-06 
00:39:30,243 StorageService.java (line 2417) Endpoint /10.10.0.24 is down and 
will not receive data for re-replication of /10.10.0.22
system.log.1: INFO [GossipStage:1] 2012-05-06 00:44:33,822 Gossiper.java (line 
818) InetAddress /10.10.0.24 is now dead.
system.log.1: INFO [GossipStage:1] 2012-05-06 04:25:23,894 Gossiper.java (line 
818) InetAddress /10.10.0.24 is now dead.
system.log.1: INFO [OptionalTasks:1] 2012-05-06 04:25:23,895 
HintedHandOffManager.java (line 179) Deleting any stored hints for /10.10.0.24
system.log.1: INFO [GossipStage:1] 2012-05-06 04:25:23,895 StorageService.java 
(line 1157) Removing token 127605887595351923798765477786913079296 for 
/10.10.0.24
system.log.1: INFO [GossipStage:1] 2012-05-09 04:26:25,015 Gossiper.java (line 
818) InetAddress /10.10.0.24 is now dead.


Maybe its the way I have removed nodes ? AFAIR I didn't used the decommission 
command. For each node I got the node down and then issue a remove token 
command.
Here is what I can find in the log about when I removed one of them:

system.log.1: INFO [GossipTasks:1] 2012-05-02 17:21:10,281 Gossiper.java (line 
818) InetAddress /10.10.0.24 is now dead.
system.log.1: INFO [HintedHandoff:1] 2012-05-02 17:21:21,496 
HintedHandOffManager.java (line 292) Endpoint /10.10.0.24 died before hint 
delivery, aborting
system.log.1: INFO [GossipStage:1] 2012-05-02 17:21:59,307 Gossiper.java (line 
818) InetAddress /10.10.0.24 is now dead.
system.log.1: INFO [HintedHandoff:1] 2012-05-02 17:31:20,336 
HintedHandOffManager.java (line 292) Endpoint /10.10.0.24 died before hint 
delivery, aborting
system.log.1: INFO [HintedHandoff:1] 2012-05-02 17:41:06,177 
HintedHandOffManager.java (line 292) Endpoint /10.10.0.24 died before hint 
delivery, aborting
system.log.1: INFO [HintedHandoff:1] 2012-05-02 17:51:18,148 
HintedHandOffManager.java (line 292) Endpoint /10.10.0.24 died before hint 
delivery, aborting
system.log.1: INFO [HintedHandoff:1] 2012-05-02 18:00:31,709 
HintedHandOffManager.java (line 292) Endpoint /10.10.0.24 died before hint 
delivery, aborting
system.log.1: INFO [HintedHandoff:1] 2012-05-02 18:11:02,521 
HintedHandOffManager.java (line 292) Endpoint /10.10.0.24 died before hint 
delivery, aborting
system.log.1: INFO [HintedHandoff:1] 2012-05-02 18:20:38,282 
HintedHandOffManager.java (line 292) Endpoint /10.10.0.24 died before hint 
delivery, aborting
system.log.1: INFO [HintedHandoff:1] 2012-05-02 18:31:09,513 
HintedHandOffManager.java (line 292) Endpoint /10.10.0.24 died before hint 
delivery, aborting
system.log.1: INFO [HintedHandoff:1] 2012-05-02 18:40:31,565 
HintedHandOffManager.java (line 292) Endpoint /10.10.0.24 died before hint 
delivery, aborting
system.log.1: INFO [HintedHandoff:1] 2012-05-02 18:51:10,566 
HintedHandOffManager.java (line 292) Endpoint /10.10.0.24 died before hint 
delivery, aborting
system.log.1: INFO [HintedHandoff:1] 2012-05-02 19:00:32,197 
HintedHandOffManager.java (line 292) Endpoint /10.10.0.24 died before hint 
delivery, aborting
system.log.1: INFO [HintedHandoff:1] 2012-05-02 19:11:17,018 
HintedHandOffManager.java (line 292) Endpoint /10.10.0.24 died before hint 
delivery, aborting
system.log.1: INFO [HintedHandoff:1] 2012-05-02 19:21:21,759 
HintedHandOffManager.java (line 292) Endpoint /10.10.0.24 died before hint 
delivery, aborting
system.log.1: INFO [GossipStage:1] 2012-05-02 20:05:57,281 Gossiper.java (line 
818) InetAddress /10.10.0.24 is now dead.
system.log.1: INFO [OptionalTasks:1] 2012-05-02 20:05:57,281 
HintedHandOffManager.java (line 179) Deleting any stored hints for /10.10.0.24
system.log.1: INFO [GossipStage:1] 2012-05-02 20:05:57,281 StorageService.java 
(line 1157) Removing token 14583

Re: Dead node still being pinged

[Nicolas Lalevée]

Le 8 juin 2012 à 20:50, aaron morton a écrit :

> Are the old machines listed in the seed list on the new ones ?

No they don't.

The first of my old node was, when I was "migrating". But not anymore.

Nicolas


> Cheers
> 
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 9/06/2012, at 12:10 AM, Nicolas Lalevée wrote:
> 
>> I had a configuration where I had 4 nodes, data-1,4. We then bought 3 bigger 
>> machines, data-5,7. And we moved all data from data-1,4 to data-5,7.
>> To move all the data without interruption of service, I added one new node 
>> at a time. And then I removed one by one the old machines via a "remove 
>> token".
>> 
>> Everything was working fine. Until there was an expected load on our 
>> cluster, the machine started to swap and become unresponsive. We fixed the 
>> unexpected load and the three new machines were restarted. After that the 
>> new cassandra machines were stating that some old token were not assigned, 
>> namely from data-2 and data-4. To fix this I issued again some "remove 
>> token" commands.
>> 
>> Everything seems to be back to normal, but on the network I still see some 
>> packet from the new cluster to the old machines. On the port 7000.
>> How I can tell cassandra to completely forget about the old machines ?
>> 
>> Nicolas
>> 
> 

Re: Setting column to null

[shashwat shriparv]

What you can do is you can define some specific variable like "NULLDATA"
some thing like that to update in columns that does have value



On Fri, Jun 8, 2012 at 11:58 PM, aaron morton wrote:

> You don't nee to set columns to null, delete the column instead.
>
> Cheers
>
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 8/06/2012, at 9:34 AM, Leonid Ilyevsky wrote:
>
> Is it possible to explicitly set a column value to null?
> ** **
> I see that if insert statement does not include a specific column, that
> column comes up as null (assuming we are creating a record with new unique
> key).
> But if we want to update a record, how we set it to null?
> ** **
> Another situation is when I use prepared cql3 statement (in Java) and send
> parameters when I execute it. If I want to leave some column unassigned, I
> need a special statement without that column.
> What I would like is, prepare one statement including all columns, and
> then be able to set some of them to null. I tried to set corresponding
> ByteBuffer parameter to null, obviously got an exception.
>
> --
> This email, along with any attachments, is confidential and may be legally
> privileged or otherwise protected from disclosure. Any unauthorized
> dissemination, copying or use of the contents of this email is strictly
> prohibited and may be in violation of law. If you are not the intended
> recipient, any disclosure, copying, forwarding or distribution of this
> email is strictly prohibited and this email and any attachments should be
> deleted immediately. This email and any attachments do not constitute an
> offer to sell or a solicitation of an offer to purchase any interest in any
> investment vehicle sponsored by Moon Capital Management LP (“Moon
> Capital”). Moon Capital does not provide legal, accounting or tax advice.
> Any statement regarding legal, accounting or tax matters was not intended
> or written to be relied upon by any person as advice. Moon Capital does not
> waive confidentiality or privilege as a result of this email.
>
>
>


-- 


∞
Shashwat Shriparv

how to compact an index CF?

[Poziombka, Wade L]

I have an index to a column IX in column family A.

How would I go about compacting that?  I have tried nodetool compact keyspace 
A.IX

But that complains "Unknown table/cf pair"

I'm sure there must be some simple magic to make this happen.  I just cannot 
tell what it is.

Re: Cassandra 1.1.1 stack overflow on an infinite loop building IntervalTree

[Omid Aladini]

Thanks. Yes it's exactly the same. Will follow up there.

-- Omid

On Fri, Jun 8, 2012 at 5:55 PM, Sylvain Lebresne wrote:

> Looks a lot like https://issues.apache.org/jira/browse/CASSANDRA-4321.
> Feel free to add a comment on there if you have any additional info.
>
> --
> Sylvain
>
> On Fri, Jun 8, 2012 at 12:06 PM, Omid Aladini 
> wrote:
> > Also looks similar to this ticket:
> >
> > https://issues.apache.org/jira/browse/CASSANDRA-4078
> >
> >
> >
> > On Thu, Jun 7, 2012 at 6:48 PM, Omid Aladini 
> wrote:
> >>
> >> Hi,
> >>
> >> One of my 1.1.1 nodes doesn't restart due to stack overflow on building
> >> the interval tree. Bumping the stack size doesn't help. Here's the stack
> >> trace:
> >>
> >> https://gist.github.com/2889611
> >>
> >> It looks more like an infinite loop on IntervalNode constructor's logic
> >> than a deep tree since DEBUG log shows looping over the same intervals:
> >>
> >> https://gist.github.com/2889862
> >>
> >> Running it with assertions enabled shows a number of sstables which the
> >> first key > last key, for example:
> >>
> >> 2012-06-07_16:12:18.18781 java.lang.AssertionError: SSTable first key
> >> DecoratedKey(2254009252149354268486114339861094,
> >> 3730343137317c3438333632333932) > last key
> >> DecoratedKey(22166106697727078019854024428005234814,
> >> 313138323637397c3432373931353435)
> >>
> >> and let's the node come up without hitting IntervalNode constructor. I
> >> wonder how invalid sstables get create in the first place? Is there a
> way to
> >> verify if other nodes in the cluster are affected as well?
> >>
> >> Speaking of a solution to get the node back up without wiping the data
> off
> >> and let it bootstrap again, I was wondering if I remove affected
> sstables
> >> and restart the node followed by a repair, will the node end up in a
> >> consistent state?
> >>
> >> SStables contain counter columns and leveled compaction is used.
> >>
> >> Thanks,
> >> Omid
> >
> >
>

RE: how to compact an index CF?

[Poziombka, Wade L]

This is reference to https://issues.apache.org/jira/browse/CASSANDRA-4314

in which Jonathan Ellis instructed me (I think me) to

"If you compact the index CF with this patch applied, that should get rid of 
the tombstones. (compacting the data CF won't do anything.)"

However, after much looking I cannot see a way to actually do this?  Is it 
automatic?

From: Poziombka, Wade L
Sent: Friday, June 08, 2012 2:22 PM
To: 'user@cassandra.apache.org'
Subject: how to compact an index CF?

I have an index to a column IX in column family A.

How would I go about compacting that?  I have tried nodetool compact keyspace 
A.IX

But that complains "Unknown table/cf pair"

I'm sure there must be some simple magic to make this happen.  I just cannot 
tell what it is.

Re: Setting column to null

[Edward Capriolo]

Your best bet is to define the column as a composite column where one part
represents is null and the other part is the data.

On Friday, June 8, 2012, shashwat shriparv 
wrote:
> What you can do is you can define some specific variable like "NULLDATA"
some thing like that to update in columns that does have value
>
>
> On Fri, Jun 8, 2012 at 11:58 PM, aaron morton 
wrote:
>
> You don't nee to set columns to null, delete the column instead.
> Cheers
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
> On 8/06/2012, at 9:34 AM, Leonid Ilyevsky wrote:
>
> Is it possible to explicitly set a column value to null?
>
> I see that if insert statement does not include a specific column, that
column comes up as null (assuming we are creating a record with new unique
key).
> But if we want to update a record, how we set it to null?
>
> Another situation is when I use prepared cql3 statement (in Java) and
send parameters when I execute it. If I want to leave some column
unassigned, I need a special statement without that column.
> What I would like is, prepare one statement including all columns, and
then be able to set some of them to null. I tried to set corresponding
ByteBuffer parameter to null, obviously got an exception.
> 
> This email, along with any attachments, is confidential and may be
legally privileged or otherwise protected from disclosure. Any unauthorized
dissemination, copying or use of the contents of this email is strictly
prohibited and may be in violation of law. If you are not the intended
recipient, any disclosure, copying, forwarding or distribution of this
email is strictly prohibited and this email and any attachments should be
deleted immediately. This email and any attachments do not constitute an
offer to sell or a solicitation of an offer to purchase any interest in any
investment vehicle sponsored by Moon Capital Management LP (“Moon
Capital”). Moon Capital does not provide legal, accounting or tax advice.
Any statement regarding legal, accounting or tax matters was not intended
or written to be relied upon by any person as advice. Moon Capital does not
waive confidentiality or privilege as a result of this email.
>
>
>
> --
>
>
> ∞
>
> Shashwat Shriparv
>
>

Re: kswapd0 causing read timeouts

[Gurpreet Singh]

Aaron, Ruslan,
I changed the disk access mode to mmap_index_only, and it has been stable
ever since, well at least for the past 20 hours. Previously, in abt 10-12
hours, as soon as the resident memory was full, the client would start
timing out on all its reads. It looks fine for now, i am going to let it
continue to see how long it lasts and if the problem comes again.

Aaron,
yes, i had turned swap off.

The total cpu utilization was at 700% roughly.. It looked like kswapd0 was
using just 1 cpu, but cassandra (jsvc) cpu utilization increased quite a
bit. top was reporting high system cpu, and low user cpu.
vmstat was not showing swapping. java heap size max is 8 gigs. while only 4
gigs was in use, so java heap was doing great. no gc in the logs. iostat
was doing ok from what i remember, i will have to reproduce the issue for
the exact numbers.

cfstats latency had gone very high, but that is partly due to high cpu
usage.

One thing was clear, that the SHR was inching higher (due to the mmap)
while buffer cache which started at abt 20-25mb reduced to 2 MB by the end,
which probably means that pagecache was being evicted by the kswapd0. Is
there a way to fix the size of the buffer cache and not let system evict it
in favour of mmap?

Also, mmapping data files would basically cause not only the data (asked
for) to be read into main memory, but also a bunch of extra pages
(readahead), which would not be very useful, right? The same thing for
index would actually be more useful, as there would be more index entries
in the readahead part.. and the index files being small wouldnt cause
memory pressure that page cache would be evicted. mmapping the data files
would make sense if the data size is smaller than the RAM or the hot data
set is smaller than the RAM, otherwise just the index would probably be a
better thing to mmap, no?. In my case data size is 85 gigs, while available
RAM is 16 gigs (only 8 gigs after heap).

/G


On Fri, Jun 8, 2012 at 11:44 AM, aaron morton wrote:

> Ruslan,
> Why did you suggest changing the disk_access_mode ?
>
> Gurpreet,
> I would leave the disk_access_mode with the default until you have a
> reason to change it.
>
> > 8 core, 16 gb ram, 6 data disks raid0, no swap configured
>>
> is swap disabled ?
>
> Gradually,
>> > the system cpu becomes high almost 70%, and the client starts getting
>> > continuous timeouts
>>
> 70% of one core or 70% of all cores ?
> Check the server logs, is there GC activity ?
> check nodetool cfstats to see the read latency for the cf.
>
> Take a look at vmstat to see if you are swapping, and look at iostats to
> see if io is the problem
> http://spyced.blogspot.co.nz/2010/01/linux-performance-basics.html
>
> Cheers
>
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 8/06/2012, at 9:00 PM, Gurpreet Singh wrote:
>
> Thanks Ruslan.
> I will try the mmap_index_only.
> Is there any guideline as to when to leave it to auto and when to use
> mmap_index_only?
>
> /G
>
> On Fri, Jun 8, 2012 at 1:21 AM, ruslan usifov wrote:
>
>> disk_access_mode: mmap??
>>
>> set to disk_access_mode: mmap_index_only in cassandra yaml
>>
>> 2012/6/8 Gurpreet Singh :
>> > Hi,
>> > I am testing cassandra 1.1 on a 1 node cluster.
>> > 8 core, 16 gb ram, 6 data disks raid0, no swap configured
>> >
>> > cassandra 1.1.1
>> > heap size: 8 gigs
>> > key cache size in mb: 800 (used only 200mb till now)
>> > memtable_total_space_in_mb : 2048
>> >
>> > I am running a read workload.. about 30 reads/second. no writes at all.
>> > The system runs fine for roughly 12 hours.
>> >
>> > jconsole shows that my heap size has hardly touched 4 gigs.
>> > top shows -
>> >   SHR increasing slowly from 100 mb to 6.6 gigs in  these 12 hrs
>> >   RES increases slowly from 6 gigs all the way to 15 gigs
>> >   buffers are at a healthy 25 mb at some point and that goes down to 2
>> mb in
>> > these 12 hrs
>> >   VIRT stays at 85 gigs
>> >
>> > I understand that SHR goes up because of mmap, RES goes up because it is
>> > showing SHR value as well.
>> >
>> > After around 10-12 hrs, the cpu utilization of the system starts
>> increasing,
>> > and i notice that kswapd0 process starts becoming more active.
>> Gradually,
>> > the system cpu becomes high almost 70%, and the client starts getting
>> > continuous timeouts. The fact that the buffers went down from 20 mb to
>> 2 mb
>> > suggests that kswapd0 is probably swapping out the pagecache.
>> >
>> > Is there a way out of this to avoid the kswapd0 starting to do things
>> even
>> > when there is no swap configured?
>> > This is very easily reproducible for me, and would like a way out of
>> this
>> > situation. Do i need to adjust vm memory management stuff like
>> pagecache,
>> > vfs_cache_pressure.. things like that?
>> >
>> > just some extra information, jna is installed, mlockall is successful.
>> there
>> > is no compaction running.
>> > would appreciate any help on this.
>> > Thanks
>> > Gurpreet
>