Re: Problem in getting data from a 2 node cluster of Cassandra

Hi Prakrati, In an ideal situation, no data should be lost when a node is added. How are you getting the statistics below. The output below looks like its from some code using Hector or Thrift..is the code to get statistics from a 1 node cluster or 2 exactly the same- with the only change bein

Problem in adding a new node

Dear all, I had a 1 node cluster of Cassandra. Then I added one more node to it and started Cassandra on it. I got the following error: INFO 12:44:49,588 Loading persisted ring state ERROR 12:44:49,613 Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main] java.io.IOError: java.io.IOException:

RE: Problem in getting data from a 2 node cluster of Cassandra

Yes the code is the same for both 1 and 2 node cluster. It's a Hector code. How do I get the number of rows and columns from Cassandra CLI as the data is very large. Thanks and Regards Prakrati -Original Message- From: Roshni Rajagopal [mailto:roshni.rajago...@wal-mart.com] Sent: Frida

RE: Problem in adding a new node

Hi, the node which you added in that (cassandra.yaml file)give the ip address of 1st node in seeds option. Regards Arshad From: Prakrati Agrawal [prakrati.agra...@mu-sigma.com] Sent: Friday, June 08, 2012 12:44 PM To: user@cassandra.apache.org Subject: Problem in

Re: Problem in getting data from a 2 node cluster of Cassandra

run nodetool -h localhost cfstats on the nodes... this gives node specific column family based data... just run this for both nodes... On Fri, Jun 8, 2012 at 12:46 PM, Prakrati Agrawal wrote: > Yes the code is the same for both 1 and 2 node cluster. It's a Hector code. > How do I get the number

kswapd0 causing read timeouts

Hi, I am testing cassandra 1.1 on a 1 node cluster. 8 core, 16 gb ram, 6 data disks raid0, no swap configured cassandra 1.1.1 heap size: 8 gigs key cache size in mb: 800 (used only 200mb till now) memtable_total_space_in_mb : 2048 I am running a read workload.. about 30 reads/second. no writes at

Re: kswapd0 causing read timeouts

disk_access_mode: mmap?? set to disk_access_mode: mmap_index_only in cassandra yaml 2012/6/8 Gurpreet Singh : > Hi, > I am testing cassandra 1.1 on a 1 node cluster. > 8 core, 16 gb ram, 6 data disks raid0, no swap configured > > cassandra 1.1.1 > heap size: 8 gigs > key cache size in mb: 800 (us

RE: Problem in adding a new node

Yes I gave the ip address of the 1st node in the seeds option Thanks and Regards Prakrati From: MOHD ARSHAD SALEEM [mailto:marshadsal...@tataelxsi.co.in] Sent: Friday, June 08, 2012 12:51 PM To: user@cassandra.apache.org Subject: RE: Problem in adding a new node Hi, the node which you added in t

Re: kswapd0 causing read timeouts

Thanks Ruslan. I will try the mmap_index_only. Is there any guideline as to when to leave it to auto and when to use mmap_index_only? /G On Fri, Jun 8, 2012 at 1:21 AM, ruslan usifov wrote: > disk_access_mode: mmap?? > > set to disk_access_mode: mmap_index_only in cassandra yaml > > 2012/6/8 Gur

Out of memory error

Dear all, When I try to list the entire data in my column family I get the following error: Using default limit of 100 Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:140)

Re: Out of memory error

Check this slide, http://www.slideshare.net/cloudera/hadoop-troubleshooting-101-kate-ting-cloudera Regards ∞ Shashwat Shriparv On Fri, Jun 8, 2012 at 2:34 PM, Prakrati Agrawal < prakrati.agra...@mu-sigma.com> wrote: > Dear all, > > ** ** > > When I try to list the entire data in my colum

Time taken to retrieve data from a 2 node cluster is more than 1 node cluster

Dear all Initially I had a one node cluster and I flooded my data into it. I then ran my Hector code to get some rows and columns. It took around 52.793 seconds. Then I added one more node to the cluster. I again ran the same code and it took around 112.065 seconds. Cassandra should perform fa

Re: about multitenant datamodel

See virtual keyspaces in Hector. Yes, at first, I tried to desigen data model like POD architecture (http://goo.gl/Uw1yD) with this. But, it is problem for me that strong consistency isn't guaranteed among metadata schemas. Every CF has a certain amount of overhead in memory. It's just not h

Re: Cassandra 1.1.1 stack overflow on an infinite loop building IntervalTree

Also looks similar to this ticket: https://issues.apache.org/jira/browse/CASSANDRA-4078 On Thu, Jun 7, 2012 at 6:48 PM, Omid Aladini wrote: > Hi, > > One of my 1.1.1 nodes do

Re: Time taken to retrieve data from a 2 node cluster is more than 1 node cluster

Is ur client code calling with asyncrhynous requests?? and whats ur replication factor and read consistency level. In any case, 2 nodes might take as much time as one, but should not be slow (unless u also doubled the data)... On Fri, Jun 8, 2012 at 2:41 PM, Prakrati Agrawal wrote: > > > Dear al

RE: Time taken to retrieve data from a 2 node cluster is more than 1 node cluster

My consistency level is 1 and I did not set any replication factor. I am using Hector API for getting data and don't know whether they are asynchronous or not. Thanks and Regards Prakrati -Original Message- From: rohit bhatia [mailto:rohit2...@gmail.com] Sent: Friday, June 08, 2012 3:58

Dead node still being pinged

I had a configuration where I had 4 nodes, data-1,4. We then bought 3 bigger machines, data-5,7. And we moved all data from data-1,4 to data-5,7. To move all the data without interruption of service, I added one new node at a time. And then I removed one by one the old machines via a "remove toke

RE Dead node still being pinged

Hi Nicolas, What does nodetool ring says ? (Ask every node) Have you checked that the list of seeds in every yaml is correct ? What version of cassandra are you using ? Samuel Nicolas Lalevée 08/06/2012 14:10 Veuillez répondre à user@cassandra.apache.org A user@cassandra.apache.org cc Ob

Re: Nodes not picking up data on repair, disk loaded unevenly

Follow-up: After adding the EBS nodes, I successfully compacted, the node that had ~1.3T is now down to about 400/500GB (some of that is compression savings). You're right about the load – lots of overwrites. I'm going to get things back off the EBS and add a couple more nodes (I've got 4 righ

Re: Problem in adding a new node

Do you use a 32 bit JVM ? If so I refer you to the following thread: http://mail-archives.apache.org/mod_mbox/cassandra-user/201204.mbox/%3ccaldd-zgthksc2bikp3h4trjxo5vcnhkl2wpwclsf+d9sqty...@mail.gmail.com%3E In short, avoids 32 bits, but if you really cannot, set commitlog_total_space_in_mb to a

Re: Cassandra 1.1.1 stack overflow on an infinite loop building IntervalTree

Looks a lot like https://issues.apache.org/jira/browse/CASSANDRA-4321. Feel free to add a comment on there if you have any additional info. -- Sylvain On Fri, Jun 8, 2012 at 12:06 PM, Omid Aladini wrote: > Also looks similar to this ticket: > > https://issues.apache.org/jira/browse/CASSANDRA-407

Re: Cassandra 1.1.1 Fails to Start

Different node same hardware now gets the stack overflow error but I found the part of the stack trace that is more interesting: at com.google.common.collect.Iterators$5.hasNext(Iterators.java:517) at com.google.common.collect.Iterators$3.hasNext(Iterators.java:114) at com

Re: Cassandra 1.1.1 crash during compaction

A JVM crash means that it's unlikely linked to Cassandra itself. We do use a few unsafe calls here and there, but nothing as changed on that front for 1.1.1 and we never had any report of this crashing the JVM. So imo there is a very good chance that this is a JVM bug. In particular, I doubt many p

Re: Cassandra 1.1.1 Fails to Start

Looks like someone beat me to it, https://issues.apache.org/jira/browse/CASSANDRA-4321 On Fri, Jun 8, 2012 at 9:06 AM, Javier Sotelo wrote: > Different node same hardware now gets the stack overflow error but I found > the part of the stack trace that is more interesting: > > > at > com.g

Re: Problem joining new node to cluster in 1.1.1

That very much look like a bug. Would you mind opening a ticket on https://issues.apache.org/jira/browse/CASSANDRA with those stack traces and maybe a little bit more precision on what you were doing when that happened? -- Sylvain On Fri, Jun 8, 2012 at 12:28 AM, Bryce Godfrey wrote: > As the ne

Re: Dead node still being pinged

Le 8 juin 2012 à 15:17, Samuel CARRIERE a écrit : > What does nodetool ring says ? (Ask every node) currently, each of new node see only the tokens of the new nodes. > Have you checked that the list of seeds in every yaml is correct ? yes, it is correct, every of my new node point to the firs

Re: Problem joining new node to cluster in 1.1.1

This sounds related to https://issues.apache.org/jira/browse/CASSANDRA-4251 On Thu, Jun 7, 2012 at 5:28 PM, Bryce Godfrey wrote: > As the new node starts up I get this error before boostrap starts: > > > > INFO 08:20:51,584 Enqueuing flush of Memtable-schema_columns@1493418651(0/0 > serialized/li

Re: Dead node still being pinged

I'm in the train but just a guess : maybe it's hinted handoff. A look in the logs of the new nodes could confirm that : look for the IP of an old node and maybe you'll find hinted handoff related messages. - Message d'origine - De : Nicolas Lalevée [nicolas.lale...@hibnet.org] Envoyé :

Re: memtable_flush_queue_size and memtable_flush_writers

Single queue for all writers. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 8/06/2012, at 12:49 AM, rohit bhatia wrote: > Hi > > I can't find this in any documentation online, so just wanted to ask > > Do all flush writers share the sa

RE: Problem joining new node to cluster in 1.1.1

https://issues.apache.org/jira/browse/CASSANDRA-4323 Not sure if it's a dupe of what Brandon sent (4251), so created the bug anyway. -Original Message- From: Sylvain Lebresne [mailto:sylv...@datastax.com] Sent: Friday, June 08, 2012 9:08 AM To: user@cassandra.apache.org Subject: Re: Prob

Re: Failing operations & repair

> I am making some cassandra presentations in Kyiv and would like to check that > I am telling people truth :) Thanks for spreading the word :) > 1) Failed (from client-side view) operation may still be applied to cluster Yes. If you fail with UnavailableException it's because from the coordina

Re: Data corruption issues with 1.1

Was this old data written using an older version of cassandra ? In the current versions UTF data is verified. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 8/06/2012, at 7:19 AM, Oleg Dulin wrote: > I can't quite describe what happened

Re: Setting column to null

You don't nee to set columns to null, delete the column instead. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 8/06/2012, at 9:34 AM, Leonid Ilyevsky wrote: > Is it possible to explicitly set a column value to null? > > I see that if in

Re: Problem in getting data from a 2 node cluster

1 Data Stax has some good documentation here http://www.datastax.com/docs/1.1/index Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 8/06/2012, at 2:50 PM, Prakrati Agrawal wrote: > What is the default replication factor? I did not set an

Re: kswapd0 causing read timeouts

Ruslan, Why did you suggest changing the disk_access_mode ? Gurpreet, I would leave the disk_access_mode with the default until you have a reason to change it. > > 8 core, 16 gb ram, 6 data disks raid0, no swap configured is swap disabled ? > Gradually, > > the system cpu bec

Re: Out of memory error

When you ask a question please include the query or function call you have made. An any other information that would help someone understand what you are trying to do. Also, please list things you have already tried to work around the problem. Cheers - Aaron Morton Freelance

Re: Dead node still being pinged

Are the old machines listed in the seed list on the new ones ? Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 9/06/2012, at 12:10 AM, Nicolas Lalevée wrote: > I had a configuration where I had 4 nodes, data-1,4. We then bought 3 bigger >

Re: Nodes not picking up data on repair, disk loaded unevenly

Nice work :) A - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 9/06/2012, at 1:48 AM, Luke Hospadaruk wrote: > Follow-up: > After adding the EBS nodes, I successfully compacted, the node that had ~1.3T > is now down to about 400/500GB (some of th

Re: kswapd0 causing read timeouts

2012/6/8 aaron morton : > Ruslan, > Why did you suggest changing the disk_access_mode ? Because this bring problems on empty seat, in any case for me mmap bring similar problem and i doesn't have find any solution to resolve it, only change disk_access_mode:-((. For me also will be interesting he

Re: Dead node still being pinged

Le 8 juin 2012 à 20:02, Samuel CARRIERE a écrit : > I'm in the train but just a guess : maybe it's hinted handoff. A look in the > logs of the new nodes could confirm that : look for the IP of an old node and > maybe you'll find hinted handoff related messages. I grepped on every node about ev

Re: Dead node still being pinged

Le 8 juin 2012 à 20:50, aaron morton a écrit : > Are the old machines listed in the seed list on the new ones ? No they don't. The first of my old node was, when I was "migrating". But not anymore. Nicolas > Cheers > > - > Aaron Morton > Freelance Developer > @aaronmorton > h

Re: Setting column to null

What you can do is you can define some specific variable like "NULLDATA" some thing like that to update in columns that does have value On Fri, Jun 8, 2012 at 11:58 PM, aaron morton wrote: > You don't nee to set columns to null, delete the column instead. > > Cheers > > - > Aaro

how to compact an index CF?

I have an index to a column IX in column family A. How would I go about compacting that? I have tried nodetool compact keyspace A.IX But that complains "Unknown table/cf pair" I'm sure there must be some simple magic to make this happen. I just cannot tell what it is.

Re: Cassandra 1.1.1 stack overflow on an infinite loop building IntervalTree

Thanks. Yes it's exactly the same. Will follow up there. -- Omid On Fri, Jun 8, 2012 at 5:55 PM, Sylvain Lebresne wrote: > Looks a lot like https://issues.apache.org/jira/browse/CASSANDRA-4321. > Feel free to add a comment on there if you have any additional info. > > -- > Sylvain > > On Fri, Ju

RE: how to compact an index CF?

This is reference to https://issues.apache.org/jira/browse/CASSANDRA-4314 in which Jonathan Ellis instructed me (I think me) to "If you compact the index CF with this patch applied, that should get rid of the tombstones. (compacting the data CF won't do anything.)" However, after much looking I

Re: Setting column to null

Your best bet is to define the column as a composite column where one part represents is null and the other part is the data. On Friday, June 8, 2012, shashwat shriparv wrote: > What you can do is you can define some specific variable like "NULLDATA" some thing like that to update in columns that

Re: kswapd0 causing read timeouts

Aaron, Ruslan, I changed the disk access mode to mmap_index_only, and it has been stable ever since, well at least for the past 20 hours. Previously, in abt 10-12 hours, as soon as the resident memory was full, the client would start timing out on all its reads. It looks fine for now, i am going to