Re: Issues running Bulkloader program on AIX server

Hi All, Sorry, my environment is as below: 1. 3 node cluster with Cassandra 1.1.9 provided with DSE 3.0 on Linux 2. We are trying to run the bulk loader from AIX 6.1 server. Java version 1.5. Regards, Praveen From: Praveen Akunuru mailto:praveen.akun...@wipro.com>> Date: Thursday, Apri

Cassandra services down frequently [Version 1.1.4]

Hi, We are running 4 nodes Cassandra cluster (1.1.4) with Replica Factor 2 (DC 1) and Replica Factor 1 (DC 2) in two differnet data cnters with network topology. Our machines are having 16GB RAM and 8 core with two hard drives. # /opt/apache-cassandra-1.1.4/bin/nodetool -h localhost ring

Re: Lost data after expanding cluster c* 1.2.3-1

Hi aaron, I ran the command "nodetool rebuild_index host keyspace cf" on all the nodes, in the log i see : INFO [RMI TCP Connection(5422)-10.34.139.xxx] 2013-04-04 08:31:53,641 ColumnFamilyStore.java (line 558) User Requested secondary index re-build for ... but nothing's happening, how can i mo

Re: Repair does not fix inconsistency

Hi Aaron, At first, before I go with a lot of logs: I'm considering a problem related to this issue: https://issues.apache.org/jira/browse/CASSANDRA-4905 Let's say the tombstone on one of the nodes (X) is gcable and was not compacted (purged) so far. After it was created we re-created this r

Re: upgrading 1.1.x to 1.2.x via sstableloader

I see, thanks for the replay! One more question: I can see that multiple nodes have same sstable names for a certain keyspace / cf. I am moving 8 nodes to a 6 nodes cluster, so at some point when putting sstables in place I would overwrite files from other node. What is the best way to solve this

Re: Repair does not fix inconsistency

> I'm considering a problem related to this issue: > https://issues.apache.org/**jira/browse/CASSANDRA-4905 > > Let's say the tombstone on one of the nodes (X) is gcable and was not > compacted (purged) so far. After it was created we re-created

Re: Repair does not fix inconsistency

Hi Michal, Let's say the tombstone on one of the nodes (X) is gcable and was not > compacted (purged) so far. After it was created we re-created this row, but > due some problems it was written only to the second node (Y), so we have > "live" data on node Y which is newer than the gcable tombstone

Re: Repair does not fix inconsistency

Hi Christian, About CASSANDRA-4905 - thanks for explaining this :-) This looks to me more like a secondary index issue. If you say the access via rowkey is always correct, then the repair works fine. I think there might be something wrong with your secondary index then. This was my first thou

Re: Repair does not fix inconsistency

Hi Sylvain, Thanks for explaination :-) However, in this case, I still do not get why this (probably) gcable tombstone on 2:1 could cause this mess. As AE ignores only the tombstone itself (which means that there are no data for this key on 2:1 node from repair's point of view), it should resu

Re: Repair does not fix inconsistency

Hi, This was my first thought too, but if you take a look at the logs I > attached to previous e-mail, you'll notice that query "by key" > (no-index.log) retrieves data from BOTH replicas, while the "by indexed > column" one (index.log) talks only to one of them (too bad it's the one > that contai

Re: nodetool status inconsistencies, repair performance and system keyspace compactions

Hi, most has been resolved - the failed to uncompress error was really a bug in cassandra (see https://issues.apache.org/jira/browse/CASSANDRA-5391) and the problem with different load reporting is a change between 1.2.1 (reports 100% for 3 replicas/3 nodes/2 DCs setup I have) and 1.2.3 which repo

Re: Repair does not fix inconsistency

Well... Strange. We have such problem with 6 users, but there's only ONE tombstone (created 8 days ago, so it's not gcable yet) in all the SSTables on 2:1 node - checked using sstable2json. Moreover, this tombstone DOES NOT belong to the row key I'm using for tests, because this user was NOT eve

Re: Repair does not fix inconsistency

Does CQL not allow CL=ONE queries? Why does it ask two nodes for the key, when you say that you are using CL=default=1? I'm a bit confused here (I'm a thrift user). Yup, that's another thing I'm curious about too (default CL is ONE for sure), but as for now it helps me to investigate my probl

Re: Repair does not fix inconsistency

> Well... Strange. We have such problem with 6 users, but there's only ONE > tombstone (created 8 days ago, so it's not gcable yet) in all the SSTables > on 2:1 node - checked using sstable2json. > Moreover, this tombstone DOES NOT belong to the row key I'm using for > tests, because this user was

Repair hangs when merkle tree request is not acknowledged

Hello, I have a cluster with 4 nodes, 2 nodes in 2 data centers. I had a hardware failure in one DC and had to replace the nodes. I'm running 1.2.3 on all of the nodes now. I was able to run nodetool rebuild on the two replacement nodes, but now I cannot finish a repair on any of them. I have 1

Re: Repair does not fix inconsistency

Repair is fine - all the data seem to be in SSTables. I've checked it and > while index tells me that I have 1 tombstone and 0 live cells for a key, I > can _see_, thanks to sstable2json, that I have 3 "live cells" (assuming a > cell is an entry in SSTable) and 0 tombstones. After being confused fo

Re: Alter table drop column seems not working

You are right, the documentation says that this action is not supported. I was surprised because the "auto completion" in cqlsh allows you to try it and, moreover, you have an example of a drop column when you use "help alter_drop". Maybe it would be nice to change at least the documentation and

Re: Repair does not fix inconsistency

Yes, maybe there are two issues here: repair not running and maybe really some index-thing. Repair is fine - all the data seem to be in SSTables. I've checked it and while index tells me that I have 1 tombstone and 0 live cells for a key, I can _see_, thanks to sstable2json, that I have 3 "l

Re: Linear scalability problems

We are using a single process with multiple threads, will look at client side delays. Thanks On Wed, Apr 3, 2013 at 9:30 AM, Tyler Hobbs wrote: > If I had to guess, I would say that your client is the bottleneck, not the > cluster. Are you inserting data with multiple threads or processes? > >

Re: Cassandra freezes

I am going to throw some info out there for you as it might help. 1. RAM usage grows with dataset size on that node(adding more nodes reduces the RAM used per node since each node has less rows). index_interval can be upped to reduce RAM usage but be careful with it. Switching to LCS and bl

Re: Repair does not fix inconsistency

W dniu 04.04.2013 15:38, horschi pisze: I'm glad to hear that. I feared my ticket might be responsible for your data loss. I could not live the guilt ;-) Seriously: I'm glad we can rule out the repair change. Haha, I didn't notice before that it was your ticket! ;-) Yes, if it works with CL=o

Re: Linear scalability problems

What was the RF before adding nodes? Sent from my iPhone On 04 Apr 2013, at 15:12, Anand Somani wrote: > We are using a single process with multiple threads, will look at client side > delays. > > Thanks > > On Wed, Apr 3, 2013 at 9:30 AM, Tyler Hobbs wrote: > If I had to guess, I would say

Really have to repair ?

Hi, I know that deleted rows can reappear if "node repair" is not run on every node before gc_grace_seconds seconds. However do we really need to obey this rule if we run "node repair" on node that are down for more than max_hint_window_in_ms milliseconds ? Thanks -- Cyril SCETBON __

Is there guidance about compaction thresholds and setting minthreshold to 2?

The default minthreshold for compactions is 4: http://www.datastax.com/docs/1.1/references/nodetool#nodetool-setcompactionthreshold Is there a reason that this value is not "2", the lowest possible value? If we change this to 2 what should we expect to see? Should we see less growth in storage lo

Re: Why do Datastax docs recommend Java 6?

I use IBM JVM 7, it is free, and for VMs over 8 GB it has a garbage collection policy that makes it almost pause-less. We also use some security libraries that eliminates use of other libs that you need for Oracle. On Thu, Apr 4, 2013 at 12:59 AM, Edward Capriolo wrote: > Hey guys. what gives!

Re: Is there guidance about compaction thresholds and setting minthreshold to 2?

One would think, but remember only "like sized" sstables compact. You want more files roughlt the same size rather then few big ones in most cases, but there are no hard fast rules. On Thu, Apr 4, 2013 at 11:36 AM, Peter Haggerty wrote: > The default minthreshold for compactions is 4: > > http:/

Re: IndexOutOfBoundsException during repair, streaming

On Wed, Apr 3, 2013 at 6:08 PM, aaron morton wrote: > We deleted and recreated those CFs before moving into > production mode. > > We have a wiener. > > The comparator is applying the current schema to the byte value read from > disk (schema on read) which describes a value with more than 2 compon

Re: Is there guidance about compaction thresholds and setting minthreshold to 2?

More importantly than CPU, you'll use more I/O. Say you have 4 (like-size) sstables, compact them all into one file (which is really what SizeTieredCompaction will try to do) will require twice as much I/O that with min_compaction=2 versus 4. On Thu, Apr 4, 2013 at 7:26 PM, Edward Capriolo wrote:

Data Modeling: How to keep track of arbitrarily inserted column names?

Hey Guys, I'm working on a project and one of the requirements is to have a schema free CF where end users can insert arbitrary key/value pairs per row. What would be the best way to know what are all the "keys" that were inserted (preferably w/o any locking). For example, Row1 => key1 -> XXX,

Re: Data Modeling: How to keep track of arbitrarily inserted column names?

You can not get only the column name (which you are calling a key) you can use get_range_slice which returns all the columns. When you specify an empty byte array (new byte[0]{}) as the start and finish you get back all the columns. From there you can return only the columns to the user in a format

Re: Linear scalability problems

RF=3. On Thu, Apr 4, 2013 at 7:08 AM, Cem Cayiroglu wrote: > What was the RF before adding nodes? > > Sent from my iPhone > > On 04 Apr 2013, at 15:12, Anand Somani wrote: > > We are using a single process with multiple threads, will look at client > side delays. > > Thanks > > On Wed, Apr 3, 2

Re: how to stop out of control compactions?

Ah, 0 is the magic? Odd email thread now I asked about the best practice of disabling compactions, greg said he set threshold = 10, you +1'd, I said I couldn't set > 32, and now we're at 0 ;-) will On Wed, Apr 3, 2013 at 8:50 PM, aaron morton wrote: > And it appears I can't set min > 3

Re: Any plans for read-before-write update operations in CQL3?

Well, a schema've just came to my mind, that looks interesting, so I want to share: 1) Actions are introduced. Each action receives unique I'd at coordinator node. Client can ask for a block of ids beforehand, to make actions idempotent. 2) Actions are applied to given row+column value. It's possib

Re: Data Modeling: How to keep track of arbitrarily inserted column names?

Hi Edward, I anticipate that the column names will be reused a lot. For example, key1 will be in many rows. So I think the number of distinct column names will be much much smaller than the number of rows. Is there a way to have a separate CF that keeps track of the column names? What I was t

gossip not working

I was in the middle of upgrade to 1.1.9. I brought one node with 1.1.9 while the other were running on 1.1.5. Once one of the node was on 1.1.9 it is no longer recognizing other nodes in the ring. On 192.168.56.10 and 11 192.168.56.10 DC1-CassRAC1Up Normal 28.06 GB50.00%

Re: Data Modeling: How to keep track of arbitrarily inserted column names?

Your reverse index of "which rows contain a column named X" will have very wide rows. You could look at cassandra's secondary indexing, or possibly look at a solandra/solr approach. Another option is you can shift the problem slightly, "which rows have column X that was added between time y and tim

Re: gossip not working

What errors are you seeing in the log files of the down nodes? Did you run upgradesstables? You need to upgradesstables when moving from < 1.1.7 to 1.1.9 On Apr 4, 2013, at 6:11 PM, S C wrote: > I was in the middle of upgrade to 1.1.9. I brought one node with 1.1.9 while > the other were runni

Re: Data Modeling: How to keep track of arbitrarily inserted column names?

I don't really need to answer "what rows contain column named X", so no need for a reverse index here. All I want is a distinct set of all the column names, so I can answer "what are all the available column names" On Apr 4, 2013, at 4:20 PM, Edward Capriolo wrote: > Your reverse index of "wh

RE: gossip not working

I am not seeing anything in the logs other than "Starting up server gossip" and there is no firewall between the nodes. From: paulsu...@gmail.com Subject: Re: gossip not working Date: Thu, 4 Apr 2013 18:49:29 -0500 To: user@cassandra.apache.org What errors are you seeing in the log files of the d

Re: Cassandra 1.0.10 to 1.2.3 upgrade "post-mortem"

On 04/04/2013 02:24, aaron morton wrote: I just wanted to share our experience of upgrading 1.0.10 to 1.2.3 In general it's dangerous to skip a major release when upgrading. True. But in that case it was supposed to be fine. ERROR [MutationStage:33] 2013-03-31 09:00:02,899 CassandraDaemon.java

RE: gossip not working

Is there a way to force gossip among the nodes? From: as...@outlook.com To: user@cassandra.apache.org Subject: RE: gossip not working Date: Thu, 4 Apr 2013 19:59:45 -0500 I am not seeing anything in the logs other than "Starting up server gossip" and there is no firewall between the nodes. Fr

Re: Cassandra services down frequently [Version 1.1.4]

On Thu, Apr 4, 2013 at 1:27 AM, wrote: > > After some time (1 hour / 2 hour) cassandra shut services on one or two > nodes with follwoing errors; > Wonder what the workload and schema is like ... We can see from below that you've tweaked and disabled many of the memory "safety valve" and other