Re: Does anybody know why Twitter stop integrate Cassandra as Twitter store?

Big thanks for all your replies

Re: CQL select not working for CF defined programatically with Hector API

Perfectly right. Sorry for not paying attention! Thanks Eric, Alex On Tue, Oct 4, 2011 at 4:19 AM, Eric Evans wrote: > On Mon, Oct 3, 2011 at 12:02 PM, Alexandru Sicoe > wrote: > > Hi, > > I am using Cassandra 0.8.5, Hector 0.8.0-2 and cqlsh (cql 1.0.3). If I > > define a CF with comparator Lo

Why is mutation stage increasing ??

Hello, I have my 3-node, RF=3 cluster acting strangely. Can someone shed a light as to what is going on ? It was stuck for a couple of hours (all clients TimedOut). nodetool tpstats showed huge increasing MutationStages (in the hundreds of thousands). I restarted one node and it took a while to rep

Re: Shrinking cluster with counters ...

Is the cluster still in use ? The safe way would be to reduce the RF to 1, and then nodetool decomission the nodes one at a time. This will cause them to stream data to the remaining node(s), and at the same time the node that takes ownership will be receiving writes for the new token range. I

Re: Weird problem with empty CF

No. It's generally only an issue with heavy delete workloads, and it's sometimes possible to design around it. cheers - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 5/10/2011, at 1:18 PM, Daning wrote: > Thanks. Do you have plan t

Re: Why is mutation stage increasing ??

Well what client are you using? And can you give a hint to your node hardware? 從我的 BlackBerry® 無線裝置 -Original Message- From: Philippe Date: Wed, 5 Oct 2011 10:33:21 To: user Reply-To: user@cassandra.apache.org Subject: Why is mutation stage increasing ?? Hello, I have my 3-node, RF=3 c

Re: ByteOrderedPartitioner token generation

When considering such an expedition it is important to quantify the relative account of the terms "large" and "short". The modern gentleman may also find respite through the application of the Bulk Loader http://www.datastax.com/dev/blog/bulk-loading I would avoid using the BOP unless you are s

Re: Why is mutation stage increasing ??

Lots of hinted handoff can give you mutations… > HintedHandoff 0 0 1798 0 > 0 1798 is somewhat high. This is the HH tasks on this node though, can you see HH running on other nodes in the cluster? What has been happening on this nod

Re: Why is mutation stage increasing ??

Thanks for the quick responses. @Yi Using Hector 0.8.0-1 Hardware is : - AMD Opteron 4174 6x 2.30+ GHz - 32 Go DDR3 - 1 Gbps Lossless @aaron I'm running 0.8.6 on all nodes, straight from the debian packages. I get hinted handoffs from time to time because of flapping, I've planning to

Re: Token != DecoratedKey assertion

A little feedback, I scrubbed on each server and I haven't seen this error again. The load on each server eems to be correct. nodetool compactionstats shows boat-load of "Scrub" at 100% on my 3rd node but not on the 2 others. I left it that way and haven't restart yet. 2011/9/26 aaron morton >

Re: Why is mutation stage increasing ??

Followup, I was mistaken in saying there weren't writes to the cluster. There's a process that's doing a couple mutations per second. I just restarted node #3 and found this message on node #1 INFO [HintedHandoff:1] 2011-10-05 12:25:08,173 HintedHandOffManager.java (line 314) Endpoint /xx.xx.xx.1

cluster repair script

simple script for running cluster - wide repairs #! /bin/sh set -e if test $# -eq 0; then echo "$0 " exit 1; fi for i in `nodetool -h $1 ring | cut -d ' ' -f 1 | grep -e '^[0-9]'`; do nodetool -h $i repair done

Re: dedicated gossip lan

Dne 4.10.2011 22:05, Sorin Julean napsal(a): Sorry for not being clear. Indeed I mean a separate LAN and interfaces for "listen_address". It needs to be 1GBit LAN, 100Mbit Ethernet is way too slow for cassandra.

RE: invalid column name length 0

Did the test again, empty database, with replication factor 3, Cassandra running in it's own jvm. All data is now stored using a separate program that connects to the database using THRIFT. At least this results in a lot less Dead/Up messages (I guess the GC had too much work handling the non-ca

Thrift transport error

I'm using Thrift 0.7 with Cassandra 0.8.6 and "Cassandra Cluster Admin" to work around my single node [testing] cluster. All seams to work fine, but I'm getting a contant error message "CustomTThreadPoolServer.java (line 197) Thrift transport e

Re: invalid column name length 0

Ok. Quick other question then. Did you issue deletion and/or used TTLs for that test ? Also, it's probably worth creating a ticket on https://issues.apache.org/jira/browse/CASSANDRA if you don't mind. -- Sylvain On Wed, Oct 5, 2011 at 2:42 PM, Desimpel, Ignace wrote: > Did the test again, empty

Re: Thrift transport error

Cassandra uses Thrift 0.6, so without being a specialist of thrift internals, that could be the source of the problem you're seeing (for info, we'll update to thrift 0.7 at some point, but not in the very close future; see https://issues.apache.org/jira/browse/CASSANDRA-3213). -- Sylvain On Wed,

RE: invalid column name length 0

TTLs : no Deletion : yes ; but I think I can avoid this and thus running the same test without deletion, just to eliminate possibilities. -Original Message- From: Sylvain Lebresne [mailto:sylv...@datastax.com] Sent: woensdag 5 oktober 2011 15:34 To: user@cassandra.apache.org Subject: Re:

Re: Can't connect to MX4J endpoint on Ubuntu

On 23/09/2011 23:55, Iwona Bialynicka-Birula wrote: I am trying to monitor Cassandra 8.0 using MX4J I was going through this stuff recently, as well. Have a look at Jolokia[1] and Jmx4Perl[2]. It's quite trivial in installation and will give you access too all of the stuff as MX4J does. Es

Thrift transport error

@Sylvain thanks for the eye opening hint on 3213 There are some critical issues with Thrift 0.6 that were fixed in 0.7 Thrift 0.6 critical issues https://issues.apache.org/jira/browse/THRIFT-788 https://issues.apache.org/jira/browse/THRIFT-1067 @Jonathan you're right, the error message is relate

Consistency level and ReadRepair

I have a 12 node cassandra cluster running with RF=3. I have severl clients ( all running on a single node ) connecting to the cluster ( fixed client - node mapping ) and try to do a insert, update , select and delete. Each client has a fixed mapping of the row-keys and always connect to the same

Re: nodetool cfstats on 1.0.0-rc1 throws an exception

I don't have access to the test system anymore. We did move to lower number of CFs and dont see this problem any more. I remember when I noticed the size in system.log it was little more than UINT_MAX (4294967295). I was able to recreate it multiple times. So I am wondering if there are any stats c

0.7.9 RejectedExecutionException

I'm getting the following exception on a 0.7.9 node before the node crashes. I don't have this problem with the other nodes running 0.7.8. Does anyone know what the problem is? ERROR [Thread-47] 2011-10-05 05:07:03,840 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[Thread

Memtable Switch Count

What is Memtable Switch Count in the cfstats output? thanks Ramesh

Question about sharding of rows and atomicity

Does Cassandra shard the columns of a single row across multiple nodes so that to read the columns of the row it may need access to multiple nodes? I'd say "no." Will a read from a given node ever return partial results or is the write to a node of a row atomic? Thanks, Don

Re: Consistency level and ReadRepair

Start with http://wiki.apache.org/cassandra/ReadRepair. Read repair count increasing just means you were doing reads at < CL.ALL, and had the CF configured to perform RR. On Wed, Oct 5, 2011 at 12:37 PM, Ramesh Natarajan wrote: > I have a 12 node cassandra cluster running with RF=3.  I have seve

Re: 0.7.9 RejectedExecutionException

"I can't schedule this task because I'm shutting down" is a symptom of your node crashing, not a cause. Is it being OOMkilled, perhaps? On Wed, Oct 5, 2011 at 12:42 PM, Ashley Martens wrote: > I'm getting the following exception on a 0.7.9 node before the node crashes. > I don't have this proble

Re: Question about sharding of rows and atomicity

On Wed, Oct 5, 2011 at 1:09 PM, Don Smith wrote: > Does Cassandra shard the columns of a single row across multiple nodes so > that to read the columns of the row it may need access to multiple nodes? > I'd say "no." Correct. >   Will a read from a given node ever return partial results or > is

Re: Question about sharding of rows and atomicity

Cassandra does not break apart a row. All of the columns of a row are kept on the same nodes. I believe that writing multiple columns of the same row is transactional, but not atomic. By which I mean that if one column is written all the other ones will be written as well, but if a read happens

Could not reach schema agreement

Hi Guys, How would I go about fixing this? (running 0.8.4) [default@unknown] connect 10.58.135.19/9160; Connected to: "Test Cluster" on 10.58.135.19/9160 [default@unknown] describe cluster; Cluster Information: Snitch: org.apache.cassandra.locator.SimpleSnitch Partitioner: org.apache.cassan

Re: Could not reach schema agreement

Did you try wiki.apache.org/cassandra/FAQ#schema_disagreement ? On Wed, Oct 5, 2011 at 1:47 PM, Ben Ashton wrote: > Hi Guys, > > How would I go about fixing this? (running 0.8.4) > > [default@unknown] connect 10.58.135.19/9160; > Connected to: "Test Cluster" on 10.58.135.19/9160 > [default@unknow

Re: Consistency level and ReadRepair

Lets assume we have 3 nodes all up and running at all times with no failures or communication problems. 1. If I have a RF=3 and writing with QUORUM, 2 nodes the change gets committed, what is the delay we should expect before the 3rd replica gets written 2. In this scenario ( no failures e.t.c )

Re: Could not reach schema agreement

Ah thats great! I was rubbing my head for a while as google only showed mailing lists posts with the same error. All working now, thanks On 5 October 2011 19:49, Jonathan Ellis wrote: > Did you try wiki.apache.org/cassandra/FAQ#schema_disagreement ? > > On Wed, Oct 5, 2011 at 1:47 PM, Ben Ashto

Re: Consistency level and ReadRepair

Do you see any errors in the logs? Is your HH enabled? On Wed, Oct 5, 2011 at 12:00 PM, Ramesh Natarajan wrote: > Lets assume we have 3 nodes all up and running at all times with no > failures or communication problems. > 1. If I have a RF=3 and writing with QUORUM,  2 nodes the change gets > com

Re: Consistency level and ReadRepair

Yes Hinted Handoff is enabled. However I don't see any counters raising against the HintedHandoff in the tpstats. thanks Ramesh On Wed, Oct 5, 2011 at 2:10 PM, Mohit Anchlia wrote: > Do you see any errors in the logs? Is your HH enabled? > > On Wed, Oct 5, 2011 at 12:00 PM, Ramesh Natarajan wro

Re: Consistency level and ReadRepair

As explained in the link in my earlier reply, "Read Repair" just means "a replica was checked in the background," not that it was out of sync. On Wed, Oct 5, 2011 at 2:00 PM, Ramesh Natarajan wrote: > Lets assume we have 3 nodes all up and running at all times with no > failures or communication

Re: 0.7.9 RejectedExecutionException

No OOM errors appear and the memory used is far below physical and Java max. I changed the JAR to 0.7.8 to see if that works. If so I'll find a way to roll out that version instead of 0.7.9.

Re: Could not reach schema agreement

oh no spoke to soon.. All me data are being gone :( /opt/apache-cassandra-0.8.4/bin/nodetool -h 10.224.55.162 repair Exception in thread "main" java.lang.AssertionError: Repairing no column families seems pointless, doesn't it at org.apache.cassandra.service.AntiEntropyService$RepairSess

Re: Consistency level and ReadRepair

Thanks for the explanation. I think i am at loss trying to understand the tpstats output.. when does the ReadRepair count get incremented? - When any read is performed with CL < ALL and RF=3 (or) - When there is a discrepency? I have 2 snapshots when i run tpstats and the counts indicate there we

Re: Why is mutation stage increasing ??

Sounds like there is a lot going on. I'm going to assume the order you showed the HH stats in is the order of the nodes. I'm guessing node 180 is node 2, but it would be easier if you could identify the nodes and identify the stats for them. In no particular order: * Have a heavily used My S

Re: Memtable Switch Count

How many times a "full" memtable was swapped for an empty one http://thelastpickle.com/2011/05/04/How-are-Memtables-measured/ Chhers - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 6/10/2011, at 7:04 AM, Ramesh Natarajan wrote: > What i

cassandra-cli: Create column family with composite column name

Using Cassandra 0.8.6, I've been trying to figure out how to use the CLI to create column families using composite keys and column names. The documentation on CompositeType seems pretty skimpy. But in the course of writing this email to ask how to do it, I figured out the proper syntax. In the hope

Re: Could not reach schema agreement

Check the data directories, including the snapshot one. Data is not deleted. If you create a CF the server will look for existing files and load them. Cheers - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 6/10/2011, at 8:35 AM, Ben As

TimedOutException and UnavailableException from multiGetSliceQuery

Hi all, I have been experiencing the unavailableException and TimedOutException on a 3-node cassandra cluster during a multiGetSliceQuery with 1000 columns. Since there are many keys involved in the query, I divided them into groups of 5000 rows and process each group individually in a for loop. b

Re: 0.7.9 RejectedExecutionException

I could be wrong. I just looked the amount of memory being used and it's huge. WTF?

Re: cassandra-cli: Create column family with composite column name

Hi Jim, The best resource I know so far is http://www.slideshare.net/edanuff/indexing-in-cassandra I just started working on a blog post about them last night, and I hope to update the wiki with some information when I am done. Feel free to mail me directly if you want to collaborate. I'm

Re: 0.7.9 RejectedExecutionException

check this http://wiki.apache.org/cassandra/FAQ#mmap Cheers - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 6/10/2011, at 9:25 AM, Ashley Martens wrote: > I could be wrong. I just looked the amount of memory being used and it's > huge.

Re: TimedOutException and UnavailableException from multiGetSliceQuery

5000 rows in a mutli get is way, way, way (did I say way ? ) to many. Whenever you get a TimedOutException check the tp stats on the nodes, you will normally see a high pending count. Every row get get turns into an message in a TP. So if you ask for 5k rows you flood the TP with 5k messages wh

Re: TimedOutException and UnavailableException from multiGetSliceQuery

Hi Aaron, thanks for the suggestion. It works again after I cut back the # of rows. On Wed, Oct 5, 2011 at 1:43 PM, aaron morton wrote: > 5000 rows in a mutli get is way, way, way (did I say way ? ) to many. > > Whenever you get a TimedOutException check the tp stats on the nodes, you > will nor

Compaction and total disk space used for highly overwritten CF

We have a very hot CF which we use essentially as a durable memory cache for our application. It is about 70MBytes in size after being fully populated. We completely overwrite this entire CF every few minutes (not delete). Our hope was that the CF would stay around 70MB in size, but it grows