My replication factor is 1 I haven't run repair until today, I'm using ONE for consistency level.
I have two servers that are load balanced (per session) which both run Cassandra and each server connects to Cassandra on localhost. Based on what you're saying, and being I'm using session (cookie) based load balancing it would be true that data is rarely read or written (per user) on a different server, that could be why data isn't replicating. Again, I'm extremely appreciative of your feedback as I haven't had the luxury of delving into all aspects of Cassandra and especially related to multi-node deployments. Thanks! -----Original Message----- From: Benjamin Coverston [mailto:ben.covers...@datastax.com] Sent: Thursday, March 24, 2011 8:59 PM To: user@cassandra.apache.org Subject: Re: URGENT HELP PLEASE! Hi Jared, Sounds like you have two nodes in the cluster. What is your replication factor set to? 1? 2? Have you ever run repair? What consistency level do you use for reads and writes? From the way you are speaking it sounds like you are sending all of your traffic to a single node (primary, secondary). If you never repair the ranges, and you read the data infrequently (or only with range slices) I can guess that your data never got replicated to your secondary node. Ben On 3/24/11 9:44 PM, Jared Laprise wrote: > Thanks for the responses. I got everything working again, and have some ideas > on why but am not completely sure. > > How I got it working again was simply bring the second node back online. I > was under the assumption that all data is replicated between nodes > (eventually). Am I incorrect? It would seem that each node stores different > data and delegates the read request to whichever node holds the data. > Although I've spent a lot of time with Cassandra in a single node environment > I think I may be lacking a bit of understanding on how Cassandra behaves in a > clustered environment. > > > -----Original Message----- > From: Jonathan Ellis [mailto:jbel...@gmail.com] > Sent: Thursday, March 24, 2011 8:22 PM > To: user@cassandra.apache.org > Cc: aaron morton > Subject: Re: URGENT HELP PLEASE! > > Right, Cassandra doesn't keep old versions around so to see an old version > you have to have uncompacted data and whack the new data -- either by blowing > away sstables or not replaying the commitlog. > > Snapshots flush before creating their hard links, which rules out any > commitlog problems. > > If you ran out of disk space you wouldn't get past the commitlog append, so > you'd never get new data in at all after that. > > Sounds like an environmental problem, not Cassandra specific. > > On Thu, Mar 24, 2011 at 9:10 PM, aaron morton<aa...@thelastpickle.com> wrote: >> Was there anything in the server logs during startup ? >> I've not heard of this happening before and it's hard think of how / >> why cassandra could revert it's data. Other than something external >> playing with the files on disk Aaron On 25 Mar 2011, at 13:49, Jared >> Laprise wrote: >> >> Hello all, I'm running 2 Cassandra 6.5 nodes and I brought down the >> secondary node and restarted the primary node. After Cassandra came >> back up all data has been reverted to several months ago. >> >> I could really use some incite here, this is a production website and >> I need to act quickly. I have a cron job that takes a snapshot every >> night, but even with that I tried to restore a snapshot on my local >> development environment and it was also missing a ton of data. >> >> Any help will be so appreciated. >> >> >> > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com -- Ben Coverston DataStax -- The Apache Cassandra Company http://www.datastax.com/
