Re: deleting rows and tombstones

2012-02-14 Thread Dominic Williams
find that repair is still as necessary now since hinted handoffs > are stored anytime a node does not ACK successfully? > > From: Dominic Williams > Reply-To: "user@cassandra.apache.org" > Date: Tue, 14 Feb 2012 12:31:45 -0800 > > To: "user@cassandra.apac

Re: deleting rows and tombstones

2012-02-14 Thread Dominic Williams
Hi Todd, Our systems do a lot of deletions and it does cause problems. Your best bet is to bring GCSeconds low and run repair religiously. The issue you can run into though is repair overloading your servers when your data load gets high, repair falling over and related problems. IMHO the need t

Re: How to bring cluster to consistency

2012-02-13 Thread Dominic Williams
Hi Nikolay, Some points that may be useful: 1/ auto_bootstrap = true is used for telling a new node to join the ring (the cluster). It has nothing to do with hinted handoff 2/ both of your nodes seem to be using the same token? The output indicates that 100% of your key range is assigned to 10.1

Re: Previously deleted rows resurrected by repair?

2011-12-28 Thread Dominic Williams
Hmm interesting could be some variation on 3510 (which caught me out). Personally I really don't like having to rely on repair to stop deletes being undone. If you agree follow this proposal for an alternative https://issues.apache.org/jira/browse/CASSANDRA-3620 which also stops tombstone build up

Re: Asymmetric load

2011-12-15 Thread Dominic Williams
Btw anyone having problems with repair might like to follow proposal for different system: https://issues.apache.org/jira/browse/CASSANDRA-3620 With this system you would only need to run repair to ensure all data has maximum redundancy across the cluster (which also increases consistency for Cons

Re: best practices for simulating transactions in Cassandra

2011-12-12 Thread Dominic Williams
nodes running on virtualization software, hypervisors et al have setup issues that make their clocks drift under load, and it is a good idea to be wary of that) Best, Dominic > Sorry for all the questions but I'm very concerned about this particular > problem :) > > Thanks, > Jo

Re: best practices for simulating transactions in Cassandra

2011-12-12 Thread Dominic Williams
that swept away >>> orphans. >>> >>> 3) Try to model your updates as idempotent column inserts instead. >>> How do you model updates as inserts? Instead of munging the value >>> directly, you could insert a column containing the operation you want to

Re: Anybody out there using 0.8 in production

2011-09-08 Thread Dominic Williams
Hi I've just migrated to 0.8.5 and from first looks it is a giant leap forward - better use of CPU and memory - able to scrub files previously unfixable on 0.7.6-2 etc On 9 September 2011 01:45, Anthony Ikeda wrote: > We plan to and have been using it in Dev and QA. There are some bugs that > hav

Re: Occasionally getting old data back with ConsistencyLevel.ALL

2011-08-22 Thread Dominic Williams
Hi there, here's my tuppence! 1. Something to look at first: If you write two different values to the same column quickly in succession, if both writes go out with the same timestamp, then it is indeterminate which one wins i.e. write order doesn't necessarily matter. 2. Don't trust PayPal (anyo

Re: Clock skew

2011-06-28 Thread Dominic Williams
Hi, yes you are correct, and this is a potential problem. IMPORTANT: If you need to serialize writes from your application servers, for example using distributed locking, then before releasing locks you must sleep for a period equal to the maximum variance between the clocks on your application se

Re: 99.999% uptime - Operations Best Practices?

2011-06-23 Thread Dominic Williams
Les, Cassandra is a good system, but it has not reached version 1.0 yet, nor has HBase etc. It is cutting edge technology and therefore in practice you are unlikely to achieve five nines immediately - even if in theory with perfect planning, perfect administration and so on, this should be achieva

Re: No Transactions: An Example

2011-06-23 Thread Dominic Williams
AJ this approach doesn't work if you need synchronized access to a value to prevent, for example, the lost update problem. Unless you synchronize access to something like bank balance transfers, you will quickly destroy your data On 22 June 2011 23:34, AJ wrote: > I think Sasha's idea is worth s

Re: No Transactions: An Example

2011-06-22 Thread Dominic Williams
Hi Trevor, I hope to post on my practical experiences in this area soon - we rely heavily on complex serialized operations in FightMyMonster.com. Probably the most simple serialized operation we do is updating nugget balances when, for example, there has been a trade of monsters. Currently we use

Re: OOM during restart

2011-06-21 Thread Dominic Williams
Hi gabe, What you need to do is the following: 1. Adjust cassandra.yaml so when this node is starting up it is not contacted by other nodes e.g. set thrift to 9061 and storage to 7001 2. Copy your commit logs to tmp sub-folder e.g. commitLog/tmp 3. Copy a small number of commit logs back into m

Re: SSTable corruption blocking compaction and scrub can't fix it

2011-06-17 Thread Dominic Williams
o poke around I can also arrange to give password as node will be decommissioned anyway? You can then take a look at the SSTable corruption too etc On 17 June 2011 18:06, Ryan King wrote: > Even without lsof, you should be able to get the data from /proc/$pid > > -ryan > > On

Re: SSTable corruption blocking compaction and scrub can't fix it

2011-06-17 Thread Dominic Williams
Unfortunately I shutdown that node and anyway lsof wasn't installed. But $ulimit gives unlimited On 17 June 2011 13:00, Sylvain Lebresne wrote: > On Fri, Jun 17, 2011 at 1:51 PM, Dominic Williams > wrote: > > As far as scrub goes that could be it. I'm already

Re: SSTable corruption blocking compaction and scrub can't fix it

2011-06-17 Thread Dominic Williams
y closes files > (https://issues.apache.org/jira/browse/CASSANDRA-2669) > so that may be part of why that happens. However, a simple fix is probably > to > raise up the file descriptor limit. > > -- > Sylvain > > > On Fri, Jun 17, 2011 at 11:31 AM, Dominic Williams >

SSTable corruption blocking compaction and scrub can't fix it

2011-06-17 Thread Dominic Williams
Hi all, Anyone experiencing this..? I noticed one of my 7.6-2 nodes had inexplicable and consistently high cpu usage. Checking the log I found that there was a some kind of SSTable corruption that was stopping a bunch of files from compacting (first trace copied below). I then tried scrub (befor

Re: Database grows 10X bigger after running nodetool repair

2011-05-25 Thread Dominic Williams
t are affecting users etc. Hope this helps On 25 May 2011 19:07, Dominic Williams wrote: > Jeepers creepers that's it Jeeves!!! Argh. > > Basically once my repair hit a big column family db size exploded until the > node ran out of disk space.. > > Firstly any ideas for

Re: Database grows 10X bigger after running nodetool repair

2011-05-25 Thread Dominic Williams
mane.comp.db.cassandra.user/16619 > <http://permalink.gmane.org/gmane.comp.db.cassandra.user/16619> > On May 25, 2011, at 4:52 PM, Dominic Williams wrote: > > Hi, > > I've got a strange problem, where the database on a node has inflated 10X > after running repair. Th

Database grows 10X bigger after running nodetool repair

2011-05-25 Thread Dominic Williams
Hi, I've got a strange problem, where the database on a node has inflated 10X after running repair. This is not the result of receiving missed data. I didn't perform repair within my usual 10 day cycle, so followed recommended practice: http://wiki.apache.org/cassandra/Operations#Dealing_with_the

Re: Pelops - a new Java client library paradigm

2010-06-14 Thread Dominic Williams
0. Have you built and/or >run Pelops against 0.6.2 or trunk? > > > > Todd > > > -- > > *From:* Dominic Williams [mailto:thedwilli...@googlemail.com] > *Sent:* Friday, June 11, 2010 6:42 AM > > *To:* user@cassandra.apache.o

Re: Pelops - a new Java client library paradigm

2010-06-14 Thread Dominic Williams
rashed node becomes available? > > > > Todd > > > -- > > *From:* Dominic Williams [mailto:thedwilli...@googlemail.com] > *Sent:* Friday, June 11, 2010 7:05 AM > *To:* user@cassandra.apache.org > *Subject:* Re: Pelops - a new Java client library parad

Re: Pelops - a new Java client library paradigm

2010-06-14 Thread Dominic Williams
trying to get a > confidence index and I am interested as well ;) > > Best, > Riyad > > > On Fri, Jun 11, 2010 at 7:04 AM, Dominic Williams < > thedwilli...@googlemail.com> wrote: > >> Hi good question. >> >> The scalability of Pelops is dependent on Cassan

Re: Pelops - a new Java client library paradigm

2010-06-14 Thread Dominic Williams
ion of BatchMutation, but the rest of > the API improvements are still TODO > Keep up the good work, competition keeps us healthy ;) > > > On Fri, Jun 11, 2010 at 4:41 PM, Dominic Williams < > thedwilli...@googlemail.com> wrote: > >> Pelops is a new high quality Jav

Re: Pelops - a new Java client library paradigm

2010-06-11 Thread Dominic Williams
d? Size of clusters and size of data? > > Ian > > > On Fri, Jun 11, 2010 at 9:41 AM, Dominic Williams < > thedwilli...@googlemail.com> wrote: > >> Pelops is a new high quality Java client library for Cassandra. >> >> It has a design that: >> * revea

Pelops - a new Java client library paradigm

2010-06-11 Thread Dominic Williams
Pelops is a new high quality Java client library for Cassandra. It has a design that: * reveals the full power of Cassandra through an elegant "Mutator and Selector" paradigm * generates better, cleaner, less bug prone code * reduces the learning curve for new users * drives rapid application deve

Locking and transactions over Cassandra operations using "Cages"

2010-05-13 Thread Dominic Williams
ansactions-over-cassandra-using-cages/ You can find more information about Cages here: http://cages.googlecode.com/ Best, Dominic Williams