Re: On 0.6.6 to 0.7.3 migration, DC-aware traffic and minimising data transfer

2011-03-14 Thread Jedd Rashbrooke
Jonathon, thank you for your answers here. To explain this bit ... On 11 March 2011 20:46, Jonathan Ellis wrote: > On Thu, Mar 10, 2011 at 6:06 AM, Jedd Rashbrooke wrote: >>  Copying a cluster between AWS DC's: >>  We have ~ 150-250GB per node, with a Replication Factor

On 0.6.6 to 0.7.3 migration, DC-aware traffic and minimising data transfer

2011-03-10 Thread Jedd Rashbrooke
Howdi, Assortment of questions relating to an upgrade combined with a possible migration between Data Centers (or perhaps a multi-DC redesign). Apologies if some of these have been asked before - I have kept half an eye on the list in recent times but haven't seen anything covering these pa

Re: monitoring with Zabbix

2011-01-10 Thread Jedd Rashbrooke
Hi Daniel, A recent web search found your earlier message, but I had a few (potentially dumb) questions - so this is very good timing! :) When you say start with Cassandra - my first dumb question is how you actually do this. Are you using packaged versions or tarball installs of Cassandr

Re: Memory leak with Sun Java 1.6 ?

2010-12-16 Thread Jedd Rashbrooke
Hi Peter, I've read through the "Very high memory utilization (not caused by mmap on sstables)" and the impact sounds similar .. alas they're using openJDK and I'm using Sun's. Of course, it's not impossible that the same bug is present in both JVM's. j.

Re: Memory leak with Sun Java 1.6 ?

2010-12-15 Thread Jedd Rashbrooke
Hey Peter, On 14 December 2010 20:19, Peter Schuller wrote: > So, now that I get that we have two different cases ;) Yup. My problem is java / environment based, occurring after several weeks, using 0.6.6 instances. Our thread hijacking friend / learned colleague's problem is 0.7 based an

Re: Memory leak with Sun Java 1.6 ?

2010-12-14 Thread Jedd Rashbrooke
On 10 December 2010 18:37, Peter Schuller wrote: > Memory-mapped files will account for both virtual and, to the extent > that they are resident in memory, to the resident size of the process. To clarify - in our storage-conf we have: mmap_index_only I know it's a matter of degree, but

Re: Memory leak with Sun Java 1.6 ?

2010-12-14 Thread Jedd Rashbrooke
Peter, Jonathon - thank you for your replies. I should probably have repeated myself in the body, but as I mentioned in the subject line, we're running Sun Java 1.6. On 10 December 2010 18:37, Peter Schuller wrote: > Memory-mapped files will account for both virtual and, to the extent > that

Memory leak with Sun Java 1.6 ?

2010-12-10 Thread Jedd Rashbrooke
Howdi, We're using Cassandra 0.6.6 - intending to wait until 0.7 before we do any more upgrades. We're running a cluster of 16 boxes of 7.1GB each, on Amazon EC2 using Ubuntu 10.04 (LTS). Today we saw one box kick its little feet up, and after investigating the other machines, it looks li

KeysCached - clarification and recommendations on settings

2010-10-18 Thread Jedd Rashbrooke
Greetings, I would like to check my understanding is accurate on how KeysCached is understood by Cassandra (0.6.5), and then get some suggestions on settings / OS FS cache interplay. First - my initial understanding was that if you set KeysCached to 100%, Cassandra would do a best effort to

Re: Dazed and confused with Cassandra on EC2 ...

2010-10-08 Thread Jedd Rashbrooke
On 7 October 2010 20:49, Peter Schuller wrote: > ... if you "waste" 10-15 gigs of RAM on the JVM heap for a > Cassandra instances which could live with e.g. 1 GB, you're actively > taking away those 10-15 gigs of RAM from the operating system to use > for the buffer cache. Particularly if you're I

Re: Hardware change of a node in the cluster

2010-10-04 Thread Jedd Rashbrooke
On 4 October 2010 10:58, Utku Can Topçu wrote: > Recently I've tried to upgrade (hw upgrade) one of the nodes in my cassandra > cluster from ec2-small to ec2-large. Something that bit me on this (I've done it with both Cassandra and Hadoop boxes, and some problems might be more Hadoopy related

Re: Dazed and confused with Cassandra on EC2 ...

2010-10-04 Thread Jedd Rashbrooke
Hi Peter, Thanks again for your time and thoughts on this problem. We think we've got a bit ahead of the problem by just scaling back (quite savagely) on the rate that we try to hit the cluster. Previously, with a surplus of optimism, we were throwing very big Hadoop jobs at Cassandra, in

Re: Dazed and confused with Cassandra on EC2 ...

2010-09-28 Thread Jedd Rashbrooke
Peter - my apologies for the slow response - we had to divert down a 'Plan B' approach last week involving MySQL, memcache, redis and various other uglies. On 20 September 2010 23:11, Peter Schuller wrote: > Are you running an old JVM by any chance? (Just grasping for straws.) JVM is Sun's 1

Re: Dazed and confused with Cassandra on EC2 ...

2010-09-20 Thread Jedd Rashbrooke
Hi Peter, We were logging the GC output as per this before, have since taken it out, but will put it back in I think. Apropos logging - I've found that with RMI to our boxes at EC2 I've had to do the ugly thing with this: -Djava.rmi.server.hostname= .. which then renders nodetool useless,

Re: Dazed and confused with Cassandra on EC2 ...

2010-09-17 Thread Jedd Rashbrooke
Hi Rob, Thanks for your suggestions. I should have been a bit more verbose in my platform description -- I'm using 64-bit instances, which I think in a Ben Black video I saw led to a sensible default usage of mmap when left at auto. Should I look at forcing this setting? > You don't mentio

Re: Dazed and confused with Cassandra on EC2 ...

2010-09-17 Thread Jedd Rashbrooke
Hi Dave, Thank you for your response. I can clarify a couple of things here: > 2. You grew from 2 nodes to 4, but the original 2 nodes have 200GB and the 2 > new ones have 40 GB.  What's the recommended practice for rebalancing (i.e., > when should you do it), what's the actual procedure, and

Dazed and confused with Cassandra on EC2 ...

2010-09-17 Thread Jedd Rashbrooke
Howdi, I've just landed in an experiment to get Cassandra going, and fed by PHP via Thrift via Hadoop, all running on EC2. I've been lurking a bit on the list for a couple of weeks, mostly reading any threads with the word 'performance' in them. Few people have anything polite to say about