Problem in using PHPcassa: Uncaught exception 'Exception' with message 'Could not connect to a cassandra server'

2010-12-20 Thread Rajkumar Gupta
I am using the following code to connect to Cassandra 0.7.0 using PHPcassa (thobbs) However I am able to connect to cassandra instance through cassandra-cli at command prompt but not through this php script. On Mon, Dec 20, 2010 at 11:10 AM, Nick Bailey wrote: > It would probably help if y

Virtual IP / hardware load balancing for cassandra nodes

2010-12-20 Thread Jonathan Colby
I was unable to find example or documentation on my question. I'd like to know what the best way to group a cluster of cassandra nodes behind a virtual ip. For example, can cassandra nodes be placed behind a Citrix Netscaler hardware load balancer? I can't imagine it being a problem, but in

Re: Problem in using PHPcassa: Uncaught exception 'Exception' with message 'Could not connect to a cassandra server'

2010-12-20 Thread Nick Bailey
Have you created Keyspace1? Looks like phpcassa requires you to set a keyspace when you connect but the cli would not. Which might explain why you are seeing problems. On Mon, Dec 20, 2010 at 5:42 AM, Rajkumar Gupta wrote: > I am using the following code to connect to Cassandra 0.7.0 using > PHPc

Re: Problem in using PHPcassa: Uncaught exception 'Exception' with message 'Could not connect to a cassandra server'

2010-12-20 Thread Rajkumar Gupta
Keyspace1 is included by default in the cassandra.yaml file and I imported the schema before running the php script so I hope that is not the problem.. I am now too much frustrated with this problem..:( and this is just the beginning of my Cassandra experiments On Mon, Dec 20, 2010 at 8:36 PM

Re: Virtual IP / hardware load balancing for cassandra nodes

2010-12-20 Thread Daniel Doubleday
You will loose part of the retry / fallback functionality offered by hector. The job of the client lib is not only load-balancing. I.e. if a node is bootstrapping it will accept TCP connections but throw an exception which will be communicated via thrift. The client lib is supposed to handle tha

Re: Cassandra Monitoring

2010-12-20 Thread Edward Capriolo
On Sun, Dec 19, 2010 at 10:37 PM, Dave Viner wrote: > Can you share the code for run_column_family_stores.sh ? > > On Sun, Dec 19, 2010 at 6:14 PM, Edward Capriolo > wrote: >> >> On Sun, Dec 19, 2010 at 2:01 PM, Ran Tavory wrote: >> > Mx4j is in process, same jvm, you just need to throw mx4j-too

Re: Virtual IP / hardware load balancing for cassandra nodes

2010-12-20 Thread Dave Viner
You can put a Cassandra cluster behind a load balancer. One thing to be cautious of is the health check. Just because the node is listening on port 9160 doesn't mean that it's healthy to serve requests. It is required, but not sufficient. The real test is the JMX values. Dave Viner On Mon, D

Re: Problem in using PHPcassa: Uncaught exception 'Exception' with message 'Could not connect to a cassandra server'

2010-12-20 Thread Nick Bailey
Just to verify the keyspace exists can you run 'use Keyspace1;' in the cli? Also it looks like you were using 0.6.8 and now you are using 0.7.0? What is the exact error you are seeing now? On Mon, Dec 20, 2010 at 10:21 AM, Rajkumar Gupta wrote: > Keyspace1 is included by default in the cassandra.

Re: Virtual IP / hardware load balancing for cassandra nodes

2010-12-20 Thread Jonathan Colby
Thanks guys. On Dec 20, 2010, at 5:44 PM, Dave Viner wrote: > You can put a Cassandra cluster behind a load balancer. One thing to be > cautious of is the health check. Just because the node is listening on port > 9160 doesn't mean that it's healthy to serve requests. It is required, but

Re: Problem in using PHPcassa: Uncaught exception 'Exception' with message 'Could not connect to a cassandra server'

2010-12-20 Thread Rajkumar Gupta
Nick, Sorry I can not currently access my computer on which cassandra is installed but yes, earlier I used the 'show keyspaces;' command & it correctly showed me the keyspace1 with corresponding column families. Yes I was using Cassandra 0.6.8 earlier and now I have switched to 0.7. Since I was us

Re: Problem in using PHPcassa: Uncaught exception 'Exception' with message 'Could not connect to a cassandra server'

2010-12-20 Thread Nick Bailey
So it looks like phpcassa doesn't log any errors. The error you are seeing is because the connection to the first server failed and there are no more servers to try. It doesn't say why it failed though. The only thing i can reccommend is on line 57 of connection.php to add a line to print out the e

Severe Reliability Problems - 0.7 RC2

2010-12-20 Thread Dan Hendry
I have been having severe and strange reliability problems within my Cassandra cluster. This weekend, all four of my nodes were down at once. Even now I am loosing one every few hours. I have attached output from all the system monitoring commands I can think of. What seems to happen is that the j

Re: Severe Reliability Problems - 0.7 RC2

2010-12-20 Thread Kani
Have you tried to send a KILL -3 to the Cassandra process before you send KILL -9? This way you will see what the threads are doing (and maybe blocking). The majority of the threads may give you the right spot where to look for the problem. I'm not much of a good linux administrator, but when some

Re: Virtual IP / hardware load balancing for cassandra nodes

2010-12-20 Thread Nate McCall
Current versions of Hector will mark the host as temporarily down if a connection times out (retrying N seconds later; defaults to 10). If you only have one host in your list, this will be an issue. It would be a good idea to use more than one VIP for this reason. Managing configurations on the ap

Re: Severe Reliability Problems - 0.7 RC2

2010-12-20 Thread Peter Schuller
> be correlated is the flushing of memtables tables. One of the strangest > stats I am getting when in this state is memory paging: 3727168.00 pages > scanned/second (see sar -B output). Occasionally, if I leave the process > alone (~1 h) it recovers (maybe 1 in 5 times), otherwise the only way to

Re: Problem in using PHPcassa: Uncaught exception 'Exception' with message 'Could not connect to a cassandra server'

2010-12-20 Thread Tyler Hobbs
Yes, Nick's suggestion is good. I've added a logging statement at that point in the latest commit to phpcassa. Perhaps you could try that and see what the error is? - Tyler On Mon, Dec 20, 2010 at 12:35 PM, Nick Bailey wrote: > So it looks like phpcassa doesn't log any errors. The error you a

RE: Severe Reliability Problems - 0.7 RC2

2010-12-20 Thread Dan Hendry
Yes, I have tried that (although only twice). Same impact as a regular kill: nothing happens and I get no stacktrace output. It is however on my list of things to try again the next time a node dies. I am also not able to attach jstack to the process. I have also tried disabling JNA (did not

Re: Severe Reliability Problems - 0.7 RC2

2010-12-20 Thread Adrian Cockcroft
What filesystem are you using? You might try EXT3 or 4 vs. XFS as another area of diversity. It sounds as if the page cache or filesystem is messed up. Are there any clues in /var/log/messages? How much swap space do you have configured? The kernel level debug stuff I know is all for Solaris un

Re: Severe Reliability Problems - 0.7 RC2

2010-12-20 Thread Peter Schuller
> There were a couple of threads on lkml recently that may be relevant, > but I have to run so I can't find the URL:s atm (todo later tonight). Ok, I cannot figure out how to find the "first" message in a thread in any of the lkml archives, but these two threads may be of interest, especially if y

Re: Severe Reliability Problems - 0.7 RC2

2010-12-20 Thread Brandon Williams
On Mon, Dec 20, 2010 at 2:13 PM, Dan Hendry wrote: > Yes, I have tried that (although only twice). Same impact as a regular > kill: nothing happens and I get no stacktrace output. It is however on my > list of things to try again the next time a node dies. I am also not able to > attach jstack to

Re: Severe Reliability Problems - 0.7 RC2

2010-12-20 Thread Chris Goffinet
What kernel version are you running? I have seen with I/O intense nodes with 2.6.18 to 2.6.24 the kernel has a bug where it locks the JVM and spins to 100%. On Mon, Dec 20, 2010 at 1:14 PM, Brandon Williams wrote: > On Mon, Dec 20, 2010 at 2:13 PM, Dan Hendry wrote: > >> Yes, I have tried that (

RE: Severe Reliability Problems - 0.7 RC2

2010-12-20 Thread Dan Hendry
Thanks for all the response, I have included requested information below. As a side note, I THINK I have fixed the problem by using "disk_access_mode: mmap_index_only". At the very least, none of the nodes has died since setting the option. > What kernel version are you running? 3/4 running

Re: Problem in using PHPcassa: Uncaught exception 'Exception' with message 'Could not connect to a cassandra server'

2010-12-20 Thread Rajkumar Gupta
I will try it out & let you know soon.. Thanks On Tue, Dec 21, 2010 at 1:20 AM, Tyler Hobbs wrote: > Yes, Nick's suggestion is good.  I've added a logging statement at that > point in the latest commit to phpcassa.  Perhaps you could try that and see > what the error is? > > - Tyler > > On Mon,

Re: [SOLVED] Very high memory utilization (not caused by mmap on sstables)

2010-12-20 Thread Zhu Han
Can anybody recommend a stable enough JDK environment for 0.6.x branch on ubuntu server? Thank you! best regards, hanzhu On Sun, Dec 19, 2010 at 10:29 AM, Zhu Han wrote: > The problem seems still like the C-heap of JVM, which leaks 70MB every day. > Here is the summary: > > on 12/19: