Thanks everyone. The issue was a missing firewall entry in the security groups that prevented Cassandra from clustering.
So Alain, initially I was using the private IP. I went to irc and someone had mentioned to use the public IP, although I wasnt doing multi region clustering. I should at some point write a santy tool that checks all the configuration options, validates ports, etc to help automate the validation process. Cheers, On Thu, Feb 4, 2016 at 6:37 PM, Alain RODRIGUEZ <arodr...@gmail.com> wrote: > Hi Richard, > > I think you just can't use EC2Snitch with public IPs. > > See > https://docs.datastax.com/en/cassandra/2.0/cassandra/architecture/architectureSnitchEC2_t.html > > Precisely "Because private IPs are used, this snitch does not work across > multiple regions" > > 54.*.*.* looks like a public one. > > You can stick with the private IPs (with limitation written above, even if > you can workaround with a VPN tunnel across Regions). In this case set > listen address to private IP and comment broadcast_address. You can also > use the EC2MultiRegionSnitch, but then be careful with broadcast_address > (public IP) and listen-address (private IP) configuration on the > cassandra.yaml files and also with ports management on AWS console. > > Also, as you nodes already bootstrapped, you might have to clean the > cassandra folder, usually something like rm -rf /var/lib/cassandra/* > *warning: *you will loose all the data, but this "cluster" doesn't look > like a running cluster, only you can know :-). > > Any suggestions on how to track down what might trigger this problem > > > This kind of issue might be due to: > > - Different cluster names > - *Bad configuration* (IPs, Snitch + configuration files, ...) <-- > probably your case > - Ports (firewall, AWS rules...) <-- telnet might be useful here > - Seeds being differents on the nodes <-- make sure that your seeds are > the same on every node > > Hope this will be enough to get you out of this, > > C*heers, > ----------------- > Alain Rodriguez > France > > The Last Pickle > http://www.thelastpickle.com > > > > 2016-02-04 16:35 GMT+00:00 Victor Chen <victor.h.c...@gmail.com>: > >> Along the lines of what Ben and Bryan suggested, what are you using to >> verify ports are open? If you do something like: >> >> node1$ nc -zv node2 9042 >> node2$ nc -zv node1 9042 >> >> does it succeed from both nodes? >> Does the first node 'know' that it is a seed? i.e. do you have first node >> listed in its own seed's list? >> What does the system.log show as both nodes are spun up? >> >> >> On Wed, Feb 3, 2016 at 7:20 PM, Bryan Cheng <br...@blockcypher.com> >> wrote: >> >>> >>> >>> >>>> On Wed, 3 Feb 2016 at 11:49 Richard L. Burton III <mrbur...@gmail.com> >>>> wrote: >>>> >>>>> >>>>> Any suggestions on how to track down what might trigger this problem? >>>>> I'm not receiving any exceptions. >>>>> >>>> >>> You're not getting "Unable to gossip with any seeds" on the second node? >>> What does nodetool status show on both machines? >>> >> >> > -- -Richard L. Burton III @rburton
