Thanks, Shasha. It looks like there are two ways to make Cassandra work across regions, changing the source code or using software like Vyatta and VIPs.
++ Changing the source code ++ Milind documents how to change the source code in 0.7.4 but warns that there are significant limitations to this, which are not really discussed in his paper. I think some people ran into a problem running nodetool repair after this change, but it worked fine for others. Also, people worried about inter-node communication not being encrypted last month, but with the release of 0.8 this isn't a problem. Are there any other problems we can run into by changing the source code this way? How to change source code in 0.7.4: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/attachment/6191521/0/CassandraInstancesindifferentregions--EC2%5B1%5D.pdf?by-user=t +++ Configuring VIPs +++ Shasha documented a method of using Vyatta (vyatta.com) and VIPs to configure Cassandra across regions. This technique configures a VIP interface (tunXXX) for each node and sets the listen address as the VIP, so all inter-node communication occurs over the VPN tunnel. The RPC address is set to 0.0.0.0 so it listens on all interfaces and does not use the VPN tunnel. - This approach seems to be better, but does anyone have instructions for doing this in EC2 without Vyatta, which is paid software and costs around $700? We're using Ubuntu 11.04 on EC2 and would prefer to use an open source, free software vendor. - Sameer On Thu, May 5, 2011 at 10:48 PM, Sasha Dolgy <sdo...@gmail.com> wrote: > > http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Re-IP-address-resolution-in-MultiDC-setup-EC2-VIP-td6306635.html > On May 6, 2011 3:07 AM, "Sameer Farooqui" <cassandral...@gmail.com> wrote: > > We're trying to set up a Cassandra 0.8.0beta1 cluster across Amazon East > & > > West regions. It does not work out of the box with the binaries and the > > nodes in different regions end up setting this own clusters. > > > > The problem is with Cassandra's Listening Address as described by Rui: > > "Using external IP causes Binding exception, because Cassandra, and java > > socket, cannot listen on the external IP, which does not belong to this > > node. Using private IP prevents nodes in other region from connecting > back > > to this node with the private IP, which has been sent to other nodes for > > cluster establishing." > > > > In Feb 2011, Peter Fales posted: "... the trick is that you have must use > > the "external" addresses in your storage-conf.xml, but since you don't > have > > a NIC that can actually bind to those addresses, you need to listen on > the > > "internal" addresses (or, more simply, all interfaces). At the time, I > was > > not able to get the cross-region cluster to work without making changes > to > > the Cassandra code. Perhaps things have evolved so that there are other > ways > > to do it now." > > > > Does anyone know if Cassandra 0.8 can be configured to run in multiple > > Amazon regions without modifying the source code? And if not, has anybody > > successfully done this with the 0.8 source code and shared the technique? > > > > > > After some research we found that other people have also ran into this > issue > > with versions 0.6.x and 0.7.x of Cassandra: > > > > Fix for 0.6.5 by Peter Fales in Sept 2010: > > http://www.mail-archive.com/dev@cassandra.apache.org/msg01832.html > > http://www.mail-archive.com/user@cassandra.apache.org/msg05692.html > > > > > > Fix for 0.7 by Rui: > > http://rliu09.wordpress.com/2011/04/23/cassandra-across-ec2-regions/ > > > > > > - Sameer > > Accenture Technology Labs >
