Hi Dan, We run RF 2 on RAID0 EBS drives. The reason we use EBS over on-instance storage is two fold;
Firstly we have a relatively small cluster ( 4 nodes ), so we're quite sensitive to any AWS issues (at the region level). If we had a larger cluster then we would definitely use ephemeral storage as ephemeral storage will provide much more consistent (*slightly higher*) throughput than EBS. I'm sure you've read a lot about how bad EBS performance is, but genuinely we see very little difference between EBS / ephemeral storage in terms of performance when in a RAID0 setup. Some numbers which are similar to our tests can be found here: http://stu.mp/2009/12/disk-io-and-throughput-benchmarks-on-amazons-ec2.html. The only way you'll know for yourself whether EBS is acceptable is running your own tests, but definitely take anything you read on blog posts etc about EBS performance with a pinch of salt. The only real disadvantage of EBS in our experience is that it's an additional moving part that will fail. You can find yourself in a position where EC2 is running normally but EBS is down, thus your nodes are down. The second reason, and the most important for us was that we didn't have time to build a good automated backup service for the ephemeral storage. The advantage of EBS is that the instance can fail and we can start a new one using the same drive, where-as data on ephemeral will get lost if the node is lost/shutdown. Using ephemeral storage would put us in a difficult position of loosing some data should a couple of instances fail for whatever reason (we don't loose instances often, however when we do we tend to loose several). If you run a larger cluster (imo 10+) then definitely use ephemeral as you shouldn't be very sensitive to loosing a node or two. The second point however doesn't hold as strong today as it did when we made our decision last year. Netflix recently open-sourced a really good tool ( https://github.com/Netflix/Priam/wiki/Backups ) which will automate the back-up of Cassandra data to S3. I'd definitely recommend checking it out to see if it will help you with AWS backups / restores, we're currently looking at rolling it out. Hope that helps, Alex. On Wed, Jul 4, 2012 at 2:56 AM, Dan Foody <dan.fo...@gmail.com> wrote: > Hi Alex, > > Can you share what replication factor you're running? > And, are you using ephemeral disks or EBS volumes? > > Thanks! > > - Dan > > > > On Jul 3, 2012, at 5:52 PM, Alex Major wrote: > > Hi Mike, > > We've run a small (4 node) cluster in the EU region since September last > year. We run across all 3 availability zones in the EU region, with 2 nodes > in one AZ and then a further node in each AZ. The latency difference > between running inside of and between AZ's has been minimal in our > experience. > > It's only when we've gone cross-region that there's been latency problem. > We temporarily ran a 9 node cluster across 3 regions, however even then > using local quoram the latency was better than the standard datacenter - > datacenter latency we're used to. > > EC2Snitch is definitely the way to go in favour of NTS in my opinion. NTS > was a pain to get setup with the internal (private) IP address setup, so > much so that we never got it safely replicating the data as we wanted. > > Alex. > > On Tue, Jul 3, 2012 at 2:16 PM, Michael Theroux <mthero...@yahoo.com>wrote: > >> Hello, >> >> We are currently running a web application utilizing Cassandra on EC2. >> Given the recent outages experienced with Amazon, we want to consider >> expanding Cassandra across availability zones sooner rather than later. >> >> We are trying to determine the optimal way to deploy Cassandra in this >> deployment. We are researching the NetworkTopologyStrategy, and the >> EC2Snitch. We are also interested in providing a high level of read or >> write consistency, >> >> My understanding is that the EC2Snitch recognizes availability zones as >> racks, and regions as data-centers. This seems to be a common >> configuration. However, if we were to want to utilize queries with a READ >> or WRITE consistency of QUORUM, would there be a high possibility that the >> communication necessary to establish a quorum, across availability zones? >> >> My understanding is that the NetworkTopologyStrategy attempts to prefer >> replicas be stored on other racks within the datacenter, which would equate >> to other availability zones in EC2. This implies to me that in order to >> have the quorum of nodes necessary to achieve consistency, that Cassandra >> will communicate with nodes across availability zones. >> >> First, is my understanding correct? Second, given the high latency that >> can sometimes exists between availability zones, is this a problem, and >> instead we should treat availability zones as data centers? >> >> Ideally, we would be able to setup a situation where we could store >> replicas across availability zones in case of failure, but establish a high >> level of read or write consistency within a single availability zone. >> >> I appreciate your responses, >> Thanks, >> -Mike >> >> >> >> > >
