An interesting hybrid that I'm coming around to seems to be using a Unix release - OmniOS has an AMI, for instance - and ZFS. With a large-enough store, I can run without EBS on my nodes, and have a single ZFS backup instance with a huge amount of slow-EBS storage for accepting ZFS snapshots.
I'm still learning all the pieces, but luckily I have a company upstairs from me that does a very similar thing with > 300TB and is willing to help me set up my ZFS backup infrastructure. Dave On Mon, Aug 12, 2013 at 10:00 PM, Brady Wetherington <br...@bespincorp.com>wrote: > I will probably stick with EBS-store for now. I don't know how comfortable > I can get with a replica that could disappear with simply an unintended > reboot (one of my nodes just did that randomly today, for example). Sure, I > would immediately start rebuilding it as soon as that were to happen, but > we could be talking a pretty huge chunk of data that would have to get > rebuilt out of the cluster. And that sounds scary. Even though, logically, > I understand that it should not be. > > I will get there; I'm just a little cautious. As I learn Riak better and > get more comfortable with it, maybe I would be able to start to move in a > direction like that. And certainly as the performance characteristics of > EBS-volumes start to bite me in the butt; that might force me to get > comfortable with instance-store real quick. I would at least hope to be > serving a decent-sized chunk of my data from memory, however. > > As for throwing my instances in one AZ - I don't feel comfortable with > that either. I'll try out the way I'm saying and will report back - do I > end up with crazy latencies all over the map, or does it seem to "just > work?" We'll see. > > In the meantime, I still feel funny about "breaking the rules" on the > 5-node cluster policy. Given my other choices as having been kinda > nailed-down for now, what do you guys think of that? > > E.g. - should I take the risk of putting a 5th instance up in the same AZ > as one of the others, or should I just "be ok" with having 4? Or should I > do something weird like changing my 'n' value to be one fewer or something > like that? (I think, as I understand it so far, I'm really liking "n=3, > w=2, r=2" - but I could change it if it made more sense with the topology > I've selected.) > > -B. > > > Date: Sun, 11 Aug 2013 18:57:11 -0600 >> From: Jared Morrow <ja...@basho.com> >> To: Jeremiah Peschka <jeremiah.pesc...@gmail.com> >> Cc: riak-users <riak-users@lists.basho.com> >> Subject: Re: Practical Riak cluster choices in AWS (number of nodes? >> AZ's?) >> Message-ID: >> < >> cacusovelpu8yfcivykexm9ztkhq-kdnowk1afvpflcsip2h...@mail.gmail.com> >> Content-Type: text/plain; charset="iso-8859-1" >> >> >> +1 to what Jeremiah said, putting a 4 or 5 node cluster in each US West >> and >> US East using MDC between them would be the optimum solution. I'm also >> not >> buying consistent latencies between AZ's, but I've also not tested it >> personally in a production environment. We have many riak-users members >> on >> AWS, so hopefully more experienced people will chime in. >> >> If you haven't seen them already, here's what I have in my "Riak on AWS" >> bookmark folder: >> >> http://media.amazonwebservices.com/AWS_NoSQL_Riak.pdf >> http://docs.basho.com/riak/latest/ops/tuning/aws/ >> http://basho.com/riak-on-aws-deployment-options/ >> >> -Jared >> >> >> >> >> On Sun, Aug 11, 2013 at 6:11 PM, Jeremiah Peschka < >> jeremiah.pesc...@gmail.com> wrote: >> >> > I'd be wary of using EBS backed nodes for Riak - with only a single >> > ethernet connection, it wil be very easy to saturate the max of 1000mbps >> > available in a single AWS NIC (unless you're using cluster compute >> > instances). I'd be more worried about temporarily losing contact with a >> > node through network saturation than through AZ failure, truthfully. >> > >> > The beauty of Riak is that a node can drop and you can replace it with >> > minimal fuss. Use that to your advantage and make every node in the >> cluster >> > disposable. >> > >> > As far as doubling up in one AZ goes - if you're worried about AZ >> failure, >> > you should treat each AZ as a separate data center and design your >> failure >> > scenarios accordingly. Yes, Amazon say you should put one Riak node in >> each >> > AZ; I'm not buying that. With no guarantee around latency, and no >> control >> > around between DCs, you need to be very careful how much of that latency >> > you're willing to introduce into your application. >> > >> > Were I in your position, I'd stand up a 5 node cluster in US-WEST-2 and >> be >> > done with it. I'd consider Riak EE for my HA/DR solution once the >> business >> > decides that off-site HA/DR is something it wants/needs. >> > >> > >> > --- >> > Jeremiah Peschka - Founder, Brent Ozar Unlimited >> > MCITP: SQL Server 2008, MVP >> > Cloudera Certified Developer for Apache Hadoop >> > >> > >> > On Sun, Aug 11, 2013 at 1:52 PM, Brady Wetherington < >> br...@bespincorp.com>wrote: >> > >> >> Hi all - >> >> >> >> I have some questions about how I want my Riak stuff to work - I've >> >> already asked these questions of some Basho people and gotten some >> answers, >> >> but thought I would toss it out into the wider world to see what you >> all >> >> have to say, too: >> >> >> >> First off - I know 5 instances is the "magic number" of instances to >> >> have. If I understand the thinking here, it's that at the default >> >> redundancy level ('n'?) of 3, it is most likely to start getting me >> some >> >> scaling (e.g., performance > just that of a single node), and yet also >> have >> >> redundancy; whereby I can lose one box and not start to take a >> performance >> >> hit. >> >> >> >> My question is - I think I can only do 4 in a way that makes sense. I >> >> only have 4 AZ's that I can use right now; AWS won't let me boot >> instances >> >> in 1a. My concern is if I try to do 5, I will be "doubling up" in one >> AZ - >> >> and in AWS you're almost as likely to lose an entire AZ as you are a >> single >> >> instance. And so, if I have instances doubled-up in one AZ (let's say >> >> us-east-1e), and then I lose 1e, I've now lost two instances. What are >> the >> >> chances that all three of my replicas of some chunk of my data are on >> those >> >> two instances? I know that it's not guaranteed that all replicas are on >> >> separate nodes. >> >> >> >> So is it better for me to ignore the recommendation of 5 nodes, and >> just >> >> do 4? Or to ignore the fact that I might be doubling-up in one AZ? >> Also, >> >> another note. These are designed to be 'durable' nodes, so if one >> should go >> >> down I would expect to bring it back up *with* its data - or, if I >> >> couldn't, I would do a force-replace or replace and rebuild it from the >> >> other replicas. I'm definitely not doing instance-store. So I don't >> know if >> >> that mitigates my need for a full 5 nodes. I would also consider >> losing one >> >> node to be "degraded" and would probably seek to fix that problem as >> soon >> >> as possible, so I wouldn't expect to be in that situation for long. I >> would >> >> probably tolerate a drop in performance during that time, too. (Not a >> >> super-severe one, but 20-30 percent? Sure.) >> >> >> >> What do you folks think? >> >> >> >> -B. >> >> >> >> >> >> _______________________________________________ >> >> riak-users mailing list >> >> riak-users@lists.basho.com >> >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> >> >> >> > >> > _______________________________________________ >> > riak-users mailing list >> > riak-users@lists.basho.com >> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> > >> > >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: < >> http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20130811/d350b1f1/attachment-0001.html >> > >> > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
