Copying the list back in, just in case someone smart has a response ;) --- Jeremiah Peschka Managing Director, Brent Ozar PLF, LLC
On Wed, Feb 22, 2012 at 6:00 PM, <char...@contentomni.com> wrote: > Thanks a lot for this Jeremiah, very insightful indeed, especially the bit > on Rackspace and I/O. I understand there will be bottlenecks with cloud > servers, but I want to build a system that is relatively low-cost but > effective (i.e. until venture capital comes into play #fingerscrossed). So > I'm just trying to be savvy with the way I start out > At least you're planning on spending the VC money on gear instead of a fancy office in SOMA (/me ducks) > > > Actuallly, I think my math was faulty, it's actually 800 billion keys, so > that's many more servers than expected. > > > > I'm guessing we can use Rackspace Cloud Servers until we get to something > like 5 million active users, then we will have to migrate to something > dedicated/hybrid. I was actually thinking about using 'smaller' servers so > that I don't incur a high cost on resources I don't need initially. Anyway, > I guess what I could do instead is take servers off the cluster, rebuild > them with higher RAM and HDD, and then join them back to the Riak > cluster...this might be a better way of doing things instead of adding > servers in a linear fashion since its RAM and HDD space that I'm looking > for. So basically, [with a partition size of 2048] I would start off with > 16 nodes of 2Gb RAM each, and then probably go to 128 nodes with 30GB of > RAM each. After this it would definitely be time to get more prodigious > hardware and get some sort of hybrid dedicated/cloud infrastructure going. > One of the advantages of Riak is that it's very easy to pull a node out of a cluster, improve the hardware, and then put it back into the cluster. Since RackSpace are likely using some kind of paravirtualization or hardware partitioning, this should be a lot less painful than you would think. I'm not sure how Riak will respond to waking up with more memory and hard disk space on a host that appears to be the same. From my experiences doing this with Windows and SQL Server (two finicky pieces of software), you should be OK. > > > Regarding migration from cloud to dedicated/hybrid [without significant > downtime], do you see any issues with this migration on the Riak-side, I > imagine by this time we would have something like 200 billion or so keys at > this point?! > Your biggest bottleneck moving between providers is going to be the time it takes to push all of your data around. One option that you do have here is standing up a Riak cluster in your new DC and then paying for the cross data center replication feature. At that point, the movement becomes automatic and you'd just have to wait for your data to move across. With sizable data volumes, it just takes a lot of time to move from one DC to another when you don't have control over the gear. Your other option, of course, is just to lease entire servers from RackSpace. They're very helpful when you get to certain levels of investment with them. > > > Thanks. > > > > -----Original Message----- > From: "Jeremiah Peschka" <jeremiah.pesc...@gmail.com> > Sent: Thursday, 23 February, 2012 12:15am > To: riak-users@lists.basho.com > Subject: Re: Riak for Messaging Project Question > > Responses inline. > --- > Jeremiah Peschka > Managing Director, Brent Ozar PLF, LLC > > > > On Wed, Feb 22, 2012 at 2:10 PM, <char...@contentomni.com> wrote: > > I'm building an online tool/app that is heavily dependent on messaging. > This > > messaging is simple text, nothing complicated, and it will take place > > between my server back-end and the desktop/device. These messages would > be > > very easy to store in Riak. > > > > > > > > Each message is created after a specific user event e.g. a user posts a > > request, etc. In turn, each message created could spawn another 200 to > 3,000 > > messages (based on some other social networking features I can't say too > > much about to keep this short). I believe, in this case, we can assume > each > > message will be a Riak Object. > > > > > > > > All tolled, from my estimation, I'm looking at 400,000 messages/objects > > generated per user per year. With an estimated active user base of 20 > > million (I hope some day), that would be 8 billion keys generated each > year. > > The size of each object is about 2Kb max. So that works out about 16 > > Terabytes of data generated per year. > > > > > > > > 1. Is Riak a good fit for this solution going up to and beyond 20 million > > users (i.e. terabytes upon terabytes added per year)? > > > > I think Riak is a good fit for this solution in terms of the ability > to handle data size. > > > > > > > 2. I plan to use 2i, which means I would be using the LevelDB backend. > Will > > this be reasonably performant for billions of keys added each year? > > > > > > LevelDB is a good backend fit, especially for when the size of your > keyspace exceeds the size of RAM. > > > > > 3. I'm using what I have here > > (http://wiki.basho.com/Cluster-Capacity-Planning.html) as my guide for > > capacity planning. I plan on using Rackspace Cloud Servers for this > specific > > project. Can I just keep adding servers as the size of my data grows?! > > > This planning guide is aimed at planning for Bitcask specifically, but > most of the advice applies > > You can keep adding servers, but you need to be careful about the > initial size of your ring. The ring size defaults to 64 virtual nodes > and it can't be changed once you put data in the cluster, so you'll > need to do some careful planning up front. Having more virtual nodes > will enabled you to safely increase the size of your ring. I believe > the current guidance is that you want no fewer 10 v-nodes per physical > server in the cluster. Also, I seem to recall reading that you want to > make sure the number of v-nodes is a power of two. Going by this, > you'll want to start with 2048 v-nodes, which could prove somewhat > problematic on a small cluster. > > > > > 4. From the guide mentioned in 3 above, it appears I will need about 400 > > [4GbRAM 160GbHDD] servers for 20 million users (assuming an n_val of 4). > > This means I would need to add 20 servers annually for each million > active > > users I add. Is it plausible to have an n_val of 4 for this many > servers?! > > Wouldn't going higher just mean I'd have to add many more servers > > needlessly?! > > > > I'm not sure I understand the question. Basically, each node in the > cluster is aware of where data belongs. When you query a Riak node, > it'll route the request to the nodes that should have the key in > question. With an n_val of 4 and say 200 servers, you'll still be > querying a maximum of 5 servers (one for whichever node coordinates > the request, and up to 4 servers sending data back). With a large > number of servers, I would be more concerned about traffic around the > ring. However, some of the changes outlined in Riak 1.1's release > notes make me think that it isn't that big of a concern. > > As an aside, 4GB of RAM and a 160GB HDD sounds like the specs on low > end cable box. You can avoid having 200+ servers by using servers with > more RAM and more drives. It's something to plan for in the long run, > but you can fit an incredible amount of storage and RAM into some > server chassis. E.g. the Dell C2200 can hold 192GB of RAM and many TB > of storage - 12 bays in chassis - and the server doesn't cost that > much in the grand scheme of things. > > > > > > > 5. Should I put all my keys in one bucket (considering I'm using 2i, > does it > > matter)?! > > > > Buckets are a logical namespace - use as many or as few as you want. > Of course, using buckets could make it easier to logically move some > of your data to another cluster if you find that one cluster can't > handle the load. > > > > > > > I'd appreciate some assistance with this. > > > > A word of warning: I/O is your enemy in shared hosting environments. > > Be wary of Rackspace's I/O pipeline. Most cloud providers are using > low end commodity servers with low end commodity storage in the back > end. That means you're going to share a host server with multiple > tenants and you'll be sharing the same single crappy Broadcom ethernet > port with everyone else on that box and you will most likely be > sharing the same Dell EqualLogic or EMC Isilon (I think Rackspace use > the Isilon unless you ask for a VMAX). > > Point is: you'll have a terribly narrow and shared pipeline to your > disk subsystem. Expect your I/O to be in the 70MB/s or lower rate. > Or... what you'd expect from a USB flash drive. > > Edit: Rackspace allege to be using local storage, so you'll be > fighting with everyone else on your server for access to the same four > 7200 RPM drives ;) Again, expect terrible performance and you won't be > disappointed. > > > > > > > Thanks. > > > > > > > > > > > > > > _______________________________________________ > > riak-users mailing list > > riak-users@lists.basho.com > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
