The 40 TB use case you heard about is probably one 40TB mysql machine that someone migrated to mongo so it would be "web scale" Cassandra is NOT good with drives that big, get a blade center or a high density chassis.
On Mon, Feb 18, 2013 at 8:00 PM, Hiller, Dean <dean.hil...@nrel.gov> wrote: > I thought about this more, and even with a 10Gbit network, it would take 40 > days to bring up a replacement node if mongodb did truly have a 42T / node > like I had heard. I wrote the below email to the person I heard this from > going back to basics which really puts some perspective on it….(and a lot of > people don't even have a 10Gbit network like we do) > > Nodes are hooked up by a 10G network at most right now where that is > 10gigabit. We are talking about 10Terabytes on disk per node recently. > > Google "10 gigabit in gigabytes" gives me 1.25 gigabytes/second (yes I could > have divided by 8 in my head but eh…course when I saw the number, I went duh) > > So trying to transfer 10 Terabytes or 10,000 Gigabytes to a node that we are > bringing online to replace a dead node would take approximately 5 days??? > > This means no one else is using the bandwidth too ;). 10,000Gigabytes * 1 > second/1.25 * 1hr/60secs * 1 day / 24 hrs = 5.555555 days. This is more > likely 11 days if we only use 50% of the network. > > So bringing a new node up to speed is more like 11 days once it is crashed. > I think this is the main reason the 1Terabyte exists to begin with, right? > > From an ops perspective, this could sound like a nightmare scenario of > waiting 10 days…..maybe it is livable though. Either way, I thought it would > be good to share the numbers. ALSO, that is assuming the bus with it's 10 > disk can keep up with 10G???? Can it? What is the limit of throughput on a > bus / second on the computers we have as on wikipedia there is a huge > variance? > > What is the rate of the disks too (multiplied by 10 of course)? Will they > keep up with a 10G rate for bringing a new node online? > > This all comes into play even more so when you want to double the size of > your cluster of course as all nodes have to transfer half of what they have > to all the new nodes that come online(cassandra actually has a very data > center/rack aware topology to transfer data correctly to not use up all > bandwidth unecessarily…I am not sure mongodb has that). Anyways, just food > for thought. > > From: aaron morton <aa...@thelastpickle.com<mailto:aa...@thelastpickle.com>> > Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" > <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> > Date: Monday, February 18, 2013 1:39 PM > To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" > <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>, Vegard Berget > <p...@fantasista.no<mailto:p...@fantasista.no>> > Subject: Re: cassandra vs. mongodb quick question > > My experience is repair of 300GB compressed data takes longer than 300GB of > uncompressed, but I cannot point to an exact number. Calculating the > differences is mostly CPU bound and works on the non compressed data. > > Streaming uses compression (after uncompressing the on disk data). > > So if you have 300GB of compressed data, take a look at how long repair takes > and see if you are comfortable with that. You may also want to test replacing > a node so you can get the procedure documented and understand how long it > takes. > > The idea of the soft 300GB to 500GB limit cam about because of a number of > cases where people had 1 TB on a single node and they were surprised it took > days to repair or replace. If you know how long things may take, and that > fits in your operations then go with it. > > Cheers > > ----------------- > Aaron Morton > Freelance Cassandra Developer > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 18/02/2013, at 10:08 PM, Vegard Berget > <p...@fantasista.no<mailto:p...@fantasista.no>> wrote: > > > > Just out of curiosity : > > When using compression, does this affect this one way or another? Is 300G > (compressed) SSTable size, or total size of data? > > .vegard, > > ----- Original Message ----- > From: > user@cassandra.apache.org<mailto:user@cassandra.apache.org> > > To: > <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> > Cc: > > Sent: > Mon, 18 Feb 2013 08:41:25 +1300 > Subject: > Re: cassandra vs. mongodb quick question > > > If you have spinning disk and 1G networking and no virtual nodes, I would > still say 300G to 500G is a soft limit. > > If you are using virtual nodes, SSD, JBOD disk configuration or faster > networking you may go higher. > > The limiting factors are the time it take to repair, the time it takes to > replace a node, the memory considerations for 100's of millions of rows. If > you the performance of those operations is acceptable to you, then go crazy. > > Cheers > > ----------------- > Aaron Morton > Freelance Cassandra Developer > New Zealand > > @aaronmorton > http://www.thelastpickle.com<http://www.thelastpickle.com/> > > On 16/02/2013, at 9:05 AM, "Hiller, Dean" > <dean.hil...@nrel.gov<mailto:dean.hil...@nrel.gov>> wrote: > > So I found out mongodb varies their node size from 1T to 42T per node > depending on the profile. So if I was going to be writing a lot but rarely > changing rows, could I also use cassandra with a per node size of +20T or is > that not advisable? > > Thanks, > Dean > >
