For a variety of reasons, we have clusters with 5 TB of disk per host as a “standard.” In our larger data clusters, it does take longer to add/remove nodes or do things like upgradesstables after an upgrade. These nodes have 3+TB of actual data on the drive. But, we were able to shrink the node count from our days of using 1 or 2 TB of disk. Lots of potential cost tradeoffs to consider – licensing/support, server cost, maintenance time, more or less servers to have failures, number of (expensive?!) switch ports used, etc.
NOTE: this is 3.x experience, not 4.x with faster streaming. Sean R. Durity INTERNAL USE From: Joe Obernberger <joseph.obernber...@gmail.com> Sent: Thursday, August 17, 2023 10:46 AM To: user@cassandra.apache.org Subject: [EXTERNAL] Re: Big Data Question Thanks for this - yeah - duh - forgot about replication in my example! So - is 2TBytes per Cassandra instance advisable? Better to use more/less? Modern 2u servers can be had with 24 3. 8TBtyte SSDs; so assume 80Tbytes per server, you could Thanks for this - yeah - duh - forgot about replication in my example! So - is 2TBytes per Cassandra instance advisable? Better to use more/less? Modern 2u servers can be had with 24 3.8TBtyte SSDs; so assume 80Tbytes per server, you could do: (1024*3)/80 = 39 servers, but you'd have to run 40 instances of Cassandra on each server; maybe 24G of heap per instance, so a server with 1TByte of RAM would work. Is this what folks would do? -Joe On 8/17/2023 9:13 AM, Bowen Song via user wrote: > Just pointing out the obvious, for 1PB of data on nodes with 2TB disk > each, you will need far more than 500 nodes. > > 1, it is unwise to run Cassandra with replication factor 1. It usually > makes sense to use RF=3, so 1PB data will cost 3PB of storage space, > minimal of 1500 such nodes. > > 2, depending on the compaction strategy you use and the write access > pattern, there's a disk space amplification to consider. For example, > with STCS, the disk usage can be many times of the actual live data size. > > 3, you will need some extra free disk space as temporary space for > running compactions. > > 4, the data is rarely going to be perfectly evenly distributed among > all nodes, and you need to take that into consideration and size the > nodes based on the node with the most data. > > 5, enough of bad news, here's a good one. Compression will save you (a > lot) of disk space! > > With all the above considered, you probably will end up with a lot > more than the 500 nodes you initially thought. Your choice of > compaction strategy and compression ratio can dramatically affect this > calculation. > > > On 16/08/2023 16:33, Joe Obernberger wrote: >> General question on how to configure Cassandra. Say I have 1PByte of >> data to store. The general rule of thumb is that each node (or at >> least instance of Cassandra) shouldn't handle more than 2TBytes of >> disk. That means 500 instances of Cassandra. >> >> Assuming you have very fast persistent storage (such as a NetApp, >> PorterWorx etc.), would using Kubernetes or some orchestration layer >> to handle those nodes be a viable approach? Perhaps the worker nodes >> would have enough RAM to run 4 instances (pods) of Cassandra, you >> would need 125 servers. >> Another approach is to build your servers with 5 (or more) SSD >> devices - one for OS, four for each instance of Cassandra running on >> that server. Then build some scripts/ansible/puppet that would >> manage Cassandra start/stops, and other maintenance items. >> >> Where I think this runs into problems is with repairs, or >> sstablescrubs that can take days to run on a single instance. How is >> that handled 'in the real world'? With seed nodes, how many would >> you have in such a configuration? >> Thanks for any thoughts! >> >> -Joe >> >> -- This email has been checked for viruses by AVG antivirus software. https://urldefense.com/v3/__http://www.avg.com__;!!M-nmYVHPHQ!JNgRIPkjVYoJBn7hBrUMEUxlXhoB0f9NUYIcGYPiexUZA5rpWWgPiLJp37dwGzdXMyMVIJJn0hzkcljb0wokF_RwMJ_g6KRPXA$<https://urldefense.com/v3/__http:/www.avg.com__;!!M-nmYVHPHQ!JNgRIPkjVYoJBn7hBrUMEUxlXhoB0f9NUYIcGYPiexUZA5rpWWgPiLJp37dwGzdXMyMVIJJn0hzkcljb0wokF_RwMJ_g6KRPXA$> ________________________________ The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
