Re: [Discuss] Raspberry PI[4,5] as infrastructure?
> ma...@mohawksoft.com wrote: >> I have a RPI5 running ZFS, PostgreSQL, a DLNA server, and a full >> development stack that compiles just about any code I have laying >> around. >> >> These things chave 8 gigs of RAM, 4 CPUs, use 15 watts of power, and >> cost >> less than a video card. My desktop is considerably bigger, but the core >> speed is only 2 or 3 times as fast as the ARM. > > My rule of thumb: humans doing normal desktop tasks (whatever is > not pushing the envelope of that computing generation) cannot > distinguish less than a 100% performance improvement. No argument there depending on what the human is doing. I'f I'm compiling a large program or doing a large query on a database, CPU performance and I/O make a difference. > > (AKA most of the time, everything is instantaneous. You only > notice when it isn't.) Hence finding the XZ exploit :-) > > ... > >> Would a stack of RPI5s, controlled by some sort of docker look-alike, >> perform better than a huge VMware server? Would it perform better than a >> large kubernetes cluster? Would they be more secure because they are >> physically separated. >> >> Thoughts? > > RPI - all of them to date - are limited by I/O. Let's suppose > the unit of computing is an 8GB RPI5, and the cost per unit to > interconnect and power and cool them is $20, so for each $100 > increment we get 4 2.4GHz cores and 8GB RAM, a GPU and two lanes > of PCIe 2.0, and another 15W. > > For some workloads this is great. If you were building a > security camera hub, for example, being able to add two cameras > worth of realtime processing for that price is very nice. > Anything where the individual tasks are reasonable but there are > a lot of them coming in seems like a good bet. > > But somewhere around the 4-8 Pi mark, it will become obvious > that you need all three of better storage, networking, and > interconnections -- and you can only solve one of those at a > time with the RPI5. The coordination overhead starts taking a > larger chunk of each unit. Pretty soon you run into Amdahl's > law: eventually, the non-parallelizable part of any process will > be the bottleneck. This is something I have encountered for so long. Everyone rejects out of hand things that do not scale infinitely. Is this why pick-up trucks are more like military troupe carriers than utility vehicles? I still assert, 99% of the computing that people actually need is easily handled with no specialize scaling architecture. Everyone is so hell bent on building the latest and greatest technologies that they over build their data infrastructure. Its wasteful. Yes, sure, if you are building room-size compute facilities that do an LLM or a social media platform, then yea, go for it, but most of the people who build in the cloud don't do anything like this. I won't call it a niche because it is a VERY big segment of the compute market, but my assertion is that a Raspberry PI5 would more than suffice for a company file server, database, internal infrastructure, etc. Think about your local supermarket. All the POS terminals could easily be handled by a raspberry PI and a USB-3 disk bay. ___ Discuss mailing list Discuss@driftwood.blu.org https://driftwood.blu.org/mailman/listinfo/discuss
Re: [Discuss] Raspberry PI[4,5] as infrastructure?
> On Sun, 9 Jun 2024 12:37:20 -0400 > ma...@mohawksoft.com wrote: > >> Would a stack of RPI5s, controlled by some sort of docker look-alike, >> perform better than a huge VMware server? Would it perform better >> than a large kubernetes cluster? Would they be more secure because >> they are physically separated. > > Not at scale. My employer's current base for our ESX cluster hardware > refreshes this year are Dell R650 rack servers with dual top of the > line Xeon processors and 1TB RAM each; redundant 25 Gbit/s Ethernet; > redundant 32 Gbit/s fibre for connection to the Unity storage system. > Minimum of 3 machines per cluster; with clusters typically 5 to 15 > machines. > > Raspberries Pi can't compete with this. Yes, that is an impressive amount of compute infrastructure, and 1TB ram and 25g ethernet is impressive. If you need that, sure, but. Here's the thing. What are you going to use it for? Did you ever see the movie "Money Ball?" Your whole design is kind of what I was pointing out. How many CPU cores do you have? Say you have 144 cores at 2.2ghz (3ghz turbo) per processor, with 288 cores total. What is your memory band-width? Is the system NUMA because I doubt the processor and motherboard are SMP. Now, that architecture will not scale 1:1 because with NUMA different ranges split between the two processors. Yes, Linux does a good job managing this these days, but it is not perfect. I assume since it is going to be an ESX server, it will be sub-divided across a number of virtual machines partitioned for in a way that each VM is configured for a specific work-load. Eight? That means you fibre and Ethernet will be divided across the various VMs. How much does each of those Xeon processors cost? $5K? How much is the RAM? How much is the server? How much power does it consume? What is your max work-load configuration? What is your typical? 72 Raspberry PI5s will cost $7500, have over 1/2 TB RAM, consume about 1KW power. etc. etc. There is a lot of compute power in each of your servers, but there are a hell of a lot of inefficiencies. Virtual machines are inefficient. Kernel scheduling and aligning processors to the VM schedule reduces physical CPU to virtual CPU throughput. A lot of virtual machines fighting for Fibre or ethernet is inefficient. If you are a cloud provider or something like that, great, you can budget this in. I don't know what business to which your servers are targeted, but I don't think it is all that common in the greater scheme of things. I think the days of these systems are numbered. They are very expensive, consume a lot of power, do not use resources (CPU/Memory) very efficiently. Not only that, the redundancy is fairly low. > > -- > \m/ (--) \m/ > ___ > Discuss mailing list > Discuss@driftwood.blu.org > https://driftwood.blu.org/mailman/listinfo/discuss > ___ Discuss mailing list Discuss@driftwood.blu.org https://driftwood.blu.org/mailman/listinfo/discuss
Re: [Discuss] Raspberry PI[4,5] as infrastructure?
On 6/9/24 10:14, Dan Ritter wrote: RPI - all of them to date - are limited by I/O. Yes, a serious limitation is their poor bandwidth to RAM. In an Intel box a lot of effort usually goes into making that high bandwidth. Servers also support ECC RAM, and even consumer boxes frequently allow installing matched pairs of RAM sticks for more bandwidth. Arm machines can do all of this, but a Raspberry Pi does not do any of it. -kb ___ Discuss mailing list Discuss@driftwood.blu.org https://driftwood.blu.org/mailman/listinfo/discuss
Re: [Discuss] Raspberry PI[4,5] as infrastructure?
On 6/10/24 06:00, ma...@mohawksoft.com wrote: (AKA most of the time, everything is instantaneous. You only notice when it isn't.) Hence finding the XZ exploit :-) Good point. -kb ___ Discuss mailing list Discuss@driftwood.blu.org https://driftwood.blu.org/mailman/listinfo/discuss
Re: [Discuss] Raspberry PI[4,5] as infrastructure?
On Sun, 9 Jun 2024 13:52:49 -0700 Kent Borg wrote: > Arm chips are starting to make inroads into real server farms, > because they offer more performance per watt than do Intel and AMD > chips, and I suspect also because they can scale their power usage so > well. But, though they are getting pretty fast, are still slower. You can get proper ARM servers: rack-mounted systems with all the trimmings you would expect from server class hardware: remote management, ECC RAM, PCIe expansion for high speed network and fibre storage connectivity, etc. We have about a dozen standalone ARM servers for product development and testing. They're mostly used as Docker hosts, and they work well enough for that. Good for when performance per watt is a higher priority than performance, but yeah, the Intel servers handily out-perform the best ARM has to offer in terms of raw performance. -- \m/ (--) \m/ ___ Discuss mailing list Discuss@driftwood.blu.org https://driftwood.blu.org/mailman/listinfo/discuss
