On Fri, May 8, 2015 at 11:53 AM, John Levine <jo...@iecc.com> wrote: > Some people I know (yes really) are building a system that will have > several thousand little computers in some racks.
Very cool-ly crazy. > Each of the > computers runs Linux and has a gigabit ethernet interface. It occurs > to me that it is unlikely that I can buy an ethernet switch with > thousands of ports, and even if I could, would I want a Linux system > to have 10,000 entries or more in its ARP table. Agreed. :) You don't really want 10,000 entries in a routing FIB table either, but I was seriously encouraged by the work going on in linux 4.0 and 4.1 to improve those lookups. https://netdev01.org/docs/duyck-fib-trie.pdf I'd love to know the actual scalability of some modern routing protocols (isis, babel, ospfv3, olsrv2, rpl) with that many nodes too.... > Most of the traffic will be from one node to another, with > considerably less to the outside. Physical distance shouldn't be a > problem since everything's in the same room, maybe the same rack. That is an awful lot of ports to fit in a rack (48 ports, 36 2U slots in the rack (and is that too high?) = 1728 ports) A thought is you could make it meshier using multiple interfaces per tiny linux box? Put, say 3-6 interfaces and have a very few switches interconnecting given clusters (and multiple paths to each switch). That would reduce your arp table (and fib table) by a lot at the cost of adding hops... > What's the rule of thumb for number of hosts per switch, cascaded > switches vs. routers, and whatever else one needs to design a dense > network like this? TIA max per vlan 4096. Still a lot. Another approach might be max density on a switch (48?) per cluster, routed (not switched) 10GigE to another 10GigE+ switch. I'd love to know the rule of thumbs here also, I imagine some rules must exist for those in the VM or VXLAN worlds. > R's, > John -- Dave Täht Open Networking needs **Open Source Hardware** https://plus.google.com/u/0/+EricRaymond/posts/JqxCe2pFr67
