Re: Thousands of hosts on a gigabit LAN, maybe not

On 10/05/2015 00:33, Karl Auer wrote: > Would be interesting to see how IPv6 performed, since is one of the > things it was supposed to be able to deliver - massively scalable links > (equivalent to an IPv4 broadcast domain) via massively reduced protocol > chatter (IPv6 multicast groups vs IPv4 br

RE: Thousands of hosts on a gigabit LAN, maybe not

Also, do you need line rate forwarding? Having 1,000 devices with 1Gb uplinks doesn't necessarily mean that full throughput is required... the clustering and the applications may be sporadic and bursty? It's definitely sporadic and bursty. There's another network for high speed traffic among

RE: Thousands of hosts on a gigabit LAN, maybe not

t will impact which vendor you choose. I'd like to hear more about this effort once you get it going. Which vendor you went with, how you tuned it, and why you selected who you did. Also, how it works. LFoD > Date: Sun, 10 May 2015 01:17:07 + > From: jo...@iecc.com > To: nanog@nano

Re: Thousands of hosts on a gigabit LAN, maybe not

On 09/05/2015 23:33, Karl Auer wrote: IPv4 ARP, for example, hits every on-subnet neighbour; the IPv6 equivalent uses multicast to hit only those neighbours that happen to share the same 24 low-end L3 address bits as the desired target - a statistically much smaller subset of on-link neighbours,

RE: Thousands of hosts on a gigabit LAN, maybe not

> Some people I know (yes really) are building a system that will have > several thousand little computers in some racks. Each of the > computers runs Linux and has a gigabit ethernet interface. It occurs > to me that it is unlikely that I can buy an ethernet switch with > thousands of ports, and

Re: Thousands of hosts on a gigabit LAN, maybe not

>To the OP please do tell us more about what you are doing, it sounds >very interesting. There's a conference paper in preparation. I'll send a pointer when I can. R's, John

Re: Thousands of hosts on a gigabit LAN, maybe not

On 2015-05-09 11:57, Baldur Norddahl wrote: The standard 48 port with 2 port uplink 1U switch is far from full depth. You put them in the back of the rack and have the small computers in the front. You might even turn the switches around, so the ports face inwards into the rack. The network cab

Re: Thousands of hosts on a gigabit LAN, maybe not

You do not mention low cost before ;) Em sábado, 9 de maio de 2015, John Levine escreveu: > In article < > cahf3uwypqn1ns_umjz-znuk3i5ufczbu9l39b-crovg6yum...@mail.gmail.com > > you write: > >Juniper OCX1100 have 72 ports in 1U. > > Yeah, too bad it costs $32,000. Other than that it'd be perf

Re: Thousands of hosts on a gigabit LAN, maybe not

In article you write: >Juniper OCX1100 have 72 ports in 1U. Yeah, too bad it costs $32,000. Other than that it'd be perfect. R's, John

Re: Thousands of hosts on a gigabit LAN, maybe not

Juniper OCX1100 have 72 ports in 1U. And you can tune Linux IPv4 neighbor: https://ams-ix.net/technical/specifications-descriptions/config-guide#11 -- Eduardo Schoedler Em sábado, 9 de maio de 2015, Lamar Owen escreveu: > On 05/08/2015 02:53 PM, John Levine wrote: > >> ... >> Most of the tra

Re: Thousands of hosts on a gigabit LAN, maybe not

On Sat, 2015-05-09 at 17:06 -0400, Lamar Owen wrote: > The effective limit on subnet size would be of course broadcast > overhead; 1,000 nodes on a /22 would likely be painfully slow due to > broadcast overhead alone. Would be interesting to see how IPv6 performed, since is one of the things it

Re: Thousands of hosts on a gigabit LAN, maybe not

On 05/08/2015 02:53 PM, John Levine wrote: ... Most of the traffic will be from one node to another, with considerably less to the outside. Physical distance shouldn't be a problem since everything's in the same room, maybe the same rack. What's the rule of thumb for number of hosts per switch,

Re: Thousands of hosts on a gigabit LAN, maybe not

The standard 48 port with 2 port uplink 1U switch is far from full depth. You put them in the back of the rack and have the small computers in the front. You might even turn the switches around, so the ports face inwards into the rack. The network cables would be very short and go directly from the

RE: Thousands of hosts on a gigabit LAN, maybe not

On 2015-05-08 18:20, Phil Bedard wrote: The real answer to this is being able to cram them into a single chassis which can multiplex the network through a backplane. Something like the HP Moonshot ARM system or the way others like Google build high density compute with integrated Ethernet switchi

Re: Thousands of hosts on a gigabit LAN, maybe not

On 9 May 2015, at 1:53, John Levine wrote: What's the rule of thumb for number of hosts per switch, cascaded switches vs. routers, and whatever else one needs to design a dense network like this? Most of the major switch vendors have design guides and other examples like this available (thi

Re: Thousands of hosts on a gigabit LAN, maybe not

On Fri, May 8, 2015 at 5:19 PM, Jima wrote: Dang. The more I think about this project, the more expensive it sounds. Naw, just use WiFi. ;) -- Joe Hamelin, W7COM, Tulalip, WA, 360-474-7474

Re: Thousands of hosts on a gigabit LAN, maybe not

On 2015-05-08 12:53, John Levine wrote: What's the rule of thumb for number of hosts per switch, cascaded switches vs. routers, and whatever else one needs to design a dense network like this? TIA I won't pretend to know best practices, but my inclination would be to connect the devices to 4

Re: Thousands of hosts on a gigabit LAN, maybe not

On Fri, May 8, 2015 at 11:53 AM, John Levine wrote: > Some people I know (yes really) are building a system that will have > several thousand little computers in some racks. Each of the > computers runs Linux and has a gigabit ethernet interface. Though a bit off-topic I ran in to this project

RE: Thousands of hosts on a gigabit LAN, maybe not

The real answer to this is being able to cram them into a single chassis which can multiplex the network through a backplane. Something like the HP Moonshot ARM system or the way others like Google build high density compute with integrated Ethernet switching. Phil -Original Message-

Re: Thousands of hosts on a gigabit LAN, maybe not

On 2015-05-08 13:53, John Levine wrote: Some people I know (yes really) are building a system that will have several thousand little computers in some racks. How many racks? How many computers per rack unit? How many computers per rack? (How are you handling power?) How big is each computer?

Re: Thousands of hosts on a gigabit LAN, maybe not

On 05/08/2015 04:17 PM, Niels Bakker wrote: * lists.na...@monmotha.net (Brandon Martin) [Fri 08 May 2015, 21:42 CEST]: [1] Purely as an example, you can cram 3x Brocade MLX-16 chassis into a 42U rack (with 0RU to spare). That gives you 48 slots for line cards. You really can't. Cables need t

Re: Thousands of hosts on a gigabit LAN, maybe not

Morrow's comment about the ARMD WG notwithstanding, there might be some useful context in https://tools.ietf.org/html/draft-karir-armd-statistics-01 Cheers, -Benson Christopher Morrow May 8, 2015 at 12:19 PM consider the pain of also ipv6's link-local gamery.

RE: Thousands of hosts on a gigabit LAN, maybe not

The first thing that came to mind was "Bitcoin farm!" then "Ask Bitmaintech" and then "I'd be more worried about the number of fans and A/C units". I promise, no bitcoins involved. R's, John

Re: Thousands of hosts on a gigabit LAN, maybe not

* lists.na...@monmotha.net (Brandon Martin) [Fri 08 May 2015, 21:42 CEST]: [1] Purely as an example, you can cram 3x Brocade MLX-16 chassis into a 42U rack (with 0RU to spare). That gives you 48 slots for line cards. You really can't. Cables need to come from the top, not from the sides, or

RE: Thousands of hosts on a gigabit LAN, maybe not

Agree with many of the other comments. Smaller subnets (the /23 suggestion sounds good) with L3 between the subnets. The first thing that came to mind was "Bitcoin farm!" then "Ask Bitmaintech" and then "I'd be more worried about the number of fans and A/C units". Brian > Date: Fri, 8 M

Re: Thousands of hosts on a gigabit LAN, maybe not

>> to have 10,000 entries or more in its ARP table. > >Agreed. :) You don't really want 10,000 entries in a routing FIB >table either, but I was seriously encouraged by the work going >on in linux 4.0 and 4.1 to improve those lookups. One obvious way to deal with that is to put some manageable num

Re: Thousands of hosts on a gigabit LAN, maybe not

Forgot to mention - you might also want to check out Beowulf clusters - there's an email list at http://www.beowulf.org/ - probably some useful info in the list archives, maybe a good place to post your query. Miles Miles Fidelman wrote: John Levine wrote: Some people I know (yes really) are

RE: Thousands of hosts on a gigabit LAN, maybe not

You may want to look at CLOS / leaf/spine architecture. This design tends to be optimized for east-west traffic, scales easily as bandwidth needs grow, and keeps thing simple, l2/l3 boundry on the ToR switch, L3 ECMP from leaf to spine. Not a lot of complexity and scale fairly high on both lea

Re: Thousands of hosts on a gigabit LAN, maybe not

Linux has a (configurable) limit on the neighbor table. I know in RHEL variants, the default has been 1024 neighbors for a while. net.ipv4.neigh.default.gc_thresh3 net.ipv4.neigh.default.gc_thresh2 net.ipv4.neigh.default.gc_thresh1 net.ipv6.neigh.default.gc_thresh3 net.ipv6.neigh.default.gc_thr

Re: Thousands of hosts on a gigabit LAN, maybe not

John Levine wrote: Some people I know (yes really) are building a system that will have several thousand little computers in some racks. Each of the computers runs Linux and has a gigabit ethernet interface. It occurs to me that it is unlikely that I can buy an ethernet switch with thousands of

Re: Thousands of hosts on a gigabit LAN, maybe not

On 05/08/2015 02:53 PM, John Levine wrote: Some people I know (yes really) are building a system that will have several thousand little computers in some racks. Each of the computers runs Linux and has a gigabit ethernet interface. It occurs to me that it is unlikely that I can buy an ethernet

Re: Thousands of hosts on a gigabit LAN, maybe not

- The more switches a packet has to go through, the higher the latency, so your response times may deteriorate if you cascade too many switches. Legend says up to 4 is a good number, any further you risk creating a big mess. - The more switches you add, the higher your bandwidth utilized by broadc

Re: Thousands of hosts on a gigabit LAN, maybe not

On Fri, May 8, 2015 at 11:53 AM, John Levine wrote: > Some people I know (yes really) are building a system that will have > several thousand little computers in some racks. Very cool-ly crazy. > Each of the > computers runs Linux and has a gigabit ethernet interface. It occurs > to me that it

Re: Thousands of hosts on a gigabit LAN, maybe not

On Fri, May 8, 2015 at 2:53 PM, John Levine wrote: > Some people I know (yes really) are building a system that will have > several thousand little computers in some racks. Each of the > computers runs Linux and has a gigabit ethernet interface. It occurs > to me that it is unlikely that I can b

RE: Thousands of hosts on a gigabit LAN, maybe not

Sounds interesting. I wouldn't do more than a /23 (assuming IPv4) per subnet. Join them all together with a fast L3 switch. I'm still trying to visualize what several thousand tiny computers in a single rack might look like. Other than a cabling nightmare. 1000 RJ-45 switch ports is a good