If you need that kind of density, I recommend a Clos fabric. Arista, Juniper,
Brocade, Big Switch BCF and Cisco all have solutions that would allow you to
build a high-density leaf/spine. You can build the Cisco solution with NXOS or
ACI, depending which models you choose. The prices on these solutions are all
somewhat in the same ballpark based on list pricing I've seen... even Cisco
(the Nexus 9k is surprisingly in the same range as branded whitebox). There is
also Pluribus which offers a fabric, but their niche is having server procs on
board the switches and it seems like your project involves physical rather than
virtual servers. Still, the Pluribus could be used without taking advantage of
the on board server compute I suppose.
I also recommend looking into a solution that supports VXLAN (or GENEVE, or
whatever overlay works for your needs) simply because MAC is carried in Layer-3
so you won't have to deal with spanning tree or monstrous mac tables. But you
don't need to do an overlay if you just segment with traditional VLANs.
I'm guessing you don't need HA (A/B uplinks utilizing LACP) for these servers?
Also, do you need line rate forwarding? Having 1,000 devices with 1Gb uplinks
doesn't necessarily mean that full throughput is required... the clustering and
the applications may be sporadic and bursty? I have seen load-testing clusters,
hadoop and data warehousing pushing high volumes but the individual NICs in the
clusters never actually hit capacity... If you need line-rate, then you need to
do a deep dive with several of the vendors because there are significant
differences in buffers on some models.
And... what support do you need? Just one spare on the shelf or full vendor
support on every switch? That will impact which vendor you choose.
I'd like to hear more about this effort once you get it going. Which vendor you
went with, how you tuned it, and why you selected who you did. Also, how it
works.
LFoD
> Date: Sun, 10 May 2015 01:17:07 +0000
> From: jo...@iecc.com
> To: nanog@nanog.org
> Subject: Re: Thousands of hosts on a gigabit LAN, maybe not
>
> In article
> <cahf3uwypqn1ns_umjz-znuk3i5ufczbu9l39b-crovg6yum...@mail.gmail.com> you
> write:
> >Juniper OCX1100 have 72 ports in 1U.
>
> Yeah, too bad it costs $32,000. Other than that it'd be perfect.
>
> R's,
> John
