On Tue, Dec 24, 2024 at 2:00 PM Phil Bedard <bedard.p...@gmail.com> wrote:
> All major vendors espouse both chassis and fabrics depending on what you
> are doing. I’m typically more of a fan of fabric based models but as
> others mentioned depends what you are doing. When I say fabric I mean
> something using Ethernet and a standard control plane, not proprietary
> interconnects and fabric encapsulation which is what you see in some BRCM
> based solutions. Those are basically virtual/multi-chassis systems, which
> also have their own pros and cons. What vendor/box you choose is mostly
> dependent on the feature set you need.
>
>
>
> There is a giant list of pros and cons between traditional chassis and
> distributed “fabrics” but here are a few.
>
>
>
> Management and control plane scale can be an issue until that gets figured
> out. Doing a 1:1 replacement of traditional large chassis with fabrics can
> add a lot of routers to a network.
>
>
>
> Power depends on the chassis and power distribution design. However, you
> can build a fabric as you need, it doesn’t require the power day one like a
> traditional chassis.
>
>
>
> Upgrading chassis switch fabrics and moving/mixing generations of line
> cards is almost always a painful experience.
>
>
>
> Phil
>
Power is a *huge* part of the equation that I think many people overlook.
When you look at what a really big chassis takes in terms of power feeds,
it's not uncommon to need relatively specialized 3-phase 240V power feeds
for the very-high-end chassis box that give you the same type of high speed
port densities that a pizza-box fabric folded Clos model can yield.
(not to pick on any vendor, but here's an example of the types of power
feeds
a large chassis can require:)
"
AC Power Distribution Modules (PDMs)
The (REDACTED MODEL #) supports connection of a single-phase or three-phase
(delta or wye) AC PDM. Four AC PDM models are available: three-phase delta,
three-phase wye, seven-feed single-phase, and nine-feed single-phase.
-
Each three-phase AC PDM requires two three-phase feeds to be connected.
Each phase from each of the two feeds is distributed among one or two PSMs.
One feed has each phase going to two PSMs, and the other feed has each
phase going to a single PSM.
-
The single-phase AC PDM provides an AC input connection from the
single-phase AC power source, and also provides an input power
interface to
the PSM through a system power midplane. The single-phase AC PDMs accept
seven or nine AC power cords from a single-phase AC source.
-
Each AC input is independent and feeds one PSM. Up to nine PSMs can
be connected through the AC PDM.
Generally speaking, you're getting a licensed electrician to run
three-phase power feeds
for them, you're not going to just ask for a couple more outlets from the
colo provider, and
the chassis listed above takes 4 power distribution modules each with two
3-phase AC feeds
for a total of 8 3-phase AC connections, 4 primary and 4 secondary feeds.
That's a lot of
custom electrical work to feed your chassis, not to mention 2x12KW of
provisioned power.
By comparison, you can get a similar amount of port density and fabric
throughput with
a folded-Clos design using 1RU 24 port 400G rack switches which each
require a redundant
10amp, 120V power feed; absolutely standard, your normal rack PDU handles
them quite
well. Start with the 12 switches for your spine, add leaf switches as
needed to scale up,
and by the time you've hit the same leaf port density as the big chassis
box, you've only
provisioned up half the power as the big chassis box. Over time, that
difference in
provisioned power can make a huge difference in operational costs.
At this point, I'd be hard-pressed to find a reason to support recommending
a big single
chassis solution to anyone other than an enterprise customer that wants to
outsource
most of its network support needs to a vendor. In that model, yes, the one
big chassis
model can make sense. But for everyone else, it's seriously time to look
at the scalability
and operational cost benefits of clustered pizza boxes.
Thanks!
Matt
