Every port has 2 costs associated with it: port itself and optical pluggable. Historically, the proportion was even bigger than 10:1, but optical people did manage to preserve their margin better. I did not try to calculate for a couple of years, but before the cost of a router port was comparable to single-mode pluggable, multi-mode was cheaper, and copper was even cheaper. (it was funny to see that a discount was asked from the networking vendor, but the biggest payment was going to the optical vendor, especially for switches)
2 hops though CLOS architecture is 4 ports and 4 pluggables – all are typically (in Telco) single-mode for unification. In the case of a chassis, the cross-bar is electrical and costs very little money. Actually, the chassis-based router has 4 ports too on the traffic path, but single-mode pluggable is only on external ports, internal ports are cost-compared to copper pluggable. Hence, the chassis-based router has a natural advantage: less cost on SFP/QSFP/etc. Strictly speaking, pluggable is not a networking vendor's business, especially for new high-speed interfaces. The actual situation is that pizza-box (even from a respected vendor, where all features are available) is cheaper than chassis-based (per port) is attributed to non-technical reasons, primarily bigger competition. Pay attention that “Modern Switch” is just a package for “Optical Pluggables”. Hence, DC people are making very strange designs (with a lot of compromises, typically oversubscription) to downgrade multi-mode pluggable to copper pluggable. Then claim it as a big achievement. The money is there. PS: I agree (with Tom) that the feature list is important even for the pizza box. Eduard From: NANOG <nanog-bounces+vasilenko.eduard=huawei....@nanog.org> On Behalf Of Tom Beecher Sent: Tuesday, December 24, 2024 18:12 To: David Sinn <ds...@uw.edu> Cc: NANOG <nanog@nanog.org> Subject: Re: Distributed Router Fabrics Much of this is right, but again with caveats. - The boxes are fungible, to a point. Differences in ASICs, buffers, etc can really create traffic problems if you mix wrong. You don't want to be yolo'ing whatever is cheap this month in there. - You're going to eventually have a feature need that commercial management software doesn't account for. Can they build it for you, and how much is that? If you built your own software to manage it, how much does it cost you to build it? - You're very correct about how initial mistakes or things you didn't know can bite you hard later. The wrong growing pain can really hurt if you're not prepared for it. - Really have to think about the internals and the design. There are some companies who have presented on how they built these things, and when you listen to their protocol design, it makes your head hurt how much overcomplication was built in. Like I said before, distributed fabrics CAN be amazing, but there are always tradeoffs. There are some things you don't have to care about with a big chassis, but you do with a DF. And the other way around as well. It's about picking which set of things you WANT to deal with, or are better for you to deal with than the other. On Tue, Dec 24, 2024 at 9:50 AM David Sinn <ds...@uw.edu<mailto:ds...@uw.edu>> wrote: From experience I can tell you that once you fully operationalize the pizza box model you will never go back to the chassis model. Why would you trade, open, standards based model for interconnect (OSPF and BGP work great at scale) for proprietary black boxes that do stupid router tricks to make a bunch of discrete components pretend to be one along with giving you the benefit of a huge blast-radius when the software inevitably breaks? Distributed ARP/ND, solved. Actually distributed BFD (not "it's all running on one line card because customers like LACP bundles spread between line cards and that's really hard to distribute reliably), solved. Pizza box models means the boxes are fungible. So you can competitively bid between multiple suppliers and pick and choose who you want to buy from depending on what is the most important thing at the time (delivery dates? price? which of them is annoying you the least at that moment in time?). They are also infinitely more scaleable(*) than any big chassis model. State of the art 5 years ago had Internet edge systems deploying with 8k of 400G ports and datacenter deployments with 65k 400G ports using the same fundamental design. The real downside: vendors don't like the flexibility that it affords the customer and the meaninglessness of differentiation between vendors that it drives the operators to avoid. David (*) - Among some of the critical things to get right from the outset is what peak scale you want to have for the fabric because recabling is not something to be taken lightly... On Dec 23, 2024, at 7:15 AM, Mike Hammett <na...@ics-il.net<mailto:na...@ics-il.net>> wrote: "Obviously, at some point, buying a big chassis..." Actually, as I read more about it and watch more videos about it, it seems like that isn't necessarily true. The claims they have at the top end surpass what any chassis platform I've seen is capable of, though I don't know that they actually have pushed the upper bounds of what's possible in the real world. ----- Mike Hammett Intelligent Computing Solutions<https://urldefense.com/v3/__http:/www.ics-il.com/__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0E8DBeZtQ$> [http://www.ics-il.com/images/fbicon.png]<https://urldefense.com/v3/__https:/www.facebook.com/ICSIL__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0HlU5g15g$>[http://www.ics-il.com/images/googleicon.png]<https://urldefense.com/v3/__https:/plus.google.com/*IntelligentComputingSolutionsDeKalb__;Kw!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0Fd4VS5Aw$>[http://www.ics-il.com/images/linkedinicon.png]<https://urldefense.com/v3/__https:/www.linkedin.com/company/intelligent-computing-solutions__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0HbHzd3kA$>[http://www.ics-il.com/images/twittericon.png]<https://urldefense.com/v3/__https:/twitter.com/ICSIL__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0Fj7RvhHQ$> Midwest Internet Exchange<https://urldefense.com/v3/__http:/www.midwest-ix.com/__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0Gzrk6nZQ$> [http://www.ics-il.com/images/fbicon.png]<https://urldefense.com/v3/__https:/www.facebook.com/mdwestix__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0F4YUnLMg$>[http://www.ics-il.com/images/linkedinicon.png]<https://urldefense.com/v3/__https:/www.linkedin.com/company/midwest-internet-exchange__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0HgozL-IA$>[http://www.ics-il.com/images/twittericon.png]<https://urldefense.com/v3/__https:/twitter.com/mdwestix__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0GiPThd0g$> The Brothers WISP<https://urldefense.com/v3/__http:/www.thebrotherswisp.com/__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0GRvdikNA$> [http://www.ics-il.com/images/fbicon.png]<https://urldefense.com/v3/__https:/www.facebook.com/thebrotherswisp__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0FzIY3xBA$>[http://www.ics-il.com/images/youtubeicon.png]<https://urldefense.com/v3/__https:/www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0GJBCPlCg$> ________________________________ From: "Mike Hammett" <na...@ics-il.net<mailto:na...@ics-il.net>> To: "NANOG" <nanog@nanog.org<mailto:nanog@nanog.org>> Sent: Friday, December 20, 2024 10:06:36 AM Subject: Distributed Router Fabrics I've noticed that the whitebox hardware vendors are pushing distributed router fabrics, where you can keep buying pizza boxes and hooking them into a larger and larger fabric. Obviously, at some point, buying a big chassis makes more sense. Does it make sense building up to that point? What are your thoughts on that direction? ----- Mike Hammett [ http://www.ics-il.com/<https://urldefense.com/v3/__http:/www.ics-il.com/__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0E8DBeZtQ$> | Intelligent Computing Solutions ] [ https://www.facebook.com/ICSIL<https://urldefense.com/v3/__https:/www.facebook.com/ICSIL__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0HlU5g15g$> ] [ https://plus.google.com/+IntelligentComputingSolutionsDeKalb<https://urldefense.com/v3/__https:/plus.google.com/*IntelligentComputingSolutionsDeKalb__;Kw!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0Fd4VS5Aw$> ] [ https://www.linkedin.com/company/intelligent-computing-solutions<https://urldefense.com/v3/__https:/www.linkedin.com/company/intelligent-computing-solutions__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0HbHzd3kA$> ] [ https://twitter.com/ICSIL<https://urldefense.com/v3/__https:/twitter.com/ICSIL__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0Fj7RvhHQ$> ] [ http://www.midwest-ix.com/<https://urldefense.com/v3/__http:/www.midwest-ix.com/__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0Gzrk6nZQ$> | Midwest Internet Exchange ] [ https://www.facebook.com/mdwestix<https://urldefense.com/v3/__https:/www.facebook.com/mdwestix__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0F4YUnLMg$> ] [ https://www.linkedin.com/company/midwest-internet-exchange<https://urldefense.com/v3/__https:/www.linkedin.com/company/midwest-internet-exchange__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0HgozL-IA$> ] [ https://twitter.com/mdwestix<https://urldefense.com/v3/__https:/twitter.com/mdwestix__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0GiPThd0g$> ] [ http://www.thebrotherswisp.com/<https://urldefense.com/v3/__http:/www.thebrotherswisp.com/__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0GRvdikNA$> | The Brothers WISP ] [ https://www.facebook.com/thebrotherswisp<https://urldefense.com/v3/__https:/www.facebook.com/thebrotherswisp__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0FzIY3xBA$> ] [ https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg<https://urldefense.com/v3/__https:/www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0GJBCPlCg$> ]
