He does claim that bandwidth for peer to peer, saying > 800 Gbps on a single link and https://www.tomshardware.com/news/800-gigabit-ethernet-gbe-spec-standard seems to confirm that, if I'm reading it right.
You seem to be saying that memory is always the bottleneck. I don't think we disagree there. I'm not sure why I/O bandwidth cannot exceed memory bandwidth. You'll still be bottle necked on memory, but I don't think anyone is claiming otherwise. Just that it changes designs now that, to quote the Dreier blog > memory located remotely across a network link can now be accessed with > no penalty in throughput (although we’ll need the right software architecture to manage latency) Maybe this old news to you? :) I found it eye opening. On Sunday, May 25, 2025 at 1:52:17 AM UTC+1 robert engels wrote: I understand, but it makes no practical sense. The 800GB/sec is the throughput the fabric, not a peer to peer rate. So yes, the network can support multiple clients at a TOTAL rate greater than the speed of an individual machine - eventually the data goes through the memory bus of a machine. On May 24, 2025, at 7:40 PM, Jason E. Aten <j.e....@gmail.com> wrote: In the video, https://www.infoq.com/presentations/redesign-oltp/ at 14:00 minutes in is what I was referring to. Greef shows a graph from Roland Dreier and cites his blog post, see Figure 1 of that blog. He points out the flip from 2020 to 2023. https://blog.enfabrica.net/the-next-step-in-high-performance-distributed-computing-systems-4f98f13064ac > By 2023, DDR advanced 50% with the transition to DDR5, but PCIe > throughput quadrupled with a two-generation jump from Gen3 to Gen5, > and Ethernet throughput increased eightfold, to 800 Gbps on a single link On Sunday, May 25, 2025 at 1:19:41 AM UTC+1 robert engels wrote: That doesn’t make sense. Memory can’t be slower than disk - disk is either physical, or memory backed - which means its upper speed limit is the memory speed limit. If the presentation means to imply that ethernet is so fast - faster than memory busses themselves - that you could fan out to achieve speed faster than memory - that doesn’t make help - because either it is DMA - going to disk (which is slower), or going to the CPU - which can only read memory at the speed of the memory bus. You could theoretically achieve “ultra speeds” with multicasting, on a super fast ethernet bus - but the source is still probably limited to memory speeds. So the abstract you provided doesn’t jive with me. On May 24, 2025, at 7:06 PM, Jason E. Aten <j.e....@gmail.com> wrote: This is an amazing talk from last year 2024 March 22 Qcon from the TigerBeetle CEO, Joran Greef. Zig has lessons for Go. "Redesigning OLTP for a New Order of Magnitude" https://www.infoq.com/presentations/redesign-oltp/ Early on the talks covers that latest trends in memory vs network vs disk, (hint: at 800GB ethernet means memory is now the bottleneck, and it is slower than disk(!)). Greef also talks about Deterministic Simulation Testing (DST) and gives a highly compelling demo for why you want it. He points out that replication algorithms like Paxos and Raft, and points out that a single sector fault on one machine can create global data loss. Yikes. (See 35 minutes in) Again, the video at 35:00 gives a very strong critique of Raft vs VSRR arguing raft too simple/inappropriate (assumes perfect disks) and for actual faulty disks can get blocked and not be fault tolerant at all. They use VSRR in TigerBeetle, plus NASA/Fortran style "pre-allocate all memory". TigerBeetle is Apache 2 open source. It is written in Zig but has lessons for software using Go throughout. His discussion of optimizing write compaction in LSM trees is fascinating, as is integrating consensus with replication (2018 best paper at FAST from Alagappan et al), and using speculative replicated state machine execution to avoid stalling on bad storage sectors -- I expect all of these to be the future of databases. Five stars. Enjoy, Jason -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/b9aab0c5-e52e-4950-a98b-11f539c56c83n%40googlegroups.com <https://groups.google.com/d/msgid/golang-nuts/b9aab0c5-e52e-4950-a98b-11f539c56c83n%40googlegroups.com?utm_medium=email&utm_source=footer> . -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/f7514604-664e-47d4-974e-8b2fee2e7e20n%40googlegroups.com <https://groups.google.com/d/msgid/golang-nuts/f7514604-664e-47d4-974e-8b2fee2e7e20n%40googlegroups.com?utm_medium=email&utm_source=footer> . -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/397f764a-29ff-43f1-b5a6-649e234e02d1n%40googlegroups.com.
