Re: [go-nuts] great talk on database/memory trends, DST, and the weaknesses of Raft

Sat, 24 May 2025 18:24:17 -0700

That last part is what is incorrect in my opinion. Even if the network bandwidth exceeds the memory bandwidth there is the IO overhead - which means that remote memory will be slower than local memory - as the remote memory still goes through the bus plus the overhead. 

On May 24, 2025, at 8:01 PM, Jason E. Aten <j.e.a...@gmail.com> wrote:


He does claim that bandwidth for peer to peer, saying
> 800 Gbps on a single link
and https://www.tomshardware.com/news/800-gigabit-ethernet-gbe-spec-standard
seems to confirm that, if I'm reading it right.

You seem to be saying that memory is always the bottleneck. I don't think we disagree there. I'm
not sure why I/O bandwidth cannot exceed memory bandwidth. You'll still be bottle necked
on memory, but I don't think anyone is claiming otherwise. Just that it changes
designs now that, to quote the Dreier blog

> memory located remotely across a network link can now be accessed with 
> no penalty in throughput (although we’ll need the right software architecture to manage latency)

Maybe this old news to you? :) I found it eye opening.

On Sunday, May 25, 2025 at 1:52:17 AM UTC+1 robert engels wrote:
I understand, but it makes no practical sense. The 800GB/sec is the throughput the fabric, not a peer to peer rate. So yes, the network can support multiple clients at a TOTAL rate greater than the speed of an individual machine - eventually the data goes through the memory bus of a machine.

On May 24, 2025, at 7:40 PM, Jason E. Aten <j.e....@gmail.com> wrote:

In the video, https://www.infoq.com/presentations/redesign-oltp/ at 14:00 minutes in
is what I was referring to. Greef shows a graph from Roland Dreier and cites his
 blog post, see Figure 1 of that blog. He points out the flip from 2020 to 2023.

https://blog.enfabrica.net/the-next-step-in-high-performance-distributed-computing-systems-4f98f13064ac

> By 2023, DDR advanced 50% with the transition to DDR5, but PCIe 
> throughput quadrupled with a two-generation jump from Gen3 to Gen5, 
> and Ethernet throughput increased eightfold, to 800 Gbps on a single link

On Sunday, May 25, 2025 at 1:19:41 AM UTC+1 robert engels wrote:
That doesn’t make sense. Memory can’t be slower than disk - disk is either physical, or memory backed - which means its upper speed limit is the memory speed limit.

If the presentation means to imply that ethernet is so fast - faster than memory busses themselves - that you could fan out to achieve speed faster than memory - that doesn’t make help - because either it is DMA - going to disk (which is slower), or going to the CPU - which can only read memory at the speed of the memory bus.

You could theoretically achieve “ultra speeds” with multicasting, on a super fast ethernet bus - but the source is still probably limited to memory speeds.

So the abstract you provided doesn’t jive with me.

On May 24, 2025, at 7:06 PM, Jason E. Aten <j.e....@gmail.com> wrote:

This is an amazing talk from last year 2024 March 22 Qcon from
the TigerBeetle CEO, Joran Greef. Zig has lessons for Go.

"Redesigning OLTP for a New Order of Magnitude"

Early on the talks covers that latest trends in memory vs network vs disk,
(hint: at 800GB ethernet means memory is now the bottleneck, and
it is slower than disk(!)).

Greef also talks about Deterministic Simulation Testing (DST) and 
gives a highly compelling demo for why you want it.

He points out that replication algorithms like Paxos and Raft, and points
out that a single sector fault on one machine can create global 
data loss. Yikes.

(See 35 minutes in)

Again, the video at 35:00 gives a very strong critique of Raft vs VSRR

arguing raft too simple/inappropriate (assumes perfect disks)

and for actual faulty disks can get blocked and not be fault

tolerant at all. They use VSRR in TigerBeetle, plus NASA/Fortran

style "pre-allocate all memory". 


TigerBeetle is Apache 2 open source. It is written in Zig but has

lessons for software using Go throughout.  His discussion of

optimizing write compaction in LSM trees is fascinating, 

as is integrating consensus with replication (2018 best paper

at FAST from Alagappan et al), and using 

speculative replicated state machine

execution to avoid stalling on bad storage sectors -- I

expect all of these to be the future of databases.


Five stars.


Enjoy,

Jason


-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/b9aab0c5-e52e-4950-a98b-11f539c56c83n%40googlegroups.com.


-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/397f764a-29ff-43f1-b5a6-649e234e02d1n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/161CC975-3F6A-4709-8C89-2C71E0558623%40ix.netcom.com.

Reply via email to