KanPlus opened a new issue, #3431: URL: https://github.com/apache/brpc/issues/3431
## Summary In bRPC RDMA, **all connections' send-serialization buffers and receive pre-posted buffers come from a single process-wide registered memory pool** (`AllocBlock`, `block_pool.cpp:426`), pinned via `ibv_reg_mr`. Unlike TCP (independent, pageable, kernel-managed per-socket buffers), this creates a **structural send/recv coupling** that can cause a **silent, whole-pool stall** under large-scale, mixed-load, under-provisioned deployments. ## Root cause (with code pointers) - Global IOBuf allocator is replaced by the registered pool: `rdma_helper.cpp:578`. - Send buffer is released only after the peer ACKs (IMM): `rdma_endpoint.cpp:965`. The peer can only receive if it has **already posted a recv buffer** — which also comes from the same pool. - On pool exhaustion, `PostRecv(1)` fails (`rdma_endpoint.cpp:983`, `AllocBlockFrom` returns NULL at `block_pool.cpp:374`). - Cycle: pool exhausted → recv re-post fails → no one can receive → in-flight sends never complete → ACK never returns → send buffers (same pool) never freed → pool never recovers → **all connections sharing the pool stall**. - The per-connection credit window (`_window_size`) is congestion control only and is **independent** of `AllocBlock`; it provides no memory isolation. ## When it happens (scenarios) Three conditions together: 1. **Mixed load** — large messages (64KB/2MB blocks) and small RPCs (8KB blocks) share one pool. 2. **High fan-out** — many concurrent connections under load. 3. **Under-provisioned pool** — especially since, by **default `buckets=4`**, the 8KB pool (`block_type 0`, carrying all normal RPC send/recv buffers) **cannot grow at runtime** (`CanExtendBlockRuntime`, `block_pool.cpp:139-141` requires `buckets==1` or `region_num<1`; `InitBlockPool` already set `region_num[0]=1`). It is locked at the initial 1GB. Typical deployments at risk: gateways/proxies mixing big transfers and small control RPCs, multi-tenant access layers, long-lived large-message streams. ## Impact - **Whole-pool stall, not single-request failure.** All RDMA connections sharing the pool hang simultaneously (availability-level). - **Silent and hard to diagnose.** Only occasional `Fail to extend region` / `Fail to allocate rbuf` in logs — no "shared-pool deadlock" signal. Easy to misattribute to network/kernel (#3202 showed `ibv_post_send: Cannot allocate memory`, rooted in kernel SQ memory, while the block pool had already triggered extend). - **No per-connection fairness/isolation.** `AllocBlockFrom` is LIFO + per-thread TLS hoarding + random bucket, biasing toward recently active (large-flow) threads. ## Questions for the community 1. Is the send/recv shared-pool coupling at scale considered expected behavior, or a real reliability risk? 2. Should docs state explicit RDMA **applicable / not-applicable** boundaries for large-scale mixed load? 3. Should `CanExtendBlockRuntime`'s `buckets==1` restriction be relaxed for `block_type 0` so the 8KB pool can grow under default config? 4. Should a per-direction (at least receive-side) reserved/guaranteed memory mechanism be introduced to avoid recv refill being starved by sends? 5. Should docs (`max_regions` default 16 vs code 3; `buckets` restriction) be aligned with code? *Static source analysis only; recommend a repro with mixed large/small flows + high fan-out + gradually shrinking pool.* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
