jimmystewpot opened a new pull request, #3752:
URL: https://github.com/apache/brpc/pull/3752
### What problem does this PR solve?
Issue Number:
Problem Summary:
This issue was discovered while profiling StarRocks (which relies heavily on
`brpc` / `bthread` / `butil` for RPC and concurrency) on ARM64 servers, where
CPU profiles showed significant time and cache-coherency stalls spent inside
legacy `ldxr`/`stxr` loops and unconditional `dmb ish` memory barriers.
`src/butil/atomicops_internals_arm64_gcc.h` has not been updated since its
original 2014 Chromium import and suffers from three key limitations on modern
ARM64 (`__aarch64__`) platforms:
1. **Inability to utilise ARMv8.1-A LSE or Outline Atomics:**
`NoBarrier_CompareAndSwap`, `NoBarrier_AtomicExchange`, and
`NoBarrier_AtomicIncrement` (both 32-bit and 64-bit) use literal `ldxr`/`stxr`
load-linked/store-conditional assembly loops. Because these are opaque assembly
strings to the compiler, neither GCC nor Clang can emit ARMv8.1-A Large System
Extensions (LSE) instructions (`cas`, `swp`, `ldadd`, `stadd`) when targeting
`-march=armv8-a+lse` / `-march=armv8.2-a+`, nor can they emit
runtime-dispatched `-moutline-atomics` helper calls (`__aarch64_cas*`,
`__aarch64_ldadd*`, `__aarch64_swp*`) on baseline `-march=armv8-a` builds.
2. **Redundant Full-System Memory Barriers (`dmb ish`):**
`Barrier_AtomicIncrement`, `Acquire_CompareAndSwap`, and
`Release_CompareAndSwap` wrap relaxed LL/SC loops in full `MemoryBarrier()`
(`dmb ish`) fences—even executing a full `dmb ish` barrier when
`Acquire_CompareAndSwap` fails—rather than using architectural C++11/ARMv8
acquire/release semantics (`ldaxr`/`stlxr` on ARMv8.0-A or
`casa`/`casl`/`ldaddal` on ARMv8.1-A+ LSE).
3. **Deprecated `<libkern/OSAtomic.h>` Fallback on Apple Silicon:**
In `src/butil/atomicops.h`, `#elif defined(OS_MACOSX)` precedes `#elif
defined(COMPILER_GCC) && defined(ARCH_CPU_ARM64)`, routing macOS ARM64 builds
to the deprecated `OSAtomic*` routines in `butil/atomicops_internals_mac.h`.
### What is changed and the side effects?
Changed:
- **`src/butil/atomicops_internals_arm64_gcc.h`:** Replaced all 32-bit
(`Atomic32`) and 64-bit (`Atomic64`) inline assembly blocks with GCC/Clang
`__atomic_*` built-ins (`__atomic_compare_exchange_n`, `__atomic_exchange_n`,
`__atomic_add_fetch`, `__atomic_store_n`, `__atomic_load_n`, and
`__atomic_thread_fence`), preserving Chromium's `Acquire_Store` and
`Release_Load` semantics.
- **`src/butil/atomicops.h` & `BUILD.bazel`:** Ordered `#elif
defined(COMPILER_GCC) && defined(ARCH_CPU_ARM64)` ahead of `#elif
defined(OS_MACOSX)` so Apple Silicon builds use
`atomicops_internals_arm64_gcc.h`, and exported the header in `macos_lib.hdrs`
for Bazel builds on macOS.
- **`test/atomicops_unittest.cc`:** Expanded unit test coverage to exercise
`Barrier_AtomicIncrement`, `Acquire_CompareAndSwap`, `Release_CompareAndSwap`
(both success and failure paths), explicit `Atomic64` instantiations, and
multi-threaded CAS/increment contention.
Side effects:
- Performance effects:
- **Disassembly (`objdump -d`):**
- Baseline `-march=armv8-a` (GCC 13.3 / Clang 21): 0 raw `ldxr`/`stxr`
instructions in caller object files; emits runtime-detected `-moutline-atomics`
helper calls (`__aarch64_cas*`, `__aarch64_ldadd*`, `__aarch64_swp*`).
- `-march=armv8-a+lse`: Emits inline `cas`, `casa`, `casl`, `swp`,
`ldadd`, and `ldaddal` instructions with 0 `dmb ish` barriers in
`Acquire_CompareAndSwap`, `Release_CompareAndSwap`, and
`Barrier_AtomicIncrement`.
- **Contention Benchmark (ARM Neoverse-V2, 4 threads contended, Ubuntu
24.04, GCC 13.3.0):**
| Benchmark (4 threads contended) | Before (`ldxr`/`stxr` + `dmb ish`) |
After (`-march=armv8-a` outline LSE) | After (`-march=armv8-a+lse` inline LSE) |
| :--- | :---: | :---: | :---: |
| `Barrier_AtomicIncrement` | 46.34 ns/op | **15.39 ns/op** (3.01x
faster) | **7.48 ns/op** (6.20x faster) |
| Contended CAS Spinlock (`Acquire_CAS` + `Release_Store`) | 68.01 ns/op
| **29.74 ns/op** (2.29x faster) | **32.71 ns/op** (2.08x faster) |
- Breaking backward compatibility: No. Preserves 100% API and ABI
compatibility for `butil::subtle::Atomic32`, `Atomic64`, and `AtomicWord`.
---
### Check List:
- Please make sure your changes are compilable.
- When providing us with a new feature, it is best to add related tests.
- Please follow [Contributor Covenant Code of
Conduct](https://github.com/apache/brpc/blob/master/CODE_OF_CONDUCT.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]