| Issue |
208405
|
| Summary |
[OpenMP][libomp] Default barrier spin-wait emits many sched_yield calls in load-imbalanced fork/join workloads — expected?
|
| Labels |
new issue
|
| Assignees |
Saieiei
|
| Reporter |
Saieiei
|
### Summary / question
In a fork/join-heavy workload with load-imbalanced parallel regions, LLVM `libomp`'s default wait policy (`KMP_BLOCKTIME=200ms`) makes threads that finish a region early spin-wait at the implicit barrier and issue many `sched_yield` calls.
Setting `KMP_BLOCKTIME=0` or `OMP_WAIT_POLICY=passive` removes the `sched_yield` calls by converting the wait behavior into futex-heavy blocking, but elapsed time does not improve and can worsen.
Is this expected for this workload shape? Is there recommended runtime guidance for imbalanced fork/join-heavy codes? I am asking as an observation/question, not claiming a defect.
### Reproducer
Attached: `libomp_forkjoin_public_issue.tar.gz`
[libomp_forkjoin_public_issue.tar.gz](https://github.com/user-attachments/files/29842778/libomp_forkjoin_public_issue.tar.gz)
It contains:
- `omp_forkjoin.c`
- `omp_forkjoin.f90`
- `README.md`
The reproducer is self-contained / CC0 and contains no third-party or proprietary application source.
The workload repeatedly enters OpenMP parallel regions. A few "heavy" threads do much more deterministic integer work, while the many "light" threads finish early and wait at the implicit barrier. The program prints a checksum for validation.
Basic C build/run:
```bash
clang -O2 -fopenmp omp_forkjoin.c -o omp_forkjoin_c
OMP_NUM_THREADS=128 OMP_PROC_BIND=close OMP_PLACES=cores ./omp_forkjoin_c
```
### Environment
- x86_64, 2×96 physical cores = 192 physical cores
- `OMP_NUM_THREADS=128`, `OMP_PROC_BIND=close`, `OMP_PLACES=cores`
- Not oversubscribed
- clang/libomp built from `llvm/llvm-project` commit `29f1a71a8d3f`
- libomp reports `LLVM OMP version 5.0.20140926, API 5.0 (201611)`
### Observed
Median of 5 normal timing runs. Syscall counts are from `strace -f -qq -c` with reduced iteration count.
| config | elapsed | system | `sched_yield` | `futex` |
|---|---:|---:|---:|---:|
| default (`KMP_BLOCKTIME=200ms`) | 1.12s | 5.51s | ~613k | ~600 |
| `KMP_BLOCKTIME=0` | 1.59s | 10.14s | ~0 | ~669k |
| `OMP_WAIT_POLICY=passive` | 1.60s | 11.18s | ~0 | ~669k |
The `sched_yield` count scales with iteration count.
### Balanced control
A balanced version of the same loop, where every thread does equal work, does not show the same normal-execution system-time behavior:
| config | system time |
|---|---:|
| imbalanced default | 5.51s |
| balanced control | 0.36s |
So the imbalance appears to be the trigger.
Note: `strace` perturbs this workload, so I am using normal `/usr/bin/time -v` for the balanced-vs-imbalanced system-time comparison. The `strace` counts are mainly used for the default-vs-`KMP_BLOCKTIME=0` / passive syscall-mix comparison.
### Notes
- The same pattern was also seen from both C/clang and Fortran/flang builds using LLVM `libomp`, so this looks like runtime behavior rather than front-end-specific behavior.
- I am not asking `libomp` to match any other runtime.
- I am trying to confirm whether this default wait behavior is expected, and what setting is recommended for imbalanced fork/join-heavy workloads.
![]()
_______________________________________________
llvm-bugs mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-bugs
