** Description changed: - As part of our efforts to have the ROCm v7.2.3 stack in Resolute this - bug has been created. + [ Impact ] + Users running ROCm 7.1.0 on Resolute have RCCL (NCCL 2.23.4 compat) + missing the following fixes and improvements present in 7.2.4 + (NCCL 2.27.7 compat): - This bug shall be updated to follow the SRU template. For now it serves - as a placeholder. + Bug fixes: + - P2P batching hang: certain AllToAll and AllGather operations over + P2P could hang indefinitely. + - Shutdown race condition in ncclCommDestroy that could cause crashes + or memory corruption when tearing down communicators. + - Wrong results in BFloat16 reduction collectives on ROCm >= 6.0 due + to incorrect type specialisation in the reduce kernel. + - Missing opCount increment for intra-node communications causing + incorrect operation ordering and potential hangs. + - Proxy thread silently ignoring progressOps errors, masking + transport failures. + - 8-byte memory leak per communicator create/destroy cycle. + - Missing memory fences in the LL protocol on gfx950 (MI350x) causing + potential data corruption. + - Null dereference in MSCCL++ split path. + - Segfault in the ext-profiler plugin. + - rcclNetP2pPolicy incorrectly applied for InfiniBand transports. + - ncclFuncAllReduceWithBias removed from the public ncclFunc_t enum + (was an RCCL-only extension that was never upstream-compatible). + - Disable BFloat16 pipelining for reduction collectives to fix + correctness issues. - PPA: https://launchpad.net/~b0b0a/+archive/ubuntu/rccl-7.2.3 - Autopkgtests: https://autopkgtest.ubuntu.com/user/b0b0a/ppa/rccl-7.2.3 - Upstream code diff: https://github.com/ROCm/rccl/compare/rocm-7.1.1...rocm-7.2.3 - abipkgdiff [librccl.so.1.0 vs librccl.so.1.0]: - ``` - === Comparing librccl1 === - Running: abipkgdiff --d1 /home/ubuntu/actions-runner/_work/bullwinkle-cicd/bullwinkle-cicd/old/librccl1-dbgsym_7.1.0-0ubuntu2_amd64.deb --d2 /home/ubuntu/actions-runner/_work/bullwinkle-cicd/bullwinkle-cicd/new/librccl1-dbgsym_7.2.3-0ubuntu1~ppa1~ubuntu26.04.1_amd64.deb /home/ubuntu/actions-runner/_work/bullwinkle-cicd/bullwinkle-cicd/old/librccl1_7.1.0-0ubuntu2_amd64.deb /home/ubuntu/actions-runner/_work/bullwinkle-cicd/bullwinkle-cicd/new/librccl1_7.2.3-0ubuntu1~ppa1~ubuntu26.04.1_amd64.deb - ================ changes of 'librccl.so.1.0'=============== - Functions changes summary: 0 Removed, 1 Changed (34 filtered out), 3 Added functions - Variables changes summary: 0 Removed, 0 Changed, 0 Added variable - Function symbols changes summary: 0 Removed, 0 Added function symbol not referenced by debug info - Variable symbols changes summary: 327 Removed, 291 Added variable symbols not referenced by debug info + New features and improvements: + - 3 new public API functions: rcclGetAlgoInfo, rcclGetAlgoName, + rcclGetProtocolName for algorithm/protocol introspection. + - Full gfx950 (MI350x) tuning: LL/LL128 thresholds, channel counts, + threadblock sizes, gdrcopy, rail-optimised tree support. + - Navi4 (gfx1151) LL enablement and tuning. + - AMD AINIC network transport support. + - NCCL 2.27.5/2.27.6/2.27.7 compatibility. - 3 Added functions: + [ Test Plan ] + 1. Build + Package successfully builds in the bug PPA as well as the Bullwinkle + team rocm-devel PPA via recipe. Bullwinkle team has pushed the + package to Debian experimental and then synced it to stonking. - [A] 'function ncclResult_t rcclGetAlgoInfo(ncclComm*, ncclFunc_t, uint64_t, ncclDataType_t, int, int, int, int*, int*, int*)' {rcclGetAlgoInfo} - [A] 'function ncclResult_t rcclGetAlgoName(int, const char**)' {rcclGetAlgoName} - [A] 'function ncclResult_t rcclGetProtocolName(int, const char**)' {rcclGetProtocolName} + 2. Install & basic smoke-test (no GPU required) + sudo apt install librccl1 librccl-dev + # Verify the library loads: + python3 -c "import ctypes; ctypes.CDLL('librccl.so.1'); print('OK')" - 1 function with some indirect sub-type change: + 3. Autopkgtest on machine with AMD GPU + Such tests have not been executed for any version that has landed in + Ubuntu as we are lacking machine with at least 2 GPUs - [C] 'method latency_profiler::CollTrace::~CollTrace()' at CollTrace.h:36:1 has some indirect sub-type changes: - . - . - . - ``` + [ Where problems could occur ] + - The ncclComm internal struct grew by 1080 bytes (3 new fields: + minMaxChannelThresholds, archName, cuCount). ncclComm_t is an opaque + pointer in the public API so no consumer embeds it directly, but any + plugin or tuner that casts the opaque handle to the internal struct + would be broken. Symptom: crash or wrong values in tuner plugins + compiled against 7.1.0 headers. + - ncclDebugLogLevel gained a new NCCL_LOG_ERROR value at ordinal 1, + shifting all existing levels up by one. Logger callback plugins + compiled against 7.1.0 would silently map log levels incorrectly. + Symptom: wrong severity in profiler/logging plugins. + - ncclFuncAllReduceWithBias was removed from ncclFunc_t. Any code + compiled against 7.1.0 headers that uses this enum value would fail + to build or misbehave at runtime. Symptom: compile error or wrong + collective dispatched. + - The NCCL 2.27.x compatibility bump changes wire protocol details. + Mixed-version clusters (some nodes on 7.1.0, others on 7.2.4) are + not supported and would produce undefined behaviour. Symptom: + collective hangs or incorrect results in multi-node jobs. + - The Navi4/gfx950 tuning changes affect throughput on those GPUs. + A regression would show as degraded bandwidth, not correctness + failures. Symptom: lower throughput in collective benchmarks. + + [ Other Info ] + * ABI: SONAME unchanged (librccl.so.1). abipkgdiff (full log attached + as part of the bug) summary: + - 0 functions removed. + - 3 functions added (rcclGetAlgoInfo, rcclGetAlgoName, + rcclGetProtocolName) — additive, backwards compatible. + - 1 function with indirect sub-type change + (latency_profiler::CollTrace::~CollTrace): caused by ncclComm + struct growth (+1080 bytes, 3 new fields) and enum ordinal + changes (ncclDebugLogLevel, ncclFunc_t). ncclComm is an opaque + handle; no public API function signatures changed. + - 327 removed / 291 added __hip_cuid_* variable symbols — internal + HIP compilation-unit IDs, not public API, expected on recompile. + * DPKG_GENSYMBOLS_CHECK_LEVEL=4 enforced in debian/rules; symbols + check passed with newly exported symbols floor-versioned at 7.2.0. + * This update is part of the coordinated ROCm 7.2.4 stack SRU. + * Initial MP to stonking that later ended in debian experimental: + https://salsa.debian.org/rocm-team/rccl/-/merge_requests/3 + * PPA: https://launchpad.net/~b0b0a/+archive/ubuntu/rccl-2154290 + https://launchpad.net/~b0b0a/+archive/ubuntu/rccl-7.2.4 + * Autopkgtests: https://autopkgtest.ubuntu.com/user/b0b0a/ppa/rccl-7.2.4 + * Upstream comparison: + https://github.com/ROCm/rccl/compare/rocm-7.1.0...rocm-7.2.4
** Summary changed: - SRU: New upstream version 7.2.3 + SRU: New upstream version 7.2.4 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2154290 Title: SRU: New upstream version 7.2.4 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rccl/+bug/2154290/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
