The GitHub Actions job "Lint" on tvm.git/main has failed. Run started by GitHub user tlopex (triggered by tlopex).
Head commit for run: 70b264006695562893afc954e95c0278ce0a835a / HuEnwei <[email protected]> [Fix][Relax][Frontend][ONNX] Fix Softplus accuracy loss from hardcoded threshold (#20212) Fixes: #20184 ## Summary The Relax ONNX frontend `Softplus._impl_v1` passed a hardcoded `threshold` (`10.0` for float16, `20.0` otherwise) to `relax.op.nn.softplus`. ONNX `Softplus` is defined as `y = log(exp(x) + 1)` with **no threshold** attribute, but the topi lowering of `relax.op.nn.softplus` clamps the result to the linear function `x` once `beta * x` exceeds the threshold: ```python # python/tvm/topi/nn/elemwise.py return tvm.tir.Select(b * value > t, value, (1 / b) * tvm.tir.log(1 + tvm.tir.exp(b * value))) ``` ## Root cause The `threshold` is chosen once for every dtype, and `20` is far too small for float64: `log(exp(x) + 1) = x + log(1 + exp(-x))` is fully representable in float64 up to `x ≈ 709`, so clamping at `x = 20` drops the `log(1 + exp(-x))` term. Differential test vs `onnx.reference` showed an observable absolute error of `1.87e-9` at `x = 20.1`: | x (float64) | `onnx.reference` | TVM (before) | |---|---|---| | 20.0 | 20.000000002061153 | 20.000000002061153 | | 20.1 | 20.10000000186501 | **20.1** (err 1.87e-9) | | 25.0 | 25.000000000013888 | **25.0** (err 1.39e-11) | (For float32/float16 the clamp is numerically invisible in ordinary ranges — `log(exp(x)+1)` already rounds to `x` for `x ≳ 15` — and near overflow both onnxruntime and TVM intentionally return a finite value, so the observable, spec-relevant defect is the float64 case.) ## Fix Lower ONNX `Softplus` as the numerically stable, threshold-free form: ```python x = inputs[0] dtype = x.ty.dtype return relax.op.add( relax.op.maximum(x, relax.const(0, dtype)), relax.op.log( relax.op.add( relax.const(1, dtype), relax.op.exp(relax.op.negative(relax.op.abs(x))), ) ), ) ``` `max(x, 0) + log(1 + exp(-|x|))` equals `log(exp(x) + 1)` mathematically for every finite `x`: it never overflows (both terms are bounded), it matches onnxruntime on float32/float16 (including near overflow, where the naive formula would overflow to `inf`), and it keeps float64 exact across the whole representable range. The torch frontend keeps using `relax.op.nn.softplus` with `beta`/`threshold` since PyTorch's `nn.Softplus` genuinely has those parameters. ## Validation Differential test: Relax (build + `VirtualMachine`) vs onnxruntime (float32/float16) and the mathematically exact stable form cross-checked with `onnx.reference` (float64; onnxruntime's CPU EP has no float64 Softplus), across 3 dtypes × full value range (incl. the old failure region `x ∈ (20, 30]` and near-overflow) × multiple shapes and opsets. | Case | Result | |---|---| | fp32 regular / threshold-region / near-overflow `[60..100]` | all OK (vs onnxrt) | | fp64 regular / `x ∈ (19..50]` (old failure region) / `[600..1000]` | all OK, max\|diff\|=0 (vs exact) | | fp16 regular / threshold-region `[10..12]` | all OK (vs onnxrt) | | opset 1 / 13 / 22 (fp32) | all OK | | fp64 `x = 20.1 / 25 / 30` (pre-fix failures) | max\|diff\| = 0.00e+00 | Total: **19 differential cases, 19 OK, 0 rejected, 0 numeric mismatches**. In-tree tests `test_softplus_large_values` (float32 up to `x = 80` vs onnxruntime) and `test_softplus_float64_accuracy` (float64 vs exact, fails before this fix) were added; the existing `test_unary[Softplus]` still passes. ## Files changed - `python/tvm/relax/frontend/onnx/onnx_frontend.py` — `Softplus._impl_v1`: lower numerically stable `max(x, 0) + log(1 + exp(-|x|))` instead of `relax.op.nn.softplus` with a hardcoded `threshold`. - `tests/python/relax/test_frontend_onnx.py` — add `test_softplus_large_values` and `test_softplus_float64_accuracy`. ## Base Based on current upstream `main` (`ad0a225074`, 2026-08-28) using the current `x.ty.dtype` API (vs the older `struct_info.dtype`). The single commit `e5eede1b5d` applies cleanly on `main` with `git am`/`git apply`. Report URL: https://github.com/apache/tvm/actions/runs/33154003214 With regards, GitHub Actions via GitBox --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
