mirror of
https://github.com/opencv/opencv.git
synced 2026-09-11 21:01:33 -05:00
ba7e716840
dnn: use the full vector width in blocked-layout pointwise kernels - #29836 Four block-layout kernels share one defect: they vectorize *across the channel block*, so they only use the vector unit while it happens to match `C0`. This PR fixes all four behind one shared helper, and along the way fixes an unrelated correctness bug found in the same function. ## 1. FP16 BatchNorm writes no output `batch_norm2_layer.cpp`, the `CV_16F` arm of the accelerated block-layout path, wraps its whole body in a condition that cannot be true: ```cpp } else if (type == CV_16F) { const hfloat* inptr = ...; if (type == CV_32F) { // false by construction ``` There is no `else`, so every vector loop in that arm is dead and **nothing is written to the output**. A half-precision BatchNorm on a blocked tensor with `C0` equal to 1, 2 or 4 times the vector width returns whatever the destination buffer already held. The `CV_32F` and `CV_16BF` arms are correct; only the redundant wrapper is removed. Confirmed with a sentinel-prefilled destination: before the fix the sentinel survives the call (`maxerr = 12346`) for all three `C0` values, after it the output is exact. ## 2. The kernels only use min(C0, VEC_SZ) lanes `C0` is the block-layout channel block, fixed at 8 (`net_impl.hpp`, `DEFAULT_C0`), so these guards decide how much of the register gets used: | kernel | guard | consequence with C0=8 | |---|---|---| | ChannelsPReLU | `C0 == VEC_SZ` | scalar at 4 lanes **and** at 16/32 | | BatchNorm | `C0 == vlanes*{4,2,1}` | scalar at 16/32 | | InstanceNorm | `c0 <= validC0 - VEC_SZ` | scalar at 16/32 | | GroupNorm | `c0 <= c0_hi - VEC_SZ` | scalar at 16/32 | PReLU is the worst case: an equality can only hold on an 8-lane build, so a default x86 SSE build (4 lanes) runs it scalar too. The other three use chunk loops that work at <= 8 lanes and fail above. These files are not in the CPU-dispatch list, so `v_float32` is whatever `CPU_BASELINE` gives -- this is not RISC-V-specific. ### Approach A block-layout plane is contiguous over `(H, W, C0)` and the coefficients repeat with period `C0`, so one register can span `vlanes/C0` pixels: replicate the coefficients across it and walk the plane flat. All three spanning call sites now go through one helper, `cpu_kernels/blocked_pointwise.hpp::blockedSpanApply()`, parameterised by the per-element operation (`BlockedAffineOp`, `BlockedPReLUOp`). PReLU additionally gains a chunked path for `C0 > VEC_SZ`, which is what restores it on 4-lane targets. BatchNorm keeps its own unrolled loops and gains a whole-vector step plus a remainder loop. Pre-existing paths are untouched, so targets that already vectorized keep the same code. **GroupNorm also gains a vector reduction.** Its mean/variance pass walked one channel at a time with a stride of `C0`, which no target vectorized at all, so that half speeds up everywhere rather than only on wide vectors. Both new GroupNorm paths are restricted to blocks owned entirely by one group; where a group boundary falls inside a block the old per-channel code still runs, because spanning would cross into channels another `parallel_for_` task is writing. ### Note for reviewers: the helper's `noinline` is load-bearing `blockedSpanApply()` carries an explicit `noinline`. Inlined, GCC 15.2 on RISC-V speculates its stores into callers whose guard is false, which silently corrupted a neighbouring group's channels in `fastNormGroupBlockF32`. The symptom was exactly half the elements wrong at VLEN=1024 in the cases where a group splits a block; a runtime trace showed the guard evaluating false on every block, and inserting any call before the `if` made it disappear. Please do not remove the attribute. ## Benchmarks SpacemiT K3, GCC 15.2, `CPU_BASELINE=RVV`, single thread, median of 11, pristine vs patched built back to back in one session. The board exposes two core types with different VLEN, so both columns are the same binary on the same machine. Speedup, VLEN=256 / VLEN=1024: | shape (NxC1xHxWxC0) | Ci | InstanceNorm | BatchNorm | GroupNorm | |---|---|---|---|---| | 1x32x56x56x8 | 256 | 1.01x / 14.38x | 0.91x / 11.62x | 1.85x / 14.56x | | 1x16x28x28x8 | 128 | 1.01x / 16.27x | 0.98x / 3.95x | 1.91x / 16.01x | | 1x8x112x112x8 | 64 | 1.02x / 12.76x | 1.03x / 1.37x | 1.98x / 12.86x | | 1x16x56x56x4 | 64 | 3.01x / 12.47x | 4.49x / 6.18x | 3.01x / 11.78x | ChannelsPReLU, measured separately the same way: | shape | Ci | VLEN=256 | VLEN=1024 | |---|---|---|---| | 1x32x56x56x8 | 256 | 1.45x | 8.37x | | 1x16x28x28x8 | 128 | 1.55x | 4.16x | | 1x64x14x14x8 | 512 | 1.46x | 4.05x | | 1x8x112x112x8 | 64 | 1.47x | 3.77x | | 1x16x56x56x4 | 64 | 2.99x | 8.47x | | 1x8x56x56x16 | 128 | 1.20x | 2.33x | The VLEN=256 columns for InstanceNorm and BatchNorm are flat by construction -- `C0 == VEC_SZ` there, so those shapes already vectorized and the code is unchanged; the 0.91-1.03x spread is measurement noise. The rows that move at 256 are `C0=4` (block narrower than the vector), PReLU (broken at every width), and GroupNorm (reduction). **Caveat on the BatchNorm numbers.** BatchNorm timings on this board are much less reproducible than the other three. The `1x8x112x112x8` case in particular measured anywhere from 1.4x to 6.9x across builds with byte-identical BatchNorm sources, and its pristine baseline moved by 26% between runs. The working set there is 3.06 MB, an exact multiple of 4096, and this hardware is sensitive to how source and destination alias in the cache; the numbers above are one back-to-back pair rather than a stable figure. InstanceNorm, GroupNorm and PReLU reproduced to within ~1% across every build. ## Testing Verified against a scalar reference over 43 shape / `C0` / `Ci` / group combinations across the four layers, at VLEN 256 and 1024, at 1 and 8 threads, clean under `MALLOC_CHECK_=3`. Cases include partial trailing blocks, odd planes (7x7, 13x11) that exercise the remainder loops, `C0` from 2 to 64, and GroupNorm configurations where a group boundary falls inside a block -- those take the fallback and match bit-exactly, which is what confirms the ownership guard. `opencv_test_dnn` was **not** run: the build used here is `BUILD_LIST=dnn`, which generates no dnn test target, and the board has no `opencv_extra` checkout. The blocked path's reachability was confirmed by inspection instead -- `ActivationLayer::getLayouts` passes the producer's layout through, and `useBlockLayout()` runs unconditionally in `finalizeGraph`, so these kernels are on the default path in real nets. ## Platform scope Nothing here is behind a RISC-V `#ifdef`; this is universal-intrinsic code that compiles into every target. | target (C0=8) | lanes | what changes | |---|---|---| | x86 SSE baseline (default) | 4 | PReLU newly vectorized; GroupNorm reduction newly vectorized | | x86 AVX2 baseline | 8 | GroupNorm reduction; BatchNorm loop rewritten (same iterations) | | x86 AVX-512 baseline | 16 | + all spanning paths go live | | ARM64 NEON | 4 | as SSE baseline | | ARMv7 NEON | 4 | `CV_SIMD_64F`=0, reduction path skipped | | RVV 256 | 8 | PReLU, GroupNorm reduction | | RVV >= 512 | 16/32 | everything | Two changes reach a **default x86 build**: PReLU, which was scalar there because `C0 == VEC_SZ` cannot hold at 4 lanes, and the GroupNorm reduction, which was scalar everywhere. The latter changes GroupNorm's numerical output on those targets, since summation order differs -- measured at ~1e-7 relative here. Measurements are RISC-V only; no x86 or ARM machine was available. The spanning branches were exercised on RVV at the same `vlanes/C0` ratios an AVX-512-baseline build would hit (2:1 and 4:1), but the x86-reachable changes above have had no x86 validation and are the part most worth checking in CI.