dnn(cuda): transformer layer coverage and CUDA Graph replay on the new engine - #30439
Follow-up to #30415 (ResNet layer coverage and cuDNN JIT conv/pooling on the new engine). Adds CUDA support on the new DNN engine
for the layers transformer models need, and brings the CUDA path to parity with ONNX Runtime.
**Companion PR** opencv/opencv_extra#1448
**Changes**
- CUDA execs for Attention, Cast, ConstantOfShape, Expand, Gather, Not, Reduce, Shape, Slice, Softmax, Tile, Transpose, Concat,
Gelu, LayerNormalization, MatMul, Split, Squeeze, Unsqueeze and the element-wise activations.
- A graph with static shapes is recorded as one CUDA Graph on the second `forward()` and replayed from then on.
- The cuDNN JIT convolution reads NCHW buffers in place; NHWC staging is kept only for FP16 grouped convolutions.
- TF32 for cuBLAS GEMMs (`OPENCV_DNN_CUDA_ALLOW_TF32`), FP32 accumulation for FP16 GEMMs
(`OPENCV_DNN_CUDA_FP16_HALF_ACCUMULATE`), and fused MatMul/Gemm post-ops.
- Batch-broadcast MatMul and Shape are safe to capture in a CUDA Graph; the unused pointer-array `cublas::gemmBatched` is
removed.
- Tests: `forward()` runs three times on batch-broadcast MatMul and Shape models; `MatMul_init_bcast` runs on CUDA again;
`AttentionSharedShapeReshape` uses the FP16 default tolerance on CUDA FP16.
- Perf: new CUDA-only `*_CUDA_pinned` tests pin the input shapes.
**Performance** (ms, median of 50, FP32 with TF32, RTX PRO 6000 Blackwell, idle GPU)
| Model | CPU | CUDA (cuDNN JIT) | CUDA (full cuDNN) | ORT 1.31 CUDA EP |
|---|---|---|---|---|
| ResNet-50 | 8.02 | 0.80 | 0.80 | 0.88 |
| ResNet-18 | 3.27 | 0.44 | 0.44 | 0.48 |
| MobileNetV2 | 1.99 | 0.28 | 0.27 | 0.40 |
| DenseNet-121 | 17.42 | 1.48 | 1.58 | 1.58 |
| EfficientNet-Lite4 | 5.03 | 0.53 | 0.54 | 0.73 |
| ViT-B/32 | 11.06 | 1.02 | 1.02 | 0.95 |
| ViT-Base/16 | 24.38 | 1.16 | 1.17 | 1.13 |
| DeiT-Tiny | 4.58 | 0.65 | 0.65 | 0.68 |
| MobileViT-XS | 6.93 | 0.72 | 0.72 | 0.77 |
| YOLOv3 | 47.21 | 2.60 | 2.59 | 2.72 |
| YOLOv4 | 109.38 | 3.58 | 3.61 | 4.18 |
| YOLOv4-tiny | 7.31 | 0.64 | 0.64 | 0.66 |
| BEiT-Base | 25.30 | 1.24 | 1.24 | 1.19 |
| BERT | 10.00 | 0.97 | 0.97 | 0.99 |
| SigLIP | 41.14 | 2.09 | 2.09 | 2.44
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [ ] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
dnn: fix ONNX conformance failures in layers and the graph engine - #30181
Fixes wrong results found by running the ONNX backend-scoreboard conformance suite (onnx 1.23.2) against the new DNN engine. No new ops are added; these are all ops OpenCV already claims to support but computed incorrectly.
Updated ONNX coverage is: `[scoreboard] passed=1740 failed=21 skipped=272 total=2033 (85.6% passed)`
**Graph engine / importer**
- Reshape+Reshape fusion now only fires when the outer shape is known and has no `0` entries, since a `0` copies a dim from the inner Reshape's output.
- Scan slices rank-1 inputs to scalars, so scalar loop-carried state keeps rank 0.
- Rank-0 tensors read from ONNX keep rank 0 instead of becoming 1x1.
- When a model is refused, `setInput()` now reports the actual import failure (including unknown ops that throw early) instead of a misleading "missing blob" error.
**Attention**
- Softcap is applied before the bias/mask, as the spec does, so masked keys no longer leak probability.
- `qk_matmul_output_mode` 1 and 2 now return the right tensors (checked against onnxruntime; the old denylist entries blaming the reference data were wrong).
- Attention-25 `left_window_size` / `right_window_size` are supported, including together with `attn_mask`.
- A fully masked row now gives all zeros instead of a uniform distribution.
**Layers**
- GRU: `direction="reverse"` is honoured.
- LSTM reverse: `Y_h` / `Y_c` are taken from the correct time step.
- SpaceToDepth: opset-28 `mode` (CRD / DCR) is supported.
- MaxPool: `storage_order` is respected for the indices output.
- OneHot: out-of-range indices produce an all-`off_value` row instead of wrapping.
- Mod with `fmod=0`: no longer truncates through int (wrong for float, lossy for int64).
- ReduceMax / ReduceMin on an empty bool set return the correct identity.
- Swish: `alpha` is respected (CPU and OpenCL).
- Quantize/DequantizeLinear: scale and zero_point may differ in shape when they're broadcast-compatible (e.g. rank-0 scale with `[1]` zero_point).
- BitShift: signed integer types are supported.
**Tests**
- 12 conformance tests removed from the denylists, a few OpenVINO filter entries added, and new 1D layer tests for the fixes.
- Full C++ conformance passes (5376/5376); `opencv_test_dnn` has no new failures compared with 5.x.
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Unify element-wise ops with the core engine - #30083
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Added support for Granite Docling 258M and PaddleOCR-VL - #29618
Adds `com.microsoft::GroupQueryAttention` to the engine-new ONNX importer, required by GraniteDocling-258M's ,end-to-end samples for GraniteDocling-258M and PaddleOCR-VL-1.5. Verified on Linux x86_64 / GCC / CPU
**companion opencv_extra PR** [Vlm ocr support engine new opencv/opencv_extra#1412](https://github.com/opencv/opencv_extra/pull/1412) for test data
upstream/5.x already lowers `GroupQueryAttention` to `AttentionOnnxAi`, but that path rejects `do_rotary=1`:
```cpp
CV_CheckEQ(params.get<int>("do_rotary", 0), 0, "GroupQueryAttention: do_rotary=1 is not supported");
```
GraniteDocling-258M's export sets it. Confirmed two ways: building a Llama checkpoint from its published `text_config` (9 heads / 3 KV heads, `rope_theta=100000`) and running onnxruntime-genai's `model_builder`; and reading the community export at `onnx-community/granite-docling-258M-ONNX`, which carries 30 `GroupQueryAttention` nodes with `do_rotary=1`, `num_heads=9`, `kv_num_heads=3`, `head_dim=64`.
## Changes
**Importer** (`onnx_importer2.cpp`, +49)
Routing as above, plus the input validation the op needs: packed QKV rejected, past_key / past_value required as a pair, `do_rotary=1` requires cos/sin, unsupported trailing inputs rejected, and `seqlens_k` shorter than the buffer recognised as a shared-buffer export rather than silently mis-attended.
**Layer** (`group_query_attention_layer.cpp`, +380)
Attention is built on `fastGemmBatch` and `fused_softmax_softcap_mask`, the same kernels `attention_onnxai_layer.cpp` uses. Grouping falls out of the batched-GEMM offset table (query head → `h / groupSize`), so no broadcast copy is needed, and the second GEMM writes straight into the `[B, S, num_heads*D]` output. Supports rotary (full and partial), sliding window, softcap, growing and preallocated caches, and optional `present_*` outputs.
**Core** (`persistence_json.cpp`, +92)
`\uXXXX` escapes are UTF-16 code units, so anything above the BMP arrives as a surrogate pair. These were encoded as two separate 3-byte sequences CESU-8, which is not valid UTF-8 - for keys and values alike. Now recombined into one 4-byte sequence, with unpaired surrogates rejected.
**Tests / perf / samples**
`test_layer_group_query_attention.cpp` (+93), `test_tokenizer.cpp` (+28), `test_tokenizer.py` (+42), `perf_layer.cpp` (+125: `Layer_GroupQueryAttention` MHA_ShortCache and Grouped_WithCache, plus `Layer_Sign`), and two samples (`granite_docling_inference.py`, `paddleocr_vl_inference.py`).
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Unified DNN graph fusion behind fuseChains - #30088
Requires: https://github.com/opencv/opencv_extra/pull/1417
### What This PR Does
Follow-up to #29656, which added the backend-agnostic chain mechanism. That PR introduced it and made pointwise fusion its first consumer; this one moves the rest of the fusion onto it.
- `Net::Impl::fuseBasic()` is gone. Its conv+BatchNorm, conv+activation and conv+residual branches now go through the chain mechanism.
- `fuseBN` and the six other passes now run from inside `fuseChains()`, in their existing order.
- `graph_fusion_basic.cpp` keeps `fuseBN()` and `fuseInstanceNormAffine()`.
- New `FusionOps::foldInputScale` folds a scale sitting before a conv directly into its weights, across the dense, depthwise, grouped and MLAS-prepacked layouts.
### Results
Measured on Intel Core i9-11900K
| Model | 5.x | this PR | speedup |
|-----------------------------|-----------|-----------|---------|
| inception_v2 | 15.04 ms | 11.80 ms | 1.27 |
| efficientnet-lite4 | 14.83 ms | 12.28 ms | 1.21 |
| pose_estimation_mediapipe | 5.66 ms | 5.25 ms | 1.08 |
| siglip_base_patch16_224 | 143.04 ms | 135.27 ms | 1.06 |
| densenet121 | 25.99 ms | 24.76 ms | 1.05 |
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Added Backend Agnostic fusion in DNN - #29656
# What This Adds
A way for any layer, on any backend, to absorb trailing per-element math — with the pass knowing nothing about either layer and holding no list of op types.
- **Description**: `LayerMath` (one layer's math) → `AdjacencyGraph` (hash-consed chain)
- **Contract**: two `bool` virtuals on `Layer`
- **CPU implementations**: three, behind that contract
Pointwise fusion is the first consumer, not the point.
## The Problem
```cpp
ActivationLayer* activ = dynamic_cast<ActivationLayer*>(layer_ptr);
Conv2Layer* conv = getLayer<Conv2Layer>(newprog, conv_layer_idx);
if (conv) conv->fuseActivation(layer);
```
Guest type, host type, method name all hardcoded in the pass. `Clip` never fused (fails the cast), `Gemm` never fused (no virtual), no 2-op chain fused at all, and no backend could fuse anything without a CPU-specific method.
## Results
| Model | 5.x | this PR | speedup |
|-----------------------|----------|----------|---------|
| MPHand | 2.34 ms | 1.03 ms | 2.27 |
| EfficientNet | 9.82 ms | 5.00 ms | 1.96 |
| MPPose | 5.14 ms | 2.81 ms | 1.83 |
| MobileNet_SSD_v1_ONNX | 12.58 ms | 8.38 ms | 1.50 |
| BlazeFace | 0.95 ms | 0.77 ms | 1.23 |
| DenseNet_121 | 20.56 ms | 19.08 ms | 1.08 |
| MobileNetv2_ONNX | 2.07 ms | 1.99 ms | 1.04 |
| MobileViT_XS | 6.33 ms | 6.11 ms | 1.04 |
| YuNet_320 | 1.33 ms | 1.27 ms | 1.04 |
| PPOCRv3 | 43.05 ms | 42.15 ms | 1.02 |
| BERT | 9.78 ms | 9.68 ms | 1.01 |
| BEiT_Base_Patch16_224 | 27.48 ms | 27.08 ms | 1.01 |
| DeiT_Tiny_Patch16_224 | 4.74 ms | 4.72 ms | 1.01 |
| MPPalm | 1.88 ms | 1.85 ms | 1.01 |
| SSD | 50.19 ms | 49.82 ms | 1.01 |
| YOLOv4_tiny | 7.10 ms | 7.03 ms | 1.01 |
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
dnn: use the full vector width in blocked-layout pointwise kernels - #29836
Four block-layout kernels share one defect: they vectorize *across the channel block*, so they only use the vector unit while it happens to match `C0`. This PR fixes all four behind one shared helper, and along the way fixes an unrelated correctness bug found in the same function.
## 1. FP16 BatchNorm writes no output
`batch_norm2_layer.cpp`, the `CV_16F` arm of the accelerated block-layout path, wraps its whole body in a condition that cannot be true:
```cpp
} else if (type == CV_16F) {
const hfloat* inptr = ...;
if (type == CV_32F) { // false by construction
```
There is no `else`, so every vector loop in that arm is dead and **nothing is written to the output**. A half-precision BatchNorm on a blocked tensor with `C0` equal to 1, 2 or 4 times the vector width returns whatever the destination buffer already held. The `CV_32F` and `CV_16BF` arms are correct; only the redundant wrapper is removed.
Confirmed with a sentinel-prefilled destination: before the fix the sentinel survives the call (`maxerr = 12346`) for all three `C0` values, after it the output is exact.
## 2. The kernels only use min(C0, VEC_SZ) lanes
`C0` is the block-layout channel block, fixed at 8 (`net_impl.hpp`, `DEFAULT_C0`), so these guards decide how much of the register gets used:
| kernel | guard | consequence with C0=8 |
|---|---|---|
| ChannelsPReLU | `C0 == VEC_SZ` | scalar at 4 lanes **and** at 16/32 |
| BatchNorm | `C0 == vlanes*{4,2,1}` | scalar at 16/32 |
| InstanceNorm | `c0 <= validC0 - VEC_SZ` | scalar at 16/32 |
| GroupNorm | `c0 <= c0_hi - VEC_SZ` | scalar at 16/32 |
PReLU is the worst case: an equality can only hold on an 8-lane build, so a default x86 SSE build (4 lanes) runs it scalar too. The other three use chunk loops that work at <= 8 lanes and fail above.
These files are not in the CPU-dispatch list, so `v_float32` is whatever `CPU_BASELINE` gives -- this is not RISC-V-specific.
### Approach
A block-layout plane is contiguous over `(H, W, C0)` and the coefficients repeat with period `C0`, so one register can span `vlanes/C0` pixels: replicate the coefficients across it and walk the plane flat.
All three spanning call sites now go through one helper, `cpu_kernels/blocked_pointwise.hpp::blockedSpanApply()`, parameterised by the per-element operation (`BlockedAffineOp`, `BlockedPReLUOp`). PReLU additionally gains a chunked path for `C0 > VEC_SZ`, which is what restores it on 4-lane targets. BatchNorm keeps its own unrolled loops and gains a whole-vector step plus a remainder loop. Pre-existing paths are untouched, so targets that already vectorized keep the same code.
**GroupNorm also gains a vector reduction.** Its mean/variance pass walked one channel at a time with a stride of `C0`, which no target vectorized at all, so that half speeds up everywhere rather than only on wide vectors. Both new GroupNorm paths are restricted to blocks owned entirely by one group; where a group boundary falls inside a block the old per-channel code still runs, because spanning would cross into channels another `parallel_for_` task is writing.
### Note for reviewers: the helper's `noinline` is load-bearing
`blockedSpanApply()` carries an explicit `noinline`. Inlined, GCC 15.2 on RISC-V speculates its stores into callers whose guard is false, which silently corrupted a neighbouring group's channels in `fastNormGroupBlockF32`. The symptom was exactly half the elements wrong at VLEN=1024 in the cases where a group splits a block; a runtime trace showed the guard evaluating false on every block, and inserting any call before the `if` made it disappear. Please do not remove the attribute.
## Benchmarks
SpacemiT K3, GCC 15.2, `CPU_BASELINE=RVV`, single thread, median of 11, pristine vs patched built back to back in one session. The board exposes two core types with different VLEN, so both columns are the same binary on the same machine.
Speedup, VLEN=256 / VLEN=1024:
| shape (NxC1xHxWxC0) | Ci | InstanceNorm | BatchNorm | GroupNorm |
|---|---|---|---|---|
| 1x32x56x56x8 | 256 | 1.01x / 14.38x | 0.91x / 11.62x | 1.85x / 14.56x |
| 1x16x28x28x8 | 128 | 1.01x / 16.27x | 0.98x / 3.95x | 1.91x / 16.01x |
| 1x8x112x112x8 | 64 | 1.02x / 12.76x | 1.03x / 1.37x | 1.98x / 12.86x |
| 1x16x56x56x4 | 64 | 3.01x / 12.47x | 4.49x / 6.18x | 3.01x / 11.78x |
ChannelsPReLU, measured separately the same way:
| shape | Ci | VLEN=256 | VLEN=1024 |
|---|---|---|---|
| 1x32x56x56x8 | 256 | 1.45x | 8.37x |
| 1x16x28x28x8 | 128 | 1.55x | 4.16x |
| 1x64x14x14x8 | 512 | 1.46x | 4.05x |
| 1x8x112x112x8 | 64 | 1.47x | 3.77x |
| 1x16x56x56x4 | 64 | 2.99x | 8.47x |
| 1x8x56x56x16 | 128 | 1.20x | 2.33x |
The VLEN=256 columns for InstanceNorm and BatchNorm are flat by construction -- `C0 == VEC_SZ` there, so those shapes already vectorized and the code is unchanged; the 0.91-1.03x spread is measurement noise. The rows that move at 256 are `C0=4` (block narrower than the vector), PReLU (broken at every width), and GroupNorm (reduction).
**Caveat on the BatchNorm numbers.** BatchNorm timings on this board are much less reproducible than the other three. The `1x8x112x112x8` case in particular measured anywhere from 1.4x to 6.9x across builds with byte-identical BatchNorm sources, and its pristine baseline moved by 26% between runs. The working set there is 3.06 MB, an exact multiple of 4096, and this hardware is sensitive to how source and destination alias in the cache; the numbers above are one back-to-back pair rather than a stable figure. InstanceNorm, GroupNorm and PReLU reproduced to within ~1% across every build.
## Testing
Verified against a scalar reference over 43 shape / `C0` / `Ci` / group combinations across the four layers, at VLEN 256 and 1024, at 1 and 8 threads, clean under `MALLOC_CHECK_=3`. Cases include partial trailing blocks, odd planes (7x7, 13x11) that exercise the remainder loops, `C0` from 2 to 64, and GroupNorm configurations where a group boundary falls inside a block -- those take the fallback and match bit-exactly, which is what confirms the ownership guard.
`opencv_test_dnn` was **not** run: the build used here is `BUILD_LIST=dnn`, which generates no dnn test target, and the board has no `opencv_extra` checkout. The blocked path's reachability was confirmed by inspection instead -- `ActivationLayer::getLayouts` passes the producer's layout through, and `useBlockLayout()` runs unconditionally in `finalizeGraph`, so these kernels are on the default path in real nets.
## Platform scope
Nothing here is behind a RISC-V `#ifdef`; this is universal-intrinsic code that compiles into every target.
| target (C0=8) | lanes | what changes |
|---|---|---|
| x86 SSE baseline (default) | 4 | PReLU newly vectorized; GroupNorm reduction newly vectorized |
| x86 AVX2 baseline | 8 | GroupNorm reduction; BatchNorm loop rewritten (same iterations) |
| x86 AVX-512 baseline | 16 | + all spanning paths go live |
| ARM64 NEON | 4 | as SSE baseline |
| ARMv7 NEON | 4 | `CV_SIMD_64F`=0, reduction path skipped |
| RVV 256 | 8 | PReLU, GroupNorm reduction |
| RVV >= 512 | 16/32 | everything |
Two changes reach a **default x86 build**: PReLU, which was scalar there because `C0 == VEC_SZ` cannot hold at 4 lanes, and the GroupNorm reduction, which was scalar everywhere. The latter changes GroupNorm's numerical output on those targets, since summation order differs -- measured at ~1e-7 relative here.
Measurements are RISC-V only; no x86 or ARM machine was available. The spanning branches were exercised on RVV at the same `vlanes/C0` ratios an AVX-512-baseline build would hit (2:1 and 4:1), but the x86-reachable changes above have had no x86 validation and are the part most worth checking in CI.
Extend data type support in elementwise, gather and scatter layers - #29786
### PR Changes:
### Support added, per layer
| Layer | Types added | Gate / kernel |
|---|---|---|
| Abs | 8U, 8S, 16U, 16S, 32U, 32S, 64U, 64S | Gate + new templated integer kernel |
| Sign | 8U, 8S, 16U, 16S, 32U, 32S, 64U, 64S | Gate + same templated kernel |
| Neg | 32S | Gate only. The kernel already had a `CV_32S` branch |
| GatherElements | 64F, 16U, 16S, 32U, 64U | Gate + element-width dispatch |
| Scatter | 64F, 16U, 16S, 32U, 64U | Gate + dispatch arms, all five reductions checked |
| ScatterND | 64F, 16U, 16S, 32U, 64U | Gate + dispatch arms, same as Scatter |
| GatherND | 64F, 16U, 16S, 32U, 64U | Gate + element-width dispatch |
| Slice2 | (fix) | Kernel. Wrong-width copy, see below |
`Abs`/`Sign` use one template over all widths with the signed/unsigned split resolved at compile time;
unsigned `abs` short-circuits to `copyTo` and unsigned `sign` reduces to `x != 0`. `GatherElements` and
`GatherND` only move elements, so their per-dtype arms collapsed to four widths. `GatherND` also unified a
target-conditional gate that split `16F`/`32F` by target in a file with no OpenCL path.
`Slice2` had two duplicated depth chains that both fell through to `run_parallel<float>`, a 4-byte copy, so
`64F`/`64U` truncated and `16U`/`16S` read and wrote past the element. Reachable only when the innermost axis
has `step != 1`, which is why the float32 slice tests passed. Now dispatches on `elemSize()`, matching
`pad2_layer.cpp`.
**Two further fixes:** signed-overflow UB in the int64 `Power`/`Neg` path (`sp[i] * scale` is undefined at
`INT64_MIN`, now multiplied through `uint64_t`), and `CV_OCL_RUN` now skips integer depths, since the OCL
activation kernels are float math and `CV_32S` would have gone through a 24-bit mantissa.
**New accuracy tests in `test_int.cpp`**: `Test_Abs_Int`, `Test_Sign_Int`, `Test_Neg_Int`, `Test_Scatter_Int`,
`Test_GatherND_Int`, with `Test_GatherElements_Int` and `Test_ScatterND_Int` widened to nine depths.
Removed `test_slice_start_out_of_bounds` from the parser denylist.
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Port of #29377 to 5.x. Wires Log, Erf, Exp, Sin, Cos, Sinh, Cosh, Tan,
Softplus, BNLL, Asinh, Acosh, Atanh into the dispatched activation_kernels
registry, plus a Layer_Activation perf test. 1.4-12.3x across
M4/A76/Threadripper/Xeon, correct to <=5e-7 vs scalar.
dnn: SIMD for transcendental activation layers (15 functors) 🧑💻🤖#2937
# dnn: SIMD for transcendental activation layers (15 functors) 🧑💻🤖
### Summary
Many `BaseDefaultFunctor`-based activation functors fall back to the scalar `BaseDefaultFunctor::apply`
— one `libm` call per element. This PR adds vectorized `apply()` overrides (universal intrinsics,
matching the existing `MishFunctor`/`GeluFunctor`/`SigmoidFunctor` pattern, each with a scalar tail) to
the **15 transcendental activations** that have a clean SIMD path:
| functor | uses | | functor | uses |
|---|---|---|---|---|
| TanH | `v_exp` (1−2/(e²ˣ+1)) | | Asinh | `v_log`,`v_sqrt` (sign·log(\|x\|+√(x²+1))) |
| Log | `v_log` | | Acosh | `v_log`,`v_sqrt` |
| Erf | `v_erf` | | Atanh | `v_log` |
| Exp | `v_exp` | | Softplus | `v_exp`,`v_log` |
| Sin | `v_sin` | | BNLL | `v_exp`,`v_log` (max(x,0)+log1p(e^−\|x\|)) |
| Cos | `v_cos` | | GeluApproximation | `v_exp` (tanh) |
| Tan | `v_sin`/`v_cos` | | Sinh / Cosh | `v_exp` |
Functors compile at the baseline SIMD width (SSE on x86, NEON on ARM — same as the existing SIMD
functors). `Sqrt`/`Floor`/`Ceil`/`Round`/`Abs`/etc. are intentionally **not** included — those are
single cheap ops the compiler already auto-vectorizes (measured `Sqrt` at 0.95×). `Asin`/`Acos`/`Atan`
are omitted (no `v_atan`/`v_asin`/`v_acos` intrinsic).
### Performance
Real `opencv_perf_dnn` `Layer_Activation` (added here), 8×256×128×100 = 26.2 M-elem CV_32F blob, A/B
vs the scalar fallback, median speedup:
| functor | M4 clang/NEON | A76 gcc/NEON | Threadripper gcc | Xeon W-2235 gcc | functor | M4 | A76 | TR | Xeon |
|---|---|---|---|---|---|---|---|---|---|
| TanH | 9.1× | 2.3× | 7.3× | 7.3× | Sinh | 3.3× | 3.2× | 5.2× | 5.0× |
| Cos | 8.8× | 3.0× | 3.3× | 3.0× | Acosh | 3.2× | 2.9× | 2.7× | 2.6× |
| Log | 7.8× | 3.0× | 2.5× | 2.4× | Cosh | 2.8× | 3.2× | 2.6× | 2.5× |
| Sin | 7.8× | 3.0× | 3.4× | 2.9× | Exp | 2.4× | 2.5× | 1.4× | 1.5× |
| GeluApprox | 7.6× | 2.4× | 6.6× | 5.8× | BNLL | 4.2× | 2.0× | 1.6× | 1.4× |
| Atanh | 7.0× | 3.6× | 6.0× | 5.2× | Softplus | 2.0× | 2.0× | 3.3× | 2.7× |
| Tan | 6.8× | 3.6× | 6.6× | 6.0× | Erf | 4.2× | 2.4× | 3.9× | 3.5× |
| Asinh | 3.9× | 2.4× | 5.8× | 5.7× | | | | | |
Every functor is a win on every tested out-of-order core (worst case 1.4×). On the **in-order
Cortex-A55** (a55-tuned build) all 15 measure 0.985–0.99× — neutral, within noise: the SIMD
polynomials are dependency-chain-bound, which an in-order pipeline can neither accelerate nor (here)
slow down. So: meaningful wins on out-of-order ARM + x86, no regression on in-order ARM.
### Accuracy
Uses the `v_exp`/`v_log`/`v_erf`/`v_sin`/`v_cos` polynomials OpenCV already ships and relies on (Gelu
via `v_erf`, Mish/Sigmoid via `v_exp`). Verified vs scalar `libm` on 320 K random elements per functor
with domain-correct inputs (Acosh x≥1, Atanh |x|<1, …): **max abs error ≤ 3e-5** (most ≤ 2e-6), zero
elements exceeding rel>1e-3 & abs>1e-4. ONNX conformance and the layer accuracy tests exercise these ops.
### Pull Request Readiness Checklist
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch (4.x)
- [ ] There is a reference to the original bug report and related work — N/A (perf improvement)
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable — perf test added (`Layer_Activation`, `SANITY_CHECK_NOTHING`); accuracy covered by existing ONNX conformance/layer tests
- [ ] The feature is well documented and sample code can be built with the project CMake — N/A (internal optimization, no new API)
Extended fusion for activation functions in new DNN engine #28750
After this fusion, we see following improvements in YOLO models:
| Model | Before (`ENGINE_NEW`) | After (`ENGINE_NEW`) | `ENGINE_ORT` | % Improvement (Before v/s After) |
| :--- | :--- | :--- | :--- | :--- |
| **YOLOv8n** | 18.89 ms| 12.06 ms| 12.15 ms| 36.16% |
| **YOLOv5n** | 17.12 ms| 9.29 ms| 9.23 ms| 45.73% |
| **YOLOX-S** | 38.78 ms| 25.56 ms| 25.16 ms| 34.09% |
Device details:
- Model name: Intel(R) Core(TM) i9-14900KS, x86, 32 Cores, ubuntu 24.04,
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Block layout-based convolution in DNN #28585
merge together with https://github.com/opencv/opencv_extra/pull/1321
Some core parts of the new engine in DNN module have been revised substantially:
1. all tests seem to pass, except for `Test_Graph_Simplifier.ResizeSubgraph`, which has been disabled because it does not take the newly added `TransformLayoutLayer` into account. The test should be reworked perhaps.
1. convolution and related operations (maxpool/avgpool) now use so-called block layout (`DATA_LAYOUT_BLOCK`), where `NxCxHxW` tensors are represented as `NxC1xHxWxC0`, where `C1=(C + C0-1)/C0` and `C0` is a power-of-two (usually 4, 8, 16 or 32).
1. graph is now pre-processed and `TransformLayoutLayer` is inserted to convert data from NCHW or NHWC layout to the block layout or vice versa. The transformations are done in a lazy way only when they are really needed. For example, in the whole Resnet only 2 transformations are performed.
1. transformer-based models and other models that do not use convolutions will run as usual, without going to block layout.
1. there is yet another graph preprocessing stage added that embeds constant weights/scale and bias into convolution and batch norm layers.
1. 'batchnorm', 'activation' and 'adding a residual' are now fused with convolution, just like in the old engine. That brings some noticeable acceleration.
1. optimized convolution kernels have been added.
* depthwise convolution, as well as maxpool and avgpool support C0=4, 8, 16 etc. _as long as_ C0 is divisible by the number of fp32 lanes in a SIMD register of the target platform (e.g. on ARM with NEON there must be `C0 % 4 == 0`, on x64 with AVX2 `C0 % 8 == 0`).
* non-depthwise convolution only supports C0=8 for now. C0=8 seems to be a sweetspot for ARM with NEON, x64 with AVX2 or RISC-V with RVV (with 128- or 256-bit registers). For some platforms with dedicated matrix accelerators C0=16 or even C0=32 might be more efficient, but we could add the respective kernels later.
* only fp32 kernels have been added. fp16/bf16 kernels might be added a little later.
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
- [x] Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
* added empty set support
* accept unnamed dynamic dims
* Fix for LSTM test failure
* removed converToND
* openvino failing test fix
* ARM CI issue fix
* ARM issue fix
Extended Reduce layer support in new DNN engine #27816
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
TFLite fixes for Face Blendshapes V2 #27307
### Pull Request Readiness Checklist
* Scalars support
* Better handling of 1D tensors
* New ops import: SUB, SQRT, DIV, NEG, SQUARED_DIFFERENCE, SUM
* Number of NHWC<->NCHW layouts compatibility improvements
resolves#27211
**Merge with extra**: https://github.com/opencv/opencv_extra/pull/1257
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
dnn: optimize activations with v_exp #25881
Merge with https://github.com/opencv/opencv_extra/pull/1191.
This PR optimizes the following activations:
- [x] Swish
- [x] Mish
- [x] Elu
- [x] Celu
- [x] Selu
- [x] HardSwish
### Performance (Updated on 2024-07-18)
#### AmLogic A311D2 (ARM Cortex A73 + A53)
```
Geometric mean (ms)
Name of Test activations activations.patch activations.patch
vs
activations
(x-factor)
Celu::Layer_Elementwise::OCV/CPU 115.859 27.930 4.15
Elu::Layer_Elementwise::OCV/CPU 27.846 27.003 1.03
Gelu::Layer_Elementwise::OCV/CPU 0.657 0.602 1.09
HardSwish::Layer_Elementwise::OCV/CPU 31.885 6.781 4.70
Mish::Layer_Elementwise::OCV/CPU 35.729 32.089 1.11
Selu::Layer_Elementwise::OCV/CPU 61.955 27.850 2.22
Swish::Layer_Elementwise::OCV/CPU 30.819 26.688 1.15
```
#### Apple M1
```
Geometric mean (ms)
Name of Test activations activations.patch activations.patch
vs
activations
(x-factor)
Celu::Layer_Elementwise::OCV/CPU 16.184 2.118 7.64
Celu::Layer_Elementwise::OCV/CPU_FP16 16.280 2.123 7.67
Elu::Layer_Elementwise::OCV/CPU 9.123 1.878 4.86
Elu::Layer_Elementwise::OCV/CPU_FP16 9.085 1.897 4.79
Gelu::Layer_Elementwise::OCV/CPU 0.089 0.081 1.11
Gelu::Layer_Elementwise::OCV/CPU_FP16 0.086 0.074 1.17
HardSwish::Layer_Elementwise::OCV/CPU 1.560 1.555 1.00
HardSwish::Layer_Elementwise::OCV/CPU_FP16 1.536 1.523 1.01
Mish::Layer_Elementwise::OCV/CPU 6.077 2.476 2.45
Mish::Layer_Elementwise::OCV/CPU_FP16 5.990 2.496 2.40
Selu::Layer_Elementwise::OCV/CPU 11.351 1.976 5.74
Selu::Layer_Elementwise::OCV/CPU_FP16 11.533 1.985 5.81
Swish::Layer_Elementwise::OCV/CPU 4.687 1.890 2.48
Swish::Layer_Elementwise::OCV/CPU_FP16 4.715 1.873 2.52
```
#### Intel i7-12700K
```
Geometric mean (ms)
Name of Test activations activations.patch activations.patch
vs
activations
(x-factor)
Celu::Layer_Elementwise::OCV/CPU 17.106 3.560 4.81
Elu::Layer_Elementwise::OCV/CPU 5.064 3.478 1.46
Gelu::Layer_Elementwise::OCV/CPU 0.036 0.035 1.04
HardSwish::Layer_Elementwise::OCV/CPU 2.914 2.893 1.01
Mish::Layer_Elementwise::OCV/CPU 3.820 3.529 1.08
Selu::Layer_Elementwise::OCV/CPU 10.799 3.593 3.01
Swish::Layer_Elementwise::OCV/CPU 3.651 3.473 1.05
```
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
* added v_erf and implemented gelu acceleration via vectorization
* remove anonymous v_erf and use v_erf from intrin_math
* enable perf for ov and cuda backend
Added bool support to dnn #25605
Added bool support to dnn pipeline (CPU, OpenVINO and CUDA pipelines).
Added bool support to these layers(CPU and OpenVINO):
- Equal, Greater, GreaterOrEqual, Less, LessOrEqual
- Not
- And, Or, Xor
- Where
Enabled all the conformance tests for these layers.
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
dnn cleanup: On-fly-quantization removal #2498
On-fly-quantization is first introduced via https://github.com/opencv/opencv/pull/20228.
We decided to remove it but keep int8 layers implementation because on-fly-quantization
is less practical given the fact that there has been so many dedicated tools for model
quantization.
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
dnn: cleanup of halide backend for 5.x #24231
Merge with https://github.com/opencv/opencv_extra/pull/1092.
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake