dnn(cuda): transformer layer coverage and CUDA Graph replay on the new engine - #30439
Follow-up to #30415 (ResNet layer coverage and cuDNN JIT conv/pooling on the new engine). Adds CUDA support on the new DNN engine
for the layers transformer models need, and brings the CUDA path to parity with ONNX Runtime.
**Companion PR** opencv/opencv_extra#1448
**Changes**
- CUDA execs for Attention, Cast, ConstantOfShape, Expand, Gather, Not, Reduce, Shape, Slice, Softmax, Tile, Transpose, Concat,
Gelu, LayerNormalization, MatMul, Split, Squeeze, Unsqueeze and the element-wise activations.
- A graph with static shapes is recorded as one CUDA Graph on the second `forward()` and replayed from then on.
- The cuDNN JIT convolution reads NCHW buffers in place; NHWC staging is kept only for FP16 grouped convolutions.
- TF32 for cuBLAS GEMMs (`OPENCV_DNN_CUDA_ALLOW_TF32`), FP32 accumulation for FP16 GEMMs
(`OPENCV_DNN_CUDA_FP16_HALF_ACCUMULATE`), and fused MatMul/Gemm post-ops.
- Batch-broadcast MatMul and Shape are safe to capture in a CUDA Graph; the unused pointer-array `cublas::gemmBatched` is
removed.
- Tests: `forward()` runs three times on batch-broadcast MatMul and Shape models; `MatMul_init_bcast` runs on CUDA again;
`AttentionSharedShapeReshape` uses the FP16 default tolerance on CUDA FP16.
- Perf: new CUDA-only `*_CUDA_pinned` tests pin the input shapes.
**Performance** (ms, median of 50, FP32 with TF32, RTX PRO 6000 Blackwell, idle GPU)
| Model | CPU | CUDA (cuDNN JIT) | CUDA (full cuDNN) | ORT 1.31 CUDA EP |
|---|---|---|---|---|
| ResNet-50 | 8.02 | 0.80 | 0.80 | 0.88 |
| ResNet-18 | 3.27 | 0.44 | 0.44 | 0.48 |
| MobileNetV2 | 1.99 | 0.28 | 0.27 | 0.40 |
| DenseNet-121 | 17.42 | 1.48 | 1.58 | 1.58 |
| EfficientNet-Lite4 | 5.03 | 0.53 | 0.54 | 0.73 |
| ViT-B/32 | 11.06 | 1.02 | 1.02 | 0.95 |
| ViT-Base/16 | 24.38 | 1.16 | 1.17 | 1.13 |
| DeiT-Tiny | 4.58 | 0.65 | 0.65 | 0.68 |
| MobileViT-XS | 6.93 | 0.72 | 0.72 | 0.77 |
| YOLOv3 | 47.21 | 2.60 | 2.59 | 2.72 |
| YOLOv4 | 109.38 | 3.58 | 3.61 | 4.18 |
| YOLOv4-tiny | 7.31 | 0.64 | 0.64 | 0.66 |
| BEiT-Base | 25.30 | 1.24 | 1.24 | 1.19 |
| BERT | 10.00 | 0.97 | 0.97 | 0.99 |
| SigLIP | 41.14 | 2.09 | 2.09 | 2.44
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [ ] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
Remove Tensorflow Importer - #29474
[PR OpenCV Extra](https://github.com/opencv/opencv_extra/pull/1390)
[PR OpenCV Contrib](https://github.com/opencv/opencv_contrib/pull/4171)
Removed tensorflow importers
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
core: reduceND, used by the DNN Reduce2 layer - #30157
Split out of #30082, which now carries only the resize half.
Adds `cv::reduceND`, which reduces an n-dimensional array over a set of axes with any of the `ReduceTypes`, extended with `REDUCE_PROD`, `REDUCE_L1`, `REDUCE_L2`, `REDUCE_LOG_SUM` and `REDUCE_LOG_SUM_EXP`. `cv::reduce` accepts n-dimensional input through it.
`sum`, `mean` and `minMaxIdx` now share one parallel driver (`stat_parallel.hpp`), so they parallelize over large arrays the same way `reduceND` does.
The new graph engine's `Reduce2` layer calls `cv::reduceND` instead of its own kernels, so `reduce2_kernels.simd.hpp` and the hand-written reduction loops in `reduce2_layer.cpp` go away.
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Reducing over more than two axes walks the reduced sub-block with an odometer
over all of them but the innermost one. When an index carried into the next
axis, the step of that axis was assigned instead of being accumulated, so the
steps of every axis above the one that wrapped were lost and the corresponding
elements were read again and again:
axes [0, 2, 3] on a 3x3x3x3 input, shape (3,3,3,3), steps (27, 9, 3, 1)
projected steps: 0 3 6 27 30 33 27 30 33 <- should end in 54 57 60
Only the first output slice happened to be right, because the carry that
introduced the wrong step is the one that moves to the second one.
The same accounting exists for the unprojected axes just below, which
subtracts the step of the wrapped axis and adds the step of the axis it carries
into; the projected loop now does the same. This fixes test_mvn_expanded and
test_mvn_expanded_ver18 (a MeanVarianceNormalization decomposed into
primitives, denied as "Wrong answer" until now) plus any model that reduces
over three or more axes, since the inner projection also feeds the unprojected
axis steps. The OpenVINO backend still answers test_mvn_expanded_ver18
incorrectly, so it is filtered in
test_onnx_conformance_layer_filter__openvino.inl.hpp.
Found by comparing against onnxruntime: 9 of 200 random (rank, shape, axes set,
keepdims, op) combinations were wrong before and none are now; the two
conformance cases above and a new layer test fail without this change.
SIMD Kernel speedup for DNN layers #28889
This PR add following speedups for Grounding Dino tiny model.
For Device: Intel(R) Core(TM) i9-14900KS, x86, 32 Cores, ubuntu 24.04,
| Model | `ENGINE_NEW (Before)` | `ENGINE_NEW (After)` | `ENGINE_ORT` |
| :--- | :--- | :--- | :--- |
| **Grounding Dino Tiny** | 3130 ms | 1872.06 ms| 1800.18 ms|
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
* added empty set support
* accept unnamed dynamic dims
* Fix for LSTM test failure
* removed converToND
* openvino failing test fix
* ARM CI issue fix
* ARM issue fix
[BUG FIX] Incorrect shape assert bug fix in 5.x #28575
Closes: https://github.com/opencv/opencv/issues/28563
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Extended Reduce layer support in new DNN engine #27816
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake