Commit Graph

11 Commits

Author SHA1 Message Date
Savya Sanchi Sharma
0f51622bac Merge pull request #30439 from SavyaSanchi-Sharma:layers
dnn(cuda): transformer layer coverage and CUDA Graph replay on the new engine - #30439

  Follow-up to #30415 (ResNet layer coverage and cuDNN JIT conv/pooling on the new engine). Adds CUDA support on the new DNN engine
  for the layers transformer models need, and brings the CUDA path to parity with ONNX Runtime.
**Companion PR** opencv/opencv_extra#1448

  **Changes**
  - CUDA execs for Attention, Cast, ConstantOfShape, Expand, Gather, Not, Reduce, Shape, Slice, Softmax, Tile, Transpose, Concat,
  Gelu, LayerNormalization, MatMul, Split, Squeeze, Unsqueeze and the element-wise activations.
  - A graph with static shapes is recorded as one CUDA Graph on the second `forward()` and replayed from then on.
  - The cuDNN JIT convolution reads NCHW buffers in place; NHWC staging is kept only for FP16 grouped convolutions.
  - TF32 for cuBLAS GEMMs (`OPENCV_DNN_CUDA_ALLOW_TF32`), FP32 accumulation for FP16 GEMMs
  (`OPENCV_DNN_CUDA_FP16_HALF_ACCUMULATE`), and fused MatMul/Gemm post-ops.
  - Batch-broadcast MatMul and Shape are safe to capture in a CUDA Graph; the unused pointer-array `cublas::gemmBatched` is
  removed.
  - Tests: `forward()` runs three times on batch-broadcast MatMul and Shape models; `MatMul_init_bcast` runs on CUDA again;
  `AttentionSharedShapeReshape` uses the FP16 default tolerance on CUDA FP16.
  - Perf: new CUDA-only `*_CUDA_pinned` tests pin the input shapes.

  **Performance** (ms, median of 50, FP32 with TF32, RTX PRO 6000 Blackwell, idle GPU)

| Model | CPU | CUDA (cuDNN JIT) | CUDA (full cuDNN) | ORT 1.31 CUDA EP |
  |---|---|---|---|---|
  | ResNet-50 | 8.02 | 0.80 | 0.80 | 0.88 |
  | ResNet-18 | 3.27 | 0.44 | 0.44 | 0.48 |
  | MobileNetV2 | 1.99 | 0.28 | 0.27 | 0.40 |
  | DenseNet-121 | 17.42 | 1.48 | 1.58 | 1.58 |
  | EfficientNet-Lite4 | 5.03 | 0.53 | 0.54 | 0.73 |
  | ViT-B/32 | 11.06 | 1.02 | 1.02 | 0.95 |
  | ViT-Base/16 | 24.38 | 1.16 | 1.17 | 1.13 |
  | DeiT-Tiny | 4.58 | 0.65 | 0.65 | 0.68 |
  | MobileViT-XS | 6.93 | 0.72 | 0.72 | 0.77 |
  | YOLOv3 | 47.21 | 2.60 | 2.59 | 2.72 |
  | YOLOv4 | 109.38 | 3.58 | 3.61 | 4.18 |
  | YOLOv4-tiny | 7.31 | 0.64 | 0.64 | 0.66 |
  | BEiT-Base | 25.30 | 1.24 | 1.24 | 1.19 |
  | BERT | 10.00 | 0.97 | 0.97 | 0.99 |
  | SigLIP | 41.14 | 2.09 | 2.09 | 2.44 
### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [ ] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
2026-11-03 10:29:32 +03:00
Prasad Ayush Kumar
36836bb4cc Merge pull request #30313 from Prasadayus:dnn-block-layout-optimization
DNN: Optimize block layout handling and fold Mul/Add into BatchNorm - #30313

Requires: https://github.com/opencv/opencv_extra/pull/1439

### PR Changes

- AveragePool runs on plain NCHW input without conversion to block layout; with an avgpool HAL hook registered, block layout is kept as before.
- NaryEltwise with a scalar constant operand runs in block layout.
- BatchNorm absorbs following Mul/Add by scalar or per-channel constants into its scale and bias; ChainFuser marks a [C,1,1] / [1,C,1,1] constant per channel once it proves the input is 4D.
- Global average/max pooling reduces block input directly.
- An elementwise op feeding a conversion to plain now runs after it, on C channels instead of C0 lanes.
- Slice2 on the channel axis keeps block layout for C0-aligned or runtime bounds.

### Performance

Measured on i9-14900KS, default threads (32).

| Model       | 5.x        | After PR   | ORT  (1.30)     | OpenCV before vs after | After PR vs ORT |
|-------------|-----------:|-----------:|-----------:|-----------------------:|----------------:|
| RAFT        | 965.67 ms  | 691.03 ms  | 604.26 ms  | 1.40x                  | 0.87x           |
| DenseNet121 | 14.14 ms   | 9.84 ms    | 12.05 ms   | 1.44x                  | 1.22x           |
| XFeat       | 5.80 ms    | 5.63 ms    | 2.95 ms    | 1.03x                  | 0.52x           |
| NAFNet      | 1304.50 ms | 1195.67 ms | 1407.77 ms | 1.09x                  | 1.18x           |
| RealESRGAN  | 422.57 ms  | 403.93 ms  | 534.29 ms  | 1.05x                  | 1.32x           |
| PPOCRv3_det | 14.25 ms   | 14.15 ms   | 16.32 ms   | 1.01x                  | 1.15x           |

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
2026-10-29 08:41:09 +03:00
Abhishek Gola
3f426d1f1b Merge pull request #30207 from abhishek-gola:core-drop-nd-suffix
drop ND prefix from the functions - #30207

Merge pull request #30207 from abhishek-gola:core-drop-nd-suffix

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
2026-10-14 11:26:22 +03:00
Abhishek Gola
20e367198c Merge pull request #30081 from abhishek-gola:core-nd-data-movement
core: unified N-D data-movement engine - #30081

closes: https://github.com/opencv/opencv/issues/29315 

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
2026-10-02 16:20:57 +03:00
Prasad Ayush Kumar
30e2e0aaff Merge pull request #29786 from Prasadayus:layers_dtype_coverage_increase
Extend data type support in elementwise, gather and scatter layers - #29786

### PR Changes:

### Support added, per layer

  | Layer | Types added | Gate / kernel |
  |---|---|---|
  | Abs | 8U, 8S, 16U, 16S, 32U, 32S, 64U, 64S | Gate + new templated integer kernel |
  | Sign | 8U, 8S, 16U, 16S, 32U, 32S, 64U, 64S | Gate + same templated kernel |
  | Neg | 32S | Gate only. The kernel already had a `CV_32S` branch |
  | GatherElements | 64F, 16U, 16S, 32U, 64U | Gate + element-width dispatch |
  | Scatter | 64F, 16U, 16S, 32U, 64U | Gate + dispatch arms, all five reductions checked |
  | ScatterND | 64F, 16U, 16S, 32U, 64U | Gate + dispatch arms, same as Scatter |
  | GatherND | 64F, 16U, 16S, 32U, 64U | Gate + element-width dispatch |
  | Slice2 | (fix) | Kernel. Wrong-width copy, see below |


  `Abs`/`Sign` use one template over all widths with the signed/unsigned split resolved at compile time;
  unsigned `abs` short-circuits to `copyTo` and unsigned `sign` reduces to `x != 0`. `GatherElements` and
  `GatherND` only move elements, so their per-dtype arms collapsed to four widths. `GatherND` also unified a
  target-conditional gate that split `16F`/`32F` by target in a file with no OpenCL path.

  `Slice2` had two duplicated depth chains that both fell through to `run_parallel<float>`, a 4-byte copy, so
  `64F`/`64U` truncated and `16U`/`16S` read and wrote past the element. Reachable only when the innermost axis
  has `step != 1`, which is why the float32 slice tests passed. Now dispatches on `elemSize()`, matching
  `pad2_layer.cpp`.

  **Two further fixes:** signed-overflow UB in the int64 `Power`/`Neg` path (`sp[i] * scale` is undefined at
  `INT64_MIN`, now multiplied through `uint64_t`), and `CV_OCL_RUN` now skips integer depths, since the OCL
  activation kernels are float math and `CV_32S` would have gone through a 24-bit mantissa.

  **New accuracy tests in `test_int.cpp`**: `Test_Abs_Int`, `Test_Sign_Int`, `Test_Neg_Int`, `Test_Scatter_Int`,
  `Test_GatherND_Int`, with `Test_GatherElements_Int` and `Test_ScatterND_Int` widened to nine depths.

  Removed `test_slice_start_out_of_bounds` from the parser denylist.

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
2026-08-27 09:26:14 +03:00
Abhishek Gola
bf0cf34963 Merge pull request #29126 from abhishek-gola:flash_attention
Attention graph fusion with MLAS FlashAttention #29126

Performance numbers for Owl-v2 model on intel i9:
```
ORT: Average inference time over 10 runs: 1411.55 ms (min 1399.75, max 1438.89)
NEW: Average inference time over 10 runs: 1078 ms (min 1048.04, max 1110.61)
```
### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
2026-05-27 09:32:27 +03:00
Naresh
e791995f10 Merge pull request #29029 from nareshmlx:fix/dnn-slice2-empty-range
[Bug Fix] Slice2 empty-range crash in new DNN engine#29029

Closes: https://github.com/opencv/opencv/issues/29023
Merge with: https://github.com/opencv/opencv_extra/pull/1363

## Summary

ENGINE_NEW asserted `CV_Assert(outsz >= 0)` in `slice2_layer.cpp` when an ONNX Slice op had `start > end` with positive step. Per ONNX spec, such slices are valid and 
produce an empty-range output (size 0). The hard assertion crashed SwinIR inference on ENGINE_NEW.

## Fix

- Replaced the fatal assertion with a graceful clamp: when `outsz < 0`, set `outsz = 0` and `end = start`.
- Moved `allEnds[axis] = end` to after the clamp so downstream shape inference receives a consistent value (the previous order propagated un-clamped garbage end values
into `normalize_axis()`, causing SIGSEGV).

## Test

Adds `Reproducibility_SwinIR_ONNX` accuracy test in `test_model.cpp`. Test data registered in opencv_extra PR with matching branch name `fix/dnn-slice2-empty-range`.

- Without fix: 3/3 FAIL with `slice2_layer.cpp:124: (-215:Assertion failed) outsz >= 0`
- With fix: 3/3 PASS (CPU, OCL, OCL_FP16)

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
    Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
2026-05-21 18:25:18 +03:00
Abhishek Gola
72e0bc2bf3 Merge pull request #28957 from abhishek-gola:fusion_block_layout_extension
Extend attention fusion for runtime QK scale and add MatMul to Gemm rewriter #28957

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
2026-05-13 08:32:00 +03:00
Abhishek Gola
c83b86eb57 Merge pull request #28811 from abhishek-gola:qlinear_support
Added QLinear layer support #28811

closes: https://github.com/opencv/opencv/issues/26310

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
2026-04-28 17:38:29 +03:00
Hanbin Bae
ca0d47c8e6 Merge pull request #28447 from Anemptyship:optimize/slice-layer
optimize(dnn): optimize Slice layer for strided inputs #28447

## Description
This PR optimizes the `SliceLayer` implementation for strided inputs (where `step > 1`). The original implementation used a recursive element-wise copy (`getSliceRecursive`) for any strided slice, which was extremely inefficient.

This PR introduces:
1.  **Parallelization**: Uses `cv::parallel_for_` to parallelize the outermost dimension of the slice operation.
2.  **Memcpy Optimization**: Automatically detects "pseudo-contiguous" blocks in strided slices (e.g., slicing an outer dimension but keeping inner dimensions intact) and uses `std::memcpy` instead of scalar loops.
3.  **Refactoring**: Replaces the recursive function with a dedicated `ParallelSlice` loop body.

## Impact
Significant performance improvement for strided slice operations (common in detection heads, strided sampling, etc.).

**Benchmark Results:**
| Test Case | Before (ms) | After (ms) | Speedup |
| :--- | :--- | :--- | :--- |
| **Strided Axis 0** `[::2, ...]` | 1.10 | **0.02** | **~55x** |
| **Strided Axis 2** `[..., ::2]` | 1.10 | **0.06** | **~18x** |
| **Contiguous** (Baseline) | 0.10 | 0.10 | 1.0x (Unchanged) |

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
2026-02-07 13:17:23 +03:00
Vadim Pisarevsky
3cd57ea09e Merge pull request #26056 from vpisarev:new_dnn_engine
New dnn engine #26056

This is the 1st PR with the new engine; CI is green and PR is ready to be merged, I think.
Merge together with https://github.com/opencv/opencv_contrib/pull/3794

---

**Known limitations:**
* [solved] OpenVINO is temporarily disabled, but is probably easy to restore (it's not a deal breaker to merge this PR, I guess)
* The new engine does not support any backends nor any targets except for the default CPU implementation. But it's possible to choose the old engine when loading a model, then all the functionality is available.
* [Caffe patch is here: #26208] The new engine only supports ONNX. When a model is constructed manually or is loaded from a file of different format (.tf, .tflite, .caffe, .darknet), the old engine is used.
* Even in the case of ONNX some layers are not supported by the new engine, such as all quantized layers (including DequantizeLinear, QuantizeLinear, QLinearConv etc.), LSTM, GRU, .... It's planned, of course, to have full support for ONNX by OpenCV 5.0 gold release. When a loaded model contains unsupported layers, we switch to the old engine automatically  (at ONNX parsing time, not at `forward()` time).
* Some layers , e.g. Expat, are only partially supported by the new engine. In the case of unsupported flavours it switches to the old engine automatically (at ONNX parsing time, not at `forward()` time).
* 'Concat' graph optimization is disabled. The optimization eliminates Concat layer and instead makes the layers that generate tensors to be concatenated to write the outputs to the final destination. Of course, it's only possible when `axis=0` or `axis=N=1`. The optimization is not compatible with dynamic shapes since we need to know in advance where to store the tensors. Because some of the layer implementations have been modified to become more compatible with the new engine, the feature appears to be broken even when the old engine is used.
* Some `dnn::Net` API is not available with the new engine. Also, shape inference may return false if some of the output or intermediate tensors' shapes cannot be inferred without running the model. Probably this can be fixed by a dummy run of the model with zero inputs.
* Some overloads of `dnn::Net::getFLOPs()` and `dnn::Net::getMemoryConsumption()` are not exposed any longer in wrapper generators; but the most useful overloads are exposed (and checked by Java tests).
* [in progress] A few Einsum tests related to empty shapes have been disabled due to crashes in the tests and in Einsum implementations. The code and the tests need to be repaired.
* OpenCL implementation of Deconvolution is disabled. It's very bad and very slow anyway; need to be completely revised.
* Deconvolution3D test is now skipped, because it was only supported by CUDA and OpenVINO backends, both of which are not supported by the new engine.
* Some tests, such as FastNeuralStyle, checked that the in the case of CUDA backend there is no fallback to CPU. Currently all layers in the new engine are processed on CPU, so there are many fallbacks. The checks, therefore, have been temporarily disabled.

---

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
2024-10-16 15:28:19 +03:00