dnn(cuda): move the CUDA backend to UMat device buffers - #29807
This PR is in continuation with #29658 and #30259
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [ ] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
dnn: speed up LSTM2 (gate math, input projection, recurrence) - #30309
Speeds up the ONNX LSTM layer (`LSTM2`) in `recurrent2_layers.cpp`:
- Default sigmoid/tanh gates are computed on raw rows with the dispatched activation kernels. Peephole, clip and custom activations keep the old path.
- The input projection runs before the per-direction loop, so the GEMM uses all threads. It goes through MLAS when available.
- Batch 1 (hidden size >= 128): hidden units are split across threads, with one `parallel_for_` per timestep.
- Batch > 1: one packed GEMM per step replaces one GEMV per row.
Ryzen 9 9950X, default threads, 5.x vs this PR:
| Model | 5.x, ms | PR, ms | x-factor |
|---|:-:|:-:|:-:|
| `DNNTestNetwork.CRNN` | 2.693 | 1.562 | 1.72 |
| CRNN EN 2021sep | 2.498 | 1.472 | 1.70 |
| crnn.onnx | 2.650 | 1.434 | 1.85 |
| LSTM2 bidirectional, batch 1, in 512, hidden 256, 24 steps (CRNN layer 1) | 0.710 | 0.212 | 3.36
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [ ] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
The fusion read only num_heads and head_dim from the QKV Reshape target,
so a graph reshaping Q/K/V to [B, T*S, H, D] fused the same way as one
reshaping to [B*T, S, H, D], and the fused layer computed the second for
both, giving a wrong result for the first.
The fusion now records the first two target entries, the batch and the
sequence length the attention runs over, each as a constant times a
product of input dims, or as inferred. It reads constants and chains of
Shape, Gather, Mul, Unsqueeze, Squeeze and Cast (as in the SAM3 decoder).
If an entry is computed some other way, or Q, K and V disagree, the block
is left unfused. The layer evaluates the record against the input shape
and runs over that grouping, which is a view of the same contiguous
tokens. Without a record, as for the ONNX Attention op, the input must be
3D.
The output has v_hidden values per token, but getMemoryShapes() sized it
from the input width, so a V wider than the input wrote past the output
buffer. Size it from v_hidden for both output ranks, and test V widths
below, equal to and above the input width.
- Attention Q/K/V projection: pack each (Q/K/V, head) weight slice with
MLAS and run all per-head GEMMs in one multithreaded
mlasSgemmPackedBatch() call; the fastGemm loop stays as the fallback.
- MLAS: split each operation into 4 tiles per thread on hybrid CPUs.
- forward(): hand the private output Mats to the caller instead of
copying them again.
- Attention: accept [..., S, C] input (SAM3 decoder).
- Move the activation clip clamp and holdFinishedRows into
recurrent_activations.hpp and call them from both GRU and RNN; drop the
redundant useClip flag, since clip > 0 is the switch.
- Select the GRU outputs positionally, matching getMemoryShapes(), instead
of guessing Y / Y_h from their shape and keeping the old single-output
may-be-Y_h policy.
- Reject direction=reverse together with two-direction weights instead of
silently running both sweeps backwards, and drop the unreachable assert.
- Collapse the four inlined GRU references into one shared helper and drop
the stale sentence from the SingleOutputIsSequenceY comment.
- Drop the hand-made hPrev allocation: hInternal.copyTo() already sizes and
types it.
- Gate the GRU reference on the input timestep rather than the loop
position, so reverse + sequence_lens skips the padding instead of the
valid steps, and cover that combination with a test.
GlobalAveragePool and GlobalMaxPool accept N x C x D1 x ... x Dn inputs, but
the layer only worked for at most two spatial dimensions: the parameter
vectors derived from the layer params hold one pad/stride entry per recognized
dimension (kernel_d/h/w), so a 5-D input hit
"CV_CheckLE(addedDims, pads_begin.size())" in getMemoryShapes and the model
could not be loaded at all.
The quantized pooling layer shares the same parameters, so it silently read
past the pad/stride vectors and produced garbage output shapes (and read
garbage kernels) in the same situation.
Fix by giving the paddings (0) and the strides (1) as many entries as the
global kernel has spatial dimensions, both in the shape computation and in
finalize. The extra entries are inserted at the front so that the parameters
stay trailing-aligned with the spatial dimensions, and a check rejects inputs
with more spatial dimensions than the layer can describe instead of reading
the per-dimension flags out of bounds.
Add squeeze/unsqueeze/flatten/reshape for N-D arrays - #30091
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Reducing over more than two axes walks the reduced sub-block with an odometer
over all of them but the innermost one. When an index carried into the next
axis, the step of that axis was assigned instead of being accumulated, so the
steps of every axis above the one that wrapped were lost and the corresponding
elements were read again and again:
axes [0, 2, 3] on a 3x3x3x3 input, shape (3,3,3,3), steps (27, 9, 3, 1)
projected steps: 0 3 6 27 30 33 27 30 33 <- should end in 54 57 60
Only the first output slice happened to be right, because the carry that
introduced the wrong step is the one that moves to the second one.
The same accounting exists for the unprojected axes just below, which
subtracts the step of the wrapped axis and adds the step of the axis it carries
into; the projected loop now does the same. This fixes test_mvn_expanded and
test_mvn_expanded_ver18 (a MeanVarianceNormalization decomposed into
primitives, denied as "Wrong answer" until now) plus any model that reduces
over three or more axes, since the inner projection also feeds the unprojected
axis steps. The OpenVINO backend still answers test_mvn_expanded_ver18
incorrectly, so it is filtered in
test_onnx_conformance_layer_filter__openvino.inl.hpp.
Found by comparing against onnxruntime: 9 of 200 random (rank, shape, axes set,
keepdims, op) combinations were wrong before and none are now; the two
conformance cases above and a new layer test fail without this change.
Softmax and LogSoftmax up to ONNX opset 12 coerce the input to the 2D tensor
[a_0*...*a_{axis-1}, a_axis*...*a_{n-1}] and reduce over its second dimension, so
axis=1 on a 2x3x4 input normalises the 12 values of the flattened tail per sample,
not the 3 values of that axis. The importer only picked the right default axis for
those opsets and the layer always reduced the single axis, which differed from
onnxruntime by up to 0.63 on such models. opset 12 is included as well: Softmax-11
is still the schema in effect there.
The layer now takes the pre-opset-13 semantics through a `coerced_2d` parameter
(the name the int8 Softmax already uses) and normalises the contiguous 2D view of
the blob. The OCL kernels reduce a single axis, so that case stays on the CPU path.
The Exp/ReduceSum/Div subgraph fold produces a Softmax node too, but it matches a
single-axis ReduceSum, which is not the pre-13 meaning, so it now marks the fused
node with coerced_2d=0 to keep the per-axis semantics of the subgraph it replaces.
Tests: a layer test comparing a 2x3x4 input against the definition, and two ONNX
models whose opset_import is 11 (data added to opencv_extra by the companion
change). Both fail on the previous code with normInf 0.63 and 4.38.
Reported-by: differential testing against ONNX Runtime 1.30 CPU.
`clip` bounds the input of the activations to [-clip, clip]. The layer never
read it, so a model asking for it computed the unclipped gates instead - off by
up to 0.56 on the cases checked here, with no warning.
Apply it to the two places the gates are activated: the z/r pair before
sigmoid() and the candidate before tanh(), i.e. after the recurrent bias has
been folded in. It stays off unless the attribute is present and positive, so
models without it keep the same code path.
Checked against ONNX Runtime 1.30 (CPU): GRU with clip agrees to ~1e-7 for
clip = 0.4 and 0.05, with and without linear_before_reset, forward, reverse and
bidirectionally, with an initial h and with sequence_lens. The new
Layer_GRU_Test_Accuracy_.Clip clamps the gate pre-activations in its reference
and fails on Y and Y_h without the patch.
The ONNX conformance GRU cases do not set clip, so nothing covered this.
The layer never read input #4, so a node with sequence_lens ran every batch
entry to the end of the padded sequence: Y was not zero-padded past each entry's
own length, and Y_h was the state after the whole sequence instead of the one
that entry finished with. RNN2 is the only recurrent layer that implemented it.
Take RNN2's semantics, which are the spec's: an entry stops updating once it
reaches its length, keeping the state it finished with, and contributes zeros to
Y from there on. That also holds for a lone reverse direction, where the scan
starts at the last timestep but the valid range still ends at each entry's own
length.
Checked against ONNX Runtime 1.30 (CPU): GRU with sequence_lens - forward,
reverse and bidirectionally, with and without linear_before_reset, with initial
h and with fewer entries than the batch - agrees to 6e-8 - 1.9e-7 after the
change (it was 0.2 - 0.9 off before). The new
Layer_GRU_Test_Accuracy_.SequenceLens covers a sample at full length, a
truncated one and an empty one, and fails on Y and Y_h without the patch.
The ONNX conformance suite does not cover this: its test_gru_seq_length and
test_rnn_seq_length cases are named for a longer test sequence but carry no
sequence_lens input at all.
The layer only ever compared `direction` against "bidirectional", so a node
with direction=reverse ran forwards: Y came out in forward order and Y_h was
the state left after the last timestep instead of the first. Walking the
sequence backwards only happened for the second direction of a bidirectional
node, while LSTM2/RNN2 honour a lone reverse direction.
Parse the attribute into a `reverse` flag and use it wherever the scan order is
picked. Checked against ONNX Runtime 1.30: the same model with
direction=reverse now matches, and Layer_GRU_Test_Accuracy_.ReverseDirection
compares Y and Y_h against the recurrence evaluated from the last timestep
down. The ONNX conformance suite has no reverse-direction GRU case.
fuseMatMulConstBToGemm() rewrites MatMul(A, const B) into a Gemm with
flatten_a=false so that A keeps its leading dims. ONNX MatMul accepts a
rank-1 A: it is promoted to (1, K) and the leading 1 is squeezed from the
result again, so Y is (N,). GemmLayerImpl::getMemoryShapes() rejected that
with "Tensor A must be n-dimensional (n >= 2)", so a model like
MatMul(A[3], const B[3,6]) only loaded while B was still a runtime input;
once the fusion fired it threw. finalize() and forward() indexed
shape_A[dims_A - 2] / shape_Y[dims_Y - 2] unconditionally and would have
read out of bounds for the same shapes.
flatten_a=false is only produced by that rewrite, so gate the relaxed rank
check on it: a plain ONNX Gemm (flatten_a defaults to true) still requires a
2-D A, which is what the op specifies. Where the extra dimension is needed,
treat the rank-1 A / (N,) Y as M == 1 and use A.view as (1, K), matching
what MatMul produced before the rewrite.
Add Test_Gemm.Rank1AFromMatMulRewrite covering fp32/fp64 with and without a
bias, plus a check that a plain Gemm still rejects a rank-1 A. It fails
before the fix with the reported assertion.
For the ONNX recurrent ops output #0 is Y, the hidden state of every
timestep, and output #1 is Y_h, the last one. A node that declares a single
output can only mean Y: asking for Y_h alone requires output #0 to be
declared as an empty name, which makes it a second output, so the declared
count never collapses to one.
The GRU nevertheless answered with Y_h in that case, while LSTM2 and RNN2
both answer Y for the same node. Give slot #0 the Y shape unconditionally
when a single output is requested, which makes resolveOutputPolicy() unused.
Add Layer_GRU_Test_Accuracy_.SingleOutputIsSequenceY, written from the ONNX
GRU equations for requiredOutputs == 1; it fails before the fix and passes
after. The existing ONNX conformance GRU cases declare ['', 'Y_h'] and are
unaffected.
MatMulHelper::updatePackedBOffsets derived the per-slice stride of the
packed constant weight as packed_B_size / batch, where `batch` is the
number of matrices in the *output*, i.e. after broadcasting the batch
dims. fastGemmPackB stores one packed matrix per batch slice actually
present in B, so when B is broadcast over the output batch dims the two
counts differ and every slice beyond the first was read at the wrong
offset. The output had the correct shape but wrong values: only the
first index along a broadcast dim matched.
Remember the number of slices stored in B (B_batch) in compute() and use
it to derive the packed stride.
Add a regression test using A(2,1,3,4) @ B(1,5,4,6); it fails before the
fix and passes after.
Generalize tokenizer loading to support method-based family dispatch in DNN - #29675
Companion PR: https://github.com/opencv/opencv_extra/pull/1402
### Changes
Added ALBERT and BERT support end-to-end, with `samples/dnn/albert_inference.py `and `samples/dnn/bert_inference.py `as validation samples, plus expanded coverage in modules/dnn/test/test_tokenizer.cpp. Required changes to `cv::dnn::dnn.hpp`, `graph_fusion_attention.cpp`, and `unicode.cpp/unicode.hpp` to support Unigram and WordPiece tokenizers.
To back ALBERT/BERT, generalized `cv::dnn::Tokenizer `from a single BPE implementation into a method-dispatched frontend, adding `core_wordpiece.cpp/hpp` (WordPiece) and `core_unigram.cpp/hpp` (Unigram) as new backends. `tokenizer.cpp` now routes by method across BPE, Gemma,, SentencePiece, Unigram, and WordPiece behind one shared interface.
Tested against the following samples and the output matches to old tokenizer:
```
gpt2_inference.py
qwen_inference.py
gemma3_inference.py
```
GPT2:
```
Preparing GPT-2 model...
Inferencing GPT-2 model...
Hello, I'm a language model, not a programming language. I'm a language model. I'm a language model. I'm a language model. I'm a language model. I'm a
```
Gemma3:
```
Preparing Gemma3 model...
Prompt:
<start_of_turn>user
What is OpenCV?<end_of_turn>
<start_of_turn>model
Inferencing Gemma3 model...
Response:
Okay, let's break down what OpenCV is.
**What is OpenCV?**
OpenCV (Open Source Computer Vision Library) is a powerful and
```
Qwen2.5:
```
Preparing Qwen2.5 model...
Prompt:
<|im_start|>user
What is OpenCV?<|im_end|>
<|im_start|>assistant
Inferencing Qwen2.5 model...
Response:
OpenCV is a set of computer vision libraries in C++ designed to be used for image and video processing. It provides a wide range of tools and functions for
```
### Tokenizer References
Byte-level BPE: [tokenizers/src/pre_tokenizers/byte_level.rs](https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/pre_tokenizers/byte_level.rs)
(This defines the byte-level mapping rules, which is used in conjunction with the [BPE model](https://www.google.com/search?q=https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/models/bpe/mod.rs))
SentencePiece BPE (Metaspace): [tokenizers/src/pre_tokenizers/metaspace.rs](https://www.google.com/search?q=https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/pre_tokenizers/metaspace.rs)
(This defines the rule for replacing whitespace with the U+2581 _ character and handling byte fallback)
Unigram: [tokenizers/src/models/unigram/mod.rs](https://www.google.com/search?q=https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/models/unigram/mod.rs)
(This contains the core logic for the Unigram lattice scoring and probabilistic tokenization rules)
WordPiece: [tokenizers/src/models/wordpiece/mod.rs](https://www.google.com/search?q=https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/models/wordpiece/mod.rs)
(This explicitly cites Schuster & Nakajima in the code comments and implements the greedy longest-match rule with the ## prefix)
### Pull Request Readiness Checklist
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on
code under GPL or another license incompatible with OpenCV.
- [x] The PR is proposed to the proper branch (`5.x`).
- [x] There is a reference to the original bug report and related work.
- [x] There is accuracy test and test data in `opencv_extra`, same
branch name (`generalized-tokenizer`) — `bert/`, `t5/` fixtures
back the new C++ tests.
- [x] The feature is documented and sample code builds with project CMake.
Remove deprecated CommaInitializer API - #29948
This goes hand in hand with https://github.com/opencv/opencv_contrib/pull/4217
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
Added Bitcast layer & extended MatMul and DFT layers support - #29594
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Dynamic KV-cache support - #29642
The core idea is: reserveKVCache() API to pre-allocate memory for attention caches upfront, which eliminates allocation overhead during token decoding. For LLM inference, simply call reserveKVCache(prompt_len + max_new_tokens) before the prefill stage so the decode loop runs without page allocations, significantly reducing per-token latency for models like Gemma3 and Qwen.
Speedups after this PR on AMD Ryzen 9 9950X 16-Core Processor device:
Qwen2.5-0.5B-Instruct, fp32, CPU, tok/s:
```
Tokens Before After Speedup
64 12.49 23.72 1.90×
128 10.37 23.14 2.23×
256 7.20 22.40 3.11×
512 4.25 21.03 4.95×
```
Gemma 3 1B-it, fp32, CPU, 512 tokens :
```
Tokens Before After Speedup
64 6.99 11.84 1.69×
128 5.84 11.72 2.01×
256 4.17 11.50 2.76×
512 2.47 11.15 4.51×
```
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Caffe importer cleanup #28678
Merge with: https://github.com/opencv/opencv_extra/pull/1324
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
- [x] The feature is well documented and sample code can be built with the project CMake
Added SDPA layer (Scaled Dot Product Attention) #29104
Merge with: https://github.com/opencv/opencv_extra/pull/1374
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Add KV cache with paged attention and prefetch #29127
Closes: https://github.com/opencv/opencv/issues/27159
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
FP32 KV Cache #28840
OpenCV Extra: https://github.com/opencv/opencv_extra/pull/1348
## This PR introduces basic (Paged ) KV-Cache to use on CPU
### Summary:
1. To ensure proper gemm-prepacking,
1.1. The Page Size of Key Cache is currently hardcoded as `FAST_GEMM_F32_NR`(which is 8, 12 or 16 depending on CPU architecture)
1.2. The Page Size of Values Cache is hardcoded as `FAST_GEMM_F32_PACKED_STRIDE_K`
2. there are two phases supported - prefill & generate.
2.1. prefill grows cache by `N` tokens and is allowed **only** for empty cache
2.2. generate grows cache by 1 token.
2.3. **Improtant**: it is currently not allowed to grow non-empty cache by more than one token at a time (thisbehaviour is sufficient for normal LLM querying, but should be extended if we want to implement speculative decoding)
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [ ] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
Fixed subgraph name scoping in new DNN engine #28971
closes: https://github.com/opencv/opencv/issues/23663, https://github.com/opencv/opencv/issues/19977
OpenCV extra: https://github.com/opencv/opencv_extra/pull/1362
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Fix dnn NaN bugs in GELU SIMD and softmax for large inputs #28836
### Bug Description
- **GELU SIMD bug:** `exp(2*inner)` overflows to `inf` for large inputs (e.g. `x=10.6` → `inner≈51`), causing `inf/inf = NaN`; fixed by clamping `inner` to [-9, 9] as `tanh` already saturates to ±1.0 beyond this range.
- **Softmax bug:** All `-inf` inputs (masked attention rows) produce `sum=0`, then `1/0 = inf` and `0*inf = NaN`;
fixed by outputting zeros when `sum == 0`.
- Add regression tests for both fixes
Depends on : https://github.com/opencv/opencv/pull/28837
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Extended fusion for activation functions in new DNN engine #28750
After this fusion, we see following improvements in YOLO models:
| Model | Before (`ENGINE_NEW`) | After (`ENGINE_NEW`) | `ENGINE_ORT` | % Improvement (Before v/s After) |
| :--- | :--- | :--- | :--- | :--- |
| **YOLOv8n** | 18.89 ms| 12.06 ms| 12.15 ms| 36.16% |
| **YOLOv5n** | 17.12 ms| 9.29 ms| 9.23 ms| 45.73% |
| **YOLOX-S** | 38.78 ms| 25.56 ms| 25.16 ms| 34.09% |
Device details:
- Model name: Intel(R) Core(TM) i9-14900KS, x86, 32 Cores, ubuntu 24.04,
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
resolves https://github.com/opencv/opencv/issues/27580
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Use MatShape instead of MatSize inside cv::Mat/cv::UMat #27757
**Merge together with https://github.com/opencv/opencv_contrib/pull/3996**
---
This PR continues cv::Mat/cv::UMat refactoring. See #26056, where `MatShape` was introduced. Now it's put inside cv::Mat/cv::UMat instead of a weird `MatSize`. MatSize is now an alias for MatShape:
**before:**
```
struct MatShape { ... };
struct MatSize { ... };
struct Mat {
...
int dims;
int rows;
int cols;
...
MatShape shape() const { ... /* constructs MatShape out of MatSize and returns it;
layout is always 'unknown', because we don't store it */ }
MatSize size; // size is not valid without the parent cv::Mat,
// because size.p may point to Mat::rows or to Mat::cols,
// depending on the dimensionality, and dims() returns Mat::dims.
MatStep step; // may allocate memory, depending on the dimensionality.
...
};
```
**after:**
```
struct MatShape { ... };
typedef MatShape MatSize; // they are now synonyms
struct Mat {
...
int dims;
int rows;
int cols;
...
MatShape shape() const { return size; } // just return the embedded shape (including the proper layout information)
MatSize size; // size is self-contained data structure that can be used without the parent cv::Mat.
// size.dims is now a copy of dims; size.p[*] contains copies of Mat::rows and Mat::cols when dims <= 2.
MatStep step; // does not allocate extra memory buffers.
...
};
```
There are several reasons to do that:
1. the main reason is to be able to store data layout (MatShape::layout) inside each cv::Mat/cv::UMat. This is necessary for the proper shape inference in DNN module. In particular, it's necessary for the next step of DNN inference optimization where we introduce block-layout-optimized convolution and other operations. Later on, we can use layout information to support non-interleaved images (e.g. RRR...GGG...BBB...) or even batches of such images in core/imgproc modules.
2. the other reason is to represent 3D/4D/5D etc. tensors as cv::Mat/cv::UMat instances more conveniently, without extra dynamic memory allocation. Before this patch we allocated some memory buffers dynamically to store shape & steps for more than 2D arrays. Now the whole cv::Mat/cv::UMat header can be stored completely on stack/in a container. Creating another copy of Mat/UMat header is now done more efficiently.
3. the third reason is to introduce the new coding pattern: `dst.create(src.size, <dst_type>);`. The pattern is suitable for most of element-wise (including cloning) and filtering operations. This pattern does not only look crisp and self-documenting, it will also automatically copy shape (including layout) from the source tensor into the destination matrix/tensor.
4. in the future we might add `colorspace` member to MatShape that will allow to distinguish RGB from BGR or NV12. `dst.create(src.size, <dst_type>);` will then copy the colorspace information as well.
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Added Size layer to new DNN engine #27656
Merge with https://github.com/opencv/opencv_extra/pull/1274
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
IfLayer add to new DNN engine #27508
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
LSTM layer for new graph engine. #26391
Merge with extra: https://github.com/opencv/opencv_extra/pull/1218
This PR updates/creates LSTM layer compatible with new graph engine. It is based on previous LSTM implementation with some modification on how initializers blobs are processed.
Note: Following tests are currently are disabled
Two following two tests are disbled since ONNNRuntime does not support `layout=1` attiribute inference. See a detailed issue #26456 on this.
- `LSTM_layout_seq`
- `LSTM_layout_batch`
Following test fails with the new engine as it is not able to deal with shapes of the form [?, C, H, W]
- `LSTM_Activations`
Works:
- [x] One directional case any batch type
- [x] Fix directional case when batch size large than 1
- [x] Add peepholes attribute
TODO with the next PRs:
- [ ] Activation support
Note:
> Currently `LSTM_layout_seq`, `LSTM_layout_batch` are disabled as the tests are incorrect. They do not comply with the ONNX standard. Particularly test outputs are of incorrect dimensionality. They produce 3-dimentinal output instead of 4-dimentional.
------
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Modified Caffe parser to support the new dnn engine #26208
Now the Caffe parser supports both the old and the new engine. It can be selected using newEngine argument in PopulateNet.
All cpu Caffe tests work fine except:
- Test_Caffe_nets.Colorization
- Test_Caffe_layers.FasterRCNN_Proposal
Both these tests doesn't work because of the bug in the new net.forward function. The function takes the name of the desired target last layer, but uses this name as the name of the desired output tensor.
Also Colorization test contains a strange model with a Silence layer in the end, so it doesn't have outputs. The old parser just ignored it. I think, the proper solution is to run this model until the (number_of_layers - 2) layer using proper net.forward arguments in the test.
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [ ] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake