* dnn: check weight, bias and past_state in CausalConvWithState
Check the ONNX shapes of weight [C, 1, k], bias [C] and past_state [B, C, k - 1] against the input in getMemoryShapes, and require them to be float in forward, since the kernel indexes all three by input channel and reads them as float.
* dnn: check CausalConvWithState input types before fp16 conversion, require rank-1 bias
dnn: derive the Resize scales from the given scales, not from the shapes - #30105
### Pull Request Readiness Checklist
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch (5.x)
- [x] There is a reference to the original bug report and related work (no pre-existing report; found by differential testing against ONNX Runtime, see *Problem*)
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable (companion PR: opencv/opencv_extra#1455)
- [x] The feature is well documented and sample code can be built with the project CMake
---
### Problem
`Resize` resamples with a step of `in/out` when the target size comes from the `scales` input,
but ONNX defines the step along an axis as `1/scale`. The two agree only while `in*scale` is an
integer, which is not the common case. A `1x2x4x5` input with `scales=[1,1,0.5,0.5]` produces a
`1x2x2x2` output, so the step along `W` is `2` (as `1/0.5`) and not `2.5` (as `5/2`):
| `Resize(nearest=False, mode=linear, coordinate_transformation_mode=half_pixel)` | ONNX Runtime 1.30 | before |
|---|---|---|
| `1x2x4x5` input, `scales=0.5`, first output row | `0.49208999 -0.11570000` | `0.28683999 -0.48815000` |
The error is not restricted to downscaling. Any non-integral `in*scale` is affected, e.g. a
7-wide input with `scale=1.5` (output 10) was sampled at `0.7` instead of `2/3`. `cubic`,
`pytorch_half_pixel`, `half_pixel_symmetric`, `tf_half_pixel_for_nn` and `asymmetric` are all
affected; the antialias filter support is stretched the same way, producing a `2.5`-pixel-wide
filter where `1/scale` asks for `2`.
### Root cause
A constant `scales` input is folded into the `zoom_factor_x`/`zoom_factor_y` layer parameters by
`ONNXImporter2::parseResize2` (`ninputs` is set to 1). `Resize2LayerImpl::forward` therefore never
saw the tensor and took the step from `updateOutSizeAndScale`, which derives it from the shapes:
```cpp
if (sizes.empty() && !scales.empty() && halfPixelCenters) // never true for folded scales
scaleHeight = 1.f / scales[hIdx];
...
scaleHeight = float(inpShape[2]) / outHeight; // in/out instead of 1/scale
```
The antialias branch had the same problem for its filter support, which it also read from
`scales` (empty in the folded case) or fell back to `outShape/inpShape`.
### Solution
The scales are resolved once, from the `scales` input or from the folded `zoom_factor_*`, and both
the step and the antialias filter use that value:
```cpp
if (sizes.empty() && !alignCorners && onnxScaleH > 0 && onnxScaleW > 0 &&
coordTransModeE != CoordTransMode::TF_CROP_AND_RESIZE)
{
scaleHeight = 1.f / onnxScaleH;
scaleWidth = 1.f / onnxScaleW;
}
```
`align_corners` keeps the step `updateOutSizeAndScale()` derives from the shapes, `(in - 1) / (out - 1)`
with the integer output length, which is what the operator definition and onnxruntime use:
| `Resize(align_corners, mode=linear)` | ONNX Runtime 1.29 | this PR |
|---|---|---|
| `1x1x2x4`, `scales=0.6`, output `1x1x1x2` | `1 4` | `1 4` |
| `1x1x2x4`, `scales=0.6`, output `1x1x3x3`, `mode=cubic` | `1 2.5 4 7 8.5 10 13 14.5 16` | same |
Two stored conformance references still carry the value produced by the ONNX reference
implementation while it divided by the unrounded `input * scale` ([onnx/onnx#8495](https://github.com/onnx/onnx/issues/8495)).
They are regenerated from onnxruntime 1.29 in opencv/opencv_extra#1455, which is what
`test_resize_downsample_scales_{linear,cubic}_align_corners` then compares against.
`tf_crop_and_resize` ignores the step entirely (its grid comes from the roi) and is untouched.
### Tests
Three accuracy tests in `modules/dnn/test/test_layers.cpp`, all on the folded-scale path with an
input whose `in*scale` is not integral, and each fails on the previous code:
| test | covers |
|---|---|
| `Layer_Test_Resize.downscale_uses_given_scales` | step = `1/scale` for `half_pixel`, bilinear |
| `Layer_Test_Resize.antialias_downscale_uses_given_scales` | antialias filter of support `1/scale` |
| `Layer_Test_Resize.align_corners_downscale` | step = `(in - 1) / (out - 1)` and exact corners |
The references are built from the operator definition: `src = (dst + 0.5)/scale - 0.5` for
`half_pixel`, the ONNX antialias triangle filter for the second, and `src = dst * (in - 1)/(out - 1)`
for `align_corners`.
### Validation
- `opencv_test_dnn`: 5204/5204, no regressions.
- `Test_ONNX_conformance.Layer_Test/test_resize_*`: 40/40 with opencv/opencv_extra#1455 applied,
including the antialias and `half_pixel_symmetric` cases that were already green.
- Differential sweep against ONNX Runtime 1.29/1.30 CPU: the failing `Resize` configurations
(`linear`/`cubic`, 6 coordinate transformation modes, `antialias`, `cubic_coeff_a` and
`exclude_outside` combinations) now agree; every other case in the sweep is unchanged.
- Not tested: Windows/MSVC, Clang, ARM/RISC-V, CUDA/OpenVINO/OpenCL targets (no access to those
configurations here). The patch only changes which scale the CPU kernels are given; the
`scaleWidth`/`scaleHeight` members are not consumed by the CUDA/CANN node builders, which keep
using `outHeight`/`outWidth` and the coordinate mode flags.
dnn(onnx): add support for ONNX Compress operator - #30576
### Description
This PR adds support for the ONNX `Compress` operator (opset 9-24) to the OpenCV DNN dynamic engine:
- Implements `CompressLayer` in `modules/dnn/src/layers/compress_layer.cpp` supporting dynamic output shapes (`dynamicOutputShapes() { return true; }`).
- Implements full ONNX `Compress` semantics:
- Slices along a specified axis when the `axis` attribute is provided (supports negative axis indexing via `normalize_axis`).
- Flattens the input tensor to 1D before compression when the `axis` attribute is omitted.
- Discards elements exceeding condition length when condition length is shorter than target axis dimension (`std::min(dimAxis, condTotal)`).
- Handles all-false condition ($K=0$) cleanly with early return after fitting the valid empty output shape (`[..., 0, ...]`).
- Enforces continuous tensor invariants with `CV_Assert(data.isContinuous())` and validates `condition` as `CV_Bool` per ONNX spec.
- Slicing optimized via OpenCV native `parallel_for_` flattening multi-dimensional iteration domains to 1D (`totalSlices = outer * K`).
- Un-denylists and passes all 4 official ONNX conformance tests in `test_onnx_conformance_layer_parser_denylist.inl.hpp`.
- Marks `Compress` conformance tests with `SKIP;` in `test_onnx_conformance_layer_filter__openvino.inl.hpp` to ensure OpenVINO CI remains green.
- Adds focused Google Test unit tests in `modules/dnn/test/test_layers.cpp` verifying empty output behavior on all-false condition both with and without `axis`.
### Test Results
All 4 ONNX conformance tests + 2 all-false unit tests pass locally:
```text
[==========] Running 6 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 2 tests from Test_Compress
[ RUN ] Test_Compress.AllFalse_WithAxis
[ OK ] Test_Compress.AllFalse_WithAxis (29 ms)
[ RUN ] Test_Compress.AllFalse_WithoutAxis
[ OK ] Test_Compress.AllFalse_WithoutAxis (0 ms)
[----------] 2 tests from Test_Compress (30 ms total)
[----------] 4 tests from Test_ONNX_conformance
[ RUN ] Test_ONNX_conformance.Layer_Test/test_compress_0_OCV_CPU
[ OK ] Test_ONNX_conformance.Layer_Test/test_compress_0_OCV_CPU (901 ms)
[ RUN ] Test_ONNX_conformance.Layer_Test/test_compress_1_OCV_CPU
[ OK ] Test_ONNX_conformance.Layer_Test/test_compress_1_OCV_CPU (465 ms)
[ RUN ] Test_ONNX_conformance.Layer_Test/test_compress_default_axis_OCV_CPU
[ OK ] Test_ONNX_conformance.Layer_Test/test_compress_default_axis_OCV_CPU (456 ms)
[ RUN ] Test_ONNX_conformance.Layer_Test/test_compress_negative_axis_OCV_CPU
[ OK ] Test_ONNX_conformance.Layer_Test/test_compress_negative_axis_OCV_CPU (559 ms)
[----------] 4 tests from Test_ONNX_conformance (2383 ms total)
[----------] Global test environment tear-down
[==========] 6 tests from 2 test cases ran. (2413 ms total)
[ PASSED ] 6 tests.
```
### Checklist
- [x] I have read the [OpenCV contribution guidelines](https://github.com/opencv/opencv/wiki/How_to_contribute).
- [x] My code follows the OpenCV coding style guidelines.
- [x] I have tested my changes locally and all relevant unit tests pass.
- [x] No unrelated files or whitespace changes are included.
DNN: Optimize block layout handling and fold Mul/Add into BatchNorm - #30313
Requires: https://github.com/opencv/opencv_extra/pull/1439
### PR Changes
- AveragePool runs on plain NCHW input without conversion to block layout; with an avgpool HAL hook registered, block layout is kept as before.
- NaryEltwise with a scalar constant operand runs in block layout.
- BatchNorm absorbs following Mul/Add by scalar or per-channel constants into its scale and bias; ChainFuser marks a [C,1,1] / [1,C,1,1] constant per channel once it proves the input is 4D.
- Global average/max pooling reduces block input directly.
- An elementwise op feeding a conversion to plain now runs after it, on C channels instead of C0 lanes.
- Slice2 on the channel axis keeps block layout for C0-aligned or runtime bounds.
### Performance
Measured on i9-14900KS, default threads (32).
| Model | 5.x | After PR | ORT (1.30) | OpenCV before vs after | After PR vs ORT |
|-------------|-----------:|-----------:|-----------:|-----------------------:|----------------:|
| RAFT | 965.67 ms | 691.03 ms | 604.26 ms | 1.40x | 0.87x |
| DenseNet121 | 14.14 ms | 9.84 ms | 12.05 ms | 1.44x | 1.22x |
| XFeat | 5.80 ms | 5.63 ms | 2.95 ms | 1.03x | 0.52x |
| NAFNet | 1304.50 ms | 1195.67 ms | 1407.77 ms | 1.09x | 1.18x |
| RealESRGAN | 422.57 ms | 403.93 ms | 534.29 ms | 1.05x | 1.32x |
| PPOCRv3_det | 14.25 ms | 14.15 ms | 16.32 ms | 1.01x | 1.15x |
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
dnn: check weights and soft labels shape in SoftmaxCrossEntropyLoss 🤖🤖🤖 - #30530
`SoftmaxCrossEntropyLossImpl::forward` reads the optional `weights` input as a float array of `C` class weights (`weightBase[y]`, and `wData[c]` on the probability-label path) but never checks its length or type. The probability-label path also reads `labels` with the scores layout (n, c, s) without checking that the labels have the scores shape. Both tensors come straight from the ONNX model, so a short weights initializer or a `[1, C]` labels tensor against `[2, C]` scores reads past the buffer.
Before: weights of shape `[1]` with `C = 64` and labels `C - 1` gives an ASan heap-buffer-overflow READ in `reduceSCEPerSample`. Probability labels `[1, 64]` against scores `[2, 64]` give a READ of size 16 in the forward lambda.
After: both cases throw from `forward`. Weights are checked against `C` and converted to `CV_32F`, which `NegativeLogLikelihoodLoss` already does. Valid models are unchanged, and all 104 `test_sce_*` / `test_nllloss_*` conformance cases pass on OCV/CPU.
Tradeoff: one extra `convertTo` of a `C`-element vector per forward call. In exchange, `float16`/`double` weights are now read with their real values instead of being reinterpreted as float.
A regression test was added to `test_layers.cpp`. It fails under ASan without the fix and passes with it.
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
Check the [1, 1, N, 7] boxes layout in getMemoryShapes so the output size and the box loop agree on every backend. Emit the extrapolation value for samples outside [0, H - 1] x [0, W - 1] and sample the box center for 1-pixel crops, on both the CPU and CUDA paths.
dnn: fix GatherND division by zero on empty indices - #30503
GatherND divided by zero in `forward_impl` when the indices tensor is empty, which kills the process with SIGFPE on x86-64 (seen with `LightGlueMatcher` when nothing matches). The layer now returns early when the output is empty.
Fixes#30501
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
dnn: skip MatMul's zero-fill when there is no bias - #30297
Without a bias, MatMul pre-zeroed the whole output with a single-threaded memset and then still passed beta = 1 into the gemm, forcing every kernel to read and multiply-add against that zeroed buffer. Passing beta = 0 instead removes the memset and lets MLAS/fastGemmThin write the product directly. Measured 31-50% faster on a small, write-heavy, no-bias shape (512x32x512, 8 threads); `MatMul.matmul/41` and the new `MatMul.matmul_dynamicB/41` cover the constant-B and dynamic-B paths.
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
dnn: enable vendored MLAS on Windows ARM64 with MSVC - #30332
Close https://github.com/opencv/opencv/issues/30330
After https://github.com/opencv/opencv/pull/30331
### Summary
Enable OpenCV DNN’s vendored MLAS implementation on Windows ARM64 using Microsoft’s C++ compiler (`cl.exe`).
This change builds on the separate MLAS target architecture detection fix.
MLAS is currently disabled on all Windows targets. Enabling ARM64 requires more than removing that restriction: the GNU-style `.S` sources cannot be used by the MSVC toolchain, and the vendored Windows sources lack the SGEMV kernel used by the single-row fast path.
This change:
- Adds the Windows ARM64 SGEMV kernel from the same ONNX Runtime revision as the existing vendored sources.
- Preprocesses the native SGEMM and SGEMV `.asm` files with `cl.exe`, then assembles them with `armasm64`.
- Includes the generated assembly objects in `opencv_dnn`, with configuration-specific output paths.
- Enables MLAS only for Microsoft `cl.exe` targeting ARM64. Other Windows toolchains and targets remain disabled.
- Adds GEMM and SDPA accuracy coverage and documents the Windows build requirements.
The implementation uses the existing upstream `MlasHGemmSupported` probe. No additional HGEMM or FP16 workaround is introduced.
### Validation
On Windows ARM64, based on upstream commit `3074f64`:
- Release and Debug DNN builds linked successfully.
- MLAS and both NEON assembly kernels were enabled.
- All **116 selected tests passed in each configuration**, covering GEMM, single-row GEMV through FlashAttention, transposition, odd-sized tails, and accumulation across FlashAttention blocks.
- CMake guard checks confirmed unsupported Windows compiler/target combinations remain disabled.
A complete Python wheel has not been rebuilt, and performance benchmarks have not been run.
### Related work
- [[opencv/opencv#29343](https://github.com/opencv/opencv/pull/29343)](https://github.com/opencv/opencv/pull/29343): temporary blanket Windows disable; this PR enables only the supported MSVC ARM64 configuration.
- [[opencv/opencv#29885](https://github.com/opencv/opencv/issues/29885)](https://github.com/opencv/opencv/issues/29885): related Windows x86/x64 and clang-cl problems, outside this PR’s scope.
- [[opencv/opencv#29362](https://github.com/opencv/opencv/pull/29362)](https://github.com/opencv/opencv/pull/29362): existing HGEMM capability-probe fix, retained unchanged.
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [ ] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
dnn(cuda): move the CUDA backend to UMat device buffers - #29807
This PR is in continuation with #29658 and #30259
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [ ] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
dnn: speed up LSTM2 (gate math, input projection, recurrence) - #30309
Speeds up the ONNX LSTM layer (`LSTM2`) in `recurrent2_layers.cpp`:
- Default sigmoid/tanh gates are computed on raw rows with the dispatched activation kernels. Peephole, clip and custom activations keep the old path.
- The input projection runs before the per-direction loop, so the GEMM uses all threads. It goes through MLAS when available.
- Batch 1 (hidden size >= 128): hidden units are split across threads, with one `parallel_for_` per timestep.
- Batch > 1: one packed GEMM per step replaces one GEMV per row.
Ryzen 9 9950X, default threads, 5.x vs this PR:
| Model | 5.x, ms | PR, ms | x-factor |
|---|:-:|:-:|:-:|
| `DNNTestNetwork.CRNN` | 2.693 | 1.562 | 1.72 |
| CRNN EN 2021sep | 2.498 | 1.472 | 1.70 |
| crnn.onnx | 2.650 | 1.434 | 1.85 |
| LSTM2 bidirectional, batch 1, in 512, hidden 256, 24 steps (CRNN layer 1) | 0.710 | 0.212 | 3.36
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [ ] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
The fusion read only num_heads and head_dim from the QKV Reshape target,
so a graph reshaping Q/K/V to [B, T*S, H, D] fused the same way as one
reshaping to [B*T, S, H, D], and the fused layer computed the second for
both, giving a wrong result for the first.
The fusion now records the first two target entries, the batch and the
sequence length the attention runs over, each as a constant times a
product of input dims, or as inferred. It reads constants and chains of
Shape, Gather, Mul, Unsqueeze, Squeeze and Cast (as in the SAM3 decoder).
If an entry is computed some other way, or Q, K and V disagree, the block
is left unfused. The layer evaluates the record against the input shape
and runs over that grouping, which is a view of the same contiguous
tokens. Without a record, as for the ONNX Attention op, the input must be
3D.
The output has v_hidden values per token, but getMemoryShapes() sized it
from the input width, so a V wider than the input wrote past the output
buffer. Size it from v_hidden for both output ranks, and test V widths
below, equal to and above the input width.
- Attention Q/K/V projection: pack each (Q/K/V, head) weight slice with
MLAS and run all per-head GEMMs in one multithreaded
mlasSgemmPackedBatch() call; the fastGemm loop stays as the fallback.
- MLAS: split each operation into 4 tiles per thread on hybrid CPUs.
- forward(): hand the private output Mats to the caller instead of
copying them again.
- Attention: accept [..., S, C] input (SAM3 decoder).
- Move the activation clip clamp and holdFinishedRows into
recurrent_activations.hpp and call them from both GRU and RNN; drop the
redundant useClip flag, since clip > 0 is the switch.
- Select the GRU outputs positionally, matching getMemoryShapes(), instead
of guessing Y / Y_h from their shape and keeping the old single-output
may-be-Y_h policy.
- Reject direction=reverse together with two-direction weights instead of
silently running both sweeps backwards, and drop the unreachable assert.
- Collapse the four inlined GRU references into one shared helper and drop
the stale sentence from the SingleOutputIsSequenceY comment.
- Drop the hand-made hPrev allocation: hInternal.copyTo() already sizes and
types it.
- Gate the GRU reference on the input timestep rather than the loop
position, so reverse + sequence_lens skips the padding instead of the
valid steps, and cover that combination with a test.
GlobalAveragePool and GlobalMaxPool accept N x C x D1 x ... x Dn inputs, but
the layer only worked for at most two spatial dimensions: the parameter
vectors derived from the layer params hold one pad/stride entry per recognized
dimension (kernel_d/h/w), so a 5-D input hit
"CV_CheckLE(addedDims, pads_begin.size())" in getMemoryShapes and the model
could not be loaded at all.
The quantized pooling layer shares the same parameters, so it silently read
past the pad/stride vectors and produced garbage output shapes (and read
garbage kernels) in the same situation.
Fix by giving the paddings (0) and the strides (1) as many entries as the
global kernel has spatial dimensions, both in the shape computation and in
finalize. The extra entries are inserted at the front so that the parameters
stay trailing-aligned with the spatial dimensions, and a check rejects inputs
with more spatial dimensions than the layer can describe instead of reading
the per-dimension flags out of bounds.
Add squeeze/unsqueeze/flatten/reshape for N-D arrays - #30091
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Reducing over more than two axes walks the reduced sub-block with an odometer
over all of them but the innermost one. When an index carried into the next
axis, the step of that axis was assigned instead of being accumulated, so the
steps of every axis above the one that wrapped were lost and the corresponding
elements were read again and again:
axes [0, 2, 3] on a 3x3x3x3 input, shape (3,3,3,3), steps (27, 9, 3, 1)
projected steps: 0 3 6 27 30 33 27 30 33 <- should end in 54 57 60
Only the first output slice happened to be right, because the carry that
introduced the wrong step is the one that moves to the second one.
The same accounting exists for the unprojected axes just below, which
subtracts the step of the wrapped axis and adds the step of the axis it carries
into; the projected loop now does the same. This fixes test_mvn_expanded and
test_mvn_expanded_ver18 (a MeanVarianceNormalization decomposed into
primitives, denied as "Wrong answer" until now) plus any model that reduces
over three or more axes, since the inner projection also feeds the unprojected
axis steps. The OpenVINO backend still answers test_mvn_expanded_ver18
incorrectly, so it is filtered in
test_onnx_conformance_layer_filter__openvino.inl.hpp.
Found by comparing against onnxruntime: 9 of 200 random (rank, shape, axes set,
keepdims, op) combinations were wrong before and none are now; the two
conformance cases above and a new layer test fail without this change.
Softmax and LogSoftmax up to ONNX opset 12 coerce the input to the 2D tensor
[a_0*...*a_{axis-1}, a_axis*...*a_{n-1}] and reduce over its second dimension, so
axis=1 on a 2x3x4 input normalises the 12 values of the flattened tail per sample,
not the 3 values of that axis. The importer only picked the right default axis for
those opsets and the layer always reduced the single axis, which differed from
onnxruntime by up to 0.63 on such models. opset 12 is included as well: Softmax-11
is still the schema in effect there.
The layer now takes the pre-opset-13 semantics through a `coerced_2d` parameter
(the name the int8 Softmax already uses) and normalises the contiguous 2D view of
the blob. The OCL kernels reduce a single axis, so that case stays on the CPU path.
The Exp/ReduceSum/Div subgraph fold produces a Softmax node too, but it matches a
single-axis ReduceSum, which is not the pre-13 meaning, so it now marks the fused
node with coerced_2d=0 to keep the per-axis semantics of the subgraph it replaces.
Tests: a layer test comparing a 2x3x4 input against the definition, and two ONNX
models whose opset_import is 11 (data added to opencv_extra by the companion
change). Both fail on the previous code with normInf 0.63 and 4.38.
Reported-by: differential testing against ONNX Runtime 1.30 CPU.
`clip` bounds the input of the activations to [-clip, clip]. The layer never
read it, so a model asking for it computed the unclipped gates instead - off by
up to 0.56 on the cases checked here, with no warning.
Apply it to the two places the gates are activated: the z/r pair before
sigmoid() and the candidate before tanh(), i.e. after the recurrent bias has
been folded in. It stays off unless the attribute is present and positive, so
models without it keep the same code path.
Checked against ONNX Runtime 1.30 (CPU): GRU with clip agrees to ~1e-7 for
clip = 0.4 and 0.05, with and without linear_before_reset, forward, reverse and
bidirectionally, with an initial h and with sequence_lens. The new
Layer_GRU_Test_Accuracy_.Clip clamps the gate pre-activations in its reference
and fails on Y and Y_h without the patch.
The ONNX conformance GRU cases do not set clip, so nothing covered this.
The layer never read input #4, so a node with sequence_lens ran every batch
entry to the end of the padded sequence: Y was not zero-padded past each entry's
own length, and Y_h was the state after the whole sequence instead of the one
that entry finished with. RNN2 is the only recurrent layer that implemented it.
Take RNN2's semantics, which are the spec's: an entry stops updating once it
reaches its length, keeping the state it finished with, and contributes zeros to
Y from there on. That also holds for a lone reverse direction, where the scan
starts at the last timestep but the valid range still ends at each entry's own
length.
Checked against ONNX Runtime 1.30 (CPU): GRU with sequence_lens - forward,
reverse and bidirectionally, with and without linear_before_reset, with initial
h and with fewer entries than the batch - agrees to 6e-8 - 1.9e-7 after the
change (it was 0.2 - 0.9 off before). The new
Layer_GRU_Test_Accuracy_.SequenceLens covers a sample at full length, a
truncated one and an empty one, and fails on Y and Y_h without the patch.
The ONNX conformance suite does not cover this: its test_gru_seq_length and
test_rnn_seq_length cases are named for a longer test sequence but carry no
sequence_lens input at all.
The layer only ever compared `direction` against "bidirectional", so a node
with direction=reverse ran forwards: Y came out in forward order and Y_h was
the state left after the last timestep instead of the first. Walking the
sequence backwards only happened for the second direction of a bidirectional
node, while LSTM2/RNN2 honour a lone reverse direction.
Parse the attribute into a `reverse` flag and use it wherever the scan order is
picked. Checked against ONNX Runtime 1.30: the same model with
direction=reverse now matches, and Layer_GRU_Test_Accuracy_.ReverseDirection
compares Y and Y_h against the recurrence evaluated from the last timestep
down. The ONNX conformance suite has no reverse-direction GRU case.
fuseMatMulConstBToGemm() rewrites MatMul(A, const B) into a Gemm with
flatten_a=false so that A keeps its leading dims. ONNX MatMul accepts a
rank-1 A: it is promoted to (1, K) and the leading 1 is squeezed from the
result again, so Y is (N,). GemmLayerImpl::getMemoryShapes() rejected that
with "Tensor A must be n-dimensional (n >= 2)", so a model like
MatMul(A[3], const B[3,6]) only loaded while B was still a runtime input;
once the fusion fired it threw. finalize() and forward() indexed
shape_A[dims_A - 2] / shape_Y[dims_Y - 2] unconditionally and would have
read out of bounds for the same shapes.
flatten_a=false is only produced by that rewrite, so gate the relaxed rank
check on it: a plain ONNX Gemm (flatten_a defaults to true) still requires a
2-D A, which is what the op specifies. Where the extra dimension is needed,
treat the rank-1 A / (N,) Y as M == 1 and use A.view as (1, K), matching
what MatMul produced before the rewrite.
Add Test_Gemm.Rank1AFromMatMulRewrite covering fp32/fp64 with and without a
bias, plus a check that a plain Gemm still rejects a rank-1 A. It fails
before the fix with the reported assertion.
For the ONNX recurrent ops output #0 is Y, the hidden state of every
timestep, and output #1 is Y_h, the last one. A node that declares a single
output can only mean Y: asking for Y_h alone requires output #0 to be
declared as an empty name, which makes it a second output, so the declared
count never collapses to one.
The GRU nevertheless answered with Y_h in that case, while LSTM2 and RNN2
both answer Y for the same node. Give slot #0 the Y shape unconditionally
when a single output is requested, which makes resolveOutputPolicy() unused.
Add Layer_GRU_Test_Accuracy_.SingleOutputIsSequenceY, written from the ONNX
GRU equations for requiredOutputs == 1; it fails before the fix and passes
after. The existing ONNX conformance GRU cases declare ['', 'Y_h'] and are
unaffected.
MatMulHelper::updatePackedBOffsets derived the per-slice stride of the
packed constant weight as packed_B_size / batch, where `batch` is the
number of matrices in the *output*, i.e. after broadcasting the batch
dims. fastGemmPackB stores one packed matrix per batch slice actually
present in B, so when B is broadcast over the output batch dims the two
counts differ and every slice beyond the first was read at the wrong
offset. The output had the correct shape but wrong values: only the
first index along a broadcast dim matched.
Remember the number of slices stored in B (B_batch) in compute() and use
it to derive the packed stride.
Add a regression test using A(2,1,3,4) @ B(1,5,4,6); it fails before the
fix and passes after.
Generalize tokenizer loading to support method-based family dispatch in DNN - #29675
Companion PR: https://github.com/opencv/opencv_extra/pull/1402
### Changes
Added ALBERT and BERT support end-to-end, with `samples/dnn/albert_inference.py `and `samples/dnn/bert_inference.py `as validation samples, plus expanded coverage in modules/dnn/test/test_tokenizer.cpp. Required changes to `cv::dnn::dnn.hpp`, `graph_fusion_attention.cpp`, and `unicode.cpp/unicode.hpp` to support Unigram and WordPiece tokenizers.
To back ALBERT/BERT, generalized `cv::dnn::Tokenizer `from a single BPE implementation into a method-dispatched frontend, adding `core_wordpiece.cpp/hpp` (WordPiece) and `core_unigram.cpp/hpp` (Unigram) as new backends. `tokenizer.cpp` now routes by method across BPE, Gemma,, SentencePiece, Unigram, and WordPiece behind one shared interface.
Tested against the following samples and the output matches to old tokenizer:
```
gpt2_inference.py
qwen_inference.py
gemma3_inference.py
```
GPT2:
```
Preparing GPT-2 model...
Inferencing GPT-2 model...
Hello, I'm a language model, not a programming language. I'm a language model. I'm a language model. I'm a language model. I'm a language model. I'm a
```
Gemma3:
```
Preparing Gemma3 model...
Prompt:
<start_of_turn>user
What is OpenCV?<end_of_turn>
<start_of_turn>model
Inferencing Gemma3 model...
Response:
Okay, let's break down what OpenCV is.
**What is OpenCV?**
OpenCV (Open Source Computer Vision Library) is a powerful and
```
Qwen2.5:
```
Preparing Qwen2.5 model...
Prompt:
<|im_start|>user
What is OpenCV?<|im_end|>
<|im_start|>assistant
Inferencing Qwen2.5 model...
Response:
OpenCV is a set of computer vision libraries in C++ designed to be used for image and video processing. It provides a wide range of tools and functions for
```
### Tokenizer References
Byte-level BPE: [tokenizers/src/pre_tokenizers/byte_level.rs](https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/pre_tokenizers/byte_level.rs)
(This defines the byte-level mapping rules, which is used in conjunction with the [BPE model](https://www.google.com/search?q=https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/models/bpe/mod.rs))
SentencePiece BPE (Metaspace): [tokenizers/src/pre_tokenizers/metaspace.rs](https://www.google.com/search?q=https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/pre_tokenizers/metaspace.rs)
(This defines the rule for replacing whitespace with the U+2581 _ character and handling byte fallback)
Unigram: [tokenizers/src/models/unigram/mod.rs](https://www.google.com/search?q=https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/models/unigram/mod.rs)
(This contains the core logic for the Unigram lattice scoring and probabilistic tokenization rules)
WordPiece: [tokenizers/src/models/wordpiece/mod.rs](https://www.google.com/search?q=https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/models/wordpiece/mod.rs)
(This explicitly cites Schuster & Nakajima in the code comments and implements the greedy longest-match rule with the ## prefix)
### Pull Request Readiness Checklist
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on
code under GPL or another license incompatible with OpenCV.
- [x] The PR is proposed to the proper branch (`5.x`).
- [x] There is a reference to the original bug report and related work.
- [x] There is accuracy test and test data in `opencv_extra`, same
branch name (`generalized-tokenizer`) — `bert/`, `t5/` fixtures
back the new C++ tests.
- [x] The feature is documented and sample code builds with project CMake.
Remove deprecated CommaInitializer API - #29948
This goes hand in hand with https://github.com/opencv/opencv_contrib/pull/4217
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
Added Bitcast layer & extended MatMul and DFT layers support - #29594
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Dynamic KV-cache support - #29642
The core idea is: reserveKVCache() API to pre-allocate memory for attention caches upfront, which eliminates allocation overhead during token decoding. For LLM inference, simply call reserveKVCache(prompt_len + max_new_tokens) before the prefill stage so the decode loop runs without page allocations, significantly reducing per-token latency for models like Gemma3 and Qwen.
Speedups after this PR on AMD Ryzen 9 9950X 16-Core Processor device:
Qwen2.5-0.5B-Instruct, fp32, CPU, tok/s:
```
Tokens Before After Speedup
64 12.49 23.72 1.90×
128 10.37 23.14 2.23×
256 7.20 22.40 3.11×
512 4.25 21.03 4.95×
```
Gemma 3 1B-it, fp32, CPU, 512 tokens :
```
Tokens Before After Speedup
64 6.99 11.84 1.69×
128 5.84 11.72 2.01×
256 4.17 11.50 2.76×
512 2.47 11.15 4.51×
```
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Caffe importer cleanup #28678
Merge with: https://github.com/opencv/opencv_extra/pull/1324
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
- [x] The feature is well documented and sample code can be built with the project CMake
Added SDPA layer (Scaled Dot Product Attention) #29104
Merge with: https://github.com/opencv/opencv_extra/pull/1374
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Add KV cache with paged attention and prefetch #29127
Closes: https://github.com/opencv/opencv/issues/27159
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
FP32 KV Cache #28840
OpenCV Extra: https://github.com/opencv/opencv_extra/pull/1348
## This PR introduces basic (Paged ) KV-Cache to use on CPU
### Summary:
1. To ensure proper gemm-prepacking,
1.1. The Page Size of Key Cache is currently hardcoded as `FAST_GEMM_F32_NR`(which is 8, 12 or 16 depending on CPU architecture)
1.2. The Page Size of Values Cache is hardcoded as `FAST_GEMM_F32_PACKED_STRIDE_K`
2. there are two phases supported - prefill & generate.
2.1. prefill grows cache by `N` tokens and is allowed **only** for empty cache
2.2. generate grows cache by 1 token.
2.3. **Improtant**: it is currently not allowed to grow non-empty cache by more than one token at a time (thisbehaviour is sufficient for normal LLM querying, but should be extended if we want to implement speculative decoding)
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [ ] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
Fixed subgraph name scoping in new DNN engine #28971
closes: https://github.com/opencv/opencv/issues/23663, https://github.com/opencv/opencv/issues/19977
OpenCV extra: https://github.com/opencv/opencv_extra/pull/1362
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Fix dnn NaN bugs in GELU SIMD and softmax for large inputs #28836
### Bug Description
- **GELU SIMD bug:** `exp(2*inner)` overflows to `inf` for large inputs (e.g. `x=10.6` → `inner≈51`), causing `inf/inf = NaN`; fixed by clamping `inner` to [-9, 9] as `tanh` already saturates to ±1.0 beyond this range.
- **Softmax bug:** All `-inf` inputs (masked attention rows) produce `sum=0`, then `1/0 = inf` and `0*inf = NaN`;
fixed by outputting zeros when `sum == 0`.
- Add regression tests for both fixes
Depends on : https://github.com/opencv/opencv/pull/28837
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Extended fusion for activation functions in new DNN engine #28750
After this fusion, we see following improvements in YOLO models:
| Model | Before (`ENGINE_NEW`) | After (`ENGINE_NEW`) | `ENGINE_ORT` | % Improvement (Before v/s After) |
| :--- | :--- | :--- | :--- | :--- |
| **YOLOv8n** | 18.89 ms| 12.06 ms| 12.15 ms| 36.16% |
| **YOLOv5n** | 17.12 ms| 9.29 ms| 9.23 ms| 45.73% |
| **YOLOX-S** | 38.78 ms| 25.56 ms| 25.16 ms| 34.09% |
Device details:
- Model name: Intel(R) Core(TM) i9-14900KS, x86, 32 Cores, ubuntu 24.04,
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake