mirror of
https://github.com/opencv/opencv.git
synced 2026-11-10 02:35:59 -06:00
0f51622bac
dnn(cuda): transformer layer coverage and CUDA Graph replay on the new engine - #30439 Follow-up to #30415 (ResNet layer coverage and cuDNN JIT conv/pooling on the new engine). Adds CUDA support on the new DNN engine for the layers transformer models need, and brings the CUDA path to parity with ONNX Runtime. **Companion PR** opencv/opencv_extra#1448 **Changes** - CUDA execs for Attention, Cast, ConstantOfShape, Expand, Gather, Not, Reduce, Shape, Slice, Softmax, Tile, Transpose, Concat, Gelu, LayerNormalization, MatMul, Split, Squeeze, Unsqueeze and the element-wise activations. - A graph with static shapes is recorded as one CUDA Graph on the second `forward()` and replayed from then on. - The cuDNN JIT convolution reads NCHW buffers in place; NHWC staging is kept only for FP16 grouped convolutions. - TF32 for cuBLAS GEMMs (`OPENCV_DNN_CUDA_ALLOW_TF32`), FP32 accumulation for FP16 GEMMs (`OPENCV_DNN_CUDA_FP16_HALF_ACCUMULATE`), and fused MatMul/Gemm post-ops. - Batch-broadcast MatMul and Shape are safe to capture in a CUDA Graph; the unused pointer-array `cublas::gemmBatched` is removed. - Tests: `forward()` runs three times on batch-broadcast MatMul and Shape models; `MatMul_init_bcast` runs on CUDA again; `AttentionSharedShapeReshape` uses the FP16 default tolerance on CUDA FP16. - Perf: new CUDA-only `*_CUDA_pinned` tests pin the input shapes. **Performance** (ms, median of 50, FP32 with TF32, RTX PRO 6000 Blackwell, idle GPU) | Model | CPU | CUDA (cuDNN JIT) | CUDA (full cuDNN) | ORT 1.31 CUDA EP | |---|---|---|---|---| | ResNet-50 | 8.02 | 0.80 | 0.80 | 0.88 | | ResNet-18 | 3.27 | 0.44 | 0.44 | 0.48 | | MobileNetV2 | 1.99 | 0.28 | 0.27 | 0.40 | | DenseNet-121 | 17.42 | 1.48 | 1.58 | 1.58 | | EfficientNet-Lite4 | 5.03 | 0.53 | 0.54 | 0.73 | | ViT-B/32 | 11.06 | 1.02 | 1.02 | 0.95 | | ViT-Base/16 | 24.38 | 1.16 | 1.17 | 1.13 | | DeiT-Tiny | 4.58 | 0.65 | 0.65 | 0.68 | | MobileViT-XS | 6.93 | 0.72 | 0.72 | 0.77 | | YOLOv3 | 47.21 | 2.60 | 2.59 | 2.72 | | YOLOv4 | 109.38 | 3.58 | 3.61 | 4.18 | | YOLOv4-tiny | 7.31 | 0.64 | 0.64 | 0.66 | | BEiT-Base | 25.30 | 1.24 | 1.24 | 1.19 | | BERT | 10.00 | 0.97 | 0.97 | 0.99 | | SigLIP | 41.14 | 2.09 | 2.09 | 2.44 ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [ ] The PR is proposed to the proper branch - [ ] There is a reference to the original bug report and related work - [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [ ] The feature is well documented and sample code can be built with the project CMake
