Commit Graph

57 Commits

Author SHA1 Message Date
Myron Rodrigues
344f8c6400 Merge pull request #27363 from MRo47:openvino-npu-support
Feature: Add OpenVINO NPU support #27363

## Why
- OpenVINO now supports inference on integrated NPU devices in intel's Core Ultra series processors.
- Sometimes as fast as GPU, but should use considerably less power.

## How
- The NPU plugin is now available as "NPU" in openvino `ov::Core::get_available_devices()`.
- Removed the guards and checks for NPU in available targets for Inference Engine backend.

## Test example

### Pre-requisites
- Intel [Core Ultra series processor](https://www.intel.com/content/www/us/en/products/details/processors/core-ultra/edge.html#tab-blade-1-0)
- [Intel NPU driver](https://github.com/intel/linux-npu-driver/releases)
- OpenVINO 2023.3.0+ (Tested on 2025.1.0)

### Example
```cpp
#include <opencv2/dnn.hpp>
#include <iostream>

int main(){
    cv::dnn::Net net = cv::dnn::readNet("../yolov8s-openvino/yolov8s.xml", "../yolov8s-openvino/yolov8s.bin");
    cv::Size net_input_shape = cv::Size(640, 480);
    std::cout << "Setting backend to DNN_BACKEND_INFERENCE_ENGINE and target to DNN_TARGET_NPU" << std::endl;
    net.setPreferableBackend(cv::dnn::DNN_BACKEND_INFERENCE_ENGINE);
    net.setPreferableTarget(cv::dnn::DNN_TARGET_NPU);

    cv::Mat image(net_input_shape, CV_8UC3);
    cv::randu(image, cv::Scalar(0, 0, 0), cv::Scalar(255, 255, 255));
    cv::Mat blob = cv::dnn::blobFromImage(
        image, 1, net_input_shape, cv::Scalar(0, 0, 0), true, false, CV_32F);
    net.setInput(blob);
    std::cout << "Running forward" << std::endl;
    cv::Mat result = net.forward();
    std::cout << "Output shape: " << result.size << std::endl; // Output shape: 1 x 84 x 6300
}
```

model files [here](https://limewire.com/d/bPgiA#BhUeSTBnMc)

docker image used to build opencv: [ghcr.io/mro47/opencv-builder](https://github.com/MRo47/opencv-builder/blob/main/Dockerfile)

Closes #26240

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
2025-05-27 14:13:49 +03:00
Dmitry Kurtaev
0e80a97f87 Hotfix ie_ngraph.cpp in Debug 2024-10-29 10:20:51 +03:00
Dmitry Kurtaev
d193554a5f OpenVINO friendly output names from non-compiled Model 2024-10-23 09:29:05 +03:00
Oleg Pipikin
6da2ddcf0e Fix for OpenVINO 2024.0
Remove support OpenVINO lower than 2022.1 release
Remove legacy InferenceEngine wrappers
2024-03-18 15:05:50 +04:00
Alexander Alekhin
efc9837df1 Merge pull request #24892 from opencv-pushbot:gitee/alalek/dnn_avoid_16s_usage
DNN: avoid CV_16S usage for FP16 #24892

**Merge after**: #24918

TODO:
- [x] measure performance changes
- [x] optimize convertTo for OpenCL: #24918

12700K iGPU:

|Name of Test|0|1|1 vs 0 (x-factor)|
|---|:-:|:-:|:-:|
|AlexNet::DNNTestNetwork::OCV/OCL_FP16|7.441|7.480|0.99|
|CRNN::DNNTestNetwork::OCV/OCL_FP16|10.776|10.736|1.00|
|DenseNet_121::DNNTestNetwork::OCV/OCL_FP16|52.762|52.833|1.00|
|EAST_text_detection::DNNTestNetwork::OCV/OCL_FP16|60.694|60.721|1.00|
|EfficientNet::DNNTestNetwork::OCV/OCL_FP16|33.373|33.173|1.01|
|FastNeuralStyle_eccv16::DNNTestNetwork::OCV/OCL_FP16|81.840|81.724|1.00|
|GoogLeNet::DNNTestNetwork::OCV/OCL_FP16|20.965|20.927|1.00|
|Inception_5h::DNNTestNetwork::OCV/OCL_FP16|22.204|22.173|1.00|
|Inception_v2_SSD_TensorFlow::DNNTestNetwork::OCV/OCL_FP16|47.115|47.460|0.99|
|MPHand::DNNTestNetwork::OCV/OCL_FP16|6.760|6.670|1.01|
|MPPalm::DNNTestNetwork::OCV/OCL_FP16|10.188|10.171|1.00|
|MPPose::DNNTestNetwork::OCV/OCL_FP16|12.510|12.561|1.00|
|MobileNet_SSD_Caffe::DNNTestNetwork::OCV/OCL_FP16|17.290|17.072|1.01|
|MobileNet_SSD_v1_TensorFlow::DNNTestNetwork::OCV/OCL_FP16|19.473|19.306|1.01|
|MobileNet_SSD_v2_TensorFlow::DNNTestNetwork::OCV/OCL_FP16|22.874|23.404|0.98|
|OpenFace::DNNTestNetwork::OCV/OCL_FP16|9.568|9.517|1.01|
|OpenPose_pose_mpi_faster_4_stages::DNNTestNetwork::OCV/OCL_FP16|539.899|539.845|1.00|
|PPHumanSeg::DNNTestNetwork::OCV/OCL_FP16|18.015|18.769|0.96|
|PPOCRv3::DNNTestNetwork::OCV/OCL_FP16|63.122|63.540|0.99|
|ResNet_50::DNNTestNetwork::OCV/OCL_FP16|34.947|34.925|1.00|
|SFace::DNNTestNetwork::OCV/OCL_FP16|10.249|10.206|1.00|
|SSD::DNNTestNetwork::OCV/OCL_FP16|213.068|213.108|1.00|
|SqueezeNet_v1_1::DNNTestNetwork::OCV/OCL_FP16|4.867|4.878|1.00|
|VIT_B_32::DNNTestNetwork::OCV/OCL_FP16|200.563|190.788|1.05|
|VitTrack::DNNTestNetwork::OCV/OCL_FP16|7.528|7.173|1.05|
|YOLOX::DNNTestNetwork::OCV/OCL_FP16|132.858|132.701|1.00|
|YOLOv3::DNNTestNetwork::OCV/OCL_FP16|209.559|208.809|1.00|
|YOLOv4::DNNTestNetwork::OCV/OCL_FP16|221.357|220.924|1.00|
|YOLOv4_tiny::DNNTestNetwork::OCV/OCL_FP16|24.446|24.382|1.00|
|YOLOv5::DNNTestNetwork::OCV/OCL_FP16|43.922|44.080|1.00|
|YOLOv8::DNNTestNetwork::OCV/OCL_FP16|64.159|63.842|1.00|
|YuNet::DNNTestNetwork::OCV/OCL_FP16|10.177|10.231|0.99|
|opencv_face_detector::DNNTestNetwork::OCV/OCL_FP16|15.121|15.445|0.98|

Co-authored-by: Alexander Alekhin <alexander.a.alekhin@gmail.com>
2024-01-26 16:34:17 +03:00
Dmitry Kurtaev
c7ec0d599a Merge pull request #23987 from dkurt:openvino_int8_backend
OpenVINO backend for INT8 models #23987

### Pull Request Readiness Checklist

TODO:
- [x] DetectionOutput layer (https://github.com/opencv/opencv/pull/24069)
- [x] Less FP32 fallbacks (i.e. Sigmoid, eltwise sum)
- [x] Accuracy, performance tests (https://github.com/opencv/opencv/pull/24039)
- [x] Single layer tests (convolution)
- [x] ~~Fixes for OpenVINO 2022.1 (https://pullrequest.opencv.org/buildbot/builders/precommit_custom_linux/builds/100334)~~


Performace results for object detection model `coco_efficientdet_lite0_v1_1.0_quant_2021_09_06.tflite`:
| backend | performance (median time) |
|---|---|
| OpenCV | 77.42ms |
| OpenVINO 2023.0 | 10.90ms |

CPU: `11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz`

Serialized model per-layer stats (note that Convolution should use `*_I8` primitives if they are quantized correctly): https://gist.github.com/dkurt/7772bbf1907035441bb5454f19f0feef

---

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
2023-09-28 16:24:43 +03:00
Dmitry Kurtaev
178fdbbda8 Merge pull request #24196 from dkurt:ov_backend_cleanups
Use ngraph::Output in OpenVINO backend wrapper #24196

### Pull Request Readiness Checklist

resolves https://github.com/opencv/opencv/issues/24102

* Use `ngraph::Output<ngraph::Node>>` insead of `std::shared_ptr<ngraph::Node>` as a backend wrapper. It lets access to multi-output nodes: 588ddf1b18/modules/dnn/src/net_openvino.cpp (L501-L504)
* All layers can be customizable with OpenVINO >= 2022.1. nGraph reference code used for default layer implementation does not required CPU plugin also (might be tested by commenting CPU plugin at `/opt/intel/openvino/runtime/lib/intel64/plugins.xml`).
* Correct inference if only intermediate blobs requested.


See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
2023-09-05 18:08:28 +03:00
Dmitry Kurtaev
0245c0cd10 Merge pull request #24072 from dkurt:openvino_cpu_tests
Remove legacy nGraph logic #24072

### Pull Request Readiness Checklist

TODO:
- [x] Test with OpenVINO 2021.4 (tested locally)

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
2023-08-02 14:39:11 +03:00
Alexander Alekhin
9a2a34f94e dnn(openvino): remove undefined status 2022-12-28 06:55:00 +00:00
Dmitry Kurtaev
8681686d8f Merge pull request #22957 from dkurt:new_openvino_api
Switch to new OpenVINO API after 2022.1 release

* Pass Layer_Test_Convolution_DLDT.Accuracy/0 test

* Pass test Test_Caffe_layers.Softmax

* Failed 136 tests

* Fix Concat. Failed 120 tests

* Custom nGraph ops. 19 failed tests

* Set and get properties from Core

* Read model from buffer

* Change MaxPooling layer output names. Restore reshape

* Cosmetic changes

* Cosmetic changes

* Override getOutputsInfo

* Fixes for OpenVINO < 2022.1

* Async inference for 2021.4 and less

* Compile model with config

* Fix serialize for 2022.1

* Asynchronous inference with 2022.1

* Handle 1d outputs

* Work with model with dynamic output shape

* Fixes with 1d output for old API

* Control outputs by nGraph function for all OpenVINO versions

* Refer inputs in PrePostProcessor by indices

* Fix cycled dependency between InfEngineNgraphNode and InfEngineNgraphNet.
Add InferRequest callback only for async inference. Do not capture InferRequest object.

* Fix tests thresholds

* Fix HETERO:GPU,CPU plugin issues with unsupported layer
2022-12-23 16:58:41 +00:00
Alexander Alekhin
b16f76eede Merge remote-tracking branch 'upstream/3.4' into merge-3.4 2022-12-03 12:39:41 +00:00
Alexander Alekhin
74d0b4cc78 dnn(openvino): fix custom layers BlockingDesc 2022-12-03 01:34:10 +00:00
Alexander Alekhin
ca7f964104 dnn: use inheritance for OpenVINO net impl 2022-09-06 18:05:00 +00:00
Alexander Alekhin
5bf3c1df24 Merge pull request #21715 from ilyachur:change_type_info_creation 2022-03-14 09:18:58 +00:00
Ilya Churaev
419918076e Changed call of NodeTypeInfo constructor 2022-03-14 10:55:33 +03:00
Alexander Alekhin
57d3002ee1 Merge remote-tracking branch 'upstream/3.4' into merge-3.4 2022-02-06 16:10:43 +00:00
Alexander Alekhin
b57ff73086 dnn(ngraph): fix outputs handling, drop 'unconnected' logic 2022-02-06 13:08:53 +00:00
Alexander Alekhin
67978b5746 dnn(ngraph): add debuging messages 2022-02-06 13:08:53 +00:00
Alexander Alekhin
aa5bc20c83 dnn(ngraph): fixup get_output_as_single_output_node() replacement patch 2022-02-06 10:35:59 +00:00
Alexander Alekhin
39b91c97f0 Merge remote-tracking branch 'upstream/3.4' into merge-3.4 2021-07-16 10:35:42 +00:00
Alexander Alekhin
fbde0c6c96 dnn(ie): fix handling of 1D and non-32F outputs of InferenceEngine 2021-07-15 21:47:05 +00:00
Alexander Alekhin
8fad85edda Merge remote-tracking branch 'upstream/3.4' into merge-3.4 2021-07-01 10:52:31 +00:00
Alexander Alekhin
f2057ce1ab dnn(ie): replace deprecated calls 2021-06-30 22:30:15 +00:00
Alexander Alekhin
170bf6d7af Merge remote-tracking branch 'upstream/3.4' into merge-3.4 2021-05-01 09:44:24 +00:00
Alexander Alekhin
71bae7c23f dnn(ie): implicit usage of IE::GPU OpenCL kernels cache 2021-04-29 12:43:22 +03:00
Alexander Alekhin
d643a90efa Merge remote-tracking branch 'upstream/3.4' into merge-3.4 2021-03-06 17:31:08 +00:00
Liubov Batanina
94533e12eb Determine layout 2021-03-04 13:05:01 +03:00
Omar Alzaibaq
a316b11aaa Merge pull request #18220 from Omar-AE:hddl-supported
* added HDDL VPU support

* changed to return True in one line if any device connected

* dnn: use releaseHDDLPlugin()

* dnn(hddl): fix conditions
2020-11-17 19:47:24 +00:00
Maksim Doronin
36f61f3879 [IE][VPU]: Refactor vpu configs
backported commit: 7fe87d9a5b
2020-10-05 20:27:52 +00:00
Maksim Doronin
7fe87d9a5b [IE][VPU]: Refactor vpu configs 2020-08-13 15:25:01 +03:00
Ilya Churaev
246de2b7f5 Replaced copy_with_new_args to clone_with_new_inputs 2020-08-03 23:08:29 +03:00
Ilya Churaev
e8c26963e9 Fixed removing is_parameter, is_constant, is_output 2020-08-03 23:08:22 +03:00
Maksim Shabunin
b698d0a6ee Merge pull request #17894 from ilyachur:feature/ichuraev/remove_copy_with_new_args 2020-07-21 08:19:42 +00:00
Ilya Churaev
aaf65bb3af Fixed removing is_parameter, is_constant, is_output 2020-07-21 08:45:16 +03:00
Ilya Churaev
0291dd3ef8 Replaced copy_with_new_args to clone_with_new_inputs 2020-07-20 16:05:01 +03:00
Ilya Churaev
a46ca98c57 Revert "Fixed removing is_parameter, is_constant, is_output" 2020-07-20 07:00:09 +03:00
Ilya Churaev
33db2ac928 Fixed removing is_parameter, is_constant, is_output 2020-07-17 14:45:11 +03:00
Dmitry Kurtaev
3bdd7bfbea Remove deprecated Inference Engine CPU extensions 2020-06-23 14:33:22 +03:00
Dmitry Kurtaev
8e3f5fb209 Remove deprecated Inference Engine CPU extensions 2020-06-23 10:10:29 +03:00
Ilya Lavrenov
676b818d6a Removed plugin dispatcher
backport of commit 74113737f3
2020-06-15 18:03:14 +00:00
Ilya Lavrenov
74113737f3 Removed plugin dispatcher 2020-06-15 16:02:52 +03:00
Alexander Alekhin
1644af841b Merge pull request #17536 from ilya-lavrenov:remove-error-listener 2020-06-12 22:18:04 +00:00
Ilya Lavrenov
9697e3ac24 Removed error lisneter usage 2020-06-12 20:29:11 +00:00
Alexander Alekhin
d2a9efd039 dnn: use OpenVINO 2020.3 defines 2020-06-12 20:24:08 +00:00
Alexander Alekhin
f0bef94a03 dnn: update network dump code, include ngraph serialization 2020-05-26 12:49:22 +00:00
Dmitry Kurtaev
908bf935f7 Flexible inputs for OpenVINO IR models 2020-04-18 20:00:22 +03:00
Dmitry Kurtaev
5e5385a814 Remove NodeVector for nGraph 2020-04-16 11:13:49 +03:00
Alexander Alekhin
9b64eadcb6 dnn(ie): separate IECore for different devices
- HETERO plugin doesn't switch well between devices
2020-03-13 18:49:28 +03:00
Alexander Alekhin
124bf8339f dnn(IE): use HAVE_DNN_IE_NN_BUILDER_2019 for NN Builder API code
- CMake option: OPENCV_DNN_IE_NN_BUILDER_2019
2020-03-03 08:07:54 +00:00
Dmitry Kurtaev
5bbe116f89 Track Hetero execution for nGraph networks using ngraph::Function 2020-02-27 17:45:28 +03:00