Use MatShape instead of MatSize inside cv::Mat/cv::UMat #27757
**Merge together with https://github.com/opencv/opencv_contrib/pull/3996**
---
This PR continues cv::Mat/cv::UMat refactoring. See #26056, where `MatShape` was introduced. Now it's put inside cv::Mat/cv::UMat instead of a weird `MatSize`. MatSize is now an alias for MatShape:
**before:**
```
struct MatShape { ... };
struct MatSize { ... };
struct Mat {
...
int dims;
int rows;
int cols;
...
MatShape shape() const { ... /* constructs MatShape out of MatSize and returns it;
layout is always 'unknown', because we don't store it */ }
MatSize size; // size is not valid without the parent cv::Mat,
// because size.p may point to Mat::rows or to Mat::cols,
// depending on the dimensionality, and dims() returns Mat::dims.
MatStep step; // may allocate memory, depending on the dimensionality.
...
};
```
**after:**
```
struct MatShape { ... };
typedef MatShape MatSize; // they are now synonyms
struct Mat {
...
int dims;
int rows;
int cols;
...
MatShape shape() const { return size; } // just return the embedded shape (including the proper layout information)
MatSize size; // size is self-contained data structure that can be used without the parent cv::Mat.
// size.dims is now a copy of dims; size.p[*] contains copies of Mat::rows and Mat::cols when dims <= 2.
MatStep step; // does not allocate extra memory buffers.
...
};
```
There are several reasons to do that:
1. the main reason is to be able to store data layout (MatShape::layout) inside each cv::Mat/cv::UMat. This is necessary for the proper shape inference in DNN module. In particular, it's necessary for the next step of DNN inference optimization where we introduce block-layout-optimized convolution and other operations. Later on, we can use layout information to support non-interleaved images (e.g. RRR...GGG...BBB...) or even batches of such images in core/imgproc modules.
2. the other reason is to represent 3D/4D/5D etc. tensors as cv::Mat/cv::UMat instances more conveniently, without extra dynamic memory allocation. Before this patch we allocated some memory buffers dynamically to store shape & steps for more than 2D arrays. Now the whole cv::Mat/cv::UMat header can be stored completely on stack/in a container. Creating another copy of Mat/UMat header is now done more efficiently.
3. the third reason is to introduce the new coding pattern: `dst.create(src.size, <dst_type>);`. The pattern is suitable for most of element-wise (including cloning) and filtering operations. This pattern does not only look crisp and self-documenting, it will also automatically copy shape (including layout) from the source tensor into the destination matrix/tensor.
4. in the future we might add `colorspace` member to MatShape that will allow to distinguish RGB from BGR or NV12. `dst.create(src.size, <dst_type>);` will then copy the colorspace information as well.
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
* improve and refactor softmax layer
* fix building error
* compatible region layer
* fix axisStep when disable SIMD
* fix dynamic array
* try to fix error
* use nlanes from VTraits
* move axisBias to srcOffset
* fix bug caused by axisBias
* remove macro
* replace #ifdef with #if for CV_SIMD
Use ngraph::Output in OpenVINO backend wrapper #24196
### Pull Request Readiness Checklist
resolves https://github.com/opencv/opencv/issues/24102
* Use `ngraph::Output<ngraph::Node>>` insead of `std::shared_ptr<ngraph::Node>` as a backend wrapper. It lets access to multi-output nodes: 588ddf1b18/modules/dnn/src/net_openvino.cpp (L501-L504)
* All layers can be customizable with OpenVINO >= 2022.1. nGraph reference code used for default layer implementation does not required CPU plugin also (might be tested by commenting CPU plugin at `/opt/intel/openvino/runtime/lib/intel64/plugins.xml`).
* Correct inference if only intermediate blobs requested.
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Add support for YOLOv4x-mish
* backport to 3.4 for supporting yolov4x-mish
* add YOLOv4x-mish test
* address review comments
Co-authored-by: Guo Xu <guoxu@1school.com.cn>
Added OpenVINO ARM target
* Added IE ARM target
* Added OpenVINO ARM target
* Delete ARM target
* Detect ARM platform
* Changed device name in ArmPlugin
* Change ARM detection
* Remove isIntel check from deep learning layers
* Remove fp16->fp32 fallbacks where it's not necessary
* Fix Kernel::run to prevent localsize > globalsize
Feature/region layer batch mode (#12249)
* Add batch mode for Darknet networks.
Swap variables in test_darknet.
Adapt reorg layer to batch mode.
Adapt region layer.
Add OpenCL implementation.
Remove trailing whitespace.
Bugifx reorg opencl implementation.
Fix bug in OpenCL reorg.
Fix modulo bug.
Fix bug.
Reorg openCL.
Restore reorg layer opencl code.
OpenCl fix.
Work on openCL reorg.
Remove whitespace.
Fix openCL region layer implementation.
Fix bug.
Fix softmax region opencl bug.
Fix opencl bug.
Fix openCL bug.
Update aff_trans.cpp
When the fullAffine parameter is set to false, the estimateRigidTransform function maybe return empty, then the _localAffineEstimate function will be called, but the bug in it will result in incorrect results.
core(libva): support YV12 too
Added to CPU path only.
OpenCL code path still expects NV12 only (according to Intel OpenCL extension)
cmake: allow to specify own libva paths
via CMake:
- `-DVA_LIBRARIES=/opt/intel/mediasdk/lib64/libva.so.2\;/opt/intel/mediasdk/lib64/libva-drm.so.2`
android: NDK17 support
tested with NDK 17b (17.1.4828580)
Enable more deep learning tests using Intel's Inference Engine backend
ts: don't pass NULL for std::string() constructor
openvino: use 2018R3 defines
experimental version++
OpenCV version++
OpenCV 3.4.3
OpenCV version '-openvino'
openvino: use 2018R3 defines
Fixed windows build with InferenceEngine
dnn: fix variance setting bug for PriorBoxLayer
- The size of second channel should be size[2] of output tensor,
- The Scalar should be {variance[0], variance[0], variance[0], variance[0]}
for _variance.size() == 1 case.
Signed-off-by: Wu Zhiwen <zhiwen.wu@intel.com>
Fix lifetime of networks which are loaded from Model Optimizer IRs
Adds a small note describing BUILD_opencv_world (#12332)
* Added a mall note describing BUILD_opencv_world cmake option to the Installation in Windows tutorial.
* Made slight changes in BUILD_opencv_world documentation.
* Update windows_install.markdown
improved grammar
Update opengl_interop.cpp
resolves#12307
java: fix LIST_GET macro
fix typo
Added option to fail on missing testdata
Fixed that object_detection.py does not work in python3.
cleanup: IPP Async (IPP_A)
except header file with conversion routines (will be removed in OpenCV 4.0)
imgcodecs: add null pointer check
Include preprocessing nodes to object detection TensorFlow networks (#12211)
* Include preprocessing nodes to object detection TensorFlow networks
* Enable more fusion
* faster_rcnn_resnet50_coco_2018_01_28 test
countNonZero function reworked to use wide universal intrinsics instead of SSE2 intrinsics
resolve#5788
imgcodecs(webp): multiple fixes
- don't reallocate passed 'img' (test fixed - must use IMREAD_UNCHANGED / IMREAD_ANYCOLOR)
- avoid memory DDOS
- avoid reading of whole file during header processing
- avoid data access after allocated buffer during header processing (missing checks)
- use WebPFree() to free allocated buffers (libwebp >= 0.5.0)
- drop unused & undefined `.close()` method
- added checks for channels >= 5 in encoder
ml: fix adjusting K in KNearest (#12358)
dnn(perf): fix and merge Convolution tests
- OpenCL tests didn't run any OpenCL kernels
- use real configuration from existed models (the first 100 cases)
- batch size = 1
dnn(test): use dnnBackendsAndTargets() param generator
Bit-exact resize reworked to use wide intrinsics (#12038)
* Bit-exact resize reworked to use wide intrinsics
* Reworked bit-exact resize row data loading
* Added bit-exact resize row data loaders for SIMD256 and SIMD512
* Fixed type punned pointer dereferencing warning
* Reworked loading of source data for SIMD256 and SIMD512 bit-exact resize
Bit-exact GaussianBlur reworked to use wide intrinsics (#12073)
* Bit-exact GaussianBlur reworked to use wide intrinsics
* Added v_mul_hi universal intrinsic
* Removed custom SSE2 branch from bit-exact GaussianBlur
* Removed loop unrolling for gaussianBlur horizontal smoothing
doc: fix English gramma in tutorial out-of-focus-deblur filter (#12214)
* doc: fix English gramma in tutorial out-of-focus-deblur filter
* Update out_of_focus_deblur_filter.markdown
slightly modified one sentence
doc: add new tutorial motion deblur filter (#12215)
* doc: add new tutorial motion deblur filter
* Update motion_deblur_filter.markdown
a few minor changes
Replace Slice layer to Crop in Faster-RCNN networks from Caffe
js: use generated list of OpenCV headers
- replaces hand-written list
imgcodecs(webp): use safe cast to size_t on Win32
* Put Version status back to -dev.
follow the common codestyle
Exclude some target engines.
Refactor formulas.
Refactor code.
* Remove unused variable.
* Remove inference engine check for yolov2.
* Alter darknet batch tests to test with two different images.
* Add yolov3 second image GT.
* Fix bug.
* Fix bug.
* Add second test.
* Remove comment.
* Add NMS on network level.
* Add helper files to dev.
* syntax fix.
* Fix OD sample.
Fix sample dnn object detection.
Fix NMS boxes bug.
remove trailing whitespace.
Remove debug function.
Change thresholds for opencl tests.
* Adapt score diff and iou diff.
* Alter iouDiffs.
* Add debug messages.
* Adapt iouDiff.
* Fix tests
* Remove a forward method in dnn::Layer
* Add a test
* Fix tests
* Mark multiple dnn::Layer::finalize methods as deprecated
* Replace back dnn's inputBlobs to vector of pointers
* Remove Layer::forward_fallback from CV_OCL_RUN scopes
Add layer forward interface with InputArrayOfArrays and
OutputArrayOfArrays parameters, it allows UMat buffer to be
processed and transferred in the layers.
Signed-off-by: Li Peng <peng.li@intel.com>