Merge pull request #26056 from vpisarev:new_dnn_engine

New dnn engine #26056 This is the 1st PR with the new engine; CI is green and PR is ready to be merged, I think. Merge together with https://github.com/opencv/opencv_contrib/pull/3794 --- **Known limitations:** * [solved] OpenVINO is temporarily disabled, but is probably easy to restore (it's not a deal breaker to merge this PR, I guess) * The new engine does not support any backends nor any targets except for the default CPU implementation. But it's possible to choose the old engine when loading a model, then all the functionality is available. * [Caffe patch is here: #26208] The new engine only supports ONNX. When a model is constructed manually or is loaded from a file of different format (.tf, .tflite, .caffe, .darknet), the old engine is used. * Even in the case of ONNX some layers are not supported by the new engine, such as all quantized layers (including DequantizeLinear, QuantizeLinear, QLinearConv etc.), LSTM, GRU, .... It's planned, of course, to have full support for ONNX by OpenCV 5.0 gold release. When a loaded model contains unsupported layers, we switch to the old engine automatically (at ONNX parsing time, not at `forward()` time). * Some layers , e.g. Expat, are only partially supported by the new engine. In the case of unsupported flavours it switches to the old engine automatically (at ONNX parsing time, not at `forward()` time). * 'Concat' graph optimization is disabled. The optimization eliminates Concat layer and instead makes the layers that generate tensors to be concatenated to write the outputs to the final destination. Of course, it's only possible when `axis=0` or `axis=N=1`. The optimization is not compatible with dynamic shapes since we need to know in advance where to store the tensors. Because some of the layer implementations have been modified to become more compatible with the new engine, the feature appears to be broken even when the old engine is used. * Some `dnn::Net` API is not available with the new engine. Also, shape inference may return false if some of the output or intermediate tensors' shapes cannot be inferred without running the model. Probably this can be fixed by a dummy run of the model with zero inputs. * Some overloads of `dnn::Net::getFLOPs()` and `dnn::Net::getMemoryConsumption()` are not exposed any longer in wrapper generators; but the most useful overloads are exposed (and checked by Java tests). * [in progress] A few Einsum tests related to empty shapes have been disabled due to crashes in the tests and in Einsum implementations. The code and the tests need to be repaired. * OpenCL implementation of Deconvolution is disabled. It's very bad and very slow anyway; need to be completely revised. * Deconvolution3D test is now skipped, because it was only supported by CUDA and OpenVINO backends, both of which are not supported by the new engine. * Some tests, such as FastNeuralStyle, checked that the in the case of CUDA backend there is no fallback to CPU. Currently all layers in the new engine are processed on CPU, so there are many fallbacks. The checks, therefore, have been temporarily disabled. --- - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [ ] There is a reference to the original bug report and related work - [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [ ] The feature is well documented and sample code can be built with the project CMake
2026-09-13 05:42:51 -05:00 · 2024-10-16 15:28:19 +03:00
parent 12738deaef
commit 3cd57ea09e
112 changed files with 11201 additions and 558 deletions
--- a/modules/dnn/src/net_impl.cpp
+++ b/modules/dnn/src/net_impl.cpp
@@ -59,11 +59,35 @@ Net::Impl::Impl()
    preferableTarget = DNN_TARGET_CPU;
    hasDynamicShapes = false;
    useWinograd = true;
+
+    ////////////// extra initialization for the new engine /////////////////
+
+    modelFormat = DNN_MODEL_GENERIC;
+    originalLayout = DATA_LAYOUT_NCHW;
+    onnx_opset = 0;
+
+    accuracy = CV_32F;
+    enableFP16 = haveFP16 = false;
+    // FP16 is not ready yet in the new DNN engine
+    // Ticket: https://github.com/opencv/opencv/issues/26196
+    /*if (checkHardwareSupport(CV_CPU_FP16)) {
+        enableFP16 = haveFP16 = true;
+    }*/
+
+    tracingMode = DNN_TRACE_NONE;
+    profilingMode = DNN_PROFILE_NONE;
+
+    dump_strm = &std::cout;
+    dump_indent = 3;
+
+    clear();
 }


 bool Net::Impl::empty() const
 {
+    if (mainGraph)
+        return false;
    return layers.size() <= 1;  // first layer is default Data layer
 }

@@ -92,6 +116,34 @@ void Net::Impl::clear()
    }
    netWasAllocated = false;
    layersTimings.clear();
+
+    /////////////// for the new inference engine //////////////////
+
+    modelFormat = DNN_MODEL_GENERIC;
+
+    dimnames = NamesHash();
+    dimnames_vec = std::vector<std::string>();
+
+    args = std::vector<ArgData>();
+    argnames = NamesHash();
+
+    __tensors__ = std::vector<Mat>();
+    bufidxs = std::vector<int>();
+    buffers = std::vector<Mat>();
+
+    mainGraph = Ptr<Graph>();
+
+    ArgData adata;
+    adata.name = "";
+    adata.kind = DNN_ARG_CONST;
+
+    args.push_back(adata);
+    argnames.insert(std::make_pair(std::string(""), 0));
+    __tensors__.push_back(Mat());
+    bufidxs.push_back(-1);
+
+    prepared = false;
+    finalizeLayers = true;
 }


@@ -208,8 +260,22 @@ void Net::Impl::setUpNet(const std::vector<LayerPin>& blobsToKeep_)

 Ptr<Layer> Net::Impl::getLayer(int layerId) const
 {
-    LayerData& ld = getLayerData(layerId);
-    return getLayerInstance(ld);
+    if (mainGraph) {
+        CV_Assert(0 <= layerId && layerId < totalLayers);
+        int graph_ofs = 0;
+        for (const Ptr<Graph>& graph : allgraphs) {
+            const std::vector<Ptr<Layer> >& prog = graph->prog();
+            int nops = (int)prog.size();
+            CV_Assert(layerId >= graph_ofs);
+            if (layerId < graph_ofs + nops)
+                return prog[layerId - graph_ofs];
+            graph_ofs += nops;
+        }
+        CV_Error_(Error::StsObjectNotFound, ("layer #%d is not found", layerId));
+    } else {
+        LayerData& ld = getLayerData(layerId);
+        return getLayerInstance(ld);
+    }
 }


@@ -351,7 +417,7 @@ int Net::Impl::addLayer(const String& name, const String& type, const int& dtype
    {
        if (!DNN_DIAGNOSTICS_RUN || type != "NotImplemented")
        {
-            CV_Error(Error::StsBadArg, "Layer \"" + name + "\" already into net");
+            CV_Error(Error::StsBadArg, "Layer \"" + name + "\" has been already added into net");
            return -1;
        }
        else
@@ -613,12 +679,23 @@ void Net::Impl::allocateLayers(const std::vector<LayerPin>& blobsToKeep_)
 }


+#define TRACE_INFERENCE 0
+
 void Net::Impl::forwardLayer(LayerData& ld)
 {
    CV_TRACE_FUNCTION();

    Ptr<Layer> layer = ld.layerInstance;

+#if TRACE_INFERENCE
+    if (layer) {
+        printf("------------------------------------------------\n");
+        printf("Running layer '%s' (%s)\n",
+               layer->name.c_str(),
+               layer->type.c_str());
+    }
+#endif
+
    if (!ld.skip)
    {
        TickMeter tm;
@@ -842,6 +919,29 @@ void Net::Impl::forwardLayer(LayerData& ld)
        tm.stop();
        int64 t = tm.getTimeTicks();
        layersTimings[ld.id] = (t > 0) ? t : t + 1;  // zero for skipped layers only
+#if TRACE_INFERENCE
+        size_t noutputs = ld.outputBlobs.size();
+        for (size_t i = 0; i < noutputs; i++) {
+            const Mat& out = ld.outputBlobs[i];
+            printf("Output %zu.\n", i);
+            printf("  Type: %s\n", typeToString(out.type()).c_str());
+            printf("  Shape: ");
+            if (out.empty()) {
+                printf("<empty>\n");
+            } else if (out.dims == 0) {
+                printf("<scalar>\n");
+            } else {
+                for (int j = 0; j < out.dims; j++) {
+                    printf("%s%d", (j == 0 ? "[" : " x "), out.size[j]);
+                }
+                printf("]\n");
+            }
+            //fflush(stdout);
+            //pprint(std::cout, out, 0, 3, 100, '[');
+            //std::cout.flush();
+            //printf("\n");
+        }
+#endif
    }
    else
    {
@@ -890,6 +990,12 @@ Mat Net::Impl::forward(const String& outputName)
    CV_Assert(!empty());
    FPDenormalsIgnoreHintScope fp_denormals_ignore_scope;

+    if (mainGraph) {
+        if (!outputName.empty())
+            CV_Error(Error::StsNotImplemented, "The new dnn engine doesn't support inference until a specified layer. If you want to run the whole model, please don't set the outputName argument in the forward() call. If you want to run the model until a specified layer, please use the old dnn engine");
+        return forwardWithSingleOutput(outputName);
+    }
+
    String layerName = outputName;

    if (layerName.empty())
@@ -912,6 +1018,9 @@ AsyncArray Net::Impl::forwardAsync(const String& outputName)
    CV_Assert(!empty());
    FPDenormalsIgnoreHintScope fp_denormals_ignore_scope;

+    if (mainGraph)
+        CV_Error(Error::StsNotImplemented, "The new dnn engine doesn't support the async inference. If you want to run the sync inference, please call forward() instead of forwardAsync(). If you want to run the async inference, please use the old dnn engine");
+
    String layerName = outputName;

    if (layerName.empty())
@@ -940,6 +1049,13 @@ void Net::Impl::forward(OutputArrayOfArrays outputBlobs, const String& outputNam
    CV_Assert(!empty());
    FPDenormalsIgnoreHintScope fp_denormals_ignore_scope;

+    if (mainGraph) {
+        if (!outputName.empty())
+            CV_Error(Error::StsNotImplemented, "The new dnn engine doesn't support inference until a specified layer. If you want to run the whole model, please don't set the outputName argument in the forward() call. If you want to run the model until a specified layer, please use the old dnn engine");
+        forwardWithMultipleOutputs(outputBlobs, {});
+        return;
+    }
+
    String layerName = outputName;

    if (layerName.empty())
@@ -1028,6 +1144,11 @@ void Net::Impl::forward(OutputArrayOfArrays outputBlobs,
    CV_Assert(!empty());
    FPDenormalsIgnoreHintScope fp_denormals_ignore_scope;

+    if (mainGraph) {
+        forwardWithMultipleOutputs(outputBlobs, outBlobNames);
+        return;
+    }
+
    std::vector<LayerPin> pins;
    for (int i = 0; i < outBlobNames.size(); i++)
    {
@@ -1266,11 +1387,18 @@ void Net::Impl::getLayerShapes(const ShapesVec& netInputShapes,
        const int layerId,
        LayerShapes& shapes)
 {
-    LayersShapesMap inOutShapes;
-    inOutShapes[0].in = netInputShapes;  // insert shape for first input layer
-    inOutShapes[0].inTypes = netInputTypes;
-    getLayerShapesRecursively(layerId, inOutShapes);
-    shapes = inOutShapes[layerId];
+    if (mainGraph) {
+        std::vector<MatShape> shapeCache;
+        std::vector<int> typeCache;
+        CV_Assert(layerId == 0);
+        tryInferShapes(netInputShapes, netInputTypes, shapes, shapeCache, typeCache);
+    } else {
+        LayersShapesMap inOutShapes;
+        inOutShapes[0].in = netInputShapes;  // insert shape for first input layer
+        inOutShapes[0].inTypes = netInputTypes;
+        getLayerShapesRecursively(layerId, inOutShapes);
+        shapes = inOutShapes[layerId];
+    }
 }

 void Net::Impl::updateLayersShapes()
@@ -1411,6 +1539,13 @@ void Net::Impl::setInput(InputArray blob, const String& name, double scalefactor
 {
    FPDenormalsIgnoreHintScope fp_denormals_ignore_scope;

+    if (mainGraph) {
+        CV_Assert(scalefactor == 1);
+        CV_Assert(mean.val[0] == 0 && mean.val[1] == 0 && mean.val[2] == 0 && mean.val[3] == 0);
+        setMainGraphInput(blob, name);
+        return;
+    }
+
    LayerPin pin;
    pin.lid = 0;
    pin.oid = resolvePinOutputName(getLayerData(pin.lid), name);
@@ -2154,13 +2289,23 @@ std::vector<Ptr<Layer>> Net::Impl::getLayerInputs(int layerId) const
 std::vector<String> Net::Impl::getLayerNames() const
 {
    std::vector<String> res;
-    res.reserve(layers.size());

-    Impl::MapIdToLayerData::const_iterator it;
-    for (it = layers.begin(); it != layers.end(); it++)
-    {
-        if (it->second.id)  // skip Data layer
-            res.push_back(it->second.name);
+    if (mainGraph) {
+        res.reserve(totalLayers);
+        for (const Ptr<Graph>& graph: allgraphs) {
+            const std::vector<Ptr<Layer> >& prog = graph->prog();
+            for (const Ptr<Layer>& layer: prog)
+                res.push_back(layer->name);
+        }
+    } else {
+        res.reserve(layers.size());
+
+        Impl::MapIdToLayerData::const_iterator it;
+        for (it = layers.begin(); it != layers.end(); it++)
+        {
+            if (it->second.id)  // skip Data layer
+                res.push_back(it->second.name);
+        }
    }

    return res;
@@ -2199,6 +2344,15 @@ std::vector<int> Net::Impl::getUnconnectedOutLayers() const
 // FIXIT drop "unconnected" API
 std::vector<String> Net::Impl::getUnconnectedOutLayersNames() /*const*/
 {
+    if (mainGraph) {
+        std::vector<std::string> outnames;
+        const std::vector<Arg>& outargs = mainGraph->outputs();
+        for (auto out: outargs) {
+            const ArgData& adata = args.at(out.idx);
+            outnames.push_back(adata.name);
+        }
+        return outnames;
+    }
    std::vector<int> ids = getUnconnectedOutLayers();
    const size_t n = ids.size();
    std::vector<String> names(n);
@@ -2368,6 +2522,20 @@ void Net::Impl::enableWinograd(bool useWinograd_)
 void Net::Impl::getLayerTypes(std::vector<String>& layersTypes) const
 {
    layersTypes.clear();
+    if (mainGraph) {
+        std::set<std::string> layersTypesSet;
+        for (const Ptr<Graph>& g: allgraphs) {
+            const std::vector<Ptr<Layer> >& prog = g->prog();
+            for (const Ptr<Layer>& layer: prog) {
+                if (!layer)
+                    continue;
+                layersTypesSet.insert(layer->type);
+            }
+        }
+        for (auto it = layersTypesSet.begin(); it != layersTypesSet.end(); ++it)
+            layersTypes.push_back(*it);
+        return;
+    }

    std::map<String, int> layers_type_map;
    for (MapIdToLayerData::const_iterator it = layers.begin(); it != layers.end(); it++)