mirror of
https://github.com/opencv/opencv.git
synced 2026-11-15 13:57:47 -06:00
vectorized and parallel DFT/DCT + matchTemplate - #30061 (Made with a help from Claude Code Fable 5.1) ## Summary This PR replaces the scalar 1D DFT/DCT kernels in `modules/core/src/dxt.cpp` with a new engine built on universal intrinsics and compiled per ISA through `ocv_add_dispatched_file`, and makes the 2D transforms, the `DFT_ROWS`/`DCT_ROWS` batches and `matchTemplate` (`crossCorr`) multi-threaded. The public API, the output formats (CCS packing of real spectra, `DFT_COMPLEX_OUTPUT` layout, DCT ordering and normalization) and the IPP/OpenCL paths are unchanged. Only floating-point rounding may differ (the new twiddle tables are more accurate than the old recurrence-generated ones, so the error goes down, not up). Measured on Apple M4 Max (16 cores, NEON), macOS 26.5.x, Xcode 26.6, Release build (`-O3`, no IPP). All numbers are the minimum over repeated runs, `5.x` vs this PR. ## Changes ### New 1D engine (`dxt.hpp`, `dxt.simd.hpp`, `dxt.cpp`) - **Split re/im layout.** The input pass gathers the data into separate `re[]`/`im[]` workspace arrays; all middle stages operate on the split arrays, so a complex multiplication is four independent vector FMAs with no lane shuffling. The output pass interleaves back (or packs the CCS / DCT layout) with the scaling fused in. - **Mixed-radix decimation-in-time with a fused first stage.** The plan chooses the first stage (radix 1/2/4/8, twiddle-free, fused with the gather and the digit-reversal permutation) from the power of two in the length, then radix-4 stages, then radix-3, radix-5 and a generic odd radix. There are no radix-2 middle stages. - **Vectorized butterflies with universal intrinsics** (`v_float32`/`v_float64`, VLA-safe): one flat loop per stage with a single clamped counter for both the data and the twiddle offsets (the back-off clamp handles spans that are not multiples of the vector width; such stages run out of place, ping-ponging between two workspace pairs, the others run in place). - **Vectorized first stage.** The radix-2/4/8 first stage reads the input columns contiguously (`v_load_deinterleave`), runs the small DFT in registers and scatters the outputs of every lane with one 4-channel interleaved store. Only odd lengths use a scalar table-driven gather. - **Vectorized edge passes**: interleave/deinterleave, the real-FFT untangling in both directions (with `v_reverse` for the mirrored half), DCT-II post-twiddle and DCT-III pre-twiddle, the Makhoul permutations. - **Inverse = conjugation trick**: the middle stages are direction-free and use forward twiddles only; the inverse conjugates on the gather and negates the imaginary scale on the output. - **Immutable plan + per-executor workspace, zero allocations per transform.** The plan (radix sequence, permutation table, per-stage split twiddle tables) is built once in `init()` and shared by all rows of a batch; every executor owns a workspace. Twiddles are generated in double precision from a two-level exact table (a few ulps, much cheaper to build than the old N-point recurrence, which matters because `cv::dft()` builds a plan on every call). - **Per-ISA dispatch**: `ocv_add_dispatched_file(dxt SSE2 AVX2 AVX512_ICL RVV)`; the kernels are reached through a function-pointer table resolved once per depth, following the pattern of `arithm.simd.hpp` / `warp_kernels.simd.hpp`. The plan is built with the vector width of the dispatched kernels (never with the baseline `VTraits`). - The legacy scalar engine, `DFTInit`, the SSE3 `DFT_VecR*` butterflies and the `noPermute`/ `factors[0] >>= 1` machinery are removed (−1400 lines). The legacy `cv_hal_dft` hook (which took the old tables) is no longer called from the 1D path; the in-tree RISC-V HAL implementation of it will be updated in a separate PR. ### Parallelism - `crossCorr` (`matchTemplate`): the correlation tiles are independent and are now distributed with `parallel_for_`; every worker creates its DFT buffers and `DFT2D` contexts once per range. (The tile loop had been sequential since the C++ rewrite in 2010, which dropped the OpenMP pragma of `icvCrossCorr`.) - 2D DFT/DCT and `DFT_ROWS`/`DCT_ROWS` batches: the row pass, the column pass and the DCT line loop distribute their lines over threads; the 1D plan is shared, each worker allocates its workspace once per range. The number of stripes is derived from the N log N cost, so a 256x256 transform stays on the calling thread and 1024x1024 gets ~20 stripes. Nested calls (the 2D DFTs inside the parallel `crossCorr` tiles) are serialized by `parallel_for_` itself. - The column pass copies columns in batches (`CopyFromColumns`/`CopyToColumns`, 4..16 adjacent columns copied row by row into contiguous buffers sized to stay within 128 KB) instead of two at a time: with power-of-two row steps the old strided copies aliased in the cache and took 60-70% of the 2D time. ### Tests - `Core_DFT.sweep_1d`, `Core_DCT.sweep_1d`: every 5-smooth length up to 2048 plus odd, prime and "awkward" lengths, both depths, C2C forward/inverse, R2C (CCS and complex output), C2R (CCS and complex input), DCT/IDCT, in-place, column vectors, round trips; compared against the naive O(N^2) references of `test_dxt.cpp` with a tolerance of `32*eps*(log2(N) + 2 + maxfactor/16)`. - `Core_DFT.sweep_rows`: `DFT_ROWS` batches sharing one plan. - `Core_DFT.nonzero_rows`: 2D transforms with `nonzero_rows` (previously uncovered). - `Core_DFT.parallel_large`: large 2D / rows / DCT transforms, multi-threaded result vs the single-threaded one. - The naive DCT reference reduces the angle exactly before `cos()` (its accuracy was limiting the comparison for n > 100 in double precision). - Perf: `perf_dft.cpp` gains 1D row/column cases for `dft` and `dct`. - Existing `opencv_test_core` (incl. `Core_DFT*`, `Core_DCT*`, `Core_MulSpectrums`, `Core_DivSpectrums`) and `opencv_test_imgproc` (phaseCorrelate, matchTemplate, filter2D) pass; the x86-64 build (SSE3 baseline + SSE4.1/AVX2/AVX512_ICL dispatch) passes the DFT/DCT tests under Rosetta 2 (AVX2/AVX-512 paths compile-checked only on this machine). ## Performance ### 1D transforms (single row, one thread), microseconds `C2C` = complex-to-complex, `R2C` = real-to-CCS, `C2R` = CCS-to-real (inverse, scaled), `32`/`64` = float/double. | n | C2C32 | C2C64 | R2C32 | C2R32 | R2C64 | DCT32 | IDCT32 | DCT64 | |---|---|---|---|---|---|---|---|---| | 256 | 0.82 → 0.55 | 0.86 → 0.74 | 0.67 → 0.53 | 0.63 → 0.52 | 0.64 → 0.56 | 0.88 → 0.56 | 0.88 → 0.56 | 0.90 → 0.66 | | 1000 | 3.84 → 1.70 | 4.08 → 2.58 | 2.86 → 1.36 | 2.88 → 1.38 | 2.77 → 1.74 | 3.95 → 1.69 | 4.02 → 1.72 | 3.97 → 2.15 | | 1024 | 3.25 → 1.65 | 3.44 → 2.40 | 2.46 → 1.23 | 2.35 → 1.27 | 2.42 → 1.59 | 3.57 → 1.57 | 3.51 → 1.59 | 3.45 → 2.02 | | 4096 | 14.9 → 6.4 | 16.4 → 11.5 | 10.7 → 4.3 | 10.2 → 4.5 | 11.0 → 6.9 | 15.4 → 6.0 | 15.3 → 6.0 | 17.0 → 9.5 | | 4725 (3³·5²·7) | 27.5 → 17.2 | 29.4 → 24.1 | 28.2 → 16.4 | 29.8 → 18.3 | 30.3 → 24.7 | – | – | – | | 65536 | 350 → 273 | 411 → 494 | 223 → 136 | 274 → 139 | 251 → 249 | 318 → 167 | 299 → 159 | 372 → 320 | Each `cv::dft()` call builds its plan anew; at n = 4096 the plan takes ~1.9 µs of the 6.4 µs (the old code spent ~4 µs of 14.9 on its tables). For n = 65536 both versions are bound by the memory traffic of the out-of-cache data, and C2C64 gets slightly slower (the split layout needs one more pass over 1 MB of data; a plan cache or larger radices could recover this later). ### 2D transforms, single thread, microseconds | size | C2C32 | R2C32 | C2R32 | R2C64 | DCT32 | IDCT32 | R2C32 rows | |---|---|---|---|---|---|---|---| | 256×256 | 297 → 140 | 162 → 75 | 165 → 77 | 178 → 126 | 259 → 166 | 290 → 174 | 72 → 29 | | 512×512 | 1660 → 569 | 765 → 306 | 723 → 317 | 896 → 551 | 1381 → 946 | 1377 → 957 | 303 → 125 | | 480×640 | 1555 → 715 | 895 → 377 | 918 → 392 | 978 → 677 | 1248 → 707 | 1262 → 695 | 389 → 147 | | 1024×1024 | 7508 → 2625 | 3781 → 1387 | 3883 → 1398 | 4964 → 3201 | 6438 → 4556 | 6805 → 4583 | 1341 → 516 | | 1080×1920 | 13141 → 5543 | 7143 → 2880 | 7269 → 2970 | 7770 → 5348 | 9735 → 5190 | 9477 → 5381 | 3116 → 1192 | | 2048×2048 | 38151 → 16822 | 18381 → 6973 | 18784 → 7294 | 25146 → 17476 | 30291 → 21631 | 30540 → 22003 | 5809 → 2242 | ### 2D transforms, default thread count (16), microseconds | size | C2C32 | R2C32 | C2R32 | R2C64 | DCT32 | IDCT32 | R2C32 rows | |---|---|---|---|---|---|---|---| | 256×256 | 309 → 138 | 157 → 75 | 161 → 77 | 174 → 126 | 265 → 167 | 293 → 170 | 71 → 29 | | 512×512 | 1660 → 162 | 785 → 155 | 788 → 160 | 866 → 256 | 1410 → 235 | 1366 → 228 | 303 → 41 | | 480×640 | 1531 → 178 | 895 → 142 | 916 → 141 | 957 → 222 | 1241 → 172 | 1248 → 173 | 393 → 46 | | 1024×1024 | 7495 → 381 | 3755 → 274 | 3767 → 269 | 4875 → 478 | 6323 → 650 | 6583 → 630 | 1343 → 94 | | 1080×1920 | 12905 → 681 | 7131 → 403 | 7240 → 382 | 7760 → 657 | 9673 → 605 | 9513 → 607 | 3124 → 143 | | 2048×2048 | 38155 → 1481 | 18324 → 826 | 18559 → 857 | 24861 → 1776 | 30120 → 2592 | 30741 → 2734 | 5890 → 274 | (5.x is single-threaded for all of these; 256×256 stays single-threaded in the new code by design.) ### matchTemplate (`opencv_perf_imgproc`, `*matchTemplate*`), milliseconds Whole suite (156 cases): 921 → 284 ms. Selected cases (8-bit image, 16×16 template unless noted): | image, template | SQDIFF | SQDIFF_NORMED | CCORR | CCORR_NORMED | CCOEFF | CCOEFF_NORMED | |---|---|---|---|---|---|---| | 128×128 | 0.15 → 0.12 | 0.14 → 0.12 | 0.09 → 0.05 | 0.13 → 0.09 | 0.11 → 0.08 | 0.13 → 0.10 | | 320×240 | 0.99 → 0.52 | 1.09 → 0.58 | 0.79 → 0.23 | 0.96 → 0.42 | 0.93 → 0.38 | 1.02 → 0.53 | | 640×480 | 3.19 → 1.17 | 3.35 → 1.22 | 2.27 → 0.29 | 3.12 → 1.04 | 2.94 → 0.90 | 3.43 → 1.38 | | 800×600 | 5.86 → 1.64 | 5.61 → 1.77 | 4.11 → 0.34 | 5.48 → 1.53 | 5.19 → 1.28 | 5.82 → 2.04 | | 1024×768 | 9.73 → 2.72 | 9.59 → 2.90 | 6.88 → 0.48 | 9.11 → 2.45 | 8.44 → 2.05 | 9.84 → 3.44 | | 1280×1024 | 14.92 → 4.33 | 15.01 → 4.58 | 10.70 → 0.64 | 14.31 → 3.97 | 13.40 → 3.33 | 15.37 → 5.58 | | 1280×1024, 1260×1000 | 13.06 → 7.90 | 13.08 → 7.74 | 10.75 → 3.58 | 12.23 → 5.03 | 11.11 → 3.94 | 12.03 → 5.05 | `TM_CCORR` is now dominated by the (still single-threaded) `integral()` and normalization passes in the other methods; those are candidates for a follow-up. ## Notes for reviewers - `modules/core/src/dxt.hpp` documents the engine layout; the plan builder (`DftPlan::build`) and the driver (`runDft`) are in `dxt.cpp`, the kernels in `dxt.simd.hpp`. - Stages whose span is smaller than the vector width run scalar (on NEON this only affects lengths of the form 2·odd; on AVX2/AVX-512 also the first radix-4 stage of 4^k lengths); handling them with narrower fixed-width vectors is a possible follow-up. - A radix-8 middle stage was implemented and measured slower on NEON than two radix-4 stages (register pressure), so it is not included. ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake