mirror of
https://github.com/opencv/opencv.git
synced 2026-11-16 06:17:53 -06:00
vectorized and parallel DFT/DCT + matchTemplate - #30061 (Made with a help from Claude Code Fable 5.1) ## Summary This PR replaces the scalar 1D DFT/DCT kernels in `modules/core/src/dxt.cpp` with a new engine built on universal intrinsics and compiled per ISA through `ocv_add_dispatched_file`, and makes the 2D transforms, the `DFT_ROWS`/`DCT_ROWS` batches and `matchTemplate` (`crossCorr`) multi-threaded. The public API, the output formats (CCS packing of real spectra, `DFT_COMPLEX_OUTPUT` layout, DCT ordering and normalization) and the IPP/OpenCL paths are unchanged. Only floating-point rounding may differ (the new twiddle tables are more accurate than the old recurrence-generated ones, so the error goes down, not up). Measured on Apple M4 Max (16 cores, NEON), macOS 26.5.x, Xcode 26.6, Release build (`-O3`, no IPP). All numbers are the minimum over repeated runs, `5.x` vs this PR. ## Changes ### New 1D engine (`dxt.hpp`, `dxt.simd.hpp`, `dxt.cpp`) - **Split re/im layout.** The input pass gathers the data into separate `re[]`/`im[]` workspace arrays; all middle stages operate on the split arrays, so a complex multiplication is four independent vector FMAs with no lane shuffling. The output pass interleaves back (or packs the CCS / DCT layout) with the scaling fused in. - **Mixed-radix decimation-in-time with a fused first stage.** The plan chooses the first stage (radix 1/2/4/8, twiddle-free, fused with the gather and the digit-reversal permutation) from the power of two in the length, then radix-4 stages, then radix-3, radix-5 and a generic odd radix. There are no radix-2 middle stages. - **Vectorized butterflies with universal intrinsics** (`v_float32`/`v_float64`, VLA-safe): one flat loop per stage with a single clamped counter for both the data and the twiddle offsets (the back-off clamp handles spans that are not multiples of the vector width; such stages run out of place, ping-ponging between two workspace pairs, the others run in place). - **Vectorized first stage.** The radix-2/4/8 first stage reads the input columns contiguously (`v_load_deinterleave`), runs the small DFT in registers and scatters the outputs of every lane with one 4-channel interleaved store. Only odd lengths use a scalar table-driven gather. - **Vectorized edge passes**: interleave/deinterleave, the real-FFT untangling in both directions (with `v_reverse` for the mirrored half), DCT-II post-twiddle and DCT-III pre-twiddle, the Makhoul permutations. - **Inverse = conjugation trick**: the middle stages are direction-free and use forward twiddles only; the inverse conjugates on the gather and negates the imaginary scale on the output. - **Immutable plan + per-executor workspace, zero allocations per transform.** The plan (radix sequence, permutation table, per-stage split twiddle tables) is built once in `init()` and shared by all rows of a batch; every executor owns a workspace. Twiddles are generated in double precision from a two-level exact table (a few ulps, much cheaper to build than the old N-point recurrence, which matters because `cv::dft()` builds a plan on every call). - **Per-ISA dispatch**: `ocv_add_dispatched_file(dxt SSE2 AVX2 AVX512_ICL RVV)`; the kernels are reached through a function-pointer table resolved once per depth, following the pattern of `arithm.simd.hpp` / `warp_kernels.simd.hpp`. The plan is built with the vector width of the dispatched kernels (never with the baseline `VTraits`). - The legacy scalar engine, `DFTInit`, the SSE3 `DFT_VecR*` butterflies and the `noPermute`/ `factors[0] >>= 1` machinery are removed (−1400 lines). The legacy `cv_hal_dft` hook (which took the old tables) is no longer called from the 1D path; the in-tree RISC-V HAL implementation of it will be updated in a separate PR. ### Parallelism - `crossCorr` (`matchTemplate`): the correlation tiles are independent and are now distributed with `parallel_for_`; every worker creates its DFT buffers and `DFT2D` contexts once per range. (The tile loop had been sequential since the C++ rewrite in 2010, which dropped the OpenMP pragma of `icvCrossCorr`.) - 2D DFT/DCT and `DFT_ROWS`/`DCT_ROWS` batches: the row pass, the column pass and the DCT line loop distribute their lines over threads; the 1D plan is shared, each worker allocates its workspace once per range. The number of stripes is derived from the N log N cost, so a 256x256 transform stays on the calling thread and 1024x1024 gets ~20 stripes. Nested calls (the 2D DFTs inside the parallel `crossCorr` tiles) are serialized by `parallel_for_` itself. - The column pass copies columns in batches (`CopyFromColumns`/`CopyToColumns`, 4..16 adjacent columns copied row by row into contiguous buffers sized to stay within 128 KB) instead of two at a time: with power-of-two row steps the old strided copies aliased in the cache and took 60-70% of the 2D time. ### Tests - `Core_DFT.sweep_1d`, `Core_DCT.sweep_1d`: every 5-smooth length up to 2048 plus odd, prime and "awkward" lengths, both depths, C2C forward/inverse, R2C (CCS and complex output), C2R (CCS and complex input), DCT/IDCT, in-place, column vectors, round trips; compared against the naive O(N^2) references of `test_dxt.cpp` with a tolerance of `32*eps*(log2(N) + 2 + maxfactor/16)`. - `Core_DFT.sweep_rows`: `DFT_ROWS` batches sharing one plan. - `Core_DFT.nonzero_rows`: 2D transforms with `nonzero_rows` (previously uncovered). - `Core_DFT.parallel_large`: large 2D / rows / DCT transforms, multi-threaded result vs the single-threaded one. - The naive DCT reference reduces the angle exactly before `cos()` (its accuracy was limiting the comparison for n > 100 in double precision). - Perf: `perf_dft.cpp` gains 1D row/column cases for `dft` and `dct`. - Existing `opencv_test_core` (incl. `Core_DFT*`, `Core_DCT*`, `Core_MulSpectrums`, `Core_DivSpectrums`) and `opencv_test_imgproc` (phaseCorrelate, matchTemplate, filter2D) pass; the x86-64 build (SSE3 baseline + SSE4.1/AVX2/AVX512_ICL dispatch) passes the DFT/DCT tests under Rosetta 2 (AVX2/AVX-512 paths compile-checked only on this machine). ## Performance ### 1D transforms (single row, one thread), microseconds `C2C` = complex-to-complex, `R2C` = real-to-CCS, `C2R` = CCS-to-real (inverse, scaled), `32`/`64` = float/double. | n | C2C32 | C2C64 | R2C32 | C2R32 | R2C64 | DCT32 | IDCT32 | DCT64 | |---|---|---|---|---|---|---|---|---| | 256 | 0.82 → 0.55 | 0.86 → 0.74 | 0.67 → 0.53 | 0.63 → 0.52 | 0.64 → 0.56 | 0.88 → 0.56 | 0.88 → 0.56 | 0.90 → 0.66 | | 1000 | 3.84 → 1.70 | 4.08 → 2.58 | 2.86 → 1.36 | 2.88 → 1.38 | 2.77 → 1.74 | 3.95 → 1.69 | 4.02 → 1.72 | 3.97 → 2.15 | | 1024 | 3.25 → 1.65 | 3.44 → 2.40 | 2.46 → 1.23 | 2.35 → 1.27 | 2.42 → 1.59 | 3.57 → 1.57 | 3.51 → 1.59 | 3.45 → 2.02 | | 4096 | 14.9 → 6.4 | 16.4 → 11.5 | 10.7 → 4.3 | 10.2 → 4.5 | 11.0 → 6.9 | 15.4 → 6.0 | 15.3 → 6.0 | 17.0 → 9.5 | | 4725 (3³·5²·7) | 27.5 → 17.2 | 29.4 → 24.1 | 28.2 → 16.4 | 29.8 → 18.3 | 30.3 → 24.7 | – | – | – | | 65536 | 350 → 273 | 411 → 494 | 223 → 136 | 274 → 139 | 251 → 249 | 318 → 167 | 299 → 159 | 372 → 320 | Each `cv::dft()` call builds its plan anew; at n = 4096 the plan takes ~1.9 µs of the 6.4 µs (the old code spent ~4 µs of 14.9 on its tables). For n = 65536 both versions are bound by the memory traffic of the out-of-cache data, and C2C64 gets slightly slower (the split layout needs one more pass over 1 MB of data; a plan cache or larger radices could recover this later). ### 2D transforms, single thread, microseconds | size | C2C32 | R2C32 | C2R32 | R2C64 | DCT32 | IDCT32 | R2C32 rows | |---|---|---|---|---|---|---|---| | 256×256 | 297 → 140 | 162 → 75 | 165 → 77 | 178 → 126 | 259 → 166 | 290 → 174 | 72 → 29 | | 512×512 | 1660 → 569 | 765 → 306 | 723 → 317 | 896 → 551 | 1381 → 946 | 1377 → 957 | 303 → 125 | | 480×640 | 1555 → 715 | 895 → 377 | 918 → 392 | 978 → 677 | 1248 → 707 | 1262 → 695 | 389 → 147 | | 1024×1024 | 7508 → 2625 | 3781 → 1387 | 3883 → 1398 | 4964 → 3201 | 6438 → 4556 | 6805 → 4583 | 1341 → 516 | | 1080×1920 | 13141 → 5543 | 7143 → 2880 | 7269 → 2970 | 7770 → 5348 | 9735 → 5190 | 9477 → 5381 | 3116 → 1192 | | 2048×2048 | 38151 → 16822 | 18381 → 6973 | 18784 → 7294 | 25146 → 17476 | 30291 → 21631 | 30540 → 22003 | 5809 → 2242 | ### 2D transforms, default thread count (16), microseconds | size | C2C32 | R2C32 | C2R32 | R2C64 | DCT32 | IDCT32 | R2C32 rows | |---|---|---|---|---|---|---|---| | 256×256 | 309 → 138 | 157 → 75 | 161 → 77 | 174 → 126 | 265 → 167 | 293 → 170 | 71 → 29 | | 512×512 | 1660 → 162 | 785 → 155 | 788 → 160 | 866 → 256 | 1410 → 235 | 1366 → 228 | 303 → 41 | | 480×640 | 1531 → 178 | 895 → 142 | 916 → 141 | 957 → 222 | 1241 → 172 | 1248 → 173 | 393 → 46 | | 1024×1024 | 7495 → 381 | 3755 → 274 | 3767 → 269 | 4875 → 478 | 6323 → 650 | 6583 → 630 | 1343 → 94 | | 1080×1920 | 12905 → 681 | 7131 → 403 | 7240 → 382 | 7760 → 657 | 9673 → 605 | 9513 → 607 | 3124 → 143 | | 2048×2048 | 38155 → 1481 | 18324 → 826 | 18559 → 857 | 24861 → 1776 | 30120 → 2592 | 30741 → 2734 | 5890 → 274 | (5.x is single-threaded for all of these; 256×256 stays single-threaded in the new code by design.) ### matchTemplate (`opencv_perf_imgproc`, `*matchTemplate*`), milliseconds Whole suite (156 cases): 921 → 284 ms. Selected cases (8-bit image, 16×16 template unless noted): | image, template | SQDIFF | SQDIFF_NORMED | CCORR | CCORR_NORMED | CCOEFF | CCOEFF_NORMED | |---|---|---|---|---|---|---| | 128×128 | 0.15 → 0.12 | 0.14 → 0.12 | 0.09 → 0.05 | 0.13 → 0.09 | 0.11 → 0.08 | 0.13 → 0.10 | | 320×240 | 0.99 → 0.52 | 1.09 → 0.58 | 0.79 → 0.23 | 0.96 → 0.42 | 0.93 → 0.38 | 1.02 → 0.53 | | 640×480 | 3.19 → 1.17 | 3.35 → 1.22 | 2.27 → 0.29 | 3.12 → 1.04 | 2.94 → 0.90 | 3.43 → 1.38 | | 800×600 | 5.86 → 1.64 | 5.61 → 1.77 | 4.11 → 0.34 | 5.48 → 1.53 | 5.19 → 1.28 | 5.82 → 2.04 | | 1024×768 | 9.73 → 2.72 | 9.59 → 2.90 | 6.88 → 0.48 | 9.11 → 2.45 | 8.44 → 2.05 | 9.84 → 3.44 | | 1280×1024 | 14.92 → 4.33 | 15.01 → 4.58 | 10.70 → 0.64 | 14.31 → 3.97 | 13.40 → 3.33 | 15.37 → 5.58 | | 1280×1024, 1260×1000 | 13.06 → 7.90 | 13.08 → 7.74 | 10.75 → 3.58 | 12.23 → 5.03 | 11.11 → 3.94 | 12.03 → 5.05 | `TM_CCORR` is now dominated by the (still single-threaded) `integral()` and normalization passes in the other methods; those are candidates for a follow-up. ## Notes for reviewers - `modules/core/src/dxt.hpp` documents the engine layout; the plan builder (`DftPlan::build`) and the driver (`runDft`) are in `dxt.cpp`, the kernels in `dxt.simd.hpp`. - Stages whose span is smaller than the vector width run scalar (on NEON this only affects lengths of the form 2·odd; on AVX2/AVX-512 also the first radix-4 stage of 4^k lengths); handling them with narrower fixed-width vectors is a possible follow-up. - A radix-8 middle stage was implemented and measured slower on NEON than two radix-4 stages (register pressure), so it is not included. ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake
120 lines
4.0 KiB
C++
120 lines
4.0 KiB
C++
#include "perf_precomp.hpp"
|
|
|
|
namespace opencv_test
|
|
{
|
|
using namespace perf;
|
|
|
|
///////////////////////////////////////////////////////dft//////////////////////////////////////////////////////////////
|
|
|
|
#define MAT_TYPES_DFT CV_32FC1, CV_32FC2, CV_64FC1
|
|
#define MAT_SIZES_DFT cv::Size(320, 480), cv::Size(800, 600), cv::Size(1280, 1024), sz1080p, sz2K
|
|
CV_ENUM(FlagsType, 0, DFT_INVERSE, DFT_SCALE, DFT_COMPLEX_OUTPUT, DFT_ROWS, DFT_INVERSE|DFT_COMPLEX_OUTPUT)
|
|
#define TEST_MATS_DFT testing::Combine(testing::Values(MAT_SIZES_DFT), testing::Values(MAT_TYPES_DFT), FlagsType::all(), testing::Values(true, false))
|
|
|
|
typedef tuple<Size, MatType, FlagsType, bool> Size_MatType_FlagsType_NzeroRows_t;
|
|
typedef perf::TestBaseWithParam<Size_MatType_FlagsType_NzeroRows_t> Size_MatType_FlagsType_NzeroRows;
|
|
|
|
PERF_TEST_P(Size_MatType_FlagsType_NzeroRows, dft, TEST_MATS_DFT)
|
|
{
|
|
Size sz = get<0>(GetParam());
|
|
int type = get<1>(GetParam());
|
|
int flags = get<2>(GetParam());
|
|
bool isNzeroRows = get<3>(GetParam());
|
|
|
|
int nonzero_rows = 0;
|
|
|
|
Mat src(sz, type);
|
|
Mat dst(sz, type);
|
|
|
|
declare.in(src, WARMUP_RNG).time(60);
|
|
|
|
if (isNzeroRows)
|
|
nonzero_rows = sz.height/2;
|
|
|
|
TEST_CYCLE() dft(src, dst, flags, nonzero_rows);
|
|
|
|
SANITY_CHECK(dst, 1e-5, ERROR_RELATIVE);
|
|
}
|
|
|
|
///////////////////////////////////////////////////////dct//////////////////////////////////////////////////////
|
|
|
|
CV_ENUM(DCT_FlagsType, 0, DCT_INVERSE , DCT_ROWS, DCT_INVERSE|DCT_ROWS)
|
|
|
|
typedef tuple<Size, MatType, DCT_FlagsType> Size_MatType_Flag_t;
|
|
typedef perf::TestBaseWithParam<Size_MatType_Flag_t> Size_MatType_Flag;
|
|
|
|
PERF_TEST_P(Size_MatType_Flag, dct, testing::Combine(
|
|
testing::Values(cv::Size(320, 240),cv::Size(800, 600),
|
|
cv::Size(1024, 768), cv::Size(1280, 1024),
|
|
sz1080p, sz2K),
|
|
testing::Values(CV_32FC1, CV_64FC1), DCT_FlagsType::all()))
|
|
{
|
|
Size sz = get<0>(GetParam());
|
|
int type = get<1>(GetParam());
|
|
int flags = get<2>(GetParam());
|
|
|
|
Mat src(sz, type);
|
|
Mat dst(sz, type);
|
|
|
|
declare
|
|
.in(src, WARMUP_RNG)
|
|
.out(dst)
|
|
.time(60);
|
|
|
|
TEST_CYCLE() dct(src, dst, flags);
|
|
|
|
SANITY_CHECK(dst, 1e-5, ERROR_RELATIVE);
|
|
}
|
|
|
|
///////////////////////////////////////////// 1D transforms /////////////////////////////////////////////
|
|
|
|
CV_ENUM(DFT1D_FlagsType, 0, DFT_INVERSE, DFT_SCALE, DFT_COMPLEX_OUTPUT)
|
|
|
|
typedef tuple<int, MatType, DFT1D_FlagsType, bool> Len_MatType_Flags_Col_t;
|
|
typedef perf::TestBaseWithParam<Len_MatType_Flags_Col_t> Len_MatType_Flags_Col;
|
|
|
|
// single row (1 x N) or single column (N x 1) vectors, N covers the 2^k, 5-smooth and odd plans
|
|
PERF_TEST_P(Len_MatType_Flags_Col, dft1d, testing::Combine(
|
|
testing::Values(256, 1000, 1024, 4096, 4725, 65536),
|
|
testing::Values(CV_32FC1, CV_32FC2, CV_64FC1, CV_64FC2),
|
|
DFT1D_FlagsType::all(), testing::Bool()))
|
|
{
|
|
int len = get<0>(GetParam());
|
|
int type = get<1>(GetParam());
|
|
int flags = get<2>(GetParam());
|
|
bool column = get<3>(GetParam());
|
|
|
|
Mat src(column ? len : 1, column ? 1 : len, type);
|
|
Mat dst;
|
|
|
|
declare.in(src, WARMUP_RNG).time(20);
|
|
|
|
TEST_CYCLE() dft(src, dst, flags);
|
|
|
|
SANITY_CHECK_NOTHING();
|
|
}
|
|
|
|
typedef tuple<int, MatType, DCT_FlagsType> Len_MatType_Flag_t;
|
|
typedef perf::TestBaseWithParam<Len_MatType_Flag_t> Len_MatType_Flag;
|
|
|
|
PERF_TEST_P(Len_MatType_Flag, dct1d, testing::Combine(
|
|
testing::Values(256, 1000, 1024, 4096, 65536),
|
|
testing::Values(CV_32FC1, CV_64FC1),
|
|
testing::Values(0, DCT_INVERSE)))
|
|
{
|
|
int len = get<0>(GetParam());
|
|
int type = get<1>(GetParam());
|
|
int flags = get<2>(GetParam());
|
|
|
|
Mat src(1, len, type);
|
|
Mat dst;
|
|
|
|
declare.in(src, WARMUP_RNG).time(20);
|
|
|
|
TEST_CYCLE() dct(src, dst, flags);
|
|
|
|
SANITY_CHECK_NOTHING();
|
|
}
|
|
|
|
} // namespace
|