Commit Graph

14 Commits

Author SHA1 Message Date
Vadim Pisarevsky
6a5623e271 Merge pull request #30061 from vpisarev:fft_vector
vectorized and parallel DFT/DCT + matchTemplate - #30061

(Made with a help from Claude Code Fable 5.1)

## Summary

This PR replaces the scalar 1D DFT/DCT kernels in `modules/core/src/dxt.cpp` with a new engine
built on universal intrinsics and compiled per ISA through `ocv_add_dispatched_file`, and makes
the 2D transforms, the `DFT_ROWS`/`DCT_ROWS` batches and `matchTemplate` (`crossCorr`) multi-threaded.

The public API, the output formats (CCS packing of real spectra, `DFT_COMPLEX_OUTPUT` layout,
DCT ordering and normalization) and the IPP/OpenCL paths are unchanged. Only floating-point
rounding may differ (the new twiddle tables are more accurate than the old recurrence-generated
ones, so the error goes down, not up).

Measured on Apple M4 Max (16 cores, NEON), macOS 26.5.x, Xcode 26.6, Release build
(`-O3`, no IPP). All numbers are the minimum over repeated runs, `5.x` vs this PR.

## Changes

### New 1D engine (`dxt.hpp`, `dxt.simd.hpp`, `dxt.cpp`)

- **Split re/im layout.** The input pass gathers the data into separate `re[]`/`im[]` workspace
  arrays; all middle stages operate on the split arrays, so a complex multiplication is four
  independent vector FMAs with no lane shuffling. The output pass interleaves back (or packs the
  CCS / DCT layout) with the scaling fused in.
- **Mixed-radix decimation-in-time with a fused first stage.** The plan chooses the first stage
  (radix 1/2/4/8, twiddle-free, fused with the gather and the digit-reversal permutation) from
  the power of two in the length, then radix-4 stages, then radix-3, radix-5 and a generic odd
  radix. There are no radix-2 middle stages.
- **Vectorized butterflies with universal intrinsics** (`v_float32`/`v_float64`, VLA-safe): one
  flat loop per stage with a single clamped counter for both the data and the twiddle offsets
  (the back-off clamp handles spans that are not multiples of the vector width; such stages run
  out of place, ping-ponging between two workspace pairs, the others run in place).
- **Vectorized first stage.** The radix-2/4/8 first stage reads the input columns contiguously
  (`v_load_deinterleave`), runs the small DFT in registers and scatters the outputs of every
  lane with one 4-channel interleaved store. Only odd lengths use a scalar table-driven gather.
- **Vectorized edge passes**: interleave/deinterleave, the real-FFT untangling in both directions
  (with `v_reverse` for the mirrored half), DCT-II post-twiddle and DCT-III pre-twiddle, the
  Makhoul permutations.
- **Inverse = conjugation trick**: the middle stages are direction-free and use forward twiddles
  only; the inverse conjugates on the gather and negates the imaginary scale on the output.
- **Immutable plan + per-executor workspace, zero allocations per transform.** The plan
  (radix sequence, permutation table, per-stage split twiddle tables) is built once in `init()`
  and shared by all rows of a batch; every executor owns a workspace. Twiddles are generated in
  double precision from a two-level exact table (a few ulps, much cheaper to build than the old
  N-point recurrence, which matters because `cv::dft()` builds a plan on every call).
- **Per-ISA dispatch**: `ocv_add_dispatched_file(dxt SSE2 AVX2 AVX512_ICL RVV)`; the kernels are
  reached through a function-pointer table resolved once per depth, following the pattern of
  `arithm.simd.hpp` / `warp_kernels.simd.hpp`. The plan is built with the vector width of the
  dispatched kernels (never with the baseline `VTraits`).
- The legacy scalar engine, `DFTInit`, the SSE3 `DFT_VecR*` butterflies and the `noPermute`/
  `factors[0] >>= 1` machinery are removed (−1400 lines). The legacy `cv_hal_dft` hook (which
  took the old tables) is no longer called from the 1D path; the in-tree RISC-V HAL
  implementation of it will be updated in a separate PR.

### Parallelism

- `crossCorr` (`matchTemplate`): the correlation tiles are independent and are now distributed
  with `parallel_for_`; every worker creates its DFT buffers and `DFT2D` contexts once per range.
  (The tile loop had been sequential since the C++ rewrite in 2010, which dropped the OpenMP
  pragma of `icvCrossCorr`.)
- 2D DFT/DCT and `DFT_ROWS`/`DCT_ROWS` batches: the row pass, the column pass and the DCT line
  loop distribute their lines over threads; the 1D plan is shared, each worker allocates its
  workspace once per range. The number of stripes is derived from the N log N cost, so a 256x256
  transform stays on the calling thread and 1024x1024 gets ~20 stripes. Nested calls (the 2D DFTs
  inside the parallel `crossCorr` tiles) are serialized by `parallel_for_` itself.
- The column pass copies columns in batches (`CopyFromColumns`/`CopyToColumns`, 4..16 adjacent
  columns copied row by row into contiguous buffers sized to stay within 128 KB) instead of two
  at a time: with power-of-two row steps the old strided copies aliased in the cache and took
  60-70% of the 2D time.

### Tests

- `Core_DFT.sweep_1d`, `Core_DCT.sweep_1d`: every 5-smooth length up to 2048 plus odd, prime and
  "awkward" lengths, both depths, C2C forward/inverse, R2C (CCS and complex output), C2R (CCS and
  complex input), DCT/IDCT, in-place, column vectors, round trips; compared against the naive
  O(N^2) references of `test_dxt.cpp` with a tolerance of `32*eps*(log2(N) + 2 + maxfactor/16)`.
- `Core_DFT.sweep_rows`: `DFT_ROWS` batches sharing one plan.
- `Core_DFT.nonzero_rows`: 2D transforms with `nonzero_rows` (previously uncovered).
- `Core_DFT.parallel_large`: large 2D / rows / DCT transforms, multi-threaded result vs the
  single-threaded one.
- The naive DCT reference reduces the angle exactly before `cos()` (its accuracy was limiting
  the comparison for n > 100 in double precision).
- Perf: `perf_dft.cpp` gains 1D row/column cases for `dft` and `dct`.
- Existing `opencv_test_core` (incl. `Core_DFT*`, `Core_DCT*`, `Core_MulSpectrums`,
  `Core_DivSpectrums`) and `opencv_test_imgproc` (phaseCorrelate, matchTemplate, filter2D) pass;
  the x86-64 build (SSE3 baseline + SSE4.1/AVX2/AVX512_ICL dispatch) passes the DFT/DCT tests
  under Rosetta 2 (AVX2/AVX-512 paths compile-checked only on this machine).

## Performance

### 1D transforms (single row, one thread), microseconds

`C2C` = complex-to-complex, `R2C` = real-to-CCS, `C2R` = CCS-to-real (inverse, scaled),
`32`/`64` = float/double.

| n | C2C32 | C2C64 | R2C32 | C2R32 | R2C64 | DCT32 | IDCT32 | DCT64 |
|---|---|---|---|---|---|---|---|---|
| 256 | 0.82 → 0.55 | 0.86 → 0.74 | 0.67 → 0.53 | 0.63 → 0.52 | 0.64 → 0.56 | 0.88 → 0.56 | 0.88 → 0.56 | 0.90 → 0.66 |
| 1000 | 3.84 → 1.70 | 4.08 → 2.58 | 2.86 → 1.36 | 2.88 → 1.38 | 2.77 → 1.74 | 3.95 → 1.69 | 4.02 → 1.72 | 3.97 → 2.15 |
| 1024 | 3.25 → 1.65 | 3.44 → 2.40 | 2.46 → 1.23 | 2.35 → 1.27 | 2.42 → 1.59 | 3.57 → 1.57 | 3.51 → 1.59 | 3.45 → 2.02 |
| 4096 | 14.9 → 6.4 | 16.4 → 11.5 | 10.7 → 4.3 | 10.2 → 4.5 | 11.0 → 6.9 | 15.4 → 6.0 | 15.3 → 6.0 | 17.0 → 9.5 |
| 4725 (3³·5²·7) | 27.5 → 17.2 | 29.4 → 24.1 | 28.2 → 16.4 | 29.8 → 18.3 | 30.3 → 24.7 | – | – | – |
| 65536 | 350 → 273 | 411 → 494 | 223 → 136 | 274 → 139 | 251 → 249 | 318 → 167 | 299 → 159 | 372 → 320 |

Each `cv::dft()` call builds its plan anew; at n = 4096 the plan takes ~1.9 µs of the 6.4 µs
(the old code spent ~4 µs of 14.9 on its tables). For n = 65536 both versions are bound by the
memory traffic of the out-of-cache data, and C2C64 gets slightly slower (the split layout needs
one more pass over 1 MB of data; a plan cache or larger radices could recover this later).

### 2D transforms, single thread, microseconds

| size | C2C32 | R2C32 | C2R32 | R2C64 | DCT32 | IDCT32 | R2C32 rows |
|---|---|---|---|---|---|---|---|
| 256×256 | 297 → 140 | 162 → 75 | 165 → 77 | 178 → 126 | 259 → 166 | 290 → 174 | 72 → 29 |
| 512×512 | 1660 → 569 | 765 → 306 | 723 → 317 | 896 → 551 | 1381 → 946 | 1377 → 957 | 303 → 125 |
| 480×640 | 1555 → 715 | 895 → 377 | 918 → 392 | 978 → 677 | 1248 → 707 | 1262 → 695 | 389 → 147 |
| 1024×1024 | 7508 → 2625 | 3781 → 1387 | 3883 → 1398 | 4964 → 3201 | 6438 → 4556 | 6805 → 4583 | 1341 → 516 |
| 1080×1920 | 13141 → 5543 | 7143 → 2880 | 7269 → 2970 | 7770 → 5348 | 9735 → 5190 | 9477 → 5381 | 3116 → 1192 |
| 2048×2048 | 38151 → 16822 | 18381 → 6973 | 18784 → 7294 | 25146 → 17476 | 30291 → 21631 | 30540 → 22003 | 5809 → 2242 |

### 2D transforms, default thread count (16), microseconds

| size | C2C32 | R2C32 | C2R32 | R2C64 | DCT32 | IDCT32 | R2C32 rows |
|---|---|---|---|---|---|---|---|
| 256×256 | 309 → 138 | 157 → 75 | 161 → 77 | 174 → 126 | 265 → 167 | 293 → 170 | 71 → 29 |
| 512×512 | 1660 → 162 | 785 → 155 | 788 → 160 | 866 → 256 | 1410 → 235 | 1366 → 228 | 303 → 41 |
| 480×640 | 1531 → 178 | 895 → 142 | 916 → 141 | 957 → 222 | 1241 → 172 | 1248 → 173 | 393 → 46 |
| 1024×1024 | 7495 → 381 | 3755 → 274 | 3767 → 269 | 4875 → 478 | 6323 → 650 | 6583 → 630 | 1343 → 94 |
| 1080×1920 | 12905 → 681 | 7131 → 403 | 7240 → 382 | 7760 → 657 | 9673 → 605 | 9513 → 607 | 3124 → 143 |
| 2048×2048 | 38155 → 1481 | 18324 → 826 | 18559 → 857 | 24861 → 1776 | 30120 → 2592 | 30741 → 2734 | 5890 → 274 |

(5.x is single-threaded for all of these; 256×256 stays single-threaded in the new code by design.)

### matchTemplate (`opencv_perf_imgproc`, `*matchTemplate*`), milliseconds

Whole suite (156 cases): 921 → 284 ms. Selected cases (8-bit image, 16×16 template unless noted):

| image, template | SQDIFF | SQDIFF_NORMED | CCORR | CCORR_NORMED | CCOEFF | CCOEFF_NORMED |
|---|---|---|---|---|---|---|
| 128×128 | 0.15 → 0.12 | 0.14 → 0.12 | 0.09 → 0.05 | 0.13 → 0.09 | 0.11 → 0.08 | 0.13 → 0.10 |
| 320×240 | 0.99 → 0.52 | 1.09 → 0.58 | 0.79 → 0.23 | 0.96 → 0.42 | 0.93 → 0.38 | 1.02 → 0.53 |
| 640×480 | 3.19 → 1.17 | 3.35 → 1.22 | 2.27 → 0.29 | 3.12 → 1.04 | 2.94 → 0.90 | 3.43 → 1.38 |
| 800×600 | 5.86 → 1.64 | 5.61 → 1.77 | 4.11 → 0.34 | 5.48 → 1.53 | 5.19 → 1.28 | 5.82 → 2.04 |
| 1024×768 | 9.73 → 2.72 | 9.59 → 2.90 | 6.88 → 0.48 | 9.11 → 2.45 | 8.44 → 2.05 | 9.84 → 3.44 |
| 1280×1024 | 14.92 → 4.33 | 15.01 → 4.58 | 10.70 → 0.64 | 14.31 → 3.97 | 13.40 → 3.33 | 15.37 → 5.58 |
| 1280×1024, 1260×1000 | 13.06 → 7.90 | 13.08 → 7.74 | 10.75 → 3.58 | 12.23 → 5.03 | 11.11 → 3.94 | 12.03 → 5.05 |

`TM_CCORR` is now dominated by the (still single-threaded) `integral()` and normalization passes
in the other methods; those are candidates for a follow-up.

## Notes for reviewers

- `modules/core/src/dxt.hpp` documents the engine layout; the plan builder (`DftPlan::build`)
  and the driver (`runDft`) are in `dxt.cpp`, the kernels in `dxt.simd.hpp`.
- Stages whose span is smaller than the vector width run scalar (on NEON this only affects
  lengths of the form 2·odd; on AVX2/AVX-512 also the first radix-4 stage of 4^k lengths);
  handling them with narrower fixed-width vectors is a possible follow-up.
- A radix-8 middle stage was implemented and measured slower on NEON than two radix-4 stages
  (register pressure), so it is not included.

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
2026-10-21 09:10:35 +03:00
Alexander Alekhin
4a297a2443 ts: refactor OpenCV tests
- removed tr1 usage (dropped in C++17)
- moved includes of vector/map/iostream/limits into ts.hpp
- require opencv_test + anonymous namespace (added compile check)
- fixed norm() usage (must be from cvtest::norm for checks) and other conflict functions
- added missing license headers
2018-02-03 19:39:47 +00:00
Alexander Alekhin
849228ac6a Merge pull request #2684 from ElenaGvozdeva:ippiDFT 2014-05-07 17:57:17 +04:00
Elena Gvozdeva
e362c3fb38 fixed 2014-05-06 13:09:45 +04:00
Elena Gvozdeva
507ea95265 fixed 2014-04-28 16:46:11 +04:00
Elena Gvozdeva
6880dbfd95 IPP: cv::dct 2014-04-28 16:05:19 +04:00
Elena Gvozdeva
2d71c094b3 IPP: CV::dft 2014-04-23 15:52:05 +04:00
vbystricky
9b6feee1a5 Change sanity check for perfomance test of dft 2014-04-07 14:26:47 +04:00
OpenCV Buildbot
81f826db2b Normalize line endings and whitespace 2012-10-17 15:57:49 +04:00
Andrey Kamaev
65f5343ed5 Refactored performance tests. TEST_CYCLE macro is renamed to TEST_CYCLE_N; SIMPLE_TEST_CYCLE is renamed to TEST_CYCLE; from now 100 iterations are default for performance tests 2011-12-29 16:46:16 +00:00
Kirill Kornyakov
70da780c8e added perf test for dot product (8U, 32S, 32F) 2011-12-28 13:29:27 +00:00
Andrey Kamaev
f937d68d60 Fixed build for x64 with sse2 disabled; several improvements in performance testing framework; enabled SANITY_CHECK() for performance tests 2011-12-27 16:37:24 +00:00
Kirill Kornyakov
177618984d dft perf test minor fix 2011-10-21 08:28:33 +00:00
Kirill Kornyakov
f2b72928cc created perf test for DFT 2011-10-20 07:32:12 +00:00