mirror of
https://chromium.googlesource.com/libyuv/libyuv
synced 2026-10-28 18:07:12 +08:00
205022c08c
Add AVX-512 VNNI (VPDPBUSD / _mm512_dpbusd_epi32) row kernels for 64-bit x86 targets alongside existing AVX-512BW implementations in both row_gcc.cc (inline assembly) and row_win.cc (C++ intrinsics): - ARGBToYMatrixRow_AVX512VNNI and ARGBToYMatrixRow_Any_AVX512VNNI: Pre-computes the 32-bit signed-pixel bias once per row and replaces vpmaddubsw + vpmaddwd + vpaddw with a single vpdpbusd per 16 pixels. - ARGBToUV444MatrixRow_AVX512VNNI and ARGBToUV444MatrixRow_Any_AVX512VNNI: Loads 64 ARGB pixels once instead of twice and computes 32-bit U and V dot products directly via vpdpbusd with signed UV matrix coefficients. - ARGBToUVMatrixRow_AVX512VNNI: Computes 2x2 box-filtered ARGB to U and V conversion using vpdpbusd with k-masked tail handling for arbitrary widths. - ARGBGrayRow_AVX512VNNI: Computes BT.601 grayscale luminance using a single vpdpbusd per 16 pixels, broadcasts Y to RGB via vpshufb, and preserves alpha via vpternlogd with k-masked tail handling. - Wire runtime dispatch via TestCpuFlag(kCpuHasAVX512VNNI) across convert.cc, convert_from_argb.cc, and planar_functions.cc. Add AVX512VNNIVBMI variants of the Y, UV444 and UV kernels. These use both AVX-512 VNNI (vpdpbusd) and AVX-512 VBMI (vpermt2b) and are only selected when the CPU has both features; there are no VBMI versions without VNNI. The AVX512VNNI kernels remain for CPUs with VNNI but no VBMI (Cascade Lake, Cooper Lake). - The 32-bit sums are always in [0, 65535], so Y, U and V are byte 1 of each int32. Two vpermt2b gather 64 results from 4 accumulators, replacing vpackusdw/vpackssdw + vpsrlw + vpackuswb + vpermd. - U and V accumulators start at ~kAddUV so they hold ~(kAddUV - dot). The inverse is folded into the vpternlogd that merges the two 256-bit halves. kAddUV is loaded from ArgbConstants rather than hard coded as 0x8000. - Y folds the source loads into vpxorq. - row_win.cc hides the vpermt2b indices from clang, which otherwise constant folds them into a longer vpshufb + vpor + vpermd sequence. Performance (1280x720, 5000 iterations, AVX512BW -> AVX512VNNI): uArch / Benchmark AVX512BW AVX512VNNI Speedup --------------------------- -------- ---------- ------- Intel Ice Lake ARGBToI420 941 ms 862 ms +9.2% Intel Ice Lake ARGBToI422 1059 ms 947 ms +11.8% Intel Ice Lake ARGBToI444 1140 ms 1038 ms +9.8% Intel Ice Lake ARGBGray 357 ms 273 ms +30.8% Intel Sapphire Rapids GrayTo 317 ms 218 ms +45.4% Intel Emerald Rapids GrayTo 310 ms 213 ms +45.5% Intel Granite Rapids GrayTo 349 ms 241 ms +44.8% AMD Zen 4 ARGBToI420 648 ms 517 ms +25.3% AMD Zen 4 ARGBToI444 695 ms 526 ms +32.1% AMD Zen 4 ARGBGray 403 ms 227 ms +77.5% AMD Zen 5 ARGBToI422 460 ms 403 ms +14.1% AMD Zen 5 ARGBToI444 382 ms 321 ms +19.0% Performance (row_gcc, AVX512VNNI -> AVX512VNNIVBMI): uArch / Benchmark AVX512VNNI VNNIVBMI Speedup ---------------------------- ---------- -------- ------- AMD Zen 6 ARGBToI420 1280x720 327 ms 284 ms +15.1% AMD Zen 6 ARGBToI422 1280x720 372 ms 330 ms +12.7% AMD Zen 6 ARGBToI444 1280x720 357 ms 334 ms +6.9% AMD Zen 6 ARGBToI400 256x24 94 ms 76 ms +23.7% AMD Zen 5 ARGBToI422 1280x720 406 ms 350 ms +16.0% AMD Zen 5 ARGBToI444 1280x720 330 ms 301 ms +9.6% AMD Zen 5 ARGBToI400 256x24 100 ms 90 ms +11.1% AMD Zen 4 ARGBToI420 1280x720 631 ms 540 ms +16.9% AMD Zen 4 ARGBToI422 1280x720 993 ms 825 ms +20.4% AMD Zen 4 ARGBToI400 256x24 245 ms 218 ms +12.4% Intel GNR ARGBToI422 256x24 654 ms 616 ms +6.2% Intel SPR ARGBToI422 256x24 653 ms 613 ms +6.5% Intel GNR, EMR, SPR and Ice Lake are within +/-3% at 1280x720. R=rrwinterton@gmail.com Bug: none Test: out/Release/libyuv_unittest --gtest_filter=*ARGBToI4*:*ARGBGray* Test: LIBYUV_DISABLE_AVX512VBMI=1 out/Release/libyuv_unittest Test: sde64 -gnr / -clx with is_msan=true (row_win.cc) Change-Id: Ia7a37b39dd2948fe4055a8a7694dd1ff8ac3e55c Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8647811 Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|---|---|---|
| .. | ||
| compare_common.cc | ||
| compare_gcc.cc | ||
| compare_hvx.cc | ||
| compare_neon64.cc | ||
| compare_neon.cc | ||
| compare_wasm.cc | ||
| compare_win.cc | ||
| compare.cc | ||
| convert_argb.cc | ||
| convert_from_argb.cc | ||
| convert_from.cc | ||
| convert_jpeg.cc | ||
| convert_to_argb.cc | ||
| convert_to_i420.cc | ||
| convert.cc | ||
| cpu_id.cc | ||
| mjpeg_decoder.cc | ||
| mjpeg_validate.cc | ||
| planar_functions.cc | ||
| rotate_any.cc | ||
| rotate_argb.cc | ||
| rotate_common.cc | ||
| rotate_gcc.cc | ||
| rotate_hvx.cc | ||
| rotate_lasx.cc | ||
| rotate_lsx.cc | ||
| rotate_neon64.cc | ||
| rotate_neon.cc | ||
| rotate_rvv.cc | ||
| rotate_sme.cc | ||
| rotate_sve.cc | ||
| rotate_wasm.cc | ||
| rotate_win.cc | ||
| rotate.cc | ||
| row_any.cc | ||
| row_common.cc | ||
| row_gcc.cc | ||
| row_hvx.cc | ||
| row_lasx.cc | ||
| row_lsx.cc | ||
| row_neon64.cc | ||
| row_neon.cc | ||
| row_rvv.cc | ||
| row_sme.cc | ||
| row_sve.cc | ||
| row_wasm.cc | ||
| row_win.cc | ||
| scale_any.cc | ||
| scale_argb.cc | ||
| scale_common.cc | ||
| scale_gcc.cc | ||
| scale_hvx.cc | ||
| scale_lsx.cc | ||
| scale_neon64.cc | ||
| scale_neon.cc | ||
| scale_rgb.cc | ||
| scale_rvv.cc | ||
| scale_sme.cc | ||
| scale_uv.cc | ||
| scale_wasm.cc | ||
| scale_win.cc | ||
| scale.cc | ||
| test.sh | ||
| video_common.cc | ||