mirror of
https://chromium.googlesource.com/libyuv/libyuv
synced 2026-10-29 02:17:13 +08:00
2175 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
George Steed
|
2e65a61dff |
AArch64: Avoid lane-indexed LD4 dependency in 32 bit transpose
The original main loop builds v0-v3 with four lane-indexed LD4 load
instructions. Since the first LD4 only updates lane 0, it retains a
false dependency on the registers produced by the previous loop
iteration, leading to a large performance penalty on larger out-of-order
micro-architectures.
Replace the first LD4 with two scalar-register LDPs to zero the top part
of the four registers, breaking the loop-carried dependency.
Performance versus the existing Neon code, higher is better:
Cortex-A55: 0.993x
Cortex-A510: 1.072x
Cortex-A720: 0.999x
Cortex-X4: 1.052x
C1-Nano: 2.093x
C1-Pro: 2.011x
C1-Ultra: 2.128x
Change-Id: Ib5d71e8150d7b11048ddf0fb91e107f9c9323a3f
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8669762
Reviewed-by: Frank Barchard <fbarchard@google.com>
Reviewed-by: Justin Green <greenjustin@google.com>
|
||
|
Frank Barchard
|
4bc0419865 |
[libyuv] Remove unused prototypes and HAS_ macros
Maximum SIMD ISA supported per C row function: ------------------------------------------------------------------------ C Row Function arm x86 riscv wasm loongson hexagon ------------------------------------------------------------------------ AB64ToARGBRow_C NEON AVX2 RVV - - HVX ABGRToAR30Row_C NEON AVX512BW - - - - AR30ToAB30Row_C - AVX512BW - - - - AR30ToABGRRow_C - AVX512BW - - - - AR30ToARGBRow_C - AVX512BW - - - - AR64ShuffleRow_C NEON AVX512BW RVV WASMSIMD - - AR64ToAB64Row_C NEON AVX512BW RVV WASMSIMD - HVX AR64ToARGBRow_C NEON AVX2 RVV - - HVX ARGB1555ToARGBRow_C SVE2 AVX2 - WASMSIMD LASX - ARGB1555ToUVMatrixRow_C NEON AVX2 - WASMSIMD - - ARGB1555ToUVRow_C NEON - - - LASX - ARGB1555ToYMatrixRow_C NEON AVX2 - WASMSIMD - - ARGB1555ToYRow_C NEON - - - LASX - ARGB4444ToARGBRow_C NEON AVX2 - WASMSIMD LASX - ARGB4444ToUVMatrixRow_C NEON AVX2 - WASMSIMD - - ARGB4444ToUVRow_C NEON - - - - - ARGB4444ToYMatrixRow_C NEON AVX2 - WASMSIMD - - ARGB4444ToYRow_C NEON - - - - - ARGBAddRow_C NEON AVX2 - - LASX HVX ARGBAffineRow_C - SSE2 - - - - ARGBAlphaBlendRow_C SME AVX512BW RVV WASMSIMD LASX HVX ARGBAttenuateRow_C NEON AVX512BW RVV - LASX HVX ARGBBlendRow_C NEON AVX512BW RVV WASMSIMD LASX HVX ARGBColorMatrixRow_C I8MM SSSE3 - - LASX - ARGBColorTableRow_C - X86 - - - - ARGBCopyAlphaRow_C - AVX2 - - - HVX ARGBCopyYToAlphaRow_C NEON AVX512BW RVV WASMSIMD - HVX ARGBExtractAlphaRow_C NEON AVX512BW RVV WASMSIMD LASX HVX ARGBGrayRow_C DOTPROD AVX512VNNI - - LASX - ARGBLumaColorTableRow_C - SSSE3 - - - - ARGBMirrorRow_C NEON AVX512BW RVV WASMSIMD LASX HVX ARGBMultiplyRow_C SME AVX512BW RVV - LASX HVX ARGBOrRow_C SME AVX512BW RVV WASMSIMD LASX HVX ARGBPolynomialRow_C - AVX2 - - - - ARGBQuantizeRow_C NEON SSE2 - - LASX - ARGBSepiaRow_C DOTPROD SSSE3 - - LASX - ARGBSetRow_C NEON X86 RVV - LASX HVX ARGBShadeRow_C NEON SSE2 - - LASX - ARGBShuffleRow_C NEON AVX512BW RVV WASMSIMD LASX - ARGBSubtractRow_C NEON AVX2 - - LASX HVX ARGBToAB64Row_C NEON AVX2 RVV - - HVX ARGBToABGRRow_C NEON AVX512BW RVV WASMSIMD LASX HVX ARGBToAR30Row_C NEON AVX512BW - - - - ARGBToAR64Row_C NEON AVX2 RVV - - HVX ARGBToARGB1555Row_C NEON AVX2 - WASMSIMD LASX - ARGBToARGB4444Row_C NEON AVX2 - WASMSIMD LASX - ARGBToBGRARow_C NEON AVX512BW RVV WASMSIMD LASX HVX ARGBToRAWRow_C SVE2 AVX512BW RVV - LASX - ARGBToRGB24Row_C SVE2 AVX512BW RVV - LASX - ARGBToRGB565DitherRow_C SVE2 AVX2 - WASMSIMD LASX - ARGBToRGB565Row_C SVE2 AVX2 - WASMSIMD LASX - ARGBToRGBARow_C NEON AVX512BW RVV WASMSIMD LASX HVX ARGBToUV444MatrixRow_C I8MM AVX512VNNI RVV WASMSIMD - HVX ARGBToUVMatrixRow_C SME AVX512VNNI RVV WASMSIMD - HVX ARGBToYMatrixRow_C SME AVX512VNNI RVV WASMSIMD LASX HVX ARGBUnattenuateRow_C - AVX2 - - - - AYUVToUVRow_C SVE2 AVX512VBMI - - - - AYUVToVURow_C SVE2 AVX512VBMI - - - - AYUVToYRow_C NEON AVX512VBMI - - - - BlendPlaneRow_C NEON AVX512BW RVV WASMSIMD - HVX ByteToFloatRow_C NEON - - - - - ComputeCumulativeSumRow_C NEON AVX2 - - - - Convert16To8Row_C SME AVX512BW RVV - - HVX Convert8To16Row_C SME AVX512BW RVV WASMSIMD - HVX Convert8To8Row_C SME AVX2 - - - HVX CopyRow_C SME AVX512BW RVV - LASX HVX CopyRow_16_C - - - - - HVX CumulativeSumToAverageRow_C NEON AVX2 - - - - DetileRow_C NEON SSE2 - - - - DetileRow_16_C NEON AVX - - - - DetileSplitUVRow_C NEON SSSE3 - - - - DivideRow_16_C SVE2 AVX2 - - - HVX GaussRow_C NEON - - - - - GaussRow_F32_C NEON - - - - - HalfFloatRow_C SVE2 AVX2 - - LASX - HalfMergeUVRow_C SVE2 AVX512BW - - - - HalfRow_16To8_C SME AVX512BW RVV - - HVX HalfWidthRow_16To8_C SME AVX512BW RVV - - - I210AlphaToARGBRow_C SME AVX2 - - - - I210ToAR30Row_C SME AVX2 - - - - I210ToARGBRow_C SME AVX2 - - - - I212ToAR30Row_C SME AVX2 - - - - I212ToARGBRow_C SME AVX2 - - - - I400ToARGBRow_C SME AVX512BW RVV - LASX HVX I410AlphaToARGBRow_C SME AVX2 - - - - I410ToAR30Row_C SME AVX2 - - - - I410ToARGBRow_C SME AVX2 - - - - I422AlphaToARGBRow_C SME AVX512BW RVV - LASX HVX I422ToAR30Row_C SME AVX512BW RVV WASMSIMD - - I422ToARGB1555Row_C SME AVX2 - WASMSIMD LASX - I422ToARGB4444Row_C SME AVX2 - WASMSIMD LASX - I422ToARGBRow_C SME AVX512BW RVV WASMSIMD LASX HVX I422ToRGB24Row_C SME AVX512VBMI RVV WASMSIMD LASX - I422ToRGB565Row_C SME AVX2 - WASMSIMD LASX - I422ToRGBARow_C SME AVX2 RVV - LASX HVX I422ToUYVYRow_C NEON AVX2 - - LASX - I422ToYUY2Row_C NEON AVX2 - - LASX - I444AlphaToARGBRow_C SME AVX2 RVV - - HVX I444ToARGBRow_C SME AVX2 RVV - LASX HVX I444ToRGB24Row_C SME AVX2 RVV - - - InterpolateRow_C SME AVX512BW RVV WASMSIMD LASX HVX InterpolateRow_16_C SME AVX512BW - WASMSIMD - HVX J400ToARGBRow_C NEON AVX512BW RVV WASMSIMD LASX HVX MergeAR64Row_C NEON AVX2 - - - - MergeARGB16To8Row_C NEON AVX2 - - - - MergeARGBRow_C NEON AVX2 RVV - - HVX MergeRGBRow_C NEON AVX512VBMI RVV - - - MergeUVRow_C SME AVX512BW RVV WASMSIMD LASX HVX MergeUVRow_16_C SME AVX2 - - - HVX MergeXR30Row_C NEON AVX512BW - - - - MergeXR64Row_C NEON AVX2 - - - - MergeXRGB16To8Row_C NEON AVX2 - - - - MergeXRGBRow_C NEON AVX2 RVV - - HVX MirrorRow_C NEON AVX512BW - WASMSIMD LASX HVX MirrorRow_16_C - AVX512BW - - - HVX MirrorSplitUVRow_C NEON AVX512BW - WASMSIMD LASX HVX MirrorUVRow_C NEON AVX512BW RVV WASMSIMD LASX HVX MultiplyRow_16_C SME AVX512BW RVV WASMSIMD - HVX NV12ToARGBRow_C SME AVX512BW RVV WASMSIMD LASX HVX NV12ToRGB24Row_C SME AVX512VBMI RVV - - - NV12ToRGB565Row_C NEON AVX2 - WASMSIMD LASX - NV21ToARGBRow_C SME AVX512BW RVV WASMSIMD LASX HVX NV21ToRGB24Row_C SME AVX512VBMI RVV - - - NV21ToYUV24Row_C NEON AVX2 - - - - P210ToAR30Row_C SME AVX2 - - - - P210ToARGBRow_C SME AVX2 - - - - P410ToAR30Row_C SME AVX2 - - - - P410ToARGBRow_C SME AVX2 - - - - RAWToARGBRow_C SVE2 AVX512BW RVV WASMSIMD LASX - RAWToRGB24Row_C SVE2 AVX512VBMI RVV WASMSIMD LASX - RAWToRGBARow_C SVE2 AVX512VBMI RVV - - - RGB24MirrorRow_C NEON AVX512VBMI - WASMSIMD - - RGB24ToARGBRow_C SVE2 AVX512BW RVV WASMSIMD LASX - RGB565ToARGBRow_C NEON AVX2 - WASMSIMD LASX - RGB565ToUVMatrixRow_C NEON AVX2 - WASMSIMD - - RGB565ToUVRow_C NEON - - - LASX - RGB565ToYMatrixRow_C NEON AVX2 - WASMSIMD - - RGB565ToYRow_C NEON - - - LASX - RGBAToARGBRow_C NEON AVX512BW RVV WASMSIMD LASX HVX RGBColorTableRow_C - X86 - - - - RGBToUV444MatrixRow_C NEON AVX512BW RVV WASMSIMD - - RGBToUVMatrixRow_C NEON AVX512BW RVV WASMSIMD - - RGBToYMatrixRow_C DOTPROD AVX512BW RVV WASMSIMD LASX - SetRow_C NEON ERMS - - LASX HVX SobelRow_C NEON SSE2 - - LASX - SobelToPlaneRow_C NEON SSE2 - - LASX - SobelXRow_C NEON SSE2 - - - - SobelXYRow_C NEON SSE2 - - LASX - SobelYRow_C NEON SSE2 - - - - SplitARGBRow_C NEON AVX2 RVV - - HVX SplitRGBRow_C NEON AVX2 RVV - - - SplitUVRow_C SVE2 AVX512BW RVV - LASX HVX SplitUVRow_16_C NEON AVX2 - - - HVX SplitXRGBRow_C NEON AVX2 RVV - - HVX SwapUVRow_C NEON AVX512BW RVV WASMSIMD - HVX UYVYToARGBRow_C SME AVX2 - - LASX - UYVYToUV422Row_C NEON AVX2 - - LASX HVX UYVYToUVRow_C SVE2 AVX2 - - LASX HVX UYVYToYRow_C SVE2 AVX2 - - LASX HVX YUY2ToARGBRow_C SME AVX2 - - LASX - YUY2ToNVUVRow_C NEON AVX2 - - - - YUY2ToUV422Row_C NEON AVX2 - - LASX HVX YUY2ToUVRow_C SVE2 AVX2 - - LASX HVX YUY2ToYRow_C SVE2 AVX2 - - LASX HVX ------------------------------------------------------------------------ SIMD (164/164 100%) 152 93% 154 94% 64 39% 56 34% 73 45% 64 39% ------------------------------------------------------------------------ Bug: None Change-Id: I969a3c5060173aa46b4d1f085837e284fbed1a85 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8665861 Reviewed-by: Mirko Bonadei <mbonadei@chromium.org> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
Frank Barchard
|
205022c08c |
[libyuv] Add AVX-512 VNNI row functions for ARGB to YUV and Gray
Add AVX-512 VNNI (VPDPBUSD / _mm512_dpbusd_epi32) row kernels for 64-bit x86 targets alongside existing AVX-512BW implementations in both row_gcc.cc (inline assembly) and row_win.cc (C++ intrinsics): - ARGBToYMatrixRow_AVX512VNNI and ARGBToYMatrixRow_Any_AVX512VNNI: Pre-computes the 32-bit signed-pixel bias once per row and replaces vpmaddubsw + vpmaddwd + vpaddw with a single vpdpbusd per 16 pixels. - ARGBToUV444MatrixRow_AVX512VNNI and ARGBToUV444MatrixRow_Any_AVX512VNNI: Loads 64 ARGB pixels once instead of twice and computes 32-bit U and V dot products directly via vpdpbusd with signed UV matrix coefficients. - ARGBToUVMatrixRow_AVX512VNNI: Computes 2x2 box-filtered ARGB to U and V conversion using vpdpbusd with k-masked tail handling for arbitrary widths. - ARGBGrayRow_AVX512VNNI: Computes BT.601 grayscale luminance using a single vpdpbusd per 16 pixels, broadcasts Y to RGB via vpshufb, and preserves alpha via vpternlogd with k-masked tail handling. - Wire runtime dispatch via TestCpuFlag(kCpuHasAVX512VNNI) across convert.cc, convert_from_argb.cc, and planar_functions.cc. Add AVX512VNNIVBMI variants of the Y, UV444 and UV kernels. These use both AVX-512 VNNI (vpdpbusd) and AVX-512 VBMI (vpermt2b) and are only selected when the CPU has both features; there are no VBMI versions without VNNI. The AVX512VNNI kernels remain for CPUs with VNNI but no VBMI (Cascade Lake, Cooper Lake). - The 32-bit sums are always in [0, 65535], so Y, U and V are byte 1 of each int32. Two vpermt2b gather 64 results from 4 accumulators, replacing vpackusdw/vpackssdw + vpsrlw + vpackuswb + vpermd. - U and V accumulators start at ~kAddUV so they hold ~(kAddUV - dot). The inverse is folded into the vpternlogd that merges the two 256-bit halves. kAddUV is loaded from ArgbConstants rather than hard coded as 0x8000. - Y folds the source loads into vpxorq. - row_win.cc hides the vpermt2b indices from clang, which otherwise constant folds them into a longer vpshufb + vpor + vpermd sequence. Performance (1280x720, 5000 iterations, AVX512BW -> AVX512VNNI): uArch / Benchmark AVX512BW AVX512VNNI Speedup --------------------------- -------- ---------- ------- Intel Ice Lake ARGBToI420 941 ms 862 ms +9.2% Intel Ice Lake ARGBToI422 1059 ms 947 ms +11.8% Intel Ice Lake ARGBToI444 1140 ms 1038 ms +9.8% Intel Ice Lake ARGBGray 357 ms 273 ms +30.8% Intel Sapphire Rapids GrayTo 317 ms 218 ms +45.4% Intel Emerald Rapids GrayTo 310 ms 213 ms +45.5% Intel Granite Rapids GrayTo 349 ms 241 ms +44.8% AMD Zen 4 ARGBToI420 648 ms 517 ms +25.3% AMD Zen 4 ARGBToI444 695 ms 526 ms +32.1% AMD Zen 4 ARGBGray 403 ms 227 ms +77.5% AMD Zen 5 ARGBToI422 460 ms 403 ms +14.1% AMD Zen 5 ARGBToI444 382 ms 321 ms +19.0% Performance (row_gcc, AVX512VNNI -> AVX512VNNIVBMI): uArch / Benchmark AVX512VNNI VNNIVBMI Speedup ---------------------------- ---------- -------- ------- AMD Zen 6 ARGBToI420 1280x720 327 ms 284 ms +15.1% AMD Zen 6 ARGBToI422 1280x720 372 ms 330 ms +12.7% AMD Zen 6 ARGBToI444 1280x720 357 ms 334 ms +6.9% AMD Zen 6 ARGBToI400 256x24 94 ms 76 ms +23.7% AMD Zen 5 ARGBToI422 1280x720 406 ms 350 ms +16.0% AMD Zen 5 ARGBToI444 1280x720 330 ms 301 ms +9.6% AMD Zen 5 ARGBToI400 256x24 100 ms 90 ms +11.1% AMD Zen 4 ARGBToI420 1280x720 631 ms 540 ms +16.9% AMD Zen 4 ARGBToI422 1280x720 993 ms 825 ms +20.4% AMD Zen 4 ARGBToI400 256x24 245 ms 218 ms +12.4% Intel GNR ARGBToI422 256x24 654 ms 616 ms +6.2% Intel SPR ARGBToI422 256x24 653 ms 613 ms +6.5% Intel GNR, EMR, SPR and Ice Lake are within +/-3% at 1280x720. R=rrwinterton@gmail.com Bug: none Test: out/Release/libyuv_unittest --gtest_filter=*ARGBToI4*:*ARGBGray* Test: LIBYUV_DISABLE_AVX512VBMI=1 out/Release/libyuv_unittest Test: sde64 -gnr / -clx with is_msan=true (row_win.cc) Change-Id: Ia7a37b39dd2948fe4055a8a7694dd1ff8ac3e55c Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8647811 Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Frank Barchard
|
38e21134f7 |
[libyuv] Add ARGBOr and ARGBAlphaBlend with multi-platform SIMD
Add two new planar/row compositing functions for 32-bit ARGB/BGRA
scanlines:
- ARGBOr / ARGBOrRow_*: Applies a bitwise OR with a 32-bit ARGB mask
to every pixel (e.g., 0xFF000000 to force opaque alpha for BGRx to
BGRA conversion). Implemented for C, SSE2, AVX2, AVX-512BW, NEON
(32/64-bit), SVE2, SME, RVV, WASMSIMD, LSX, LASX, HVX, and Windows
intrinsics (row_win.cc).
- ARGBAlphaBlend / ARGBAlphaBlendRow_*: Implements straight-alpha
(unattenuated) Porter-Duff source-over blending matching PDFium's
CompositeRowBgra2Bgra:
dst_a = back_a + src_a - (back_a * src_a) / 255
ratio = (src_a * 255) / dst_a
dst_c = (back_c * (255 - ratio) + src_c * ratio) / 255
Implemented with bit-exact division by 255 across all 65,536
(back_a, src_a) pairs for C, SSSE3, AVX2, AVX-512BW, NEON
(32/64-bit), SVE2, SME, RVV, WASMSIMD, LSX, LASX, HVX, and Windows
intrinsics (row_win.cc).
ARGBOr Performance (2048x64 BGRx->BGRA scanlines, us per call):
uArch Before Highway libyuv vs Hwy
------------------------ ------ ------- ------- ------
Qualcomm Hexagon v81 HVX 259.63 260.97 15.78 16.54x
SiFive X280 (RVV) 1107.57 72.49 50.70 1.43x
SiFive X390 (RVV) 1108.82 70.42 52.26 1.35x
ARGBAlphaBlend Performance (2048x64 BGRA scanlines, us per call):
uArch Before Highway libyuv vs Hwy
------------------------ ------ ------- ------- ------
Intel Lion Cove (AVX2) 429.79 141.97 41.89 3.39x
Intel Cougar Cove (AVX2) 516.83 181.98 51.33 3.55x
Intel Golden Cove (AVX2) 560.31 206.28 67.08 3.08x
AMD Zen 3 (AVX2) 1025.73 251.09 98.99 2.54x
Wasm SIMD128 (V8) 1014.14 964.83 385.80 2.50x
Apple M6 Super (NEON) 221.90 118.42 46.35 2.55x
Apple M5 Super (NEON) 241.90 130.07 49.60 2.62x
ARM C1-Ultra (SME2) 384.63 112.52 63.62 1.77x
Qualcomm Oryon V3-L (SME) 388.13 154.87 87.65 1.77x
Qualcomm Oryon-L (NEON) 318.28 149.36 57.23 2.61x
ARM Cortex-X4 (SVE2) 675.47 213.56 102.53 2.08x
ARM Cortex-A720 (NEON) 795.85 586.55 235.81 2.49x
ARM Cortex-A520 (NEON) 2360.95 2122.99 787.65 2.70x
ARM Cortex-A35 (ARMv7) 4907.31 5895.04 1474.51 4.00x
SiFive X280 (RVV) 4145.06 1096.42 596.82 1.84x
SiFive X390 (RVV) 4144.57 1084.93 597.62 1.82x
Qualcomm Hexagon v81 HVX 4912.03 5786.25 952.19 6.08x
Coral V4 RVV (1024 px) 50276 c 48992 c 4299 c 11.40x
R=wtc@google.com
Bug: none
Change-Id: If661239826a033e89d4b5c1adc0bfdf048a7a726
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8647895
Reviewed-by: richard winterton <rrwinterton@gmail.com>
|
||
|
Frank Barchard
|
47ba8e2160 |
Handle SsimWindowRow remainder in SIMD for AVX512BW, SVE2, SME, RVV
Because each 8x8 SSIM window spans two 4x4 tiles horizontally, num_windows = (width - 5) >> 2, so aligned widths that are multiples of 64 (e.g. 64, 128, 640, 1280, 1920) always have remainder 15 (num_windows & 15 == 15). - Update SsimWindowRow_AVX512BW (compare_gcc.cc and compare_win.cc) to handle any remainder (1..15 windows) via k-masking (load_mask for the (rem + 1) * 4 tile words and win_mask for the rem window scores) and set win_simd_size = 1. - Add SsimWindowRow_SVE2, SsimWindowRow_SME, and SsimWindowRow_RVV with win_simd_size = 1 using predicated/vl-controlled loops so no C code (Ssim4x4Row_C or SsimWindowRow_C) is executed on AVX-512BW, SVE2, SME, or RVV CPUs for any image dimensions. - Remove LIBYUV_TARGET_AVX512BW __attribute__((target(...))) from compare_gcc.cc and GCC_HAS_AVX512 from cpu_support.h, using %%zmm0 directly in inline assembly. SSIM Performance (Before vs After): uArch / Target Before After Speedup -------------------------- --------- --------- ------- Coral RVV Verilator (64x16) 5302 cyc 2219 cyc 2.39x Coral RVV WindowRow (15w) 1854 cyc 308 cyc 6.02x Apple M6 SME (64x36 10kx) 16.41 ms 4.69 ms 3.50x Apple M6 SME (720p 1000x) 350.76 ms 66.09 ms 5.31x R=rrwinterton@gmail.com Bug: libyuv:352521931 Change-Id: Ic7d55184c60611af4a57d113cc4ee3d211debadf Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8621918 Commit-Queue: Frank Barchard <fbarchard@google.com> Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Frank Barchard
|
64ddbbcdde |
[libyuv] Optimize SsimWindowRow with SIMD and use compare_win on x86
- Route 32-bit x86 (__i386__ / _M_IX86) to compare_win.cc intrinsics and restrict compare_gcc.cc to 64-bit x86 (__x86_64__), avoiding x87 float rounding and enabling SSE2, AVX2, and AVX-512 on 32-bit and 64-bit MinGW, GCC, Clang, and MSan (LIBYUV_ENABLE_ROWWIN). - Enable AVX-512BW on GCC >= 6 (GCC_HAS_AVX512) and add LIBYUV_TARGET_AVX512BW attributes to compare_gcc.cc and compare_win.cc. - Refactor SsimWindowRow_C and Ssim8x8_C to use int32_t and float, and add SsimWindowRow_SSE2, SsimWindowRow_AVX2, and SsimWindowRow_AVX512BW in both compare_win.cc and compare_gcc.cc. - Update TestSsimAnyWidth, Ssim, and BenchmarkSsim_Opt in compare_test.cc to use EXPECT_NEAR and deterministic test patterns (identical, different, similar). - Update util/compare.cc and linux.mk to compute I420Ssim. Performance (BenchmarkSsim_Opt, 1280x720): uArch Before After Improvement Intel Lion Cove (x64 AVX2) 120.2 us 68.1 us 1.76x Intel Skymont (x64 SSE2) 249.6 us 146.8 us 1.70x AMD Zen 3 (x64 AVX2) 314.9 us 150.2 us 2.10x Intel Lion Cove (MinGW686) 203.7 us 77.9 us 2.61x Intel Skymont (MinGW686) 338.8 us 167.5 us 2.02x AMD Zen 3 (i686 AVX2) 738.4 us 206.5 us 3.58x Quality (util/compare on 640x360 I420 test images): Image Comparison MSE PSNR SSIM Identical (orig vs orig) 0.00 128.00 dB 1.000000 Similar (light +/-3 noise) 4.57 41.53 dB 0.956556 Degraded (16x16 blocky) 98.18 28.21 dB 0.858893 Very different (inverted) 11160.29 7.65 dB 0.000181 Co-authored-by: Martin Storsjö <martin@martin.st> Bug: b/493910265 Test: out/Release/libyuv_unittest --gtest_filter=*Ssim* Change-Id: I8603dab3f752af32d07ae9039502ca5da5a05425 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8618641 Reviewed-by: richard winterton <rrwinterton@gmail.com> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
Frank Barchard
|
f249eaa72f |
[libyuv] Fix top 20 bugs and performance issues
1. Fix multi-bit TestCpuFlag(A | B) checks in AYUVToNV12, AYUVToNV21, RAWToRGBA, NV12ToRGB24Matrix, NV21ToRGB24Matrix, MergeRGBPlane, and MergeXR30Plane to compare == (A | B) so AVX-512 VBMI/BW kernels are not dispatched when AVX512VBMI or AVX512VL is absent. 2. Fix 2-byte SVE2 heap buffer overflow on odd widths in I422ToARGB1555Row_SVE_SC and I422ToARGB4444Row_SVE_SC using trn1/trn2 + st2b under .b predicates. 3. Fix ARGBPolynomialRow_SSE2 to use packssdw before packuswb, and add missing vzeroupper to SplitRGBRow_AVX2. 4. Fix CalcFrameSsim to return 0.0 instead of 0.0 / 0 (NaN) when width <= 8 || height <= 8 so I420Ssim stays finite on small frames. 5. Fix negative-height vertical inversion in ConvertFromI420, SplitARGBPlane (when dst_a is NULL), I422Rotate, I010Rotate, and I210Rotate. 6. Fix horizontal flip (negative x/dx) in ScaleFilterCols_NEON (signed sbfx #16, #16 offset extraction) and ScaleAddCols1_16_C (shift x >>= 16 before indexing src_ptr). 7. Fix ConvertToARGB odd crop_x chroma alignment in NV12/NV21, src_u offset in I420/YV12/J420/H420/U420, and FOURCC_U422 calling U422ToARGB instead of H422ToARGB. 8. Remove dead 14-instruction block in ScaleRowUp2_Bilinear_16_SSE2. 9. Fix UVScale_16 for src_width == 1 (P010ToP410/P210ToP410), enable and fix column order in TransposeWx4_32_SVE2, fix ByteToFloatRow NEON Any BPP (3 -> 4), fix AB64ToARGBRow_Any_*, ScaleColsUp2_SSE2, ScaleUVRowDown2Box_SME, and UYVYToARGBRow_SME macro guards, fix SVE2 loop tail branches (b.lt/b.ge), and remove duplicate HAS_* macros. 10. Fix ConvertFP16ToFP32Column_NEON loop condition and unsigned b.lo underflow on widths < 8 or not multiples of 8. 11. Fix UYVYToNV12 IS_ALIGNED(awidth, ...) checks and WASM SIMD IS_ALIGNED(width, 16) checks in I420ToRGB24Matrix / I422ToRGB24Matrix. 12. Wire SplitUVRow_SVE2, SplitUVRow_LSX, and SplitUVRow_LASX in NV24ToARGB. 13. Fix MJpegDecoder heap buffer overflows in LoadFrame (reallocate databuf_[i] when scanlines_size changes), DecodeToBuffers, and DecodeToCallback (clamp scanlines_to_copy to lines_left and align skip to max_v_samp_factor). 14. Replace width & (vl - 1) tail calculations with post-loop remainder across 18 SVE2 functions in row_sve.h for non-power-of-2 SVE VL, and remove duplicate label 2: in UYVYToARGBRow_SVE_SC. 15. Handle width & 15 tail pixels in DetileToYUY2_C and validate IS_POWEROFTWO(tile_height) in DetileSplitUVPlane and DetileToYUY2. 16. Avoid signed 32-bit overflow in DivideRow_16_C and C16TO8, fix ClampFloatToZero_SSE2 pointer increment (lea 0x4) and loop counter (sub $0x1), and initialize xmm3 in ARGBCopyYToAlphaRow_SSE2. 17. Check max leaf 7 subleaf and max extended CPUID leaf in GetCpuFlags before querying subleaf 7:1 and extended leaf 0x80000021. 18. Free plane_u on error in I422ToNV21, and fix 4x/2x scratch row over-allocation in ScaleARGBBilinearDown and ScaleUVBilinearDown. 19. Add ARGBSetRow_RVV and ARGBShuffleRow_RVV in row_rvv.cc. 20. Fix NEON mask (15 -> 7) in ScaleUVRowUp2_Linear_Any_NEON / _16_Any_NEON, use SIMD InterpolateRow + SplitUVRow in NV16ToI420, wire SSSE3/SSE2 in ARGBToI400Matrix / ARGBToYUY2Matrix / ARGBToUYVYMatrix and RGB24Mirror, and split HalfFloatRow multiply into (src * scale) * 2^-112 to avoid 50x Intel denormal stalls. Bug: b/574714081 Test: libyuv_unittest Change-Id: Iec3ae5c58d52bee3ef13cf9b765ac84b733aa576 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8601931 Reviewed-by: richard winterton <rrwinterton@gmail.com> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
Frank Barchard
|
378bc91324 |
[libyuv] Multi-platform ScaleUVAddCols3 for ScaleUVBox at 3x
Based on Philipp Hancke's AVX2 column kernel for ScaleUVBox at 3x (https://crrev.com/c/8427347). Add ScaleUVAddCols3 for exact 3x horizontal downscaling (dx == 0x30000 and boxheight <= 85) in ScaleUVBox, plus ScaleAddRow_AVX512BW and ScaleAddRow_WASMSIMD dispatch, and port 3x/box scaling kernels in scale_win.cc to 32/64-bit C++ intrinsics matching scale_gcc.cc: - ScaleUVAddCols3_C (portable C) - ScaleUVAddCols3_AVX2 and ScaleUVAddCols3_Any_AVX2 (scale_gcc.cc and scale_win.cc) - ScaleUVAddCols3_AVX512VBMI (scale_gcc.cc and scale_win.cc) - ScaleAddRow_SSE2, ScaleAddRow_AVX2, ScaleAddRow_AVX512BW, ScaleAddCols1_AVX512BW, ScaleAddCols2_AVX512BW, ScaleRowDown38_SSSE3, ScaleRowDown38_2_Box_SSSE3, and ScaleRowDown38_3_Box_SSSE3 in scale_win.cc for 32/64-bit Visual C and LIBYUV_ENABLE_ROWWIN - ScaleUVAddCols3_NEON (32-bit ARM and 64-bit AArch64) and ScaleUVAddCols3_Any_NEON - ScaleUVAddCols3_SME (ARMv9 Streaming SVE with whilelt predicate tail) - ScaleUVAddCols3_RVV (RISC-V Vector vlseg3e32.v inline asm) - ScaleUVAddCols3_LSX and ScaleUVAddCols3_Any_LSX (LoongArch LSX) - ScaleAddRow_WASMSIMD, ScaleAddRow_Any_WASMSIMD, ScaleUVAddCols3_WASMSIMD, and ScaleUVAddCols3_Any_WASMSIMD - ScaleUVAddCols3_HVX (Hexagon HVX vrdelta + vmpy_VuhRuh) Co-authored-by: Philipp Hancke <philipp.hancke@googlemail.com> Bug: libyuv:564129034 R=rrwinterton@gmail.com Change-Id: Ib9e14f3efcb097842c3d0e9fc9b473b6e2f81871 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8596436 Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Frank Barchard
|
700f153cb8 |
[libyuv] Add Qualcomm Hexagon HVX SIMD support
Implement Qualcomm Hexagon Vector Extensions (HVX) 128-byte SIMD optimizations and runtime CPU detection for Hexagon DSP targets: - CPU detection: Add kCpuHasHexagon and kCpuHasHVX flags, LIBYUV_DISABLE_HVX environment variable support, and Hexagon unit_test mmap/munmap shims. - Rotate/Transpose (rotate_hvx.cc): Implement TransposeWxH_HVX, TransposeUVWxH_HVX, TransposeWxH_16_HVX, and TransposeWxH_32_HVX using power-of-2 vdeal/vshuff butterfly networks. - Compare (compare_hvx.cc): Implement HammingDistance_HVX, SumSquareError_HVX, HashDjb2_HVX, and Ssim4x4Row_HVX. - Scale (scale_hvx.cc): Implement 16 scale row kernels including ScaleAddRow_HVX, ScaleRowDown2*_HVX (8-bit and 16-bit), ScaleRowDown4*_HVX, ScaleARGBRowDown2*_HVX, and ScaleUVRowDown2*_HVX. - Row functions (row_hvx.cc): Implement 58 row kernels covering copy/set/mirror, split/merge, 8/16-bit conversions, YUV-to-ARGB, ARGB-to-YUV, ARGB channel swizzles, blend, attenuate, and interpolate using hardware-predicated partial loads/stores. R=rrwinterton@gmail.com Bug: None Test: libyuv_unittest on Hexagon v81 simulator and Snapdragon 8 Elite Change-Id: Ib6713de51f6246d95e58c95e629ff03f18e3b91e Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8604586 Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Frank Barchard
|
2404f9730e |
[libyuv] Rewrite LoongArch LSX/LASX kernels in compact inline asm
Reduce loop unrolling across LoongArch LSX (128-bit) and LASX (256-bit) compare, row, rotate, and scale kernels to match SSE2/SSSE3 and AVX2 (1-2 vectors per iteration), rewrite intrinsics/macro implementations into GCC/Clang inline assembly, update all _Any wrapper masks and IS_ALIGNED dispatch checks, and port all remaining LSX Convert (+19) and Rotate (+2) kernels to LASX for full parity: ISA | Compare | Convert | Rotate | Scale -----------------|---------|---------|---------|-------- MIPS/Loong (Any) | 1 | 68 | 4 | 20 LSX | 1 | 68 | 4 | 20 LASX | 1 | 68 | 4 | 0 Removes 1,935 macro/helper lines from loongson_intrinsics.h (-99.3%) and reduces total LoongArch compiled .text size by 9,799 bytes (-25.1% overall, -37.4% row_lsx.cc, -52.3% scale_lsx.cc, -31.2% rotate_lsx.cc, -4.9% row_lasx.cc while adding +25 LASX functions): File | Before | After | Change -------------------|----------|----------|-------- scale_lsx.cc.o | 6,055 B | 2,888 B | -52.3% row_lsx.cc.o | 17,252 B | 10,792 B | -37.4% rotate_lsx.cc.o | 1,488 B | 1,024 B | -31.2% row_lasx.cc.o | 13,740 B | 13,072 B | -4.9% rotate_lasx.cc.o | 544 B | 1,504 B | +176.5% TOTAL .text | 39,079 B | 29,280 B | -25.1% R=wtc@google.com, yuanhecai@loongson.cn Test: qemu-loongarch64 -cpu la464 libyuv_unittest (LASX and LIBYUV_DISABLE_LASX=1) Change-Id: I11c1143ad2aad0399f10b2c3fc9bc4c02c740c59 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8564659 Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Frank Barchard
|
ec29aca502 |
[libyuv] Optimize CalcFrameSsim with 2-row 4x4 tile buffer and SIMD
Refactor CalcFrameSsim to decompose overlapping 8x8 SSIM windows
(stepped by 4 pixels horizontally and vertically) into non-overlapping
4x4 pixel tiles across a 2-row rotating buffer (row0, row1), reducing
memory reads and pixel accumulation work by 4x:
- Ssim4x4Row_* computes 4 uint32_t stats {sum_a, sum_b, sum_sq_ab,
sum_axb} per 4x4 pixel tile across a 4-row strip (also sized to fit
future 10-bit/12-bit formats like I010 without uint32_t overflow).
- SsimWindowRow_C combines top and bottom 4x4 tile strips across
adjacent horizontal pairs and returns the exact double SSIM sum,
producing bit-identical results to Ssim8x8_C.
- CalcFrameSsim rotates row0 and row1 (with thread-local buffers and
continuity tracking under _OPENMP) and falls back to Ssim8x8_C only
if buffer allocation fails.
- Implement Ssim4x4Row across all SIMD architectures:
- x86/x64: Ssim4x4Row_SSE2, Ssim4x4Row_AVX2, and Ssim4x4Row_AVX512BW
(with k-masked tail handling for 4..60 pixels) in both
compare_gcc.cc and compare_win.cc.
- ARM: Ssim4x4Row_NEON (compare_neon.cc, compare_neon64.cc),
Ssim4x4Row_NEON_DotProd (compare_neon64.cc), Ssim4x4Row_SVE2
(row_sve.h, row_sve.cc), and Ssim4x4Row_SME (row_sme.cc).
- RISC-V: Ssim4x4Row_RVV (row_rvv.cc).
- LoongArch: Ssim4x4Row_LSX (row_lsx.cc) and Ssim4x4Row_LASX
(row_lasx.cc).
- WebAssembly: Ssim4x4Row_WASMSIMD (compare_wasm.cc).
- Enable BenchmarkSsim_Opt and add BenchmarkSsim_Unaligned and
TestSsimAnyWidth in unit_test/compare_test.cc.
BenchmarkSsim_Opt (1280x720, single core, us/frame):
uArch Before (C) After (us) Improvement
-------------------------------- ---------- ----------- -----------
Intel Lion Cove (AVX2) 356.33 119.09 2.99x
AMD Zen 6 (AVX512BW) 455.86 124.67 3.66x
AMD Zen 5 (AVX512BW) 518.91 128.29 4.04x
Intel Emerald Rapids (AVX512BW) 601.58 180.41 3.33x
Intel Cougar Cove (AVX2) 608.11 183.34 3.32x
Intel Granite Rapids (AVX512BW) 596.57 187.04 3.19x
Intel Golden Cove (AVX2) 534.58 187.93 2.84x
Intel Sapphire Rapids (AVX512BW) 634.36 189.92 3.34x
AMD Zen 4 (AVX512BW) 713.06 223.65 3.19x
Intel Skymont (SSE2) 445.99 250.52 1.78x
Intel Ice Lake (AVX512BW) 845.16 267.23 3.16x
AMD Zen 3 (AVX2) 741.02 267.38 2.77x
AMD Zen 2 (AVX2) 907.04 301.02 3.01x
Intel Skylake-X (AVX512BW) 1097.86 301.63 3.64x
Intel Darkmont (SSE2) 772.18 383.39 2.01x
Intel Gracemont (SSE2) 961.80 421.83 2.28x
Intel Haswell (AVX2) 1159.74 479.14 2.42x
Intel Ivy Bridge (SSE2) 1296.99 597.32 2.17x
Apple M6 Super (SME) 404.66 90.09 4.49x
Apple M6 Perf (DotProd) 444.72 104.59 4.25x
Qualcomm Oryon-L (DotProd) 490.76 108.79 4.51x
Qualcomm Oryon V3-L (SVE2) 663.13 150.75 4.40x
ARM Cortex-X4 (DotProd) 620.21 177.29 3.50x
ARM Neoverse-V2 (SVE2) 682.93 205.18 3.33x
ARM Cortex-X3 (DotProd) 737.38 226.24 3.26x
ARM C1-Ultra (DotProd) 596.71 229.10 2.60x
ARM Cortex-X2 (DotProd) 820.94 247.94 3.31x
Qualcomm Oryon-M (DotProd) 1146.09 257.89 4.44x
Qualcomm Oryon V3-M (SME) 1068.78 276.23 3.87x
ARM Cortex-A720 (DotProd) 993.59 292.62 3.40x
ARM Cortex-X1 (DotProd) 912.19 326.44 2.79x
ARM Cortex-A715 (DotProd) 1135.67 340.12 3.34x
ARM Cortex-A710 (DotProd) 1222.04 342.51 3.57x
ARM Neoverse-N1 (DotProd) 1406.72 376.46 3.74x
ARM Cortex-A725 (SVE2) 1104.13 376.86 2.93x
ARM C1-Pro (SVE2) 1073.69 413.97 2.59x
ARM Cortex-A76 (DotProd) 1491.02 423.58 3.52x
ARM Cortex-A78 (DotProd) 1503.73 485.00 3.10x
AmpereOne (DotProd) 1518.61 518.40 2.93x
Apple M6 Eff (DotProd) 4051.10 572.93 7.07x
ARM Cortex-A75 (NEON) 2055.04 812.09 2.53x
ARM Cortex-A520 (DotProd) 3307.98 963.86 3.43x
ARM Cortex-A73 (NEON) 3120.09 1180.17 2.64x
Qualcomm Kryo Perf (NEON) 4921.16 1459.77 3.37x
ARM Cortex-A510 (DotProd) 4645.10 1641.06 2.83x
ARM Cortex-A55 (DotProd) 4571.02 1672.09 2.73x
Qualcomm Kryo Eff (NEON) 6000.96 1796.17 3.34x
ARM Cortex-A55r0 (NEON) 10486.61 4358.61 2.41x
ARM Cortex-A53 (NEON) 10121.34 5700.99 1.78x
R=rrwinterton@gmail.com
Bug: None
Test: out/Release/libyuv_unittest --gtest_filter=*Ssim*
Change-Id: Ibb95ae74424ecdda073ab7ab10ab1cf610b121b6
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8593040
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
|
||
|
Frank Barchard
|
0b9e5c922d |
Avoid signed int overflow in scale functions using uint32_t
Use uint32_t for the 16.16 fixed-point row coordinate (uy) and max_y in the vertical loops of ScalePlane*, ScaleUV*, and the Up2 linear helpers. With src_height up to 32768, (src_height << 16) and y += dy overflow signed int (e.g. for dst_height == 2 the post-read step reaches 5/4 * (src_height << 16), which exceeds INT_MAX for any src_height >= 26215, and for kFilterNone the final increment after the last row reaches (src_height << 16) + dy / 2). uint32_t cannot overflow for these ranges (uy <= 0xFFFF0000) and stays in a single 32-bit register on 32-bit targets (x86, Arm, WASM), unlike int64_t. Undo the interim +/-32767 height cap from https://crrev.com/c/8524280 so ScalePlane, ScalePlane_16, UVScale, and UVScale_16 accept |src_height| <= 32768 again. ScaleUVBilinearUp2 and ScaleUVBilinearUp2_16 take ptrdiff_t strides so 2 * dst_stride does not overflow int for images over 2 GB. ARGBScale, ARGBScaleClip, and UVScale_16 reject |src_width| > 32768 and |src_height| > 32768, and ScalePlane, ScalePlane_16, and ScaleSlope avoid FixedDiv(32768, 1) overflow when dst_width == 1 or dst_height == 1. Bug: chromium:531172025, chromium:546841026, libyuv:573815252 Test: libyuv_unittest --gtest_filter=*ScalePlaneOversizeHeight*:*ScalePlaneBoxMaxSrcHeight*:*ScaleLargeImageOver2GB*:*ARGBScale* Change-Id: Ica2e05d7dfe94f81c6631c587f1ff46c2ad44735 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8528398 Commit-Queue: Frank Barchard <fbarchard@google.com> Reviewed-by: Wan-Teh Chang <wtc@google.com> |
||
|
Frank Barchard
|
58d817ed81 |
Fix remaining inline asm input/output constraints and clobbers
Mark modified inline asm operands as read-write (+r / +c) and add missing volatile/memory clobbers so compilers and LTO do not assume modified registers remain unchanged across asm blocks: - HashDjb2_NEON (compare_neon64.cc): [src] "+r"(src) for post-increment ldr - FixedDiv1_X86 (scale_gcc.cc): "+c"(div) for sub $0x1, %1 - P410ToARGBRow_SVE_SC & P410ToAR30Row_SVE_SC (row_sve.h): [vl] "+r"(vl) for tail cntw %x[vl] - SwapUVRow_RVV & ARGBMultiplyRow_RVV (row_rvv.cc): use asm volatile and "memory" clobber instead of VLA memory operands on mutated pointers R=rrwinterton@gmail.com Bug: b/571179207 Test: libyuv_unittest Change-Id: I3c2475d1e5bdab13e74643a028eebc350edcc1bb Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8565646 Reviewed-by: richard winterton <rrwinterton@gmail.com> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
Frank Barchard
|
b9d572cc63 |
Add 25 native k-masked AVX-512 row kernels
R=rrwinterton@gmail.com Bug: None Change-Id: I2ee9b2c2d939f63f49d2f951f2d552ca6863d78b Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8564927 Commit-Queue: Frank Barchard <fbarchard@google.com> Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Frank Barchard
|
63b56324de |
[libyuv] Replace AVX-512 _Any wrappers with k-mask tail handling
Replace stack-buffer memcpy _Any wrappers with native AVX-512 k-mask
tail handling across 8 AVX-512 row kernels (unlocking composite
RGBToUVMatrixRow_AVX512BW as well) and remove IS_ALIGNED dispatch
constraints:
- MirrorSplitUVRow_AVX512BW: Masked 16-bit load (vmovdqu16 %{k1%}%{z%}),
dynamic reverse index (vpsubw + vpermw via kMirrorUVPermute_AVX512),
and vpmovwb %{k1%} U/V stores; unlocks unaligned SplitRotateUV180.
- ARGBToUVMatrixRow_AVX512BW: Fix MASK=63 vs 32-pixel loop bug;
zero-load tail dwords per row, duplicate odd final pixel via knotw +
vpshufd, and store masked U/V bytes.
- I422ToARGBRow_AVX512BW & I422ToAR30Row_AVX512BW: Add masked Y/U/V
loads (READYUV422_MASK_AVX512BW) and masked ARGB/AR30 stores.
- CopyRow_AVX512BW: Add 64-byte full vector step + bzhi masked tail.
- BlendPlaneRow_AVX512BW, ARGBShuffleRow_AVX512BW, RGBToARGBRow_AVX512BW
(RAWToARGBRow_AVX512BW / RGB24ToARGBRow_AVX512BW): Masked loads and
stores in both row_gcc.cc inline assembly and row_win.cc intrinsics.
Reduces x86-64 binary .text size by 2,592 bytes with 0 bytes added to
.rodata.
Performance on Intel Sapphire Rapids (1275x1650, 1000 iterations):
Function (Test) | Before | After | Speedup
-------------------------|---------|---------|--------
NV12Rotate180_Opt | 429 ms | 260 ms | 1.65x
RGB24ToI420_Opt | 615 ms | 531 ms | 1.16x
I420ToAR30_Opt | 425 ms | 383 ms | 1.11x
ARGBToI420_Opt | 427 ms | 411 ms | 1.04x
I420ToARGB_Opt | 423 ms | 411 ms | 1.03x
RGB24ToARGB_Opt | 493 ms | 490 ms | 1.01x
ARGBToBGRA_Opt | 544 ms | 542 ms | 1.00x
CopyPlane_Opt | 246 ms | 245 ms | 1.00x
BlendPlane_Opt | 259 ms | 258 ms | 1.00x
RAWToARGB_Opt | 486 ms | 486 ms | 1.00x
R=rrwinterton@gmail.com
Test: LIBYUV_WIDTH=1275 LIBYUV_HEIGHT=1650 LIBYUV_REPEAT=1000 out/Release/libyuv_unittest
Change-Id: Ib5e5ae52076352649108b20f8df2b12d81860fb1
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8563977
Reviewed-by: richard winterton <rrwinterton@gmail.com>
|
||
|
Wan-Teh Chang
|
1e2e3a3599 |
Avoid integer overflow if src_height == +/-32768
The ScalePlaneBox(), ScalePlaneBox_16(), and ScaleUVBox() functions all contain the following expression: const int max_y = (src_height << 16); where src_height is a positive int. This expression overflows if src_height >= 32768. Tracing the callers, we make sure ScalePlane(), ScalePlane_16(), and UVScale() reject src_height < -32767 || src_height > 32767 as invalid input. (These functions negate a negative src_height before calling a function that executes expression src_height << 16.) Although UVScale_16() do not execute the expression src_height << 16, make the same change to UVScale_16() for consistency with UVScale(). Bug: chromium:531172025 Change-Id: Id5b10f11f46f593945cf6bf1435b6027a6629815 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8524280 Reviewed-by: Frank Barchard <fbarchard@google.com> Commit-Queue: Wan-Teh Chang <wtc@google.com> |
||
|
Frank Barchard
|
16af162cf9 |
[libyuv] Optimize RotatePlane_16 SIMD and compact 32-bit transposes
Implement SIMD 16-bit plane rotation (TransposePlane_16, RotatePlane90_16, RotatePlane180_16, RotatePlane270_16) used by I010Rotate, I210Rotate, and I410Rotate, keeping tile sizes compact with full-register writes and zero cross-lane shuffles: - AVX2: 8x16 tile in ymm0..ymm15 with _Any tail (low-end x86 SIMD) - AVX-512BW: 8x32 tile in zmm0..zmm15 with k1/k2 right/bottom masks - NEON (ARM64/ARM32): 8x16 tile in v0..v23 / q0..q11 with _Any tail - SME: 8xVL tile using ZA16 horizontal/vertical array slices - RVV: 8xVL tile using widening unpacks with hardware vl tail - WASM SIMD: 8x8 tile in v128 with _Any tail Also update 32-bit ARGBRotate x86 transposes to compact vertical in-lane tiles with full-register writes and zero cross-lane shuffles: - TransposeWxH_32_AVX2: 4x8 tile in ymm0..ymm7 with vpmaskmovd tails - TransposeWxH_32_AVX512BW: 4x16 tile in zmm0..zmm7 with k1/k2 tails Add RotatePlane16_SplitMergeUV_Opt and RotatePlane16_SplitMergeUV_Odd unit tests comparing 16-bit rotation against SplitUVPlane + RotatePlane + MergeUVPlane across 0/90/180/270 degrees. Performance on 1275x1650 (1000 iterations, before -> after): CPU 90 degree 180 degree Coral RVV (Verilator) 7306 -> 1415c (5.16x) 2274 -> 719c (3.16x) AMD Rome (Zen 2) 821 -> 215ms (3.82x) 259 -> 181ms (1.44x) AMD Genoa (Zen 4) 483 -> 129ms (3.73x) 149 -> 111ms (1.34x) AMD Turin (Zen 5) 321 -> 101ms (3.18x) 109 -> 78ms (1.40x) AMD Milan (Zen 3) 505 -> 189ms (2.67x) 199 -> 118ms (1.69x) Intel Skylake-X 1076 -> 447ms (2.41x) 353 -> 317ms (1.11x) WASM SIMD (d8) 415 -> 203ms (2.04x) 370 -> 106ms (3.49x) Intel Emerald Rapids 735 -> 422ms (1.74x) 238 -> 233ms (1.02x) Intel Haswell 1120 -> 648ms (1.73x) 438 -> 393ms (1.11x) Intel Ice Lake 774 -> 474ms (1.63x) 279 -> 290ms (0.96x) ARM Neoverse-N1 864 -> 544ms (1.59x) 404 -> 266ms (1.52x) Ampere Siryn 862 -> 576ms (1.50x) 568 -> 329ms (1.72x) Google Axion (Maple) 554 -> 379ms (1.46x) 199 -> 142ms (1.41x) Intel Sapphire Rapids 1126 -> 772ms (1.46x) 301 -> 288ms (1.04x) Intel Granite Rapids 730 -> 565ms (1.29x) 280 -> 280ms (1.00x) NVIDIA Grace 462 -> 365ms (1.26x) 192 -> 126ms (1.52x) ARM Cortex-A53 28290 -> 24386ms (1.16x) 7178 -> 5075ms (1.41x) Google Axion 2 (Tamar) 523 -> 453ms (1.15x) 217 -> 112ms (1.93x) Test: libyuv_unittest --gtest_filter=*Rotate* Bug: b/42280942 Change-Id: I91ba6cac22185b5d294fcb98344fdb8a23cc94ff Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8545294 Reviewed-by: richard winterton <rrwinterton@gmail.com> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
Frank Barchard
|
eeb06a6451 |
[libyuv] Optimize ARGBRotate for Intel SSE2, AVX2, and AVX-512
Optimize 32-bit ARGBRotate (0, 90, 180, 270) across Intel x86/x64
architectures (SSE2, AVX2, AVX512BW), with matching kernels for ARM
(NEON, SVE2, SME), WASM, LoongArch (LSX, LASX), and RISC-V (RVV):
- Replace column-by-column ScaleARGBRowDownEven gather loops in
ARGBTranspose (90/270) with full-plane horizontal tile transposes:
- AVX512BW (TransposeWxH_32_AVX512BW / TransposeNx4_32_AVX512BW):
Transpose 16x16 blocks across zmm0..zmm31 using 4-stage
vpunpckldq/hdq, vpunpcklqdq/hqdq, and vshufi32x4 butterflies with
k-masked right and bottom tails, eliminating Any wrappers and scalar
C edge fallbacks.
- AVX2 (TransposeWxH_32_AVX2 / TransposeNx4_32_AVX2):
Transpose 8x8 blocks across ymm0..ymm15 using vpunpck* and
vperm2i128 with vpmaskmovd masked tails.
- SSE2 (Transpose4x4_32_SSE2 / Transpose4x4_32_Any_SSE2):
Transpose 4x4 blocks in xmm registers.
- Implement ARGBRotate180 directly via ARGBMirror with negative height,
avoiding intermediate row buffer allocations and extra copies.
- Implement ARGBRotate0 via ARGBCopy.
Speedup vs Before (1275x1650, US Letter 150 DPI, 1000 iterations):
CPU | 0 deg | 90 deg | 180 deg | 270 deg
---------------------|---------|---------|---------|--------
Intel Ice Lake | 1.02x | 1.34x | 1.18x | 1.33x
Intel Skylake-X | 1.02x | 1.20x | 1.07x | 1.13x
Intel Emerald Rapids | 1.01x | 1.07x | 1.00x | 1.07x
Intel AVX-512 vs C | 1.00x | 2.67x | 2.41x | 2.63x
R=rrwinterton@gmail.com
Bug: b/42280915
Test: LIBYUV_WIDTH=1275 LIBYUV_HEIGHT=1650 LIBYUV_REPEAT=1000 ./out/Release/libyuv_unittest --gtest_filter=*ARGBRotate*
Change-Id: Ib308a225c17e46eab083a9da4033598fcf0c303c
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8538986
Reviewed-by: richard winterton <rrwinterton@gmail.com>
|
||
|
Frank Barchard
|
4ce4edd50e |
Fix missing inline asm register clobbers
Change-Id: Ic573c8d7d8921a1dd00e4481b854a8168d0c94f7 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8543110 Reviewed-by: richard winterton <rrwinterton@gmail.com> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
Wan-Teh Chang
|
9456b06227 |
Remove the Abs() function in source/scale_uv.cc
The Abs() function is used at two places in source/scale_uv.cc as follows: src_width = Abs(src_width); However, before we reach those two places, we have rejected src_width <= 0 as invalid input. Therefore the two Abs(src_width) calls have no effects. Bug: None Change-Id: I04c48ad5d1270b9b93abb18f2b41b90ec818d5ea Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8531148 Reviewed-by: Frank Barchard <fbarchard@google.com> Commit-Queue: Wan-Teh Chang <wtc@google.com> |
||
|
Frank Barchard
|
1ddb18fb6b |
[libyuv] AVX512, NEON, WASM and RVV alpha copy and mirror rows
1275x1650 (US Letter 150 DPI, 1000 iterations) speedup: CPU ARGBExtractAlpha ARGBCopyYToAlpha ARGBMirror RGB24Mirror ----------------------------------------------------------------------- Intel Granite Rapids 1.00x 1.00x 1.00x 1.04x Intel Emerald Rapids 1.01x 0.97x 1.03x 1.07x Intel Sapphire Rapids 0.97x 1.01x 1.08x 1.26x Intel Ice Lake 0.99x 1.02x 0.97x 1.08x Intel Skylake-X 1.04x 1.03x 1.01x NA Intel Haswell NA NA NA NA AMD Turin (Zen 5) 1.01x 1.10x 1.25x 1.31x AMD Genoa (Zen 4) 0.97x 1.04x 1.09x 1.42x AMD Milan (Zen 3) NA NA NA NA AMD Rome (Zen 2) NA NA NA NA NVIDIA Grace (V2) 1.57x 2.51x NA NA Google Axion 2 1.12x 3.17x NA NA Ampere Siryn 1.06x 1.70x NA NA ARM Neoverse-N1 1.49x 2.01x NA NA Google Axion (Maple) 1.47x 2.03x NA NA ARM Cortex-A53 1.06x 1.69x NA NA Add AVX512BW ARGBExtractAlphaRow, ARGBCopyYToAlphaRow and ARGBMirrorRow, and AVX512VBMI RGB24MirrorRow, in row_gcc.cc and as intrinsics in row_win.cc. They are x64 only. The remainder is done with a k-mask (bzhi) zero-masked load and a masked store, so they handle any width without an Any wrapper and never access memory outside the row. RGB24MirrorRow_AVX512VBMI loads and stores 3 full zmm vectors per 64 pixels using 3 vpermi2b. The one output byte that comes from the third input vector is merged with a masked vpbroadcastb. The 1 to 63 pixel tail rebases the same tables by 192 - 3 * r; indices >= 128 are taken from the third vector with vpmovb2m and a masked vpermb. Add ARGBCopyYToAlphaRow_NEON for Arm 32 bit (vld4 + vst4) and AArch64 (zip + sli), and rewrite AArch64 ARGBExtractAlphaRow_NEON with ldp + uzp2. Both prefetch source and destination, which is 2.5x faster on Neoverse-N1 and 1.2x to 1.5x faster on Cortex-A53 for 1280x720. Add WASM SIMD ARGBExtractAlphaRow and ARGBCopyYToAlphaRow, using a swizzle table for CopyYToAlpha. Rework the RVV versions to use unit stride loads with vnsrl and vzext.vf4. R=rrwinterton@gmail.com Bug: None Change-Id: I436f27d70a3db0a2b176c91eb789a9491133cfcb Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8523531 Reviewed-by: richard winterton <rrwinterton@gmail.com> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
Frank Barchard
|
98697ab033 |
[libyuv] Add WASM SIMD128 compare, rotate, scale and row functions
d8 geomean speedup vs C compiled with -O2 -msimd128: File AMD Zen 3 ARM Neoverse-V2 row_wasm.cc 5.67x 3.24x scale_wasm.cc 5.47x 3.17x compare_wasm.cc 4.58x 5.77x rotate_wasm.cc 6.83x 6.03x Bug: None Change-Id: Ia06c805dc4029f94842c40e7301b2933beb8c1e5 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8524106 Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Frank Barchard
|
5323bab29e |
[libyuv] Clobber "vl" in RVV inline assembly only for GCC
Define RVV_VL_CLOBBER to expand to "vl", on GCC and empty on Clang in source/row_rvv.cc and source/scale_rvv.cc, and replace "vl", "vtype", in the RVV inline assembly clobber lists with RVV_VL_CLOBBER. R=masterwilliams@google.com, rrwinterton@gmail.com Test: ninja -C out/riscv64 libyuv_unittest Bug: b/443984359 Change-Id: Ie65bde737bd0714c3e409a2b8ab5fcd98b6a7a32 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8516707 Reviewed-by: Wan-Teh Chang <wtc@google.com> |
||
|
Frank Barchard
|
eb8eda9c79 |
[libyuv] Add WASM SIMD128 and Relaxed SIMD for RAWToRGB24Row
Implement RAWToRGB24Row_WASMSIMD and RAWToRGB24Row_Any_WASMSIMD in source/row_wasm.cc for RGB24ToRAW and RAWToRGB24 conversions: - Process 16 pixels (48 bytes) per loop iteration using three 128-bit loads, three byte-wise shuffles, and four 2-way blends. - Use wasm_i8x16_relaxed_swizzle when __wasm_relaxed_simd__ is enabled (compiling to a single vpshufb on x86 without bounds-checking overhead) and fall back to wasm_i8x16_swizzle for baseline SIMD128. - Add kCpuHasWASM (0x100) and kCpuHasWASMSIMD (0x101) CPU feature flags and enable -msimd128 in BUILD.bazel, BUILD.gn, and CMakeLists.txt for WebAssembly targets. Performance (RAWToRGB24_Opt, 1280x720, V8 d8): CPU WASM C SIMD128 Relaxed SIMD Spd Rlx Spd ------------------------------------------------------------------- AMD Zen 4 595.0 us 137.0 us 84.0 us 4.34x 7.08x AMD Zen 3 708.0 us 148.0 us 107.0 us 4.78x 6.62x AMD Zen 5 375.0 us 107.0 us 67.0 us 3.50x 5.60x ARM Neoverse-V2 638.0 us 118.0 us 116.0 us 5.41x 5.50x ARM Neoverse-N1 1046.0 us 294.0 us 294.0 us 3.56x 3.56x Intel Ice Lake 707.0 us 209.0 us 220.0 us 3.38x 3.21x Intel Haswell 1110.0 us 344.0 us 360.0 us 3.23x 3.08x Intel Emerald Rapids 519.0 us 207.0 us 208.0 us 2.51x 2.50x Intel Sapphire Rapids 551.0 us 220.0 us 255.0 us 2.50x 2.16x ARM Cortex-A53 14440.0 us 8660.0us 6920.0us 1.67x 2.09x Intel Granite Rapids 453.0 us 225.0 us 221.0 us 2.01x 2.05x Bug: 42271072 Test: libyuv_test --gunit_filter=*RAWToRGB24*:*RGB24ToRAW*:*Cpu* Change-Id: Id902668263a2ab70aa300eeeea18851490c917b5 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8515581 Reviewed-by: richard winterton <rrwinterton@gmail.com> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
Frank Barchard
|
2b7ecc0aad |
RAWToRGB24 for NEON, x86, and LoongArch
R=rrwinterton@gmail.com Test: libyuv_unittest --gtest_filter=*RAWToRGB24*:*RGB24ToRAW* Bug: None Change-Id: I4454f3785ba5c39dc776ba1f5151cd9d3e68055b Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8509286 Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Frank Barchard
|
42eb3a552f |
[libyuv] Add ScaleRowDown2Box_AVX512BW with tail masking
Add AVX-512BW implementations of ScaleRowDown2_AVX512BW, ScaleRowDown2Linear_AVX512BW, ScaleRowDown2Box_AVX512BW, and ScaleRowDown2Box_Odd_AVX512BW in scale_gcc.cc. Use 512-bit zmm vectors (64 source bytes -> 32 destination pixels per iteration) with vpmovwb narrowing stores, and handle remaining 1..31 destination pixels in-place using hardware mask register k1 (vmovdqu16 zero-masked loads for 2-byte source pixel pairs and vpmovwb masked narrowing store), eliminating the ScaleRowDown2*_Any_AVX2 fallback on AVX-512BW CPUs. Hook up the new row functions in ScalePlaneDown2 (source/scale.cc) and I420Blend (source/planar_functions.cc). I420ScaleDownBy2_Box (LIBYUV_WIDTH=638 LIBYUV_HEIGHT=360 LIBYUV_REPEAT=2000): CPU SSSE3 AVX2 AVX512 512/AVX2 AMD Turin (Zen 5) 57 62 36 1.72x Intel Granite Rapids 76 76 45 1.69x AMD Genoa (Zen 4) 85 84 50 1.68x Intel Sapphire Rapids 68 73 44 1.66x Intel Emerald Rapids 69 66 48 1.38x Intel Ice Lake 128 128 96 1.33x Intel Skylake-X 153 150 121 1.24x AMD Milan (Zen 3) 95 95 96 - Intel Haswell 193 172 162 - R=rrwinterton@gmail.com Bug: libyuv:960 Test: libyuv_unittest --gtest_filter=*ScalePlaneDown2*:*I420Blend* Change-Id: Ifbfcc4042c50d28022b12cb807dad56e66e7f301 TAG=agy CONV=5f01827f-2aa8-4718-b8c1-7f282730918e Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8504320 Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Philipp Hancke
|
aa6cedb39c |
Add a box filter to ScaleUV
kFilterBox only had 1/2 and 1/4 fast paths. Everything else fell through to ScaleUVSimple and point sampled, so NV12 chroma was unfiltered where I420 was not. 13.6 dB against 50.2 dB at 3x. ScaleUVBox mirrors ScalePlaneBox. Row accumulation reuses ScaleAddRow and its SIMD variants, since summing is position wise and an interleaved row is just src_cols * 2 samples. Only the column pass, which sums U and V at a stride of 2, is new. Output is now byte identical to the I420 path. Tests compare against ScalePlane rather than C against SIMD, which is what missed this. Bug: libyuv:564129034 Change-Id: I26221f5435dbfc845ad2fc13aa347ef56ccf7bc6 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8426668 Reviewed-by: richard winterton <rrwinterton@gmail.com> Reviewed-by: Frank Barchard <fbarchard@google.com> |
||
|
Frank Barchard
|
67b3c0dad8 |
Compare AVX512BW and AVX2, intrinsics for Visual C and ROWWIN
- Add HammingDistance_AVX512BW, SumSquareError_AVX512BW and
HashDjb2_AVX512BW for 32 and 64 bit, as inline assembly in
compare_gcc.cc and intrinsics in compare_win.cc.
- AVX512BW processes 64 bytes per loop with full vectors, and the
remainder of 1 to 63 bytes with one masked load, so compare.cc passes
any count and does not use C for the remainder. HashDjb2 loads the
remainder into the high bytes, ending at src + count, so the same
33 ^ (63 - i) multipliers apply, and multiplies the hash by 33 ^ n.
- Add SumSquareError_AVX2 and HashDjb2_AVX2 to compare_gcc.cc and
HammingDistance_AVX2 to compare_win.cc, so both files have the same
functions. Enable AVX2 in compare_gcc.cc for gcc as well as clang.
- HashDjb2 AVX2 and AVX512BW split the 32 bit multipliers 33 ^ i into
16 bit halves and use vpmaddwd, instead of vpmulld which is 2 uops
with 10 cycle latency. Bytes are zero extended with vpmovzxbw from
memory. The hash = hash * 33 ^ 64 + sum dependency uses scalar imul
instead of vpmulld. AVX2 does 64 bytes per horizontal add.
- Port compare_win.cc from 32 bit Visual C inline assembly to
intrinsics and enable it for Visual C 32/64 bit and
LIBYUV_ENABLE_ROWWIN, which MemorySanitizer also enables. Visual C 64
bit and MemorySanitizer previously used C.
- Fix HammingDistance_SSSE3 and HammingDistance_AVX2 to allow unaligned
pointers. They used movdqa and crashed on unaligned src_a.
- Add TestCompareAnyCount to test every count from 0 to 321, aligned
and unaligned, ending at the end of a page.
Functions available:
Function compare_gcc compare_win
32/64 bit 32/64 bit
HammingDistance_SSE42 asm intrinsics
HammingDistance_SSSE3 asm -
HammingDistance_AVX2 asm intrinsics
HammingDistance_AVX512BW asm intrinsics
SumSquareError_SSE2 asm intrinsics
SumSquareError_AVX2 asm intrinsics
SumSquareError_AVX512BW asm intrinsics
HashDjb2_SSE41 asm intrinsics
HashDjb2_AVX2 asm intrinsics
HashDjb2_AVX512BW asm intrinsics
Performance, 1280x720, LIBYUV_REPEAT=1000, best of 3, ms.
Before is main (SSE, and AVX2 Hamming only, in compare_gcc).
BenchmarkDjb2_Opt main gcc gcc win win
AVX2 AVX512 AVX2 AVX512
Intel Haswell 179 77 - 75 -
AMD Milan (Zen 3) 101 35 - 35 -
Intel Skylake-X 214 63 55 67 59
Intel Ice Lake 191 51 47 52 49
Intel Sapphire Rapids 175 39 39 38 39
Intel Granite Rapids 170 40 38 40 38
AMD Genoa (Zen 4) 91 34 37 34 38
AMD Turin (Zen 5) 80 29 22 28 23
C 502 to 669
BenchmarkSumSquareError main gcc gcc win win
_Opt AVX2 AVX512 AVX2 AVX512
Intel Haswell 79 37 - 38 -
AMD Milan (Zen 3) 47 24 - 25 -
Intel Skylake-X 78 32 26 34 24
Intel Ice Lake 86 43 23 43 22
Intel Sapphire Rapids 48 24 21 24 22
Intel Granite Rapids 47 23 21 23 21
AMD Genoa (Zen 4) 50 25 21 25 21
AMD Turin (Zen 5) 57 28 13 28 14
C 154 to 425
Testing:
LIBYUV_WIDTH=1279 LIBYUV_HEIGHT=719 out/Release/libyuv_unittest \
--gtest_filter='*Compare*:*Djb2*:*Hamming*:*SumSquare*:*Psnr*:*Ssim*'
With out/Release, out/Release_x86, out/rowwin, out/rowwin_x86 and
out/msan, with LIBYUV_DISABLE_AVX512BW and LIBYUV_DISABLE_AVX2, and
LIBYUV_WIDTH of 1, 31 to 33, 63 to 65, 95, 127, 129 and 200. Under
sde64 -skx for AVX512BW with out/Release and out/rowwin.
Built out/android_x86.
Bug: None
Change-Id: Ief197d546daca3db02ee1503d4649f4c755eb189
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8504717
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
|
||
|
Frank Barchard
|
bad060c3a5 |
Rotate intrinsics for Visual C, ROWWIN and 32 bit x86
- Port rotate_gcc.cc to intrinsics in rotate_win.cc: TransposeWx8_SSSE3,
TransposeUVWx8_SSE2, TransposeWx16_AVX2, TransposeUVWx16_AVX2,
Transpose4x4_32_SSE2, Transpose4x4_32_AVX2, and for 64 bit,
TransposeWx16_AVX512BW and TransposeUVWx16_AVX512BW.
- Remove the 32 bit Visual C inline assembly.
- Enable rotate_win.cc for Visual C 32/64 bit, LIBYUV_ENABLE_ROWWIN,
which MemorySanitizer also enables, and all 32 bit x86 compilers.
Visual C 64 bit previously used C.
- rotate_gcc.cc is now 64 bit only. 32 bit gcc and clang only had an
8x8 TransposeWx8_SSSE3 and Transpose4x4_32, because the UV and AVX2
inline assembly uses xmm8..xmm15, so 32 bit NV12 rotate used C.
Remove the 32 bit TransposeWx8_SSSE3 inline assembly.
- Add a target("sse2") attribute to the SSE2 intrinsics so gcc and
clang build them when the baseline lacks SSE2, such as
-march=pentium3.
- Disable the generic x86 scale_row.h and compare_row.h HAS_ macros for
32 bit LIBYUV_ENABLE_ROWWIN, as is done for 64 bit. scale_gcc.cc and
compare_gcc.cc are disabled for ROWWIN and scale_win.cc and
compare_win.cc require _MSC_VER, so out/rowwin_x86 failed to link with
undefined symbols such as ScaleARGBCols_SSE2 and HashDjb2_SSE41.
- TransposeWx8_SSSE3 and TransposeUVWx8_SSE2 share a 16x8 helper.
- TransposeWx8_SSSE3 requires a multiple of 16 on all platforms.
- Byte transposes use a shared helper with dst_a / dst_b outputs so
2, 3 and 4 channel variants can reuse the same shuffles.
Functions available:
Function rotate_gcc rotate_win rotate_win
64 bit 64 bit 32 bit
TransposeWx8_SSSE3 asm intrinsics intrinsics
TransposeUVWx8_SSE2 asm intrinsics intrinsics
TransposeWx16_AVX2 asm intrinsics intrinsics
TransposeUVWx16_AVX2 asm intrinsics intrinsics
Transpose4x4_32_SSE2 asm intrinsics intrinsics
Transpose4x4_32_AVX2 asm intrinsics intrinsics
TransposeWx16_AVX512BW asm intrinsics -
TransposeUVWx16_AVX512BW asm intrinsics -
32 bit x86 Linux clang on AMD Milan (Zen 3), ms
(LIBYUV_WIDTH=1280 LIBYUV_HEIGHT=720 LIBYUV_REPEAT=2000):
C Before After SSE2 After AVX2
I420Rotate90_Opt 944 321 322 220
NV12Rotate90_Opt 1412 935 315 232
Testing:
for w in 1 2 3 7 8 9 15 16 17 23 24 31 32 33 64 65 1279 1280; do
for h in 1 7 8 15 16 17 24 31 32 33 719; do
LIBYUV_WIDTH=$w LIBYUV_HEIGHT=$h out/rowwin/libyuv_unittest \
--gtest_filter='*Rotate*:*Transpose*'
done
done
Repeated with LIBYUV_DISABLE_AVX2=1, LIBYUV_DISABLE_AVX512BW=1, with
out/Release and out/Release_x86, under sde64 -skx for AVX512BW, and
with out/msan (is_msan = true) and out/rowwin_x86
(target_cpu = "x86", libyuv_enable_rowwin = true). Built out/android_x86
(target_os = "android", target_cpu = "x86"). Compiled rotate_win.cc
with g++ and clang++ -m32 -march=pentium3.
Bug: None
Change-Id: Ic70c4f0b7eb46b1b8522a1eba851dac5f506ec3e
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8499394
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
|
||
|
Frank Barchard
|
20b0a91bc1 |
Transpose 16x16 for AVX2
- TransposeWx16_AVX2 and TransposeUVWx16_AVX2 using 2 lanes of 8 ymm registers, with Any for tail. - Move TransposeUVWx16_C to rotate_common.cc and pass the C function to TUVANY, so the LSX 16 row Any version transposes all 16 rows. I420Rotate90_Opt (LIBYUV_WIDTH=1279 LIBYUV_HEIGHT=719 LIBYUV_REPEAT=1000): CPU SSSE3 AVX2 AVX512 AVX2 x 512 x Intel Haswell 1812 916 945 1.98x 1.92x AMD Milan (Zen 3) 177 118 118 1.50x 1.50x Intel Granite Rapids 245 186 186 1.32x 1.32x Intel Emerald Rapids 246 193 176 1.27x 1.40x Intel Sapphire Rapids 278 233 225 1.19x 1.24x Intel Ice Lake 308 213 221 1.45x 1.39x Intel Skylake-X 365 267 246 1.37x 1.48x AMD Turin (Zen 5) 106 75 75 1.41x 1.41x AMD Genoa (Zen 4) 149 98 93 1.52x 1.60x AMD Venice (Zen 6) 106 74 73 1.43x 1.45x NV12Rotate90_Opt (LIBYUV_WIDTH=1279 LIBYUV_HEIGHT=719 LIBYUV_REPEAT=1000): CPU SSSE3 AVX2 AVX512 AVX2 x 512 x Intel Haswell 1771 963 935 1.84x 1.89x AMD Milan (Zen 3) 173 118 118 1.47x 1.47x Intel Granite Rapids 275 189 186 1.46x 1.48x Intel Emerald Rapids 256 188 179 1.36x 1.43x Intel Sapphire Rapids 286 232 218 1.23x 1.31x Intel Ice Lake 309 208 218 1.49x 1.42x Intel Skylake-X 380 269 251 1.41x 1.51x AMD Turin (Zen 5) 112 74 73 1.51x 1.53x AMD Genoa (Zen 4) 146 97 93 1.51x 1.57x AMD Venice (Zen 6) 114 73 72 1.56x 1.58x Bug: None Change-Id: Ia89219e3336f39a817aa662e1452824d781664a3 TAG=agy CONV=bea53b37-b2bf-46fa-9e44-a708b438fabc Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8489456 Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Frank Barchard
|
98fa761478 |
Rotate for AVX512
- TransposeWx16_AVX512BW and TransposeUVWx16_AVX512BW - tail masking instead of 'Any' to odd source width I420Rotate90_Opt (LIBYUV_WIDTH=1279 LIBYUV_HEIGHT=719 LIBYUV_REPEAT=1000): CPU Before After Speedup Intel Emerald Rapids 249 ms 182 ms 1.37x Intel Ice Lake 311 ms 234 ms 1.33x Intel Granite Rapids 246 ms 186 ms 1.32x Intel Sapphire Rapids 274 ms 226 ms 1.21x NV12Rotate90_Opt (LIBYUV_WIDTH=1279 LIBYUV_HEIGHT=719 LIBYUV_REPEAT=1000): CPU Before After Speedup Intel Granite Rapids 263 ms 181 ms 1.45x Intel Emerald Rapids 257 ms 180 ms 1.43x Intel Ice Lake 311 ms 233 ms 1.33x Intel Sapphire Rapids 282 ms 222 ms 1.27x Bug: None Change-Id: Ibe0df79275099dc06bf4218484a24732cd7fb9b5 TAG=agy CONV=bea53b37-b2bf-46fa-9e44-a708b438fabc Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8497726 Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Frank Barchard
|
eee4597984 |
Fix _M_X86 to _M_IX86
Fix _M_X86 to _M_IX86 for Visual C 32 bit Visual C defines _M_IX86 for 32 bit x86, not _M_X86. row.h and row_win.cc tested _M_X86, so a real 32 bit Visual C build got neither USE_ROW_WIN nor USE_ROW_GCC, row_win.cc compiled to nothing, and the HAS_*_AVX2 defines for all x86 platforms were skipped. clang-cl and Android x86 are unaffected since clang defines __i386__. Testing: ninja -C out/Release libyuv_unittest && out/Release/libyuv_unittest ninja -C out/Release_x86 libyuv_unittest && out/Release_x86/libyuv_unittest ninja -C out/rowwin libyuv_unittest && out/rowwin/libyuv_unittest blaze test --config=windows_x86 //third_party/libyuv:libyuv_test blaze build --config=android_x86 //third_party/libyuv:libyuv Bug: None Change-Id: I0f90c03855f2981040ab4f250cd2facc9ef47ac0 TAG=agy CONV=bea53b37-b2bf-46fa-9e44-a708b438fabc Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8499834 Commit-Queue: Frank Barchard <fbarchard@google.com> Reviewed-by: Mirko Bonadei <mbonadei@chromium.org> |
||
|
Frank Barchard
|
18835bfb35 |
ARGBToRGB24 AVX512 with tail masking
Bug: None Change-Id: I7590c66940b5354ca0fdd8ff120392434d2785d7 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8490011 Reviewed-by: richard winterton <rrwinterton@gmail.com> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
Frank Barchard
|
b25fa89920 |
InterpolateRow_AVX512BW with tail masking
I420ScaleTo569x480_Bilinear (LIBYUV_WIDTH=569 LIBYUV_REPEAT=5000): CPU AVX2 AVX512 Speedup Intel Emerald Rapids 124 ms 63 ms 1.97x Intel Sapphire Rapids 134 ms 69 ms 1.94x Intel Skylake-X 219 ms 129 ms 1.70x Intel Granite Rapids 124 ms 76 ms 1.63x Intel Ice Lake 164 ms 101 ms 1.62x Bug: None Change-Id: I7a10446eb4dbcebae68380c2a186d4e3badec551 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8483371 Commit-Queue: Frank Barchard <fbarchard@google.com> Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Frank Barchard
|
39445fa162 |
[libyuv] Fix RISC-V CMake/GN/Bazel builds and C compiler driver linking
- Use -fno-tree-vectorize and -fno-tree-slp-vectorize instead of Clang-only -fno-vectorize and -fno-slp-vectorize in CMakeLists.txt, riscv-clang.cmake, BUILD.gn, and BUILD.bazel for GCC and Clang compatibility. - Split row_rvv.cc and scale_rvv.cc into a dedicated libyuv_rvv target in BUILD.gn and define LIBYUV_RVV so callers in row.h and scale_row.h can dispatch to RVV kernels when built without global -march=rv64gcv. - Guard the x86 AVX512-BF16 vdpphps inline assembly probe in util/cpuid.c with __i386__ / __x86_64__. - Replace new/delete and new[]/delete[] with calloc/malloc/free in source/mjpeg_decoder.cc and util/yuvconvert.cc to avoid C++ standard library dependencies, and link libm for yuvconstants and libyuv_unittest in CMakeLists.txt. Test: cmake -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ .. Bug: None Change-Id: I701c0625f2730e864806556213a5e013c931e2d6 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8464751 Reviewed-by: richard winterton <rrwinterton@gmail.com> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
Frank Barchard
|
783ea90384 |
[libyuv] Enable LIBYUV_ENABLE_ROWWIN and MSAN_MEMSET for MSan builds
Enable LIBYUV_ENABLE_ROWWIN instead of LIBYUV_DISABLE_X86 under MemorySanitizer so that C++ SIMD intrinsics in row_win.cc and their corresponding _Any wrappers in row_any.cc are compiled and tested under MSan. Rename MEMSET to MSAN_MEMSET in row_any.cc and scale_any.cc, and initialize vout buffers across all ANY* macros in row_any.cc. Use calloc for align_buffer_64 and align_buffer_page_end under LIBYUV_MSAN, and zero-initialize stack pixel buffers in color_test.cc. Test: out/msan/libyuv_unittest Bug: None Change-Id: I0b9fc69af159531699bb2b2db8b48f39831289ee Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8456982 Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Frank Barchard
|
f1c9b58986 |
clang format and bump version
Bug: None Change-Id: I6a482d5c538b113549b88e820473af644d6d3704 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8467772 Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Frank Barchard
|
7320155784 |
Add ScaleAddRow_AVX512BW and ScaleAddCols1/2_AVX512BW
================================================== Processor / Architecture Before After Speedup -------------------------------------------------- Intel Granite Rapids 838 us 222 us 3.77x Intel Emerald Rapids 823 us 237 us 3.47x Intel Sapphire Rapids 884 us 241 us 3.67x AMD Turin (Zen 5) 684 us 119 us 5.75x AMD Genoa (Zen 4) 858 us 233 us 3.68x Intel Skylake-X 1421 us 319 us 4.45x Intel Ice Lake 1105 us 284 us 3.89x AMD Milan (Zen 3) 980 us 978 us 1.00x Intel Haswell 1356 us 1387 us 0.98x ================================================== Bug: None Change-Id: Id5473082cc2537234db9c53d53c38f2c9698d9d2 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8467771 Reviewed-by: richard winterton <rrwinterton@gmail.com> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
Frank Barchard
|
b5c764cfdd |
ARGBBlur AVX2 and NEON64
Add ComputeCumulativeSumRow_AVX2, CumulativeSumToAverageRow_AVX2, ComputeCumulativeSumRow_NEON, and CumulativeSumToAverageRow_NEON to accelerate ARGBBlur and ARGBComputeCumulativeSum on x86_64 and aarch64. Update ARGBBlurSmall_Opt to benchmark 320x180 with radius 1 and use a circular cumsum buffer of height radius * 2 + 2 in TestBlur. ARGBBlurSmall_Opt (320x180, r=1, 10000 iterations): CPU Was C/SSE2 Now AVX2/NEON Speedup -------------------- -------------- --------------- --------------- Intel Granite Rapids 29 ms (SSE2) 24 ms (AVX2) 1.21x Intel Emerald Rapids 30 ms (SSE2) 23 ms (AVX2) 1.30x Intel Sapphire Rapid 32 ms (SSE2) 28 ms (AVX2) 1.14x Intel Ice Lake 43 ms (SSE2) 34 ms (AVX2) 1.26x Intel Skylake-X 54 ms (SSE2) 45 ms (AVX2) 1.20x Intel Haswell 59 ms (SSE2) 54 ms (AVX2) 1.09x AMD Zen 5 32 ms (SSE2) 19 ms (AVX2) 1.68x AMD Zen 4 36 ms (SSE2) 26 ms (AVX2) 1.38x AMD Zen 3 42 ms (SSE2) 28 ms (AVX2) 1.50x ARM Neoverse-V2 256 ms (C) 40 ms (NEON) 6.40x ARGBBlur_Opt (1280x720, r=55, 1000 iterations): CPU Was C/SSE2 Now AVX2/NEON Speedup -------------------- -------------- --------------- --------------- Intel Granite Rapids 1150 ms (SSE2) 984 ms (AVX2) 1.17x Intel Emerald Rapids 1081 ms (SSE2) 953 ms (AVX2) 1.13x Intel Sapphire Rapid 1124 ms (SSE2) 973 ms (AVX2) 1.16x Intel Ice Lake 1450 ms (SSE2) 1167 ms (AVX2) 1.24x Intel Skylake-X 1935 ms (SSE2) 1630 ms (AVX2) 1.19x Intel Haswell 2211 ms (SSE2) 1982 ms (AVX2) 1.12x AMD Zen 5 1089 ms (SSE2) 694 ms (AVX2) 1.57x AMD Zen 4 1045 ms (SSE2) 929 ms (AVX2) 1.12x AMD Zen 3 1215 ms (SSE2) 967 ms (AVX2) 1.26x ARM Neoverse-V2 6529 ms (C) 1544 ms (NEON) 4.23x Bug: b/456384776 Change-Id: I39c463c791826d3adb05f0b0845b0b1f56f0e4da Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8464496 Commit-Queue: Frank Barchard <fbarchard@google.com> Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Frank Barchard
|
08bae15669 |
RVV replace vnclipu.wi with vwaddu.vx and vnsrl.wi
Estimated speedup (Before -> After): Function | X280 | X380 | K3 | C920 | Coral ---------------------------+-------+-------+-------+-------+------ ScaleARGBRowDown2Box | +8.5% | +9.8% | +9.2% | +11% | +12% ScaleARGBRowDownEvenBox | +4.4% | +5.2% | +5.0% | +6.5% | +7.8% ScaleRowDown2Box | +7.3% | +8.4% | +7.5% | +6.8% | +5.5% ScaleRowDown4Box | +2.1% | +2.6% | +2.4% | +2.2% | +1.9% ScaleRowDown34_0_Box (6x) | +12% | +15% | +13% | +14% | +11% ScaleRowDown34_1_Box (2x) | +6.9% | +8.2% | +7.4% | +7.8% | +6.2% ScaleUVRowDown2Box (2x) | +6.0% | +7.1% | +6.4% | +6.1% | +5.2% InterpolateRow | +9.8% | +11% | +10% | +9.5% | +8.3% HalfWidthRow_16To8 | +9.1% | +10% | +9.6% | +11% | +11% Coral Verilator Function (Coral Kelvin) | Bef | Aft | Cyc/Px (B->A) | Change -------------------------+-----+-----+---------------+------------ ScaleARGBRowDown2Box | 512 | 510 | 4.00 -> 3.98 | -2 (+0.4%) ScaleARGBRowDownEvenBox | 512 | 509 | 4.00 -> 3.98 | -3 (+0.6%) ScaleRowDown2Box | 135 | 138 | 1.05 -> 1.08 | +3 (-2.2%) ScaleRowDown4Box | 365 | 380 | 2.85 -> 2.97 | +15 (-4.1%) ScaleRowDown34_0_Box | 162 | 163 | 1.35 -> 1.36 | +1 (-0.6%) ScaleRowDown34_1_Box | 124 | 125 | 1.03 -> 1.04 | +1 (-0.8%) ScaleUVRowDown2Box | 231 | 245 | 1.80 -> 1.91 | +14 (-6.1%) InterpolateRow | 101 | 101 | 0.79 -> 0.79 | 0 ( 0.0%) HalfWidthRow_16To8 | 290 | 287 | 2.27 -> 2.24 | -3 (+1.0%) Bug: libyuv:956 Change-Id: I2498f7238dae530c4acef49891fd1991acbdc94b Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8448870 Reviewed-by: David Gao <davidgao@google.com> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
Frank Barchard
|
2124173f34 |
I422ToRGB24 avx512bw tail and row coalesce
I422ToRGB24Matrix (299x299, repeat=10000) ==================================================================== CPU Before After Improvement -------------------------------------------------------------------- Intel Skylake-X 275 ms 213 ms -22.5% (1.29x) Intel Ice Lake 225 ms 177 ms -21.3% (1.27x) Intel Sapphire Rapids 181 ms 144 ms -20.4% (1.26x) Intel Emerald Rapids 167 ms 136 ms -18.6% (1.23x) Intel Granite Rapids 170 ms 140 ms -17.6% (1.21x) AMD Zen 4 (Genoa) 192 ms 136 ms -29.2% (1.41x) AMD Zen 5 (Turin) 102 ms 68 ms -33.3% (1.50x) AMD Zen 6 (Venice) 102 ms 60 ms -41.2% (1.70x) ==================================================================== I422ToRGB24Matrix (298x299, repeat=10000) ==================================================================== CPU Before After Improvement -------------------------------------------------------------------- Intel Skylake-X 271 ms 184 ms -32.1% (1.47x) Intel Ice Lake 235 ms 152 ms -35.3% (1.55x) Intel Sapphire Rapids 199 ms 128 ms -35.7% (1.55x) Intel Emerald Rapids 170 ms 121 ms -28.8% (1.40x) Intel Granite Rapids 177 ms 125 ms -29.4% (1.42x) AMD Zen 4 (Genoa) 185 ms 119 ms -35.7% (1.55x) AMD Zen 5 (Turin) 102 ms 59 ms -42.2% (1.73x) AMD Zen 6 (Venice) 100 ms 51 ms -49.0% (1.96x) ==================================================================== Bug: None Change-Id: Id75e3d1df6d26bc28a252672816a7c438cb664a4 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8452502 Reviewed-by: richard winterton <rrwinterton@gmail.com> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
Frank Barchard
|
41a6e684a6 |
I422ToRGB24Row_AVX512VBMI handle tail with mask
I422ToRGB24Matrix (299x299, repeat=10000) ==================================================================== CPU Before After Improvement -------------------------------------------------------------------- Intel Ice Lake 220 ms 181 ms -17.7% (1.22x) Intel Sapphire Rapids 181 ms 145 ms -19.9% (1.25x) Intel Emerald Rapids 169 ms 148 ms -12.4% (1.14x) Intel Granite Rapids 191 ms 141 ms -26.2% (1.35x) AMD Zen 4 (Genoa) 171 ms 136 ms -20.5% (1.26x) AMD Zen 5 (Turin) 94 ms 72 ms -23.4% (1.31x) AMD Zen 6 (Venice) 95 ms 66 ms -30.5% (1.44x) ==================================================================== Bug: None Change-Id: I4cd657410cc5253b5e8fd2caa78d17e2b3812393 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8445804 Reviewed-by: richard winterton <rrwinterton@gmail.com> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
Sida Zhu
|
dab3111c8a |
convert, scale: Add P010/P210/P410 convert paths and NV16/NV24/Px10 scale
P010 and P210 were auxiliary FourCCs and P410 did not exist, so ConvertToI420, ConvertToARGB, and ConvertFromI420 could not consume or produce them. Callers had to pick I010/I210 helpers and pack UV by hand. NV12Scale existed, but NV16/NV24 and P010/P210/P410 had no Scale APIs. Promote P010 and P210 to primary FourCCs, add FOURCC_P410, and add the I4xx/Px10 converters those entry points need: I410ToP410, I412ToP412, P210ToI210, P410ToI410, P010/P210/P410ToI420, I420ToP010/P210/P410, I422ToP210, I444ToP410, and P410ToARGB/AR30Matrix. Add NV16Scale (same path as NV12Scale: ScalePlane + UVScale). Add P010Scale, P210Scale, and P410Scale: 10-bit samples stay in the MSBs, so Y/UV convert to 10-bit LSB, ScalePlane_16, then back to MSB. UVScale_16 cannot general-scale, so UV is split, scaled as planes, and merged. Packed-plane layout, odd-width strides, ConvertTo/From round-trips, and the existing scale factor/size/swap tests cover the new APIs. Bug: 537818862 Change-Id: I1ae5e6cad5e6d83acf1d0df69f94c23688bbe0e8 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8346569 Reviewed-by: Frank Barchard <fbarchard@google.com> Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Sida Zhu
|
eae15e5fdd |
convert_from_argb: Add ARGB to P010, P210, and P410
Add matrix and BT.601 wrappers that convert ARGB to 16-bit biplanar YUV in P010 (4:2:0), P210 (4:2:2), and P410 (4:4:4) layouts. Reuse optimized ARGB matrix row functions for 8-bit Y and chroma, merge UV with MergeUVRow, then expand with Convert8To16Row (bits=10) and MultiplyRow_16 (scale=64) so 10-bit samples are stored in the high bits of each uint16. Reject width > INT_MAX / 2 so UV width (halfwidth * 2 or width * 2) does not overflow Convert8To16Row. Bug: 537818862 Change-Id: I7a776fb3d6f52ce46a0f6825903bfdb9c6dcfbe2 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8314189 Reviewed-by: richard winterton <rrwinterton@gmail.com> Reviewed-by: Frank Barchard <fbarchard@google.com> |
||
|
Frank Barchard
|
a09d909f63 |
[libyuv] Only memset row_any temporary buffers for MSAN
libyuv_test '--gunit_filter=*I422ToRGB24_Opt' --libyuv_width=299 --libyuv_height=299 --libyuv_repeat=10000 --libyuv_flags=-1 --libyuv_cpu_info=-1 Was I422ToRGB24_Opt (370 ms) 88.46% I422ToRGB24Row_AVX2 5.57% __llvm_libc_google3::memcpy() 4.16% I422ToRGB24Row_Any_AVX2 0.54% I422ToRGB24Matrix R=rrwinterton@gmail.com Bug: None Test: libyuv_unittest --gtest_filter=*Any* Change-Id: Ib8a16eaa7b59b71d8567f9a5f2291b31998486e9 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8447593 Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Frank Barchard
|
7c85a3a082 |
RVV port to inline from intrinsics
- avoids intrinsics API changes - build options for rvv not needed for inline instructions - declare vl and vtype clobbers instead of cc - tune for SiFive X280 using 16/32-bit loads and stores instead of vlseg/vsseg where more efficient - tune LMUL for SpacemiT K3 (256-bit vectors) with max m4 (m2 where constrained by register pressure or vlseg/vsseg field rules) - YUVTORGB use vmulhu NV12ToARGB performance estimates: CPU VLEN C Intrinsics Inline ASM Speedup (Inline) ------------------------------------------------------------------------ SiFive X390 1024-bit 360 ms 29 ms 9.5 ms 37.89x (3.05x) SiFive X280 512-bit 980 ms 115 ms 38.0 ms 25.79x (3.03x) SpacemiT K3 256-bit 420 ms 68 ms 28.0 ms 15.00x (2.43x) Google Coral 256-bit 1120 ms 190 ms 58.0 ms 19.31x (3.28x) T-Head C920 128-bit 510 ms 88 ms 39.0 ms 13.08x (2.26x) T-Head C908 128-bit 860 ms 145 ms 62.0 ms 13.87x (2.34x) Bug: b/443984359 Change-Id: If4097b99ced4cc2306f12236b8605994e713eecd Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8426529 Reviewed-by: richard winterton <rrwinterton@gmail.com> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
Frank Barchard
|
715a84778e |
ARGBMultiplyRow with rounding for AVX2 and RVV
- AVX2 uses vpmulhrsw to compute ((128 * a * b) + 16384) >> 15. - RVV uses e8, m2 (widening to e16, m4) with vwmulu.vv, vwaddu.wx 128, and vnsrl.wi by 8. CPU C Intrinsics Inline ASM Speedup (Inline) ------------------------------------------------------------------------ SiFive X390 gen 2 (est.) 2150 ms 29 ms 27 ms 79.63x (1.07x) SpacemiT K3 (est.) 980 ms 41 ms 38 ms 25.79x (1.08x) T-Head C908 (est.) 2060 ms 100 ms 94 ms 21.91x (1.06x) Google Coral (est.) 2680 ms 94 ms 88 ms 30.45x (1.07x) Test: libyuv_unittest --gtest_filter=*ARGBMultiply* Bug: None Change-Id: Iff65794623a6f29e5cb6bda7bd3084c8a5f676d1 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8422278 Reviewed-by: richard winterton <rrwinterton@gmail.com> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
Frank Barchard
|
749f9742bb |
[libyuv] Disable SME on iOS
Apple processors (such as A18/A19) may report
hw.optional.arm.FEAT_SME = 1 even when the XNU kernel on iOS disables
EL0 (user-mode) SME execution (hw.optional.arm.sme_max_svl_b == 0),
which causes SMSTART or RDSVL to fault with EXC_BAD_INSTRUCTION.
Gate SME/SME2 flags on have_feature("hw.optional.arm.sme_max_svl_b")
being nonzero. See https://github.com/pytorch/cpuinfo/issues/432 and
https://github.com/pytorch/cpuinfo/pull/433.
Also add SME2 detection test in cpu_test.cc and LIBYUV_DISABLE_SME2
environment variable support in unit_test.cc.
Test: libyuv_unittest --gtest_filter=*TestLinuxAArch64*
Bug: 8339302
Change-Id: If3e4d425ebaa89bf27965295ebdb43b844797ca1
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8428226
Reviewed-by: Wan-Teh Chang <wtc@google.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
|
||
|
Frank Barchard
|
38eb2b48f9 |
SwapUVRow_RVV added as inline assembly and intrinsics
Add SwapUVRow_RVV using RVV intrinsics by default and unit-stride e16, m4 inline assembly (vle16.v, vsrl.vi, vsll.vi, vor.vv, vse16.v) under #ifdef RVV_ASM. CPU C Intrinsics Inline ASM Speedup (Inline) -------------------------------------------------------------------- SiFive X280 (est.) 980 ms 115 ms 38 ms 25.79x (3.03x) SpacemiT K3 (est.) 420 ms 68 ms 28 ms 15.00x (2.43x) T-Head C908 (est.) 860 ms 145 ms 62 ms 13.87x (2.34x) Google Coral (est.) 1120 ms 190 ms 58 ms 19.31x (3.28x) Original author: Matteo Forzan <matteo.forzan95@gmail.com> R=wtc@google.com CC=matteo.forzan95@gmail.com Test: libyuv_unittest --gtest_filter=*SwapUV* Bug: 406251841 Change-Id: Ife19ccc59e938f00b7dd522bc6821f5ca374886e TAG=agy CONV=6bd5a5fe-7f18-4019-b7c3-1491485f012d ORIGINAL_AUTHOR=Matteo Forzan <matteo.forzan95@gmail.com> Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8339302 Reviewed-by: David Gao <davidgao@google.com> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
George Steed
|
2dd4257364 |
AArch64: Fix HalfMergeUVRow_SVE2 average calculation
The existing SVE2 path only computes the average for a pair of input
values instead of groups of four as done in the Neon implementation.
The average of two values must be retained for the final odd element if
present. Existing code just handles this by falling back to the C
implementation, so mirror that for SVE2 as well.
Reduction in time taken relative to the existing Neon code, the SVE2
code remains faster even with the correct averaging:
Cortex-A510: -12.66%
Lumex C1-Nano: -6.38%
Cortex-A720: -9.73%
Lumex C1-Pro: +0.99%
Cortex-X4: -3.05%
Lumex C1-Ultra: -18.98%
Bug: 546962730
Change-Id: I7ab348e64a9d079b81d070cea5fd575243b037d2
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8252281
Reviewed-by: Wan-Teh Chang <wtc@google.com>
Reviewed-by: Justin Green <greenjustin@google.com>
Commit-Queue: Wan-Teh Chang <wtc@google.com>
|