mirror of
https://chromium.googlesource.com/libyuv/libyuv
synced 2026-11-09 14:49:57 +08:00
2190 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
George Steed
|
9e8c1ef826 |
AArch64: Avoid LD4 in ScaleRowDown4_NEON
ScaleRowDown4_NEON uses LD4 to extract the third entry from each four,
however LD4 is known to be slow on some micro-architectures and implies
overhead for performing the de-interleaving even for the unused vectors
v0/v1/v3.
Replace this with a single LD1 to load the vectors without
de-interleaving and then select the bytes we want with UZP instead.
Speedup versus the existing Neon implementation (higher is better):
Cortex-A55: 1.004
Cortex-X4: 1.873
Cortex-A720: 1.285
C1-Nano: 1.330
C1-Pro: 1.246
C1-Ultra: 1.219
Change-Id: Ib7d8633635fa149b692e433062fc4b9717a32594
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8692146
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Reviewed-by: Frank Barchard <fbarchard@google.com>
|
||
|
Frank Barchard
|
78277c25e6 |
[libyuv] Fix R/B swap in Wasm/LoongArch split/merge and x86 bugs
x86 fixes:
- Prefer CopyRow_AVX512BW and SetRow_AVX2 over ERMS rep movsb/stosb
in CopyPlane, SetPlane and RotatePlane. ERMS was checked last and
replaced the vector rows on all modern CPUs; on Zen 4 CopyRow_AVX512BW
is 1.2x to 5x faster than rep movsb for 64 to 3840 byte rows.
- Only report AMX-INT8 and ACEv1 when XCR0 enables tile state,
CPUID reports AMX-TILE and, on Linux, the process has XTILEDATA
permission (ARCH_GET_XCOMP_PERM). Without permission tile
instructions raise SIGILL. libyuv does not request the permission.
- Detect AVX512-BMM from AMD leaf 0x80000021 EAX[23] instead of basic
leaf 0x21, and detect ERMS from leaf 0x80000021 without requiring AVX.
- Allow AVX10 CPUs without the legacy AVX512F bit when CPUID leaf 0x24
reports 512-bit vectors.
- Remove unused kCpuHasAVX512VBITALG and reuse its bit for kCpuHasAPX
(CPUID.7.1:EDX[21]).
- Restrict CLANG_HAS_AVX512 to x86_64 and mark xgetbv asm volatile
with a plain "edx" clobber.
- Use 64-bit division in FixedDiv_X86 and FixedDiv1_X86 on x86_64 so
quotients >= 2^31 no longer raise SIGFPE.
- Sample the center pixel when scaling 32768 or more pixels down to 1
instead of stretching the destination and sampling pixel 0.
- Check InterpolateRow_16_AVX2 alignment against 16 pixels, matching
its loop, instead of 32.
- ARGBBlur: reject a NULL cumulative sum buffer, check the result of
ARGBComputeCumulativeSum, and use uint32_t arithmetic in the C
cumulative sum rows to avoid signed overflow on 4K frames.
Fix red and blue channels being swapped by SplitARGBRow, MergeARGBRow,
SplitXRGBRow and MergeXRGBRow for WebAssembly SIMD128 and LoongArch
LSX/LASX. libyuv ARGB is B, G, R, A in memory. The Wasm kernels stored
byte 0 to dst_r and interleaved (r, g) first. The LoongArch kernels
named their pointer parameters in B, G, R order while callers pass
R, G, B. Split and Merge had matching errors, so they cancelled out.
The SplitARGBPlane, MergeARGBPlane, SplitXRGBPlane, MergeXRGBPlane,
SplitRGBPlane and MergeRGBPlane tests only compared a full split and
merge round trip against C, which hides a channel swap that is
mirrored in both directions. The Split tests now compare each plane
against C, the Merge tests merge the C planes with both paths, and the
new SplitMergePlane_ChannelOrder test checks fixed channel values
without relying on the C code.
Bump version to 1980.
Test: libyuv_unittest --gtest_filter=*Split*:*Merge*:*ChannelOrder*
Test: libyuv_unittest (x86_64, x86_32, LIBYUV_ENABLE_ROWWIN, GCC,
MSan, UBSan)
Bug: libyuv:573815252
Change-Id: I61df6a542bc95a00f7e69441ee5e012f1c00eae7
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8749626
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
|
||
|
Frank Barchard
|
d4074c4e18 |
[libyuv] Expand WebAssembly SIMD128 and Relaxed SIMD support
Expand WebAssembly SIMD128 (_WASMSIMD) and WebAssembly Relaxed SIMD (_RELAXEDSIMD) optimizations across compare, rotate, convert, and scale row functions. Add a distinct kCpuHasRELAXEDSIMD (0x200) capability flag and runtime WasmCpuCaps() detection modeled after XNNPACK, probing whether wasm_i8x16_relaxed_swizzle or wasm_f32x4_relaxed_madd exhibit hardware-specific Relaxed SIMD behavior at runtime. Support disabling Relaxed SIMD alone (LIBYUV_DISABLE_RELAXEDSIMD), Wasm SIMD128 alone (LIBYUV_DISABLE_WASMSIMD), or all Wasm SIMD (LIBYUV_DISABLE_WASM) via compile-time macros or runtime environment variables. Compare (6 functions): C Function x86 Arm Wasm ----------------- ---------- ------------ ----------- HammingDistance_C AVX512VNNI NEON_DOTPROD RELAXEDSIMD SumSquareError_C AVX512VNNI NEON_DOTPROD WASMSIMD HashDjb2_C AVX512BW NEON WASMSIMD Ssim4x4Row_C AVX512BW SME WASMSIMD SsimWindowRow_C AVX512BW SME WASMSIMD ARGBDetectRow_C AVX2 NEON WASMSIMD ----------------- ---------- ------------ ----------- Count (6) 6/6 6/6 6/6 Percentage 100.0% 100.0% 100.0% Rotate (7 functions): C Function x86 Arm Wasm ----------------- -------- ------ -------- TransposeWx8_C AVX2 NEON WASMSIMD TransposeWx16_C AVX512BW NEON WASMSIMD TransposeUVWx8_C AVX2 NEON WASMSIMD TransposeUVWx16_C AVX512BW - WASMSIMD TransposeWx8_16_C AVX2 SVE2 WASMSIMD TransposeWxH_32_C AVX2 SME - TransposeWx4_32_C AVX2 SVE2 WASMSIMD ----------------- -------- ------ -------- Count (7) 7/7 6/7 6/7 Percentage 100.0% 85.7% 85.7% Convert (171 functions): C Function x86 Arm Wasm --------------------------- ---------- ------------ ----------- ARGBToYMatrixRow_C AVX512VNNI SME WASMSIMD ARGBToUVMatrixRow_C AVX512VNNI SME RELAXEDSIMD RGBToYMatrixRow_C AVX512BW NEON_DOTPROD RELAXEDSIMD RGBToUVMatrixRow_C AVX512BW NEON RELAXEDSIMD RGBToUV444MatrixRow_C AVX512BW NEON RELAXEDSIMD RGB565ToYMatrixRow_C AVX2 NEON WASMSIMD ARGB1555ToYMatrixRow_C AVX2 NEON WASMSIMD ARGB1555ToUVMatrixRow_C AVX2 NEON WASMSIMD ARGB4444ToYMatrixRow_C AVX2 NEON WASMSIMD ARGB4444ToUVMatrixRow_C AVX2 NEON WASMSIMD RGB565ToUVMatrixRow_C AVX2 NEON WASMSIMD ARGBToUV444MatrixRow_C AVX512VNNI NEON_I8MM RELAXEDSIMD RGB565ToYRow_C AVX2 NEON - ARGB1555ToYRow_C AVX2 NEON - ARGB4444ToYRow_C AVX2 NEON - RGB565ToUVRow_C AVX2 NEON - ARGB1555ToUVRow_C AVX2 NEON - ARGB4444ToUVRow_C AVX2 NEON - MirrorRow_C AVX512BW NEON RELAXEDSIMD MirrorUVRow_C AVX512BW NEON RELAXEDSIMD MirrorSplitUVRow_C AVX512BW NEON RELAXEDSIMD MirrorRow_16_C AVX512BW NEON - ARGBMirrorRow_C AVX512BW NEON WASMSIMD RGB24MirrorRow_C AVX512VBMI NEON RELAXEDSIMD SplitUVRow_C AVX512BW SME WASMSIMD DetileRow_C AVX2 NEON WASMSIMD DetileRow_16_C AVX2 NEON WASMSIMD DetileSplitUVRow_C AVX2 NEON WASMSIMD DetileToYUY2_C AVX2 NEON WASMSIMD UnpackMT2T_C AVX2 NEON - MergeUVRow_C AVX512BW SME WASMSIMD HalfMergeUVRow_C AVX512BW SME WASMSIMD SplitRGBRow_C AVX2 NEON RELAXEDSIMD MergeRGBRow_C AVX512VBMI NEON RELAXEDSIMD MergeARGBRow_C AVX2 NEON WASMSIMD SplitARGBRow_C AVX2 NEON WASMSIMD MergeXRGBRow_C AVX2 NEON WASMSIMD SplitXRGBRow_C AVX2 NEON WASMSIMD MergeXR30Row_C AVX2 NEON - MergeAR64Row_C AVX2 NEON - MergeARGB16To8Row_C AVX2 NEON - MergeXR64Row_C AVX2 NEON - MergeXRGB16To8Row_C AVX2 NEON - MergeUVRow_16_C AVX512BW SME WASMSIMD SplitUVRow_16_C AVX2 NEON WASMSIMD MultiplyRow_16_C AVX512BW SME WASMSIMD DivideRow_16_C AVX512BW SME WASMSIMD Convert8To16Row_C AVX512BW SME WASMSIMD Convert16To8Row_C AVX512BW SME WASMSIMD HalfRow_C AVX2 SME WASMSIMD HalfRow_16_C AVX2 SME WASMSIMD HalfRow_16To8_C AVX512BW SME WASMSIMD HalfWidthRow_16To8_C AVX512BW SME WASMSIMD HalfWidthRow_16To8_Odd_C AVX2 NEON - Convert8To8Row_C AVX512BW SME WASMSIMD CopyRow_C AVX512BW SME WASMSIMD CopyRow_16_C AVX2 NEON WASMSIMD ARGBCopyAlphaRow_C AVX2 NEON WASMSIMD ARGBExtractAlphaRow_C AVX512BW NEON WASMSIMD ARGBCopyYToAlphaRow_C AVX512BW NEON RELAXEDSIMD SetRow_C AVX2 NEON WASMSIMD ARGBSetRow_C AVX2 NEON WASMSIMD ARGBShuffleRow_C AVX512BW NEON RELAXEDSIMD RGB24ToARGBRow_C AVX512BW SME RELAXEDSIMD RAWToARGBRow_C AVX512BW SME RELAXEDSIMD RAWToRGBARow_C AVX512VBMI SME RELAXEDSIMD RAWToRGB24Row_C AVX512VBMI SME RELAXEDSIMD RGB565ToARGBRow_C AVX2 NEON WASMSIMD ARGB1555ToARGBRow_C AVX512BW SME WASMSIMD ARGB4444ToARGBRow_C AVX2 NEON WASMSIMD AR30ToARGBRow_C AVX512BW NEON WASMSIMD AR30ToABGRRow_C AVX512BW NEON WASMSIMD ARGBToAR30Row_C AVX512BW NEON WASMSIMD AR30ToAB30Row_C AVX512BW NEON WASMSIMD ARGBToRGB565DitherRow_C AVX512BW SME WASMSIMD ARGBToABGRRow_C - - - ARGBToBGRARow_C - - - ARGBToRGBARow_C - - - ARGBToRGB24Row_C AVX512BW SME RELAXEDSIMD ARGBToRAWRow_C AVX512BW SME RELAXEDSIMD ARGBToRGB565Row_C AVX512BW SME WASMSIMD ARGBToARGB1555Row_C AVX2 NEON WASMSIMD ARGBToARGB4444Row_C AVX2 NEON WASMSIMD ABGRToAR30Row_C AVX512BW NEON WASMSIMD ARGBToAR64Row_C AVX2 NEON WASMSIMD ARGBToAB64Row_C AVX2 NEON WASMSIMD AR64ToARGBRow_C AVX2 NEON WASMSIMD AB64ToARGBRow_C AVX2 NEON WASMSIMD AR64ToAB64Row_C - - - RGBAToARGBRow_C - - - AR64ShuffleRow_C - - - J400ToARGBRow_C AVX512BW NEON RELAXEDSIMD I444ToARGBRow_C AVX512BW SME WASMSIMD I444ToRGB24Row_C AVX512VBMI SME RELAXEDSIMD I422ToARGBRow_C AVX512BW SME WASMSIMD I422ToAR30Row_C AVX512BW SME WASMSIMD I210ToAR30Row_C AVX512BW SME WASMSIMD I210ToARGBRow_C AVX512BW SME WASMSIMD I212ToAR30Row_C AVX512BW SME - I212ToARGBRow_C AVX512BW SME - I410ToAR30Row_C AVX512BW SME WASMSIMD I410ToARGBRow_C AVX512BW SME WASMSIMD I210AlphaToARGBRow_C AVX512BW SME WASMSIMD I410AlphaToARGBRow_C AVX512BW SME WASMSIMD I444AlphaToARGBRow_C AVX512BW SME WASMSIMD I422AlphaToARGBRow_C AVX512BW SME WASMSIMD NV12ToARGBRow_C AVX512BW SME WASMSIMD NV12ToRGB565Row_C AVX2 NEON WASMSIMD NV21ToARGBRow_C AVX512BW SME WASMSIMD NV12ToRGB24Row_C AVX512VBMI SME RELAXEDSIMD NV21ToRGB24Row_C AVX512VBMI SME RELAXEDSIMD NV21ToYUV24Row_C AVX512VBMI NEON - YUY2ToARGBRow_C AVX512BW SME WASMSIMD UYVYToARGBRow_C AVX512BW SME WASMSIMD P210ToARGBRow_C AVX512BW SME - P410ToARGBRow_C AVX512BW SME - P210ToAR30Row_C AVX512BW SME - P410ToAR30Row_C AVX512BW SME - I422ToRGBARow_C AVX512BW SME WASMSIMD I422ToRGB24Row_C AVX512VBMI SME RELAXEDSIMD I422ToARGB4444Row_C AVX512BW SME WASMSIMD I422ToARGB1555Row_C AVX512BW SME WASMSIMD I422ToRGB565Row_C AVX512BW SME WASMSIMD I400ToARGBRow_C AVX512BW SME RELAXEDSIMD ARGBBlendRow_C AVX512BW NEON RELAXEDSIMD BlendPlaneRow_C AVX512BW NEON WASMSIMD ARGBMultiplyRow_C AVX512BW SME WASMSIMD ARGBAddRow_C AVX2 NEON WASMSIMD ARGBSubtractRow_C AVX2 NEON WASMSIMD ARGBOrRow_C AVX512BW SME WASMSIMD ARGBAlphaBlendRow_C AVX512BW SME WASMSIMD YUY2ToYRow_C AVX512BW SME WASMSIMD YUY2ToUVRow_C AVX512BW SME WASMSIMD YUY2ToNVUVRow_C AVX2 NEON WASMSIMD YUY2ToUV422Row_C AVX512BW NEON WASMSIMD UYVYToYRow_C AVX512BW SME WASMSIMD UYVYToUVRow_C AVX512BW SME WASMSIMD UYVYToUV422Row_C AVX512BW NEON WASMSIMD SwapUVRow_C AVX512BW NEON WASMSIMD AYUVToYRow_C AVX512VBMI NEON WASMSIMD AYUVToUVRow_C AVX512VBMI SME RELAXEDSIMD AYUVToVURow_C AVX512VBMI SME RELAXEDSIMD I422ToYUY2Row_C AVX2 NEON WASMSIMD I422ToUYVYRow_C AVX2 NEON WASMSIMD ARGBAttenuateRow_C AVX512BW NEON RELAXEDSIMD ARGBUnattenuateRow_C AVX2 NEON - ARGBGrayRow_C AVX512VNNI NEON_DOTPROD RELAXEDSIMD ARGBSepiaRow_C AVX512VNNI NEON_DOTPROD RELAXEDSIMD ARGBColorMatrixRow_C AVX512VNNI NEON_I8MM WASMSIMD ARGBColorTableRow_C AVX2 NEON - RGBColorTableRow_C AVX2 NEON - ARGBQuantizeRow_C AVX2 NEON WASMSIMD ARGBShadeRow_C AVX2 NEON WASMSIMD CumulativeSumToAverageRow_C AVX2 NEON - ComputeCumulativeSumRow_C AVX2 NEON - ARGBAffineRow_C AVX2 NEON - InterpolateRow_C AVX512BW SME WASMSIMD InterpolateRow_16_C AVX512BW SME WASMSIMD SobelXRow_C AVX2 NEON WASMSIMD SobelYRow_C AVX2 NEON WASMSIMD SobelRow_C AVX2 NEON RELAXEDSIMD SobelToPlaneRow_C AVX2 NEON WASMSIMD SobelXYRow_C AVX2 NEON WASMSIMD ARGBPolynomialRow_C AVX2 NEON WASMSIMD HalfFloatRow_C AVX512BW SME WASMSIMD ByteToFloatRow_C AVX2 NEON WASMSIMD ARGBLumaColorTableRow_C AVX2 NEON WASMSIMD GaussRow_F32_C AVX2 NEON - GaussCol_F32_C AVX2 NEON - GaussRow_C AVX2 NEON - GaussCol_C AVX2 NEON - --------------------------- ---------- ------------ ----------- Count (171) 165/171 165/171 134/171 Percentage 96.5% 96.5% 78.4% Scale (86 functions): C Function x86 Arm Wasm ------------------------------- ---------- ------ ----------- FixedDiv_C - - - FixedDiv1_C - - - ScaleRowDown2_C AVX512BW SME WASMSIMD ScaleRowDown2_16_C AVX2 SME - ScaleRowDown2Linear_C AVX512BW SME WASMSIMD ScaleRowDown2Linear_16_C AVX2 SME - ScaleRowDown2Box_C AVX512BW SME WASMSIMD ScaleRowDown2Box_Odd_C AVX512BW NEON WASMSIMD ScaleRowDown2Box_16_C AVX2 SME - ScaleRowDown4_C AVX2 NEON WASMSIMD ScaleRowDown4_16_C - - - ScaleRowDown4Box_C AVX2 NEON WASMSIMD ScaleRowDown4Box_16_C - - - ScaleRowDown34_C AVX2 NEON RELAXEDSIMD ScaleRowDown34_16_C - - - ScaleRowDown34_0_Box_C AVX2 NEON RELAXEDSIMD ScaleRowDown34_0_Box_16_C - - - ScaleRowDown34_1_Box_C AVX2 NEON RELAXEDSIMD ScaleRowDown34_1_Box_16_C - - - ScaleRowUp2_Linear_C AVX2 NEON RELAXEDSIMD ScaleRowUp2_Bilinear_C AVX2 NEON RELAXEDSIMD ScaleRowUp2_Linear_16_C AVX2 NEON - ScaleRowUp2_Bilinear_16_C AVX2 NEON - ScaleRowUp2_Linear_Any_C AVX2 NEON RELAXEDSIMD ScaleRowUp2_Bilinear_Any_C AVX2 NEON RELAXEDSIMD ScaleRowUp2_Linear_16_Any_C AVX2 NEON - ScaleRowUp2_Bilinear_16_Any_C AVX2 NEON - ScaleCols_C - - - ScaleCols_16_C - - - ScaleColsUp2_C AVX2 - - ScaleColsUp2_16_C - - - ScaleFilterCols_C AVX2 NEON WASMSIMD ScaleFilterCols_16_C - - - ScaleFilterCols64_C - - - ScaleFilterCols64_16_C - - - ScaleRowDown38_C AVX2 NEON RELAXEDSIMD ScaleRowDown38_16_C - - - ScaleRowDown38_3_Box_C AVX2 NEON RELAXEDSIMD ScaleRowDown38_3_Box_16_C - - - ScaleRowDown38_2_Box_C AVX2 NEON RELAXEDSIMD ScaleRowDown38_2_Box_16_C - - - ScaleAddRow_C AVX512BW NEON WASMSIMD ScaleAddRow_16_C - - - ScaleARGBRowDown2_C AVX2 SME WASMSIMD ScaleARGBRowDown2Linear_C AVX2 SME WASMSIMD ScaleARGBRowDown2Box_C AVX2 SME WASMSIMD ScaleARGBRowDownEven_C AVX2 NEON WASMSIMD ScaleARGBRowDownEvenBox_C AVX2 NEON WASMSIMD ScaleARGBCols_C AVX2 NEON WASMSIMD ScaleARGBCols64_C - - - ScaleARGBColsUp2_C AVX2 - WASMSIMD ScaleARGBFilterCols_C AVX2 NEON WASMSIMD ScaleARGBFilterCols64_C - - - ScaleUVRowDown2_C AVX2 SME WASMSIMD ScaleUVRowDown2_16_C - - - ScaleUVRowDown2Linear_C AVX2 SME WASMSIMD ScaleUVRowDown2Linear_16_C - - - ScaleUVRowDown2Box_C AVX2 SME WASMSIMD ScaleUVRowDown2Box_16_C - - - ScaleUVRowDownEven_C AVX2 NEON WASMSIMD ScaleUVRowDownEven_16_C - - - ScaleUVRowDownEvenBox_C - - - ScaleUVRowDownEvenBox_16_C - - - ScaleUVRowUp2_Linear_C AVX2 NEON RELAXEDSIMD ScaleUVRowUp2_Bilinear_C AVX2 NEON RELAXEDSIMD ScaleUVRowUp2_Linear_Any_C AVX2 NEON RELAXEDSIMD ScaleUVRowUp2_Bilinear_Any_C AVX2 NEON RELAXEDSIMD ScaleUVRowUp2_Linear_16_C AVX2 NEON - ScaleUVRowUp2_Bilinear_16_C AVX2 NEON - ScaleUVRowUp2_Linear_16_Any_C AVX2 NEON - ScaleUVRowUp2_Bilinear_16_Any_C AVX2 NEON - ScaleUVCols_C - - - ScaleUVCols_16_C - - - ScaleUVCols64_C - - - ScaleUVCols64_16_C - - - ScaleUVColsUp2_C - - - ScaleUVColsUp2_16_C - - - ScaleUVFilterCols_C - - - ScaleUVFilterCols_16_C - - - ScaleUVFilterCols64_C - - - ScaleUVFilterCols64_16_C - - - ScaleUVAddCols3_C AVX512VBMI SME WASMSIMD ScaleUVAddCols3_16_C - - - ScaleMaxSamples_C AVX2 NEON WASMSIMD ScaleSumSamples_C AVX2 NEON WASMSIMD ScaleSamples_C AVX2 NEON WASMSIMD ------------------------------- ---------- ------ ----------- Count (86) 50/86 48/86 38/86 Percentage 58.1% 55.8% 44.2% Bug: 42280942 Change-Id: I4420a570327a95cc998c6aa023286a0b3552f742 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8724066 Reviewed-by: richard winterton <rrwinterton@gmail.com> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
Frank Barchard
|
d2a6496353 |
[libyuv] Add LoongArch LASX parity, cpucfg CPUID, and 38 convert rows
Follow up on the LoongArch *_la.cc consolidation with runtime CPU detection, build/test disable flags, full LASX parity for all LSX functions, and 38 new LSX/LASX convert and planar row functions: - Update LoongArchCpuCaps() in cpu_id.cc to detect LSX and LASX via the unprivileged cpucfg instruction combined with getauxval(AT_HWCAP) OS support, and wire GetCpuFlags() for __loongarch__. - Support LIBYUV_DISABLE_LSX, LIBYUV_DISABLE_LASX, and LIBYUV_DISABLE_ASM at build time in cpu_support.h and CMakeLists.txt, and fix the __loongarch__ environment variable check in unit_test.cc. - Port all 24 LSX-only Rotate and Scale kernels to 256-bit LASX so every LSX function has a matching LASX implementation. - Move ScaleMaxSamples, ScaleSumSamples, and ScaleSamples from row_* to scale_* across all platforms and add LSX and LASX implementations in scale_la.cc. - Add 38 LSX and LASX row implementations (and _Any wrappers): SwapUVRow, ARGBCopyYToAlphaRow, HalfMergeUVRow, Convert16To8Row, Convert8To16Row, MultiplyRow_16, DivideRow_16, MergeUVRow_16, SplitUVRow_16, CopyRow_16, HalfRow, HalfRow_16, ARGBCopyAlphaRow, YUY2ToNVUVRow, AR64ToARGBRow, AB64ToARGBRow, ARGBToAR64Row, ARGBToAB64Row, Convert8To8Row, HalfRow_16To8, AR30ToAB30Row, ARGBToAR30Row, ABGRToAR30Row, AR30ToARGBRow, AR30ToABGRRow, AYUVToYRow, AYUVToUVRow, AYUVToVURow, DetileRow, DetileRow_16, MergeARGBRow, MergeXRGBRow, SplitARGBRow, SplitXRGBRow, RAWToRGBARow, RGB24MirrorRow, BlendPlaneRow, and InterpolateRow_16. Compare SIMD ISA per *_C: ------------------------------------------------------------------------ C Function x86 LoongArch ------------------------------------------------------------------------ ARGBDetectRow_C AVX2 LSX+LASX HammingDistance_C SSSE3+AVX2+AVX512BW+AVX512VNNI LSX+LASX HashDjb2_C SSE41+AVX2+AVX512BW LSX+LASX Ssim4x4Row_C SSE2+AVX2+AVX512BW LSX+LASX SsimWindowRow_C SSE2+AVX2+AVX512BW LSX+LASX SumSquareError_C SSE2+AVX2+AVX512BW+AVX512VNNI LSX+LASX ------------------------------------------------------------------------ Compare: 6 (100%) 6 (100%) 6 (100%) ------------------------------------------------------------------------ Convert SIMD ISA per *_C: ------------------------------------------------------------------------ C Function x86 LoongArch ------------------------------------------------------------------------ AB64ToARGBRow_C SSSE3+AVX2 LSX+LASX ABGRToAR30Row_C SSSE3+AVX2+AVX512BW LSX+LASX AR30ToAB30Row_C AVX2+AVX512BW LSX+LASX AR30ToABGRRow_C AVX2+AVX512BW LSX+LASX AR30ToARGBRow_C AVX2+AVX512BW LSX+LASX AR64ShuffleRow_C SSSE3+AVX2+AVX512BW LSX+LASX AR64ToAB64Row_C SSSE3+AVX2+AVX512BW LSX+LASX AR64ToARGBRow_C SSSE3+AVX2 LSX+LASX ARGB1555ToARGBRow_C AVX2+AVX512BW LSX+LASX ARGB1555ToUVMatrixRow_C AVX2 ARGB1555ToUVRow_C AVX2 LSX+LASX ARGB1555ToYMatrixRow_C AVX2 ARGB1555ToYRow_C AVX2 LSX+LASX ARGB4444ToARGBRow_C AVX2 LSX+LASX ARGB4444ToUVMatrixRow_C AVX2 ARGB4444ToUVRow_C AVX2 ARGB4444ToYMatrixRow_C AVX2 ARGB4444ToYRow_C AVX2 ARGBAddRow_C SSE2+AVX2 LSX+LASX ARGBAffineRow_C AVX2 ARGBAlphaBlendRow_C SSSE3+AVX2+AVX512BW LSX+LASX ARGBAttenuateRow_C SSSE3+AVX2+AVX512BW LSX+LASX ARGBBlendRow_C AVX2+AVX512BW LSX+LASX ARGBColorMatrixRow_C AVX2+AVX512VNNI LSX+LASX ARGBColorTableRow_C AVX2 ARGBCopyAlphaRow_C SSE2+AVX2 LSX+LASX ARGBCopyYToAlphaRow_C SSE2+AVX2+AVX512BW LSX+LASX ARGBExtractAlphaRow_C SSE2+AVX2+AVX512BW LSX+LASX ARGBGrayRow_C AVX2+AVX512BW+AVX512VNNI LSX+LASX ARGBLumaColorTableRow_C AVX2 ARGBMirrorRow_C SSE2+AVX2+AVX512BW LSX+LASX ARGBMultiplyRow_C SSE2+AVX2+AVX512BW LSX+LASX ARGBOrRow_C SSE2+AVX2+AVX512BW LSX+LASX ARGBPolynomialRow_C SSE2+AVX2 ARGBQuantizeRow_C AVX2 LSX+LASX ARGBSepiaRow_C AVX2+AVX512VNNI LSX+LASX ARGBSetRow_C AVX2 LSX+LASX ARGBShadeRow_C AVX2 LSX+LASX ARGBShuffleRow_C SSSE3+AVX2+AVX512BW LSX+LASX ARGBSubtractRow_C SSE2+AVX2 LSX+LASX ARGBToAB64Row_C SSSE3+AVX2 LSX+LASX ARGBToABGRRow_C SSSE3+AVX2+AVX512BW LSX+LASX ARGBToAR30Row_C SSSE3+AVX2+AVX512BW LSX+LASX ARGBToAR64Row_C SSSE3+AVX2 LSX+LASX ARGBToARGB1555Row_C AVX2 LSX+LASX ARGBToARGB4444Row_C AVX2 LSX+LASX ARGBToBGRARow_C SSSE3+AVX2+AVX512BW LSX+LASX ARGBToRAWRow_C SSSE3+AVX2+AVX512BW LSX+LASX ARGBToRGB24Row_C SSSE3+AVX2+AVX512BW LSX+LASX ARGBToRGB565DitherRow_C AVX2+AVX512BW LSX+LASX ARGBToRGB565Row_C AVX2+AVX512BW LSX+LASX ARGBToRGBARow_C SSSE3+AVX2+AVX512BW LSX+LASX ARGBToUV444MatrixRow_C SSSE3+AVX2+AVX512BW+AVX512VNNI+AVX512VNNIVBMI ARGBToUVMatrixRow_C SSSE3+AVX2+AVX512BW+AVX512VNNI+AVX512VNNIVBMI ARGBToYMatrixRow_C SSSE3+AVX2+AVX512BW+AVX512VNNI+AVX512VNNIVBMI LSX+LASX ARGBUnattenuateRow_C SSE2+AVX2 AYUVToUVRow_C AVX2+AVX512VBMI LSX+LASX AYUVToVURow_C AVX2+AVX512VBMI LSX+LASX AYUVToYRow_C AVX2+AVX512VBMI LSX+LASX BlendPlaneRow_C SSSE3+AVX2+AVX512BW LSX+LASX ByteToFloatRow_C AVX2 ComputeCumulativeSumRow_C AVX2 Convert16To8Row_C SSSE3+AVX2+AVX512BW LSX+LASX Convert8To16Row_C SSE2+AVX2+AVX512BW LSX+LASX Convert8To8Row_C AVX2+AVX512BW LSX+LASX CopyRow_16_C AVX2 LSX+LASX CopyRow_C ERMS+SSE2+AVX+AVX512BW LSX+LASX CumulativeSumToAverageRow_C AVX2 DetileRow_16_C AVX2 LSX+LASX DetileRow_C AVX2 LSX+LASX DetileSplitUVRow_C AVX2 DetileToYUY2_C AVX2 DivideRow_16_C AVX2+AVX512BW LSX+LASX GaussCol_C AVX2 GaussCol_F32_C AVX2 GaussRow_C AVX2 GaussRow_F32_C AVX2 HalfFloatRow_C F16C+AVX2+AVX512BW LSX+LASX HalfMergeUVRow_C SSSE3+AVX2+AVX512BW LSX+LASX HalfRow_16To8_C SSSE3+AVX2+AVX512BW LSX+LASX HalfRow_16_C AVX2 LSX+LASX HalfRow_C AVX2 LSX+LASX HalfWidthRow_16To8_C SSSE3+AVX2+AVX512BW HalfWidthRow_16To8_Odd_C AVX2 I210AlphaToARGBRow_C SSSE3+AVX2+AVX512BW I210ToAR30Row_C SSSE3+AVX2+AVX512BW I210ToARGBRow_C SSSE3+AVX2+AVX512BW I212ToAR30Row_C SSSE3+AVX2+AVX512BW I212ToARGBRow_C SSSE3+AVX2+AVX512BW I400ToARGBRow_C SSE2+AVX2+AVX512BW LSX+LASX I410AlphaToARGBRow_C SSSE3+AVX2+AVX512BW I410ToAR30Row_C SSSE3+AVX2+AVX512BW I410ToARGBRow_C SSSE3+AVX2+AVX512BW I422AlphaToARGBRow_C SSSE3+AVX2+AVX512BW LSX+LASX I422ToAR30Row_C SSSE3+AVX2+AVX512BW I422ToARGB1555Row_C SSSE3+AVX2+AVX512BW LSX+LASX I422ToARGB4444Row_C SSSE3+AVX2+AVX512BW LSX+LASX I422ToARGBRow_C SSSE3+AVX2+AVX512BW LSX+LASX I422ToRGB24Row_C SSSE3+AVX2+AVX512BW+AVX512VBMI LSX+LASX I422ToRGB565Row_C SSSE3+AVX2+AVX512BW LSX+LASX I422ToRGBARow_C SSSE3+AVX2+AVX512BW LSX+LASX I422ToUYVYRow_C SSE2+AVX2 LSX+LASX I422ToYUY2Row_C SSE2+AVX2 LSX+LASX I444AlphaToARGBRow_C SSSE3+AVX2+AVX512BW I444ToARGBRow_C SSSE3+AVX2+AVX512BW LSX+LASX I444ToRGB24Row_C SSSE3+AVX2+AVX512VBMI InterpolateRow_16_C AVX2+AVX512BW LSX+LASX InterpolateRow_C AVX2+AVX512BW LSX+LASX J400ToARGBRow_C AVX2+AVX512BW LSX+LASX MergeAR64Row_C AVX2 MergeARGB16To8Row_C AVX2 MergeARGBRow_C SSE2+AVX2 LSX+LASX MergeRGBRow_C AVX2+AVX512VBMI MergeUVRow_16_C AVX2+AVX512BW LSX+LASX MergeUVRow_C SSE2+AVX2+AVX512BW LSX+LASX MergeXR30Row_C AVX2 MergeXR64Row_C AVX2 MergeXRGB16To8Row_C AVX2 MergeXRGBRow_C SSE2+AVX2 LSX+LASX MirrorRow_16_C SSSE3+AVX2+AVX512BW LSX+LASX MirrorRow_C SSSE3+AVX2+AVX512BW LSX+LASX MirrorSplitUVRow_C SSSE3+AVX2+AVX512BW LSX+LASX MirrorUVRow_C SSSE3+AVX2+AVX512BW LSX+LASX MultiplyRow_16_C AVX2+AVX512BW LSX+LASX NV12ToARGBRow_C SSSE3+AVX2+AVX512BW LSX+LASX NV12ToRGB24Row_C SSSE3+AVX2+AVX512VBMI NV12ToRGB565Row_C SSSE3+AVX2 LSX+LASX NV21ToARGBRow_C SSSE3+AVX2+AVX512BW LSX+LASX NV21ToRGB24Row_C SSSE3+AVX2+AVX512VBMI NV21ToYUV24Row_C SSSE3+AVX2+AVX512VBMI P210ToAR30Row_C SSSE3+AVX2+AVX512BW P210ToARGBRow_C SSSE3+AVX2+AVX512BW P410ToAR30Row_C SSSE3+AVX2+AVX512BW P410ToARGBRow_C SSSE3+AVX2+AVX512BW RAWToARGBRow_C SSSE3+AVX2+AVX512BW LSX+LASX RAWToRGB24Row_C SSSE3+AVX2+AVX512BW+AVX512VBMI LSX+LASX RAWToRGBARow_C AVX2+AVX512VBMI LSX+LASX RAWToUVRow LSX+LASX RGB24MirrorRow_C SSSE3+AVX2+AVX512VBMI LSX+LASX RGB24ToARGBRow_C SSSE3+AVX2+AVX512BW LSX+LASX RGB24ToUVRow LSX+LASX RGB565ToARGBRow_C AVX2 LSX+LASX RGB565ToUVMatrixRow_C AVX2 RGB565ToUVRow_C AVX2 LSX+LASX RGB565ToYMatrixRow_C AVX2 RGB565ToYRow_C AVX2 LSX+LASX RGBAToARGBRow_C SSSE3+AVX2+AVX512BW LSX+LASX RGBColorTableRow_C AVX2 RGBToUV444MatrixRow_C SSSE3+AVX2+AVX512BW RGBToUVMatrixRow_C SSSE3+AVX2+AVX512BW RGBToYMatrixRow_C SSSE3+AVX2+AVX512BW LSX+LASX SetRow_C ERMS+AVX2 LSX+LASX SobelRow_C AVX2 LSX+LASX SobelToPlaneRow_C AVX2 LSX+LASX SobelXRow_C AVX2 SobelXYRow_C AVX2 LSX+LASX SobelYRow_C AVX2 SplitARGBRow_C SSE2+SSSE3+AVX2 LSX+LASX SplitRGBRow_C SSSE3+SSE41+AVX2 SplitUVRow_16_C AVX2 LSX+LASX SplitUVRow_C SSE2+AVX2+AVX512BW LSX+LASX SplitXRGBRow_C SSE2+SSSE3+AVX2 LSX+LASX SwapUVRow_C SSSE3+AVX2+AVX512BW LSX+LASX UYVYToARGBRow_C SSSE3+AVX2+AVX512BW LSX+LASX UYVYToUV422Row_C SSE2+AVX2+AVX512BW LSX+LASX UYVYToUVRow_C SSE2+AVX2+AVX512BW LSX+LASX UYVYToYRow_C SSE2+AVX2+AVX512BW LSX+LASX UnpackMT2T_C AVX2 YUY2ToARGBRow_C SSSE3+AVX2+AVX512BW LSX+LASX YUY2ToNVUVRow_C SSE2+AVX2 LSX+LASX YUY2ToUV422Row_C SSE2+AVX2+AVX512BW LSX+LASX YUY2ToUVRow_C SSE2+AVX2+AVX512BW LSX+LASX YUY2ToYRow_C SSE2+AVX2+AVX512BW LSX+LASX ------------------------------------------------------------------------ Convert: 173 (100%) 171 (99%) 116 (67%) ------------------------------------------------------------------------ Rotate SIMD ISA per *_C: ------------------------------------------------------------------------ C Function x86 LoongArch ------------------------------------------------------------------------ TransposeUVWx16_C AVX2+AVX512BW LSX+LASX TransposeUVWx8_C AVX2 LSX+LASX TransposeWx16_C AVX2+AVX512BW LSX+LASX TransposeWx4_32_C AVX2 LSX+LASX TransposeWx8_16_C AVX2 LSX+LASX TransposeWx8_C AVX2 LSX+LASX TransposeWxH_32_C AVX2 ------------------------------------------------------------------------ Rotate: 7 (100%) 7 (100%) 6 (86%) ------------------------------------------------------------------------ Scale SIMD ISA per *_C: ------------------------------------------------------------------------ C Function x86 LoongArch ------------------------------------------------------------------------ FixedDiv1_C X86 FixedDiv_C X86 ScaleARGBCols64_C ScaleARGBColsUp2_C AVX2 ScaleARGBCols_C AVX2 LSX+LASX ScaleARGBFilterCols64_C ScaleARGBFilterCols_C AVX2 LSX+LASX ScaleARGBRowDown2Box_C AVX2 LSX+LASX ScaleARGBRowDown2Linear_C AVX2 LSX+LASX ScaleARGBRowDown2_C AVX2 LSX+LASX ScaleARGBRowDownEvenBox_C AVX2 LSX+LASX ScaleARGBRowDownEven_C AVX2 LSX+LASX ScaleAddCols0_C ScaleAddCols1_16_C ScaleAddCols1_C AVX512BW ScaleAddCols2_16_C ScaleAddCols2_C AVX512BW ScaleAddRow_16_C ScaleAddRow_C SSE2+AVX2+AVX512BW LSX+LASX ScaleColsUp2_16_C ScaleColsUp2_C AVX2 ScaleCols_16_C ScaleCols_C ScaleFilterCols64_16_C ScaleFilterCols64_C ScaleFilterCols_16_C ScaleFilterCols_C AVX2 LSX+LASX ScaleMaxSamples_C AVX2 LSX+LASX ScaleRowDown2Box_16_C AVX2 ScaleRowDown2Box_C SSSE3+AVX2+AVX512BW LSX+LASX ScaleRowDown2Box_Odd_C SSSE3+AVX2+AVX512BW ScaleRowDown2Linear_16_C AVX2 ScaleRowDown2Linear_C SSSE3+AVX2+AVX512BW LSX+LASX ScaleRowDown2_16_C AVX2 ScaleRowDown2_C SSSE3+AVX2+AVX512BW LSX+LASX ScaleRowDown34_0_Box_16_C ScaleRowDown34_0_Box_C AVX2 LSX+LASX ScaleRowDown34_16_C ScaleRowDown34_1_Box_16_C ScaleRowDown34_1_Box_C AVX2 LSX+LASX ScaleRowDown34_C AVX2 LSX+LASX ScaleRowDown38_16_C ScaleRowDown38_2_Box_16_C ScaleRowDown38_2_Box_C AVX2 LSX+LASX ScaleRowDown38_3_Box_16_C ScaleRowDown38_3_Box_C AVX2 LSX+LASX ScaleRowDown38_C AVX2 LSX+LASX ScaleRowDown4Box_16_C ScaleRowDown4Box_C SSSE3+AVX2 LSX+LASX ScaleRowDown4_16_C ScaleRowDown4_C SSSE3+AVX2 LSX+LASX ScaleRowUp2_Bilinear_16_Any_C SSE2+AVX2 ScaleRowUp2_Bilinear_16_C SSE2+AVX2 ScaleRowUp2_Bilinear_Any_C SSE2+SSSE3+AVX2 ScaleRowUp2_Bilinear_C SSE2+SSSE3+AVX2 ScaleRowUp2_Linear_16_Any_C SSE2+AVX2 ScaleRowUp2_Linear_16_C SSE2+AVX2 ScaleRowUp2_Linear_Any_C SSE2+SSSE3+AVX2 ScaleRowUp2_Linear_C SSE2+SSSE3+AVX2 ScaleSamples_C AVX2 LSX+LASX ScaleSumSamples_C AVX2 LSX+LASX ScaleUVAddCols1_16_C ScaleUVAddCols1_C ScaleUVAddCols2_16_C ScaleUVAddCols2_C ScaleUVAddCols3_16_C ScaleUVAddCols3_C AVX2+AVX512VBMI LSX+LASX ScaleUVCols64_16_C ScaleUVCols64_C ScaleUVColsUp2_16_C ScaleUVColsUp2_C ScaleUVCols_16_C ScaleUVCols_C ScaleUVFilterCols64_16_C ScaleUVFilterCols64_C ScaleUVFilterCols_16_C ScaleUVFilterCols_C ScaleUVRowDown2Box_16_C ScaleUVRowDown2Box_C AVX2 ScaleUVRowDown2Linear_16_C ScaleUVRowDown2Linear_C AVX2 ScaleUVRowDown2_16_C ScaleUVRowDown2_C AVX2 ScaleUVRowDownEvenBox_16_C ScaleUVRowDownEvenBox_C ScaleUVRowDownEven_16_C ScaleUVRowDownEven_C AVX2 ScaleUVRowUp2_Bilinear_16_Any_C SSE41+AVX2 ScaleUVRowUp2_Bilinear_16_C SSE41+AVX2 ScaleUVRowUp2_Bilinear_Any_C SSSE3+AVX2 ScaleUVRowUp2_Bilinear_C SSSE3+AVX2 ScaleUVRowUp2_Linear_16_Any_C SSE41+AVX2 ScaleUVRowUp2_Linear_16_C SSE41+AVX2 ScaleUVRowUp2_Linear_Any_C SSSE3+AVX2 ScaleUVRowUp2_Linear_C SSSE3+AVX2 ------------------------------------------------------------------------ Scale: 95 (100%) 54 (57%) 24 (25%) ------------------------------------------------------------------------ Bug: 42280942 Change-Id: I663f277db5e8a6a73319df7e3153f9f78a147a7a Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8718861 Reviewed-by: richard winterton <rrwinterton@gmail.com> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
Frank Barchard
|
c6700f48dd |
[libyuv] Consolidate LoongArch LSX and LASX inline asm into *_la.cc
Merge 64-bit LoongArch LSX and LASX inline assembly implementations from rotate_lsx.cc, rotate_lasx.cc, row_lsx.cc, row_lasx.cc, and scale_lsx.cc into compare_la.cc, rotate_la.cc, row_la.cc, and scale_la.cc, and remove the obsolete loongson_intrinsics.h header. Because all LSX and LASX kernels use inline assembly rather than compiler intrinsics, separate libyuv_lsx/libyuv_lasx build targets and -mlsx/-mlasx compiler flags are no longer required. Guard HAS_*_LSX and HAS_*_LASX macros with defined(__loongarch_lp64) so runtime CPU dispatch enables LASX and LSX without requiring global -mlasx/-mlsx flags, and wrap every function and helper in *_la.cc with its corresponding #ifdef HAS_* guard. Also fix bugs found during LoongArch verification: - Replace illegal vldi/xvldi -1 immediate encodings (si13 = 0x1fff, which raises SIGILL at runtime) in ARGBDetectRow_LSX and ARGBDetectRow_LASX with vseqi.b/xvseqi.b -1. - Set simd_size to 16 for LSX and 32 for LASX in ComputeHammingDistance and ComputeSumSquareError. - Remove orphaned dead BGRAToUVRow, ABGRToUVRow, RGBAToUVRow, and XRGBToUVHelper functions in row_la.cc. - Remove dead alpha shifts in ARGB1555ToYRow/UVRow_LSX/LASX and hoist channel masks out of the loop in RGB565ToUVRow/ARGB1555ToUVRow. Compare SIMD ISA per *_C: ------------------------------------------------------------------------ C Function LSX LASX ------------------------------------------------------------------------ ARGBDetectRow_C LSX LASX HammingDistance_C LSX LASX HashDjb2_C LSX LASX Ssim4x4Row_C LSX LASX SsimWindowRow_C LSX LASX SumSquareError_C LSX LASX ------------------------------------------------------------------------ Compare: 6 (100%) 6 (100%) 6 (100%) ------------------------------------------------------------------------ Convert SIMD ISA per *_C: ------------------------------------------------------------------------ C Function LSX LASX ------------------------------------------------------------------------ AB64ToARGBRow_C - - ABGRToAR30Row_C - - AR30ToAB30Row_C - - AR30ToABGRRow_C - - AR30ToARGBRow_C - - AR64ShuffleRow_C LSX LASX AR64ToAB64Row_C LSX LASX AR64ToARGBRow_C - - ARGB1555ToARGBRow_C LSX LASX ARGB1555ToUVMatrixRow_C - - ARGB1555ToUVRow_C LSX LASX ARGB1555ToYMatrixRow_C - - ARGB1555ToYRow_C LSX LASX ARGB4444ToARGBRow_C LSX LASX ARGB4444ToUVMatrixRow_C - - ARGB4444ToUVRow_C - - ARGB4444ToYMatrixRow_C - - ARGB4444ToYRow_C - - ARGBAddRow_C LSX LASX ARGBAffineRow_C - - ARGBAlphaBlendRow_C LSX LASX ARGBAttenuateRow_C LSX LASX ARGBBlendRow_C LSX LASX ARGBColorMatrixRow_C LSX LASX ARGBColorTableRow_C - - ARGBCopyAlphaRow_C - - ARGBCopyYToAlphaRow_C - - ARGBExtractAlphaRow_C LSX LASX ARGBGrayRow_C LSX LASX ARGBLumaColorTableRow_C - - ARGBMirrorRow_C LSX LASX ARGBMultiplyRow_C LSX LASX ARGBOrRow_C LSX LASX ARGBPolynomialRow_C - - ARGBQuantizeRow_C LSX LASX ARGBSepiaRow_C LSX LASX ARGBSetRow_C LSX LASX ARGBShadeRow_C LSX LASX ARGBShuffleRow_C LSX LASX ARGBSubtractRow_C LSX LASX ARGBToAB64Row_C - - ARGBToABGRRow_C LSX LASX ARGBToAR30Row_C - - ARGBToAR64Row_C - - ARGBToARGB1555Row_C LSX LASX ARGBToARGB4444Row_C LSX LASX ARGBToBGRARow_C LSX LASX ARGBToRAWRow_C LSX LASX ARGBToRGB24Row_C LSX LASX ARGBToRGB565DitherRow_C LSX LASX ARGBToRGB565Row_C LSX LASX ARGBToRGBARow_C LSX LASX ARGBToUV444MatrixRow_C - - ARGBToUVMatrixRow_C - - ARGBToYMatrixRow_C LSX LASX ARGBUnattenuateRow_C - - AYUVToUVRow_C - - AYUVToVURow_C - - AYUVToYRow_C - - BlendPlaneRow_C - - ByteToFloatRow_C - - ComputeCumulativeSumRow_C - - Convert16To8Row_C - - Convert8To16Row_C - - Convert8To8Row_C - - CopyRow_16_C - - CopyRow_C LSX LASX CumulativeSumToAverageRow_C - - DetileRow_16_C - - DetileRow_C - - DetileSplitUVRow_C - - DetileToYUY2_C - - DivideRow_16_C - - GaussCol_C - - GaussCol_F32_C - - GaussRow_C - - GaussRow_F32_C - - HalfFloatRow_C LSX LASX HalfMergeUVRow_C - - HalfRow_16To8_C - - HalfRow_16_C - - HalfRow_C - - HalfWidthRow_16To8_C - - HalfWidthRow_16To8_Odd_C - - I210AlphaToARGBRow_C - - I210ToAR30Row_C - - I210ToARGBRow_C - - I212ToAR30Row_C - - I212ToARGBRow_C - - I400ToARGBRow_C LSX LASX I410AlphaToARGBRow_C - - I410ToAR30Row_C - - I410ToARGBRow_C - - I422AlphaToARGBRow_C LSX LASX I422ToAR30Row_C - - I422ToARGB1555Row_C LSX LASX I422ToARGB4444Row_C LSX LASX I422ToARGBRow_C LSX LASX I422ToRGB24Row_C LSX LASX I422ToRGB565Row_C LSX LASX I422ToRGBARow_C LSX LASX I422ToUYVYRow_C LSX LASX I422ToYUY2Row_C LSX LASX I444AlphaToARGBRow_C - - I444ToARGBRow_C LSX LASX I444ToRGB24Row_C - - InterpolateRow_16_C - - InterpolateRow_C LSX LASX J400ToARGBRow_C LSX LASX MergeAR64Row_C - - MergeARGB16To8Row_C - - MergeARGBRow_C - - MergeRGBRow_C - - MergeUVRow_16_C - - MergeUVRow_C LSX LASX MergeXR30Row_C - - MergeXR64Row_C - - MergeXRGB16To8Row_C - - MergeXRGBRow_C - - MirrorRow_16_C LSX LASX MirrorRow_C LSX LASX MirrorSplitUVRow_C LSX LASX MirrorUVRow_C LSX LASX MultiplyRow_16_C - - NV12ToARGBRow_C LSX LASX NV12ToRGB24Row_C - - NV12ToRGB565Row_C LSX LASX NV21ToARGBRow_C LSX LASX NV21ToRGB24Row_C - - NV21ToYUV24Row_C - - P210ToAR30Row_C - - P210ToARGBRow_C - - P410ToAR30Row_C - - P410ToARGBRow_C - - RAWToARGBRow_C LSX LASX RAWToRGB24Row_C LSX LASX RAWToRGBARow_C - - RGB24MirrorRow_C - - RGB24ToARGBRow_C LSX LASX RGB565ToARGBRow_C LSX LASX RGB565ToUVMatrixRow_C - - RGB565ToUVRow_C LSX LASX RGB565ToYMatrixRow_C - - RGB565ToYRow_C LSX LASX RGBAToARGBRow_C LSX LASX RGBColorTableRow_C - - RGBToUV444MatrixRow_C - - RGBToUVMatrixRow_C - - RGBToYMatrixRow_C LSX LASX ScaleMaxSamples_C - - ScaleSamples_C - - ScaleSumSamples_C - - SetRow_C LSX LASX SobelRow_C LSX LASX SobelToPlaneRow_C LSX LASX SobelXRow_C - - SobelXYRow_C LSX LASX SobelYRow_C - - SplitARGBRow_C - - SplitRGBRow_C - - SplitUVRow_16_C - - SplitUVRow_C LSX LASX SplitXRGBRow_C - - SwapUVRow_C - - UYVYToARGBRow_C LSX LASX UYVYToUV422Row_C LSX LASX UYVYToUVRow_C LSX LASX UYVYToYRow_C LSX LASX UnpackMT2T_C - - YUY2ToARGBRow_C LSX LASX YUY2ToNVUVRow_C - - YUY2ToUV422Row_C LSX LASX YUY2ToUVRow_C LSX LASX YUY2ToYRow_C LSX LASX ------------------------------------------------------------------------ Convert: 174 (100%) 76 (44%) 76 (44%) ------------------------------------------------------------------------ Rotate SIMD ISA per *_C: ------------------------------------------------------------------------ C Function LSX LASX ------------------------------------------------------------------------ TransposeUVWx16_C - LASX TransposeUVWx8_C LSX - TransposeWx16_C - LASX TransposeWx4_32_C LSX LASX TransposeWx8_16_C LSX LASX TransposeWx8_C LSX - TransposeWxH_32_C - - ------------------------------------------------------------------------ Rotate: 7 (100%) 4 (57%) 4 (57%) ------------------------------------------------------------------------ Scale SIMD ISA per *_C: ------------------------------------------------------------------------ C Function LSX LASX ------------------------------------------------------------------------ FixedDiv1_C - - FixedDiv_C - - ScaleARGBCols64_C - - ScaleARGBColsUp2_C - - ScaleARGBCols_C LSX - ScaleARGBFilterCols64_C - - ScaleARGBFilterCols_C LSX - ScaleARGBRowDown2Box_C LSX - ScaleARGBRowDown2Linear_C LSX - ScaleARGBRowDown2_C LSX - ScaleARGBRowDownEvenBox_C LSX - ScaleARGBRowDownEven_C LSX - ScaleAddCols0_C - - ScaleAddCols1_16_C - - ScaleAddCols1_C - - ScaleAddCols2_16_C - - ScaleAddCols2_C - - ScaleAddRow_16_C - - ScaleAddRow_C LSX - ScaleColsUp2_16_C - - ScaleColsUp2_C - - ScaleCols_16_C - - ScaleCols_C - - ScaleFilterCols64_16_C - - ScaleFilterCols64_C - - ScaleFilterCols_16_C - - ScaleFilterCols_C LSX - ScaleRowDown2Box_16_C - - ScaleRowDown2Box_C LSX - ScaleRowDown2Box_Odd_C - - ScaleRowDown2Linear_16_C - - ScaleRowDown2Linear_C LSX - ScaleRowDown2_16_C - - ScaleRowDown2_C LSX - ScaleRowDown34_0_Box_16_C - - ScaleRowDown34_0_Box_C LSX - ScaleRowDown34_16_C - - ScaleRowDown34_1_Box_16_C - - ScaleRowDown34_1_Box_C LSX - ScaleRowDown34_C LSX - ScaleRowDown38_16_C - - ScaleRowDown38_2_Box_16_C - - ScaleRowDown38_2_Box_C LSX - ScaleRowDown38_3_Box_16_C - - ScaleRowDown38_3_Box_C LSX - ScaleRowDown38_C LSX - ScaleRowDown4Box_16_C - - ScaleRowDown4Box_C LSX - ScaleRowDown4_16_C - - ScaleRowDown4_C LSX - ScaleRowUp2_Bilinear_16_Any_C - - ScaleRowUp2_Bilinear_16_C - - ScaleRowUp2_Bilinear_Any_C - - ScaleRowUp2_Bilinear_C - - ScaleRowUp2_Linear_16_Any_C - - ScaleRowUp2_Linear_16_C - - ScaleRowUp2_Linear_Any_C - - ScaleRowUp2_Linear_C - - ScaleUVAddCols1_16_C - - ScaleUVAddCols1_C - - ScaleUVAddCols2_16_C - - ScaleUVAddCols2_C - - ScaleUVAddCols3_16_C - - ScaleUVAddCols3_C LSX - ScaleUVCols64_16_C - - ScaleUVCols64_C - - ScaleUVColsUp2_16_C - - ScaleUVColsUp2_C - - ScaleUVCols_16_C - - ScaleUVCols_C - - ScaleUVFilterCols64_16_C - - ScaleUVFilterCols64_C - - ScaleUVFilterCols_16_C - - ScaleUVFilterCols_C - - ScaleUVRowDown2Box_16_C - - ScaleUVRowDown2Box_C - - ScaleUVRowDown2Linear_16_C - - ScaleUVRowDown2Linear_C - - ScaleUVRowDown2_16_C - - ScaleUVRowDown2_C - - ScaleUVRowDownEvenBox_16_C - - ScaleUVRowDownEvenBox_C - - ScaleUVRowDownEven_16_C - - ScaleUVRowDownEven_C - - ScaleUVRowUp2_Bilinear_16_Any_C - - ScaleUVRowUp2_Bilinear_16_C - - ScaleUVRowUp2_Bilinear_Any_C - - ScaleUVRowUp2_Bilinear_C - - ScaleUVRowUp2_Linear_16_Any_C - - ScaleUVRowUp2_Linear_16_C - - ScaleUVRowUp2_Linear_Any_C - - ScaleUVRowUp2_Linear_C - - ------------------------------------------------------------------------ Scale: 92 (100%) 21 (23%) 0 (0%) ------------------------------------------------------------------------ Test: qemu-loongarch64 -cpu la464 ./libyuv_unittest Bug: none Change-Id: I7990bc8c2524dc9434441b5379cf839012e73d5f Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8701144 Reviewed-by: richard winterton <rrwinterton@gmail.com> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
Frank Barchard
|
d088b8f2bd |
[libyuv] Add GCC 15 SME support, SVE2/SME ports, and next 10 SIMD fixes
1. Enable ARMv9 SVE2 and SME compilation with GCC 15 and Clang across
Android.bp, BUILD.gn, CMakeLists.txt, and linux.mk using portable
LIBYUV_SVE_ATTR, LIBYUV_SME_ATTR, and STREAMING_COMPATIBLE macros.
2. Port 46 SVE2 kernels to streaming-compatible SVE_SC functions in
row_sve.h and expose _SME entry points for processors with SME but
no non-streaming SVE2. Upgrade 15 1x SVE2 kernels to 2x unrolled,
use full-vector st1b in ARGBToRGB24Row/ARGBToRAWRow_SVE2, contiguous
ld1h/st1h in HalfFloatRow_SVE_SC, and frecpe+frecps reciprocal in
ARGBAlphaBlendRow_SVE_SC.
3. Add missing kCpuHasNEON runtime checks in planar_test.cc and define
missing HAS_*_NEON macros in row.h for existing NEON kernels.
4. Handle odd width directly in AYUVToUVRow_SVE_SC and
AYUVToVURow_SVE_SC and remove redundant width alignment guards in
AYUVToNV12 and AYUVToNV21.
5. Implement TransposeWx8_NEON for AArch64 in rotate_neon64.cc so
TransposePlane and RotatePlane90/270 cascade 16x -> 8x -> C tail.
6. Add _Any_ wrappers for 9 ARGB/float filter functions (ARGBGrayTo,
ARGBGray, ARGBSepia, ARGBColorMatrix, ARGBQuantize, ARGBShade,
GaussRow_F32, GaussCol_F32, and ARGBLumaColorTable).
7. Add HalfRow and HalfRow_16 wrappers for AVX2, NEON, SVE2, and SME
delegating to InterpolateRow{,_16} with fraction 0x80.
8. Add YUY2ToUV422Row_AVX512BW and UYVYToUV422Row_AVX512BW in
row_gcc.cc and row_win.cc.
9. Add HalfWidthRow_16To8_Odd_NEON in row_any.cc and wire into
I41xToI420 and Px10ToI420 for odd chroma widths.
10. Fix inline asm clobbers and vzeroupper placement in
TransposeWx32_16_AVX512BW and TransposeWx16_32_AVX512BW, and
re-enable 36 passing unit tests.
Test: libyuv_unittest
Bug: 579659306
Change-Id: I6b808f8b03ed424d3af4bb7941c34af05c1cb774
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8701301
Commit-Queue: Frank Barchard <fbarchard@google.com>
Reviewed-by: richard winterton <rrwinterton@gmail.com>
|
||
|
Frank Barchard
|
c15967905d |
[libyuv] Implement 8, 16, and 32-bit transpose using x86 ACEv1
Add kCpuHasACEV1 detection to CpuId using CPUID leaves 7.1, 0x1D.2, and 0x24.1 along with XCR0 state validation (0x001600e6). Add CLANG_HAS_ACEV1, GCC_HAS_ACEV1, and LIBYUV_TARGET_ACEV1 macros for 64-bit x86 builds on Clang >= 11 and GCC >= 11. Implement TransposeWxH_ACEV1, TransposeUVWxH_ACEV1, TransposeWxH_16_ACEV1, and TransposeWxH_32_ACEV1 in rotate_gcc.cc using Palette 2 16x16 dword tiles (ldtilecfg, tilemovcol, tilemovrow, and tilerelease) with k-register tail masking for arbitrary widths and heights. Bug: none Test: sde64 -future-ag -- ./libyuv_unittest --gtest_filter=*Transpose* Change-Id: I0091ac00b173a59183f84ff9555920caef965293 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8704075 Commit-Queue: Frank Barchard <fbarchard@google.com> Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Frank Barchard
|
38ca05f1eb |
[libyuv] Fix color matrix, blend, NV layout and null pointer bugs
Correctness fixes:
- U010ToAB30 and U210ToAB30 swap U and V but passed
kYuv2020Constants; use kYvu2020Constants as I010ToAB30 does.
- H420ToAR30 passed kYvuH709Constants without swapping U and V,
so red and blue were swapped; use kYuvH709Constants.
- I420Blend with negative height flipped Y but not U and V.
- BlendPlaneRow_WASMSIMD processes 16 pixels per loop but was
dispatched for widths that are a multiple of 8.
- I420Blend used ScaleRowDown2Box_RVV for odd widths, reading 1
byte past the row. Add ScaleRowDown2Box_Odd_RVV.
- ConvertFromI420 to NV12/NV21/NV16 with odd width used a UV
stride of width, overlapping UV rows and writing 1 byte past
the plane. Use ((width + 1) / 2) * 2.
- ConvertToI420 (NV16/NV24) and ConvertToARGB (NV12/NV21/NV16/
NV24) computed the Y plane size with the aligned width; the Y
stride is src_width. NV24 UV stride is src_width * 2.
- I420Rect and I420Interpolate skipped a UV row for negative odd
heights.
- DetilePlane and DetilePlane_16 accepted tile_height 0, reading
past the source.
- Add null checks to NV12ToNV21, IxxxToPxxx, I422ToI210,
I422ToNV21, YUY2ToI420, UYVYToI420, AYUVToNV12, AYUVToNV21,
DetileSplitUVPlane and DetileToYUY2, and avoid pointer
arithmetic on optional NULL src_y/dst_y with negative height.
MT2TToP010 now returns -1 for NULL src_y with non-NULL dst_y.
- Replace unaligned uint32_t stores in row_common.cc with memcpy.
- Add LIBYUV_TARGET_AVX512VNNI to HammingDistance_AVX512VNNI so
GCC accepts the k1 clobber on non-AVX512 builds.
- Check kCpuHasWASMSIMD instead of kCpuHasWASM in scale.cc and
scale_uv.cc.
- Avoid the 16.16 fixed point NEON/LSX/RVV column scalers for
src_width >= 32768, matching the existing AVX2 guard.
Performance:
- NV24ToI420 uses InterpolateRow, ScaleUVRowDown2Linear and
SplitUVRow instead of a scalar loop (bit exact).
- Add HalfMergeUVRow_Any_NEON and MirrorSplitUVRow_Any_{NEON,
LSX,LASX,WASMSIMD} so non-multiple-of-16 widths use SIMD.
- Enable BlendPlaneRow_SSSE3/AVX2 for row_win builds.
- Add missing LSX/LASX/RVV/SME dispatch for YUY2ToY,
I422ToYUY2/UYVY, ARGBToY/UVMatrix, RAWToNVMatrix, AR64Shuffle,
ARGBSobelize and Convert16To8Row in I010ToNV12, and coalesce
rows in ARGBToI400Matrix.
Cleanup:
- UYVYToYRow_Any_AVX2 uses the same UVSHIFT/SBPP as YUY2.
- Add #ifdef HAS_* guards in compare_gcc.cc and row_gcc.cc.
- Remove unused clobbers and dead cnth/cmp/subs instructions in
row_neon.cc, row_neon64.cc, row_sve.h, row_sve.cc and
row_sme.cc, and order clobbers "memory", "cc" first.
- Re-enable the I420Blend_Any unit test.
Test: libyuv_unittest (x86-64 Release, rowwin, ASan, UBSan, MSan;
Arm64 SME, SVE2 and NEON on Samsung S26)
Bug: 579659306
Change-Id: Ic1afddbf961f7ef7deeba124a55907a31e09822d
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8702355
Reviewed-by: richard winterton <rrwinterton@gmail.com>
|
||
|
Frank Barchard
|
79ca9187e5 |
[libyuv] Fix x86 SIMD OOB, math, alignment, MSan & GCC bugs
Address x86 SIMD correctness, sanitizer, alignment, and compiler guard
bugs across row_gcc.cc, row_win.cc, row_any.cc, row_common.cc, and
caller dispatchers:
- MSan & row_win.cc: Replace _mm512_ternarylogic_epi32 in row_win.cc
(ARGBGrayRow_AVX512VNNI, ARGBToUV444MatrixRow_AVX512VNNIVBMI, and
I422ToAR30Row_AVX512BW) with bitwise intrinsics modeled bit-by-bit by
LLVM MSan, document the MSan intrinsic rules in GEMINI.md, and fix
standalone MSan ELF PT_INTERP in BUILD.gn. Add missing USE_ROW_WIN
DivideRow_16_AVX2, HalfMergeUVRow_AVX2, and Convert8To8Row_AVX2.
- Out-of-bounds & AVX-512 opmask tails: Fix RGBToUVMatrixRow_Any_SSSE3
MASK from 7 to 15 (24-byte overread), guard UnpackMT2T_C and
UnpackMT2T_AVX2 against partial < 80-byte tail blocks, add inline
k1-k7 opmask tails to all 17 remaining AVX-512 row kernels (removing
all *_Any_AVX512* wrappers), and enable opmask-tailed
NV21ToYUV24Row_AVX512VBMI.
- Math, saturation & pointer bugs: Fix MirrorRow_16_AVX512BW 64-bit
width sign extension, ARGBPolynomialRow_AVX2 vpackssdw clamping,
CumulativeSumToAverageRow_AVX2 small-area (area <= 1) handling and
vdivss precision, HalfWidthRow_16To8_{SSSE3,AVX2,AVX512BW} unsigned
pair summation for samples >= 0x8000, HalfRow_16To8_AVX512BW
non-power-of-two scale multiplication, DivideRow_16 SIMD
scale == 65536 (16-bit depth) truncation, ARGBLumaColorTableRow_AVX2
unsigned luma weight accumulation, Merge{ARGB,XRGB}16To8Row_AVX2
depth == 8 clamping, custom ArgbConstants::kAddUV loading, 16-bit YUV
unsigned shifts/ORs, and null-pointer checks in NV16ToI420/NV24ToI420.
- Alignment (#GP), ISA (#UD), UBSan & 32-bit x86: Use unaligned loads
for ArgbConstants, YuvConstants, and ARGBPolynomialRow_SSE2 float
coefficients; remove AVX2 instructions from HalfFloat{,1}Row_F16C;
fix Merge{AR,XR}64Row_AVX2 depth == 16 shift UB and ByteToFloatRow_C /
TestByteToFloat float alignment UB; and fix 32-bit -m32 -fPIC ASan
inline asm register pressure and operand suffixes.
- GCC & compiler guards: Enable GCC_HAS_AVX512 for GCC >= 9 on x86_64
with per-function target attributes, gate HAS_*_AVX2 and HAS_*_AVX512*
macros on compiler version macros, add missing #ifdef HAS_* guards in
row_gcc.cc, fix _Any / IS_ALIGNED step mismatches, wire missing
dispatchers (ARGBSobelize, SplitRotateUV180, HalfMergeUVPlane,
ARGBPolynomial), and silence GCC 15 -Werror=stringop-overflow=.
Convert SIMD ISA per *_C (* = uncalled in library):
------------------------------------------------------------------------
C Function arm x86 riscv hexagon wasm loong
------------------------------------------------------------------------
AB64ToARGBRow_C NEON AVX2 RVV HVX - -
ABGRToAR30Row_C NEON AVX512BW - - - -
AR30ToAB30Row_C NEON AVX512BW - - - -
AR30ToABGRRow_C NEON AVX512BW - - - -
AR30ToARGBRow_C NEON AVX512BW - - - -
AR64ShuffleRow_C NEON AVX512BW RVV - RELAXED LASX
AR64ToAB64Row_C NEON AVX512BW RVV HVX RELAXED LASX
AR64ToARGBRow_C NEON AVX2 RVV HVX - -
ARGB1555ToARGBRow_C SVE2 AVX512BW - - SIMD LASX
ARGB1555ToUVMatrixRow_C NEON AVX2 - - SIMD -
ARGB1555ToUVRow_C * NEON AVX2 - - - LASX
ARGB1555ToYMatrixRow_C NEON AVX2 - - SIMD -
ARGB1555ToYRow_C * NEON AVX2 - - - LASX
ARGB4444ToARGBRow_C NEON AVX2 - - SIMD LASX
ARGB4444ToUVMatrixRow_C NEON AVX2 - - SIMD -
ARGB4444ToUVRow_C * NEON AVX2 - - - -
ARGB4444ToYMatrixRow_C NEON AVX2 - - SIMD -
ARGB4444ToYRow_C * NEON AVX2 - - - -
ARGBAddRow_C NEON AVX2 - HVX - LASX
ARGBAffineRow_C * NEON AVX2 - - - -
ARGBAlphaBlendRow_C SME AVX512BW RVV HVX SIMD LASX
ARGBAttenuateRow_C NEON AVX512BW RVV HVX - LASX
ARGBBlendRow_C NEON AVX512BW RVV HVX RELAXED LASX
ARGBColorMatrixRow_C I8MM AVX512VNNI - - - LASX
ARGBColorTableRow_C NEON AVX2 - - - -
ARGBCopyAlphaRow_C NEON AVX2 - HVX - -
ARGBCopyYToAlphaRow_C NEON AVX512BW RVV HVX RELAXED -
ARGBExtractAlphaRow_C NEON AVX512BW RVV HVX SIMD LASX
ARGBGrayRow_C DOTPROD AVX512VNNI - - - LASX
ARGBLumaColorTableRow_C NEON AVX2 - - - -
ARGBMirrorRow_C NEON AVX512BW RVV HVX SIMD LASX
ARGBMultiplyRow_C SME AVX512BW RVV HVX - LASX
ARGBOrRow_C SME AVX512BW RVV HVX SIMD LASX
ARGBPolynomialRow_C NEON AVX2 - - - -
ARGBQuantizeRow_C NEON AVX2 - - - LASX
ARGBSepiaRow_C DOTPROD AVX512VNNI - - - LASX
ARGBSetRow_C NEON AVX2 RVV HVX - LASX
ARGBShadeRow_C NEON AVX2 - - - LASX
ARGBShuffleRow_C NEON AVX512BW RVV - RELAXED LASX
ARGBSubtractRow_C NEON AVX2 - HVX - LASX
ARGBToAB64Row_C NEON AVX2 RVV HVX - -
ARGBToABGRRow_C NEON AVX512BW RVV HVX RELAXED LASX
ARGBToAR30Row_C NEON AVX512BW - - - -
ARGBToAR64Row_C NEON AVX2 RVV HVX - -
ARGBToARGB1555Row_C NEON AVX2 - - SIMD LASX
ARGBToARGB4444Row_C NEON AVX2 - - SIMD LASX
ARGBToBGRARow_C NEON AVX512BW RVV HVX RELAXED LASX
ARGBToRAWRow_C SVE2 AVX512BW RVV - - LASX
ARGBToRGB24Row_C SVE2 AVX512BW RVV - - LASX
ARGBToRGB565DitherRow_C SVE2 AVX512BW - - SIMD LASX
ARGBToRGB565Row_C SVE2 AVX512BW - - SIMD LASX
ARGBToRGBARow_C NEON AVX512BW RVV HVX RELAXED LASX
ARGBToUV444MatrixRow_C I8MM AVX512VNNI RVV HVX RELAXED -
ARGBToUVMatrixRow_C SME AVX512VNNI RVV HVX SIMD -
ARGBToYMatrixRow_C SME AVX512VNNI RVV HVX SIMD LASX
ARGBUnattenuateRow_C NEON AVX2 - - - -
AYUVToUVRow_C SVE2 AVX512VBMI - - - -
AYUVToVURow_C SVE2 AVX512VBMI - - - -
AYUVToYRow_C NEON AVX512VBMI - - - -
BlendPlaneRow_C NEON AVX512BW RVV HVX SIMD -
ByteToFloatRow_C NEON AVX2 - - - -
ComputeCumulativeSumRow_C NEON AVX2 - - - -
Convert16To8Row_C SME AVX512BW RVV HVX - -
Convert8To16Row_C SME AVX512BW RVV HVX SIMD -
Convert8To8Row_C SME AVX512BW - HVX - -
CopyRow_16_C NEON AVX2 - HVX - -
CopyRow_C SME AVX512BW RVV HVX - LASX
CumulativeSumToAverageRow_C NEON AVX2 - - - -
DetileRow_16_C NEON AVX2 - - - -
DetileRow_C NEON AVX2 - - - -
DetileSplitUVRow_C NEON AVX2 - - - -
DetileToYUY2_C NEON AVX2 - - - -
DivideRow_16_C SVE2 AVX512BW - HVX - -
GaussCol_C * NEON AVX2 - - - -
GaussCol_F32_C NEON AVX2 - - - -
GaussRow_C * NEON AVX2 - - - -
GaussRow_F32_C NEON AVX2 - - - -
HalfFloatRow_C SVE2 AVX512BW - - - LASX
HalfMergeUVRow_C SVE2 AVX512BW - - - -
HalfRow_16To8_C SME AVX512BW RVV HVX - -
HalfRow_16_C - AVX2 - - - -
HalfRow_C - AVX2 - - - -
HalfWidthRow_16To8_C SME AVX512BW RVV - - -
HalfWidthRow_16To8_Odd_C - AVX2 - - - -
I210AlphaToARGBRow_C SME AVX512BW - - - -
I210ToAR30Row_C SME AVX512BW - - - -
I210ToARGBRow_C SME AVX512BW - - - -
I212ToAR30Row_C SME AVX512BW - - - -
I212ToARGBRow_C SME AVX512BW - - - -
I400ToARGBRow_C SME AVX512BW RVV HVX - LASX
I410AlphaToARGBRow_C SME AVX512BW - - - -
I410ToAR30Row_C SME AVX512BW - - - -
I410ToARGBRow_C SME AVX512BW - - - -
I422AlphaToARGBRow_C SME AVX512BW RVV HVX - LASX
I422ToAR30Row_C SME AVX512BW RVV - SIMD -
I422ToARGB1555Row_C SME AVX512BW - - SIMD LASX
I422ToARGB4444Row_C SME AVX512BW - - SIMD LASX
I422ToARGBRow_C SME AVX512BW RVV HVX SIMD LASX
I422ToRGB24Row_C SME AVX512VBMI RVV - RELAXED LASX
I422ToRGB565Row_C SME AVX512BW - - SIMD LASX
I422ToRGBARow_C SME AVX512BW RVV HVX - LASX
I422ToUYVYRow_C NEON AVX2 - - - LASX
I422ToYUY2Row_C NEON AVX2 - - - LASX
I444AlphaToARGBRow_C SME AVX512BW RVV HVX - -
I444ToARGBRow_C SME AVX512BW RVV HVX - LASX
I444ToRGB24Row_C SME AVX512VBMI RVV - - -
InterpolateRow_16_C SME AVX512BW - HVX SIMD -
InterpolateRow_C SME AVX512BW RVV HVX SIMD LASX
J400ToARGBRow_C NEON AVX512BW RVV HVX RELAXED LASX
MergeAR64Row_C NEON AVX2 - - - -
MergeARGB16To8Row_C NEON AVX2 - - - -
MergeARGBRow_C NEON AVX2 RVV HVX - -
MergeRGBRow_C NEON AVX512VBMI RVV - - -
MergeUVRow_16_C SME AVX512BW - HVX - -
MergeUVRow_C SME AVX512BW RVV HVX SIMD LASX
MergeXR30Row_C NEON AVX512BW - - - -
MergeXR64Row_C NEON AVX2 - - - -
MergeXRGB16To8Row_C NEON AVX2 - - - -
MergeXRGBRow_C NEON AVX2 RVV HVX - -
MirrorRow_16_C NEON AVX512BW RVV HVX RELAXED LASX
MirrorRow_C NEON AVX512BW - HVX RELAXED LASX
MirrorSplitUVRow_C NEON AVX512BW - HVX RELAXED LASX
MirrorUVRow_C NEON AVX512BW RVV HVX RELAXED LASX
MultiplyRow_16_C SME AVX512BW RVV HVX SIMD -
NV12ToARGBRow_C SME AVX512BW RVV HVX SIMD LASX
NV12ToRGB24Row_C SME AVX512VBMI RVV - - -
NV12ToRGB565Row_C NEON AVX2 - - SIMD LASX
NV21ToARGBRow_C SME AVX512BW RVV HVX SIMD LASX
NV21ToRGB24Row_C SME AVX512VBMI RVV - - -
NV21ToYUV24Row_C NEON AVX512VBMI - - - -
P210ToAR30Row_C SME AVX512BW - - - -
P210ToARGBRow_C SME AVX512BW - - - -
P410ToAR30Row_C SME AVX512BW - - - -
P410ToARGBRow_C SME AVX512BW - - - -
RAWToARGBRow_C SVE2 AVX512BW RVV - RELAXED LASX
RAWToRGB24Row_C SVE2 AVX512VBMI RVV - RELAXED LASX
RAWToRGBARow_C SVE2 AVX512VBMI RVV - - -
RGB24MirrorRow_C NEON AVX512VBMI - - RELAXED -
RGB24ToARGBRow_C SVE2 AVX512BW RVV - RELAXED LASX
RGB565ToARGBRow_C NEON AVX2 - - SIMD LASX
RGB565ToUVMatrixRow_C NEON AVX2 - - SIMD -
RGB565ToUVRow_C * NEON AVX2 - - - LASX
RGB565ToYMatrixRow_C NEON AVX2 - - SIMD -
RGB565ToYRow_C * NEON AVX2 - - - LASX
RGBAToARGBRow_C NEON AVX512BW RVV HVX RELAXED LASX
RGBColorTableRow_C NEON AVX2 - - - -
RGBToUV444MatrixRow_C NEON AVX512BW RVV - RELAXED -
RGBToUVMatrixRow_C NEON AVX512BW RVV - RELAXED -
RGBToYMatrixRow_C DOTPROD AVX512BW RVV - RELAXED LASX
ScaleMaxSamples_C * NEON AVX2 - - - -
ScaleSamples_C * NEON AVX2 - - - -
ScaleSumSamples_C * NEON AVX2 - - - -
SetRow_C NEON AVX2 - HVX - LASX
SobelRow_C NEON AVX2 - - - LASX
SobelToPlaneRow_C NEON AVX2 - - - LASX
SobelXRow_C NEON AVX2 - - - -
SobelXYRow_C NEON AVX2 - - - LASX
SobelYRow_C NEON AVX2 - - - -
SplitARGBRow_C NEON AVX2 RVV HVX - -
SplitRGBRow_C NEON AVX2 RVV - - -
SplitUVRow_16_C NEON AVX2 - HVX - -
SplitUVRow_C SVE2 AVX512BW RVV HVX - LASX
SplitXRGBRow_C NEON AVX2 RVV HVX - -
SwapUVRow_C NEON AVX512BW RVV HVX SIMD -
UYVYToARGBRow_C SME AVX512BW - - - LASX
UYVYToUV422Row_C NEON AVX2 - HVX - LASX
UYVYToUVRow_C SVE2 AVX512BW - HVX - LASX
UYVYToYRow_C SVE2 AVX512BW - HVX - LASX
UnpackMT2T_C NEON AVX2 - - - -
YUY2ToARGBRow_C SME AVX512BW - - - LASX
YUY2ToNVUVRow_C NEON AVX2 - - - -
YUY2ToUV422Row_C NEON AVX2 - HVX - LASX
YUY2ToUVRow_C SVE2 AVX512BW - HVX - LASX
YUY2ToYRow_C SVE2 AVX512BW - HVX - LASX
------------------------------------------------------------------------
SIMD (174 convert) 171 98% 174 100% 65 37% 64 37% 57 33% 76 44%
------------------------------------------------------------------------
Rotate SIMD ISA per *_C (* = uncalled in library):
------------------------------------------------------------------------
C Function arm x86 riscv hexagon wasm loong
------------------------------------------------------------------------
TransposeUVWx16_C - AVX512BW - - SIMD LASX
TransposeUVWx8_C NEON AVX512BW - HVX SIMD LASX
TransposeWx16_C NEON AVX512BW - - SIMD LASX
TransposeWx4_32_C * SVE2 AVX512BW RVV HVX SIMD LASX
TransposeWx8_16_C SVE2 AVX512BW RVV HVX SIMD LASX
TransposeWx8_C NEON AVX512BW - HVX SIMD LASX
TransposeWxH_32_C NEON AVX2 RVV HVX - -
------------------------------------------------------------------------
SIMD (7 rotate) 6 86% 7 100% 3 43% 5 71% 6 86% 6 86%
------------------------------------------------------------------------
Compare SIMD ISA per *_C:
------------------------------------------------------------------------
C Function arm x86 riscv hexagon wasm loong
------------------------------------------------------------------------
ARGBDetectRow_C - AVX2 - - - -
HammingDistance_C DOTPROD AVX512BW - HVX SIMD -
HashDjb2_C NEON AVX512BW - HVX SIMD -
Ssim4x4Row_C SME AVX512BW RVV HVX SIMD LASX
SsimWindowRow_C SME AVX512BW RVV - - -
SumSquareError_C DOTPROD AVX512VNNI - HVX SIMD -
------------------------------------------------------------------------
SIMD (6 compare) 5 83% 6 100% 2 33% 4 67% 4 67% 1 17%
------------------------------------------------------------------------
Scale SIMD ISA per *_C:
------------------------------------------------------------------------
C Function arm x86 riscv hexagon wasm loong
------------------------------------------------------------------------
FixedDiv1_C - X86 - - - -
FixedDiv_C - X86 - - - -
ScaleARGBCols64_C - - - - - -
ScaleARGBColsUp2_C - AVX2 - - SIMD -
ScaleARGBCols_C NEON AVX2 - - SIMD LSX
ScaleARGBFilterCols64_C - - - - - -
ScaleARGBFilterCols_C NEON AVX2 RVV - SIMD LSX
ScaleARGBRowDown2Box_C SME AVX2 RVV HVX SIMD LSX
ScaleARGBRowDown2Linear_C SME AVX2 RVV HVX SIMD LSX
ScaleARGBRowDown2_C SME AVX2 RVV HVX SIMD LSX
ScaleARGBRowDownEvenBox_C NEON AVX2 RVV - SIMD LSX
ScaleARGBRowDownEven_C NEON AVX2 - - SIMD LSX
ScaleAddCols0_C - - - - - -
ScaleAddCols1_16_C - - - - - -
ScaleAddCols1_C - AVX512BW - - - -
ScaleAddCols2_16_C - - - - - -
ScaleAddCols2_C - AVX512BW - - - -
ScaleAddRow_16_C - - - - - -
ScaleAddRow_C NEON AVX512BW RVV HVX SIMD LSX
ScaleColsUp2_16_C - - - - - -
ScaleColsUp2_C - AVX2 - - - -
ScaleCols_16_C - - - - - -
ScaleCols_C - - - - - -
ScaleFilterCols64_16_C - - - - - -
ScaleFilterCols64_C - - - - - -
ScaleFilterCols_16_C - - - - - -
ScaleFilterCols_C NEON AVX2 - - SIMD LSX
ScaleRowDown2Box_16_C SME AVX2 - HVX - -
ScaleRowDown2Box_C SME AVX512BW RVV HVX SIMD LSX
ScaleRowDown2Box_Odd_C - AVX512BW - HVX - -
ScaleRowDown2Linear_16_C SME AVX2 - HVX - -
ScaleRowDown2Linear_C SME AVX512BW RVV HVX SIMD LSX
ScaleRowDown2_16_C SME AVX2 - HVX - -
ScaleRowDown2_C SME AVX512BW RVV HVX SIMD LSX
ScaleRowDown34_0_Box_16_C - - - - - -
ScaleRowDown34_0_Box_C NEON AVX2 RVV - RELAXED LSX
ScaleRowDown34_16_C - - - - - -
ScaleRowDown34_1_Box_16_C - - - - - -
ScaleRowDown34_1_Box_C NEON AVX2 RVV - RELAXED LSX
ScaleRowDown34_C NEON AVX2 RVV - RELAXED LSX
ScaleRowDown38_16_C - - - - - -
ScaleRowDown38_2_Box_16_C - - - - - -
ScaleRowDown38_2_Box_C NEON AVX2 RVV - RELAXED LSX
ScaleRowDown38_3_Box_16_C - - - - - -
ScaleRowDown38_3_Box_C NEON AVX2 RVV - RELAXED LSX
ScaleRowDown38_C NEON AVX2 RVV - RELAXED LSX
ScaleRowDown4Box_16_C - - - - - -
ScaleRowDown4Box_C NEON AVX2 RVV HVX SIMD LSX
ScaleRowDown4_16_C - - - - - -
ScaleRowDown4_C NEON AVX2 RVV HVX SIMD LSX
ScaleRowUp2_Bilinear_16_Any_C NEON AVX2 - - - -
ScaleRowUp2_Bilinear_16_C NEON AVX2 - - - -
ScaleRowUp2_Bilinear_Any_C NEON AVX2 RVV - - -
ScaleRowUp2_Bilinear_C NEON AVX2 RVV - - -
ScaleRowUp2_Linear_16_Any_C NEON AVX2 - - - -
ScaleRowUp2_Linear_16_C NEON AVX2 - - - -
ScaleRowUp2_Linear_Any_C NEON AVX2 RVV - - -
ScaleRowUp2_Linear_C NEON AVX2 RVV - - -
ScaleUVAddCols1_16_C - - - - - -
ScaleUVAddCols1_C - - - - - -
ScaleUVAddCols2_16_C - - - - - -
ScaleUVAddCols2_C - - - - - -
ScaleUVAddCols3_16_C - - - - - -
ScaleUVAddCols3_C SME AVX512VBMI RVV HVX SIMD LSX
ScaleUVCols64_16_C - - - - - -
ScaleUVCols64_C - - - - - -
ScaleUVColsUp2_16_C - - - - - -
ScaleUVColsUp2_C - - - - - -
ScaleUVCols_16_C - - - - - -
ScaleUVCols_C - - - - - -
ScaleUVFilterCols64_16_C - - - - - -
ScaleUVFilterCols64_C - - - - - -
ScaleUVFilterCols_16_C - - - - - -
ScaleUVFilterCols_C - - - - - -
ScaleUVRowDown2Box_16_C - - - - - -
ScaleUVRowDown2Box_C SME AVX2 RVV HVX - -
ScaleUVRowDown2Linear_16_C - - - - - -
ScaleUVRowDown2Linear_C SME AVX2 RVV HVX - -
ScaleUVRowDown2_16_C - - - - - -
ScaleUVRowDown2_C SME AVX2 RVV HVX - -
ScaleUVRowDownEvenBox_16_C - - - - - -
ScaleUVRowDownEvenBox_C - - - - - -
ScaleUVRowDownEven_16_C - - - - - -
ScaleUVRowDownEven_C NEON AVX2 RVV - - -
ScaleUVRowUp2_Bilinear_16_Any_C NEON AVX2 - - - -
ScaleUVRowUp2_Bilinear_16_C NEON AVX2 - - - -
ScaleUVRowUp2_Bilinear_Any_C NEON AVX2 RVV - - -
ScaleUVRowUp2_Bilinear_C NEON AVX2 RVV - - -
ScaleUVRowUp2_Linear_16_Any_C NEON AVX2 - - - -
ScaleUVRowUp2_Linear_16_C NEON AVX2 - - - -
ScaleUVRowUp2_Linear_Any_C NEON AVX2 RVV - - -
ScaleUVRowUp2_Linear_C NEON AVX2 RVV - - -
------------------------------------------------------------------------
SIMD (92 scale) 44 48% 51 55% 30 33% 17 18% 22 24% 21 23%
------------------------------------------------------------------------
Test: libyuv_unittest (Release, row_win, x86_32, Sanitizers, GCC 15)
Bug: 388428508, 578615529, 579659306
Change-Id: Id4eaf5f7fd85c75a60984a3b1559bfdb07b21afc
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8686033
Reviewed-by: richard winterton <rrwinterton@gmail.com>
|
||
|
Frank Barchard
|
bb310b4e7e |
[libyuv] Add 100% SIMD coverage across all compare functions
Implement SIMD kernels across all 6 compare row functions
(ARGBDetectRow, HammingDistance, HashDjb2, Ssim4x4Row, SsimWindowRow,
and SumSquareError) so that ARM (32-bit and 64-bit NEON), x86 (AVX2 and
AVX-512), RISC-V (RVV), Hexagon (HVX), WebAssembly (WASMSIMD and
RelaxedSIMD), and LoongArch (LSX and LASX) each reach 6 / 6 (100%)
coverage:
* HammingDistance:
- Add HammingDistance_AVX512VNNI in compare_gcc.cc and compare_win.cc
using vpshufb nibble popcount lookup tables accumulated into 32-bit
counters via vpdpbusd with k1 masked tail loads (1.39x faster than
HammingDistance_AVX512BW).
- Update HammingDistance_WASMSIMD in compare_wasm.cc to accumulate
per-byte popcounts into 32-bit counters using
wasm_i32x4_relaxed_dot_i8x16_i7x16_add when __wasm_relaxed_simd__ is
enabled.
- Add HammingDistance_RVV, HammingDistance_LSX, and
HammingDistance_LASX.
* SumSquareError:
- Add SumSquareError_RVV (vwsubu.vv + vwmacc.vv), SumSquareError_LSX,
and SumSquareError_LASX (xvabsd.bu + xvmulwev/wod.h.bu).
* HashDjb2:
- Add 32-bit ARM HashDjb2_NEON in compare_neon.cc, HashDjb2_RVV (with
arbitrary tail length support), HashDjb2_LSX, and HashDjb2_LASX.
* SsimWindowRow:
- Add SsimWindowRow_NEON (32-bit and 64-bit), SsimWindowRow_WASMSIMD,
SsimWindowRow_HVX, SsimWindowRow_LSX, and SsimWindowRow_LASX.
* ARGBDetectRow:
- Add ARGBDetectRow_NEON (32-bit and 64-bit), ARGBDetectRow_RVV,
ARGBDetectRow_WASMSIMD, ARGBDetectRow_HVX, ARGBDetectRow_LSX, and
ARGBDetectRow_LASX, and add 1..7 pixel SIMD tail handling to
ARGBDetectRow_AVX2 in compare_win.cc.
* Remainder & Test Coverage:
- Handle tail counts/widths in HashDjb2, ComputeHammingDistance,
ComputeSumSquareError, and CalcFrameSsim via padded SIMD buffers
when a SIMD kernel is active, using zero-numerator padding in
SsimWindowRow_Rem to avoid float cancellation.
- Add TestARGBDetectAnyWidth, TestCompareAnyCount, and
TestSsimAnyWidth in compare_test.cc.
Compare SIMD ISA per *_C:
------------------------------------------------------------------------
C Function arm x86 riscv hexagon wasm loong
------------------------------------------------------------------------
ARGBDetectRow_C NEON AVX2 RVV HVX SIMD LASX
HammingDistance_C DOTPROD AVX512VNNI RVV HVX RELAXED LASX
HashDjb2_C NEON AVX512BW RVV HVX SIMD LASX
Ssim4x4Row_C SME AVX512BW RVV HVX SIMD LASX
SsimWindowRow_C SME AVX512BW RVV HVX SIMD LASX
SumSquareError_C DOTPROD AVX512VNNI RVV HVX SIMD LASX
------------------------------------------------------------------------
SIMD (6 compare) 6 100% 6 100% 6 100% 6 100% 6 100% 6 100%
------------------------------------------------------------------------
Test: libyuv_unittest --gtest_filter=*Compare*
Bug: none
Change-Id: I72a026ca5d9b3da373e32c01396dddba8c494089
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8692272
Reviewed-by: richard winterton <rrwinterton@gmail.com>
|
||
|
Frank Barchard
|
59fb9e18a8 |
[libyuv] Fix MSan AVX-512 intrinsics and standalone dynamic linker
Clang MemorySanitizer builds define LIBYUV_ENABLE_ROWWIN and compile
row_win.cc and scale_win.cc instead of inline assembly. Replace three
families of AVX-512 intrinsics that LLVM MSan does not model bitwise or
lane-by-lane:
- _mm512_ternarylogic_epi32 (@llvm.x86.avx512.pternlog.d.512) in
ARGBGrayRow_AVX512VNNI, ARGBToUV444MatrixRow_AVX512VNNIVBMI, and
I422ToAR30Row_AVX512BW, which raises false use-of-uninitialized-value
errors whenever any bit of any operand is uninitialized (such as the
unused alpha channel of an RGBX buffer). Use _mm512_and_si512,
_mm512_andnot_si512, _mm512_or_si512, _mm512_xor_si512, and
_mm512_shuffle_i64x2, which LLVM MSan models bit-by-bit.
- _mm512_permutexvar_epi{8,16,32,64} and
_mm512_mask_permutexvar_epi{8,32} (@llvm.x86.avx512.permvar.*) across
row_win.cc and ScaleUVAddCols3_AVX512VBMI in scale_win.cc, which OR
input shadows without permuting them and poison initialized output
lanes whenever any lane of the input vector is uninitialized. Route
through PermutexvarEpi* and MaskPermutexvarEpi* helpers that use
portable lane-by-lane C loops under LIBYUV_MSAN and the native
AVX-512 intrinsics otherwise.
- _mm512_cvttps_epi32 (@llvm.x86.avx512.mask.cvttps2dq.512) in
ARGBAlphaBlendRow_AVX512BW. Route through CvttpsEpi32, which uses a
portable lane-by-lane C loop under LIBYUV_MSAN and _mm512_cvttps_epi32
otherwise.
In standalone MSan builds (!build_with_chromium &&
prebuilt_instrumented_libraries_available && is_msan), override the
relative ELF PT_INTERP from //third_party/instrumented_libs with the
absolute path to the instrumented ld-linux-x86-64.so.2 so test and
utility binaries launch cleanly from any working directory (fixing the
LUCI Linux MSan test runner FileNotFoundError).
Add MSan regression tests with poisoned RGBX alpha channels, poisoned
tail padding, and poisoned destination buffers in planar_test.cc, and
document the AVX-512 MSan intrinsic guidelines in GEMINI.md.
Test: out/msan/libyuv_unittest
Bug: libyuv:388428508
Change-Id: I6a002393db2d2e629cd7a3cf2bf4d98ee9daf79d
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8686036
Reviewed-by: richard winterton <rrwinterton@gmail.com>
|
||
|
George Steed
|
0b8dd23180 |
AArch64: Avoid LD4 in ARGBToARGB4444Row_NEON
The existing ARGBToARGB4444Row_NEON implementation uses half-vector LD4s
to load the four channels before using the ARGBTOARGB4444 macro to
re-combine them. LD4 is known to be slow on some micro-architectures so
should be avoided. Use LD2 instead to deinterleave the even and odd
bytes directly, then combine them with SRI.
Speedup versus the existing Neon implementation (higher is better):
Cortex-A510: 1.822x
Cortex-A55: 1.213x
Cortex-A720: 2.302x
Cortex-X4: 1.092x
C1-Nano: 1.505x
C1-Pro: 1.714x
C1-Ultra: 1.302x
Change-Id: If7d90f2d34d3ef146e5f41716a60d6f6436bde1d
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8671363
Reviewed-by: Frank Barchard <fbarchard@google.com>
Reviewed-by: Justin Green <greenjustin@google.com>
|
||
|
Frank Barchard
|
8d440c181c |
[libyuv] Upgrade x86 SIMD to AVX2/AVX-512 with 32-bit & row_win parity
Upgrade all remaining SSE2 and SSSE3 max-ISA kernels across row, scale,
rotate, and compare to AVX2 while removing superseded pre-AVX2 versions.
Implement AVX2 for ARGBDetectRow, HalfRow, HalfRow_16,
HalfWidthRow_16To8_Odd, AYUVTo{Y,UV,VU}Row, AR30To{ARGB,ABGR,AB30}Row,
and TransposeWx8_16, and port all remaining Arm NEON (or higher) kernels
to AVX2 (TransposeWxH_32, ScaleRowDown2*_16, ScaleUVRowDown2*,
ScaleUVRowDownEven, ScaleColsUp2, GaussCol*, Scale*Samples, and
UnpackMT2T).
Port 32-bit x86 (-m32 -fPIC) inline assembly across row_gcc.cc,
rotate_gcc.cc, and compare_gcc.cc (including 5-GPR PIC support for
I422ToAR30Row and I444/I422/I210/I410AlphaToARGBRow) so x86_32 matches
x64 SIMD coverage across all 187 supported row/rotate/compare kernels.
Handle 1..7 remainder elements directly in inline assembly without
calling *_C in ARGBDetectRow_AVX2, ARGBSepiaRow_AVX2,
ARGBColorMatrixRow_AVX2, ARGBQuantizeRow_AVX2, ARGBShadeRow_AVX2,
ARGBAffineRow_AVX2, GaussRow*_AVX2, GaussCol*_AVX2, and
*ColorTableRow_AVX2. Fix unsigned 16-bit packing in
ScaleRowUp2_{Linear,Bilinear}_16_SSE2.
Add 64-bit AVX-512 (AVX512BW, AVX512VBMI, and AVX512VNNI, including
SumSquareError_AVX512VNNI) implementations in *_gcc.cc and *_win.cc
wherever Arm64 exceeds NEON, and handle arbitrary tail widths via
opmasks in Mirror*_AVX512BW without _Any_ wrappers. Bring row_win.cc,
scale_win.cc, rotate_win.cc, and compare_win.cc C++ intrinsics (and
64-bit VISUALC_HAS_AVX2) to 100% parity with *_gcc.cc for MSVC and
MemorySanitizer.
Wire YUY2ToUVRow_HVX and UYVYToUVRow_HVX in YUY2ToI420 and UYVYToI420,
implement all 13 missing ScaleUV*_16_C kernels so UVScale_16 supports
downscaling and general upscaling, make Ssim8x8, TransposeWxH,
TransposeUVWxH, and TransposeWxH_16 static internal helpers, clean up
HAS_* guards, enable direct row unittests so all 279 *_C functions have
active test coverage, and add Intel AMX-TF32 (kCpuHasAMXTF32, CPUID
leaf 7 subleaf 1 EAX bit 4, LIBYUV_DISABLE_AMXTF32) detection.
Mark 16-bit RGB formats (RGB565, ARGB1555, and ARGB4444) and legacy
fixed-matrix 16-bit RGB-to-YUV row functions as deprecated in headers
and GEMINI.md, using two-step ARGB wrapper row functions in
row_common.cc.
AMD Zen 6 Benchmark (1280 width, 10000 iterations):
----------------------------------------------------------------
Function Before (ms) After (ms) Improvement
----------------------------------------------------------------
ARGB1555ToARGBRow 0.83 0.49 1.69x
ARGB1555ToUVRow 16.00 3.21 4.99x
ARGB1555ToYRow 7.82 1.38 5.66x
ARGB4444ToUVRow 15.04 2.58 5.83x
ARGB4444ToYRow 7.42 1.07 6.94x
ARGBAffineRow 3.51 2.75 1.28x
ARGBColorMatrixRow 4.09 1.13 3.63x
ARGBColorTableRow 11.21 8.93 1.26x
ARGBLumaColorTableRow 10.01 10.21 0.98x
ARGBQuantizeRow 2.23 1.21 1.84x
ARGBSepiaRow 3.29 0.81 4.04x
ARGBSetRow 0.21 0.21 1.02x
ARGBShadeRow 1.66 0.87 1.91x
ARGBToRGB565DitherRow 1.22 0.50 2.43x
ARGBToRGB565Row 1.09 0.44 2.47x
ByteToFloatRow 0.88 0.41 2.14x
Convert8To8Row 0.21 0.11 1.88x
CopyRow_16 0.11 0.11 0.97x
DetileRow 0.21 0.12 1.75x
DetileRow_16 0.23 0.14 1.67x
DetileSplitUVRow 0.29 0.21 1.38x
DivideRow_16 0.14 0.11 1.30x
GaussRow 3.29 1.27 2.59x
GaussRow_F32 1.93 0.89 2.17x
HalfFloatRow 1.16 0.74 1.57x
I210AlphaToARGBRow 1.83 0.92 2.00x
I210ToAR30Row 2.21 0.96 2.30x
I210ToARGBRow 1.75 0.87 2.00x
I212ToAR30Row 2.21 0.96 2.31x
I212ToARGBRow 1.75 0.87 2.01x
I410AlphaToARGBRow 1.73 0.86 2.00x
I410ToAR30Row 2.04 0.89 2.28x
I410ToARGBRow 1.64 0.82 2.00x
I422ToARGB1555Row 2.94 2.27 1.30x
I422ToARGB4444Row 2.94 2.27 1.29x
I422ToRGB565Row 2.63 1.27 2.07x
I422ToRGBARow 1.51 0.82 1.83x
I444AlphaToARGBRow 1.52 0.87 1.73x
I444ToARGBRow 1.48 0.81 1.82x
I444ToRGB24Row 2.13 0.72 2.98x
MergeUVRow_16 0.87 0.32 2.74x
P210ToAR30Row 1.77 0.70 2.54x
P210ToARGBRow 1.38 0.62 2.24x
P410ToAR30Row 1.94 0.85 2.28x
P410ToARGBRow 1.45 0.77 1.90x
RGB565ToUVRow 15.84 2.88 5.49x
RGB565ToYRow 7.82 1.22 6.42x
RGBColorTableRow 8.42 6.95 1.21x
SetRow 0.11 0.11 1.04x
SobelRow 0.80 0.43 1.87x
SobelToPlaneRow 0.27 0.12 2.25x
SobelXRow 1.58 0.85 1.85x
SobelXYRow 0.80 0.55 1.44x
SobelYRow 1.58 0.85 1.85x
UYVYToARGBRow 1.28 0.65 1.96x
UYVYToUVRow 0.38 0.21 1.78x
UYVYToYRow 0.21 0.11 1.87x
YUY2ToARGBRow 1.28 0.65 1.96x
YUY2ToUVRow 0.40 0.21 1.87x
YUY2ToYRow 0.21 0.11 1.88x
----------------------------------------------------------------
Row/Rotate/Compare SIMD ISA per *_C (* = uncalled in library):
------------------------------------------------------------------------
C Function arm64 x86_32 x64 row_win
------------------------------------------------------------------------
AB64ToARGBRow_C NEON AVX2 AVX2 AVX2
ABGRToAR30Row_C NEON AVX2 AVX512BW AVX512BW
AR30ToAB30Row_C NEON AVX2 AVX512BW AVX512BW
AR30ToABGRRow_C NEON AVX2 AVX512BW AVX512BW
AR30ToARGBRow_C NEON AVX2 AVX512BW AVX512BW
AR64ShuffleRow_C NEON AVX2 AVX512BW AVX512BW
AR64ToAB64Row_C NEON AVX2 AVX512BW AVX512BW
AR64ToARGBRow_C NEON AVX2 AVX2 AVX2
ARGB1555ToARGBRow_C SVE2 AVX2 AVX512BW AVX512BW
ARGB1555ToUVMatrixRow_C NEON AVX2 AVX2 AVX2
ARGB1555ToUVRow_C * NEON AVX2 AVX2 AVX2
ARGB1555ToYMatrixRow_C NEON AVX2 AVX2 AVX2
ARGB1555ToYRow_C * NEON AVX2 AVX2 AVX2
ARGB4444ToARGBRow_C NEON AVX2 AVX2 AVX2
ARGB4444ToUVMatrixRow_C NEON AVX2 AVX2 AVX2
ARGB4444ToUVRow_C * NEON AVX2 AVX2 AVX2
ARGB4444ToYMatrixRow_C NEON AVX2 AVX2 AVX2
ARGB4444ToYRow_C * NEON AVX2 AVX2 AVX2
ARGBAddRow_C NEON AVX2 AVX2 AVX2
ARGBAffineRow_C * NEON AVX2 AVX2 AVX2
ARGBAlphaBlendRow_C SME AVX2 AVX512BW AVX512BW
ARGBAttenuateRow_C NEON AVX2 AVX512BW AVX512BW
ARGBBlendRow_C NEON AVX2 AVX512BW AVX512BW
ARGBColorMatrixRow_C I8MM AVX2 AVX512VNNI AVX512VNNI
ARGBColorTableRow_C NEON AVX2 AVX2 AVX2
ARGBCopyAlphaRow_C NEON AVX2 AVX2 AVX2
ARGBCopyYToAlphaRow_C NEON AVX2 AVX512BW AVX512BW
ARGBDetectRow_C - AVX2 AVX2 AVX2
ARGBExtractAlphaRow_C NEON AVX2 AVX512BW AVX512BW
ARGBGrayRow_C DOTPROD AVX2 AVX512VNNI AVX512VNNI
ARGBLumaColorTableRow_C NEON AVX2 AVX2 AVX2
ARGBMirrorRow_C NEON AVX2 AVX512BW AVX512BW
ARGBMultiplyRow_C SME AVX2 AVX512BW AVX512BW
ARGBOrRow_C SME AVX2 AVX512BW AVX512BW
ARGBPolynomialRow_C NEON AVX2 AVX2 AVX2
ARGBQuantizeRow_C NEON AVX2 AVX2 AVX2
ARGBSepiaRow_C DOTPROD AVX2 AVX512VNNI AVX512VNNI
ARGBSetRow_C NEON AVX2 AVX2 AVX2
ARGBShadeRow_C NEON AVX2 AVX2 AVX2
ARGBShuffleRow_C NEON AVX2 AVX512BW AVX512BW
ARGBSubtractRow_C NEON AVX2 AVX2 AVX2
ARGBToAB64Row_C NEON AVX2 AVX2 AVX2
ARGBToABGRRow_C NEON AVX2 AVX512BW AVX512BW
ARGBToAR30Row_C NEON AVX2 AVX512BW AVX512BW
ARGBToAR64Row_C NEON AVX2 AVX2 AVX2
ARGBToARGB1555Row_C NEON AVX2 AVX2 AVX2
ARGBToARGB4444Row_C NEON AVX2 AVX2 AVX2
ARGBToBGRARow_C NEON AVX2 AVX512BW AVX512BW
ARGBToRAWRow_C SVE2 AVX2 AVX512BW AVX512BW
ARGBToRGB24Row_C SVE2 AVX2 AVX512BW AVX512BW
ARGBToRGB565DitherRow_C SVE2 AVX2 AVX512BW AVX512BW
ARGBToRGB565Row_C SVE2 AVX2 AVX512BW AVX512BW
ARGBToRGBARow_C NEON AVX2 AVX512BW AVX512BW
ARGBToUV444MatrixRow_C I8MM AVX2 AVX512VNNI AVX512VNNI
ARGBToUVMatrixRow_C SME AVX2 AVX512VNNI AVX512VNNI
ARGBToYMatrixRow_C SME AVX2 AVX512VNNI AVX512VNNI
ARGBUnattenuateRow_C NEON AVX2 AVX2 AVX2
AYUVToUVRow_C SVE2 AVX2 AVX512VBMI AVX512VBMI
AYUVToVURow_C SVE2 AVX2 AVX512VBMI AVX512VBMI
AYUVToYRow_C NEON AVX2 AVX512VBMI AVX512VBMI
BlendPlaneRow_C NEON AVX2 AVX512BW AVX512BW
ByteToFloatRow_C NEON AVX2 AVX2 AVX2
ComputeCumulativeSumRow_C NEON AVX2 AVX2 AVX2
Convert16To8Row_C SME AVX2 AVX512BW AVX512BW
Convert8To16Row_C SME AVX2 AVX512BW AVX512BW
Convert8To8Row_C SME AVX2 AVX512BW AVX512BW
CopyRow_16_C NEON AVX2 AVX2 AVX2
CopyRow_C SME AVX AVX512BW AVX512BW
CumulativeSumToAverageRow_C NEON AVX2 AVX2 AVX2
DetileRow_16_C NEON AVX2 AVX2 AVX2
DetileRow_C NEON AVX2 AVX2 AVX2
DetileSplitUVRow_C NEON AVX2 AVX2 AVX2
DetileToYUY2_C NEON AVX2 AVX2 AVX2
DivideRow_16_C SVE2 AVX2 AVX512BW AVX512BW
GaussCol_C * NEON AVX2 AVX2 AVX2
GaussCol_F32_C NEON AVX2 AVX2 AVX2
GaussRow_C * NEON AVX2 AVX2 AVX2
GaussRow_F32_C NEON AVX2 AVX2 AVX2
HalfFloatRow_C SVE2 AVX2 AVX512BW AVX512BW
HalfMergeUVRow_C SVE2 AVX2 AVX512BW AVX512BW
HalfRow_16To8_C SME AVX2 AVX512BW AVX512BW
HalfRow_16_C - AVX2 AVX2 AVX2
HalfRow_C - AVX2 AVX2 AVX2
HalfWidthRow_16To8_C SME AVX2 AVX512BW AVX512BW
HalfWidthRow_16To8_Odd_C - AVX2 AVX2 AVX2
HammingDistance_C DOTPROD AVX2 AVX512BW AVX512BW
HashDjb2_C NEON AVX2 AVX512BW AVX512BW
I210AlphaToARGBRow_C SME AVX2 AVX512BW AVX512BW
I210ToAR30Row_C SME AVX2 AVX512BW AVX512BW
I210ToARGBRow_C SME AVX2 AVX512BW AVX512BW
I212ToAR30Row_C SME AVX2 AVX512BW AVX512BW
I212ToARGBRow_C SME AVX2 AVX512BW AVX512BW
I400ToARGBRow_C SME AVX2 AVX512BW AVX512BW
I410AlphaToARGBRow_C SME AVX2 AVX512BW AVX512BW
I410ToAR30Row_C SME AVX2 AVX512BW AVX512BW
I410ToARGBRow_C SME AVX2 AVX512BW AVX512BW
I422AlphaToARGBRow_C SME AVX2 AVX512BW AVX512BW
I422ToAR30Row_C SME AVX2 AVX512BW AVX512BW
I422ToARGB1555Row_C SME AVX2 AVX512BW AVX512BW
I422ToARGB4444Row_C SME AVX2 AVX512BW AVX512BW
I422ToARGBRow_C SME AVX2 AVX512BW AVX512BW
I422ToRGB24Row_C SME AVX2 AVX512VBMI AVX512VBMI
I422ToRGB565Row_C SME AVX2 AVX512BW AVX512BW
I422ToRGBARow_C SME AVX2 AVX512BW AVX512BW
I422ToUYVYRow_C NEON AVX2 AVX2 AVX2
I422ToYUY2Row_C NEON AVX2 AVX2 AVX2
I444AlphaToARGBRow_C SME AVX2 AVX512BW AVX512BW
I444ToARGBRow_C SME AVX2 AVX512BW AVX512BW
I444ToRGB24Row_C SME AVX2 AVX512VBMI AVX512VBMI
InterpolateRow_16_C SME AVX2 AVX512BW AVX512BW
InterpolateRow_C SME AVX2 AVX512BW AVX512BW
J400ToARGBRow_C NEON AVX2 AVX512BW AVX512BW
MergeAR64Row_C NEON AVX2 AVX2 AVX2
MergeARGB16To8Row_C NEON AVX2 AVX2 AVX2
MergeARGBRow_C NEON AVX2 AVX2 AVX2
MergeRGBRow_C NEON AVX2 AVX512VBMI AVX512VBMI
MergeUVRow_16_C SME AVX2 AVX512BW AVX512BW
MergeUVRow_C SME AVX2 AVX512BW AVX512BW
MergeXR30Row_C NEON AVX2 AVX512BW AVX512BW
MergeXR64Row_C NEON AVX2 AVX2 AVX2
MergeXRGB16To8Row_C NEON AVX2 AVX2 AVX2
MergeXRGBRow_C NEON AVX2 AVX2 AVX2
MirrorRow_16_C NEON AVX2 AVX512BW AVX512BW
MirrorRow_C NEON AVX2 AVX512BW AVX512BW
MirrorSplitUVRow_C NEON AVX2 AVX512BW AVX512BW
MirrorUVRow_C NEON AVX2 AVX512BW AVX512BW
MultiplyRow_16_C SME AVX2 AVX512BW AVX512BW
NV12ToARGBRow_C SME AVX2 AVX512BW AVX512BW
NV12ToRGB24Row_C SME AVX2 AVX512VBMI AVX512VBMI
NV12ToRGB565Row_C NEON AVX2 AVX2 AVX2
NV21ToARGBRow_C SME AVX2 AVX512BW AVX512BW
NV21ToRGB24Row_C SME AVX2 AVX512VBMI AVX512VBMI
NV21ToYUV24Row_C NEON AVX2 AVX2 AVX2
P210ToAR30Row_C SME AVX2 AVX512BW AVX512BW
P210ToARGBRow_C SME AVX2 AVX512BW AVX512BW
P410ToAR30Row_C SME AVX2 AVX512BW AVX512BW
P410ToARGBRow_C SME AVX2 AVX512BW AVX512BW
RAWToARGBRow_C SVE2 AVX2 AVX512BW AVX512BW
RAWToRGB24Row_C SVE2 AVX2 AVX512VBMI AVX512VBMI
RAWToRGBARow_C SVE2 AVX2 AVX512VBMI AVX512VBMI
RGB24MirrorRow_C NEON AVX2 AVX512VBMI AVX512VBMI
RGB24ToARGBRow_C SVE2 AVX2 AVX512BW AVX512BW
RGB565ToARGBRow_C NEON AVX2 AVX2 AVX2
RGB565ToUVMatrixRow_C NEON AVX2 AVX2 AVX2
RGB565ToUVRow_C * NEON AVX2 AVX2 AVX2
RGB565ToYMatrixRow_C NEON AVX2 AVX2 AVX2
RGB565ToYRow_C * NEON AVX2 AVX2 AVX2
RGBAToARGBRow_C NEON AVX2 AVX512BW AVX512BW
RGBColorTableRow_C NEON AVX2 AVX2 AVX2
RGBToUV444MatrixRow_C NEON AVX2 AVX512BW AVX512BW
RGBToUVMatrixRow_C NEON AVX2 AVX512BW AVX512BW
RGBToYMatrixRow_C DOTPROD AVX2 AVX512BW AVX512BW
ScaleMaxSamples_C * NEON AVX2 AVX2 AVX2
ScaleSamples_C * NEON AVX2 AVX2 AVX2
ScaleSumSamples_C * NEON AVX2 AVX2 AVX2
SetRow_C NEON AVX2 AVX2 AVX2
SobelRow_C NEON AVX2 AVX2 AVX2
SobelToPlaneRow_C NEON AVX2 AVX2 AVX2
SobelXRow_C NEON AVX2 AVX2 AVX2
SobelXYRow_C NEON AVX2 AVX2 AVX2
SobelYRow_C NEON AVX2 AVX2 AVX2
SplitARGBRow_C NEON AVX2 AVX2 AVX2
SplitRGBRow_C NEON AVX2 AVX2 AVX2
SplitUVRow_16_C NEON AVX2 AVX2 AVX2
SplitUVRow_C SVE2 AVX2 AVX512BW AVX512BW
SplitXRGBRow_C NEON AVX2 AVX2 AVX2
Ssim4x4Row_C SME AVX2 AVX512BW AVX512BW
SsimWindowRow_C SME AVX2 AVX512BW AVX512BW
SumSquareError_C DOTPROD AVX2 AVX512VNNI AVX512VNNI
SwapUVRow_C NEON AVX2 AVX512BW AVX512BW
TransposeUVWx16_C - AVX2 AVX512BW AVX512BW
TransposeUVWx8_C NEON AVX2 AVX2 AVX2
TransposeWx16_C NEON AVX2 AVX512BW AVX512BW
TransposeWx4_32_C * SVE2 AVX2 AVX512BW AVX512BW
TransposeWx8_16_C SVE2 AVX2 AVX512BW AVX512BW
TransposeWx8_C - AVX2 AVX2 AVX2
TransposeWxH_32_C NEON AVX2 AVX2 AVX2
UYVYToARGBRow_C SME AVX2 AVX512BW AVX512BW
UYVYToUV422Row_C NEON AVX2 AVX2 AVX2
UYVYToUVRow_C SVE2 AVX2 AVX512BW AVX512BW
UYVYToYRow_C SVE2 AVX2 AVX512BW AVX512BW
UnpackMT2T_C NEON AVX2 AVX2 AVX2
YUY2ToARGBRow_C SME AVX2 AVX512BW AVX512BW
YUY2ToNVUVRow_C NEON AVX2 AVX2 AVX2
YUY2ToUV422Row_C NEON AVX2 AVX2 AVX2
YUY2ToUVRow_C SVE2 AVX2 AVX512BW AVX512BW
YUY2ToYRow_C SVE2 AVX2 AVX512BW AVX512BW
------------------------------------------------------------------------
SIMD (187 row/rot/cmp) 181 97% 187 100% 187 100% 187 100%
------------------------------------------------------------------------
Scale SIMD ISA supported per *_C function:
------------------------------------------------------------------------
C Function arm64 x86_32 x64 row_win
------------------------------------------------------------------------
FixedDiv1_C - X86 X86 X86
FixedDiv_C - X86 X86 X86
ScaleARGBCols64_C - - - -
ScaleARGBColsUp2_C - AVX2 AVX2 AVX2
ScaleARGBCols_C NEON AVX2 AVX2 AVX2
ScaleARGBFilterCols64_C - - - -
ScaleARGBFilterCols_C NEON AVX2 AVX2 AVX2
ScaleARGBRowDown2Box_C SME AVX2 AVX2 AVX2
ScaleARGBRowDown2Linear_C SME AVX2 AVX2 AVX2
ScaleARGBRowDown2_C SME AVX2 AVX2 AVX2
ScaleARGBRowDownEvenBox_C NEON AVX2 AVX2 AVX2
ScaleARGBRowDownEven_C NEON AVX2 AVX2 AVX2
ScaleAddCols0_C - - - -
ScaleAddCols1_16_C - - - -
ScaleAddCols1_C - - AVX512BW AVX512BW
ScaleAddCols2_16_C - - - -
ScaleAddCols2_C - - AVX512BW AVX512BW
ScaleAddRow_16_C - - - -
ScaleAddRow_C NEON AVX2 AVX512BW AVX512BW
ScaleColsUp2_16_C - - - -
ScaleColsUp2_C - AVX2 AVX2 AVX2
ScaleCols_16_C - - - -
ScaleCols_C - - - -
ScaleFilterCols64_16_C - - - -
ScaleFilterCols64_C - - - -
ScaleFilterCols_16_C - - - -
ScaleFilterCols_C NEON AVX2 AVX2 AVX2
ScaleRowDown2Box_16_C SME AVX2 AVX2 AVX2
ScaleRowDown2Box_C SME AVX2 AVX512BW AVX512BW
ScaleRowDown2Box_Odd_C - - AVX512BW AVX512BW
ScaleRowDown2Linear_16_C SME AVX2 AVX2 AVX2
ScaleRowDown2Linear_C SME AVX2 AVX512BW AVX512BW
ScaleRowDown2_16_C SME AVX2 AVX2 AVX2
ScaleRowDown2_C SME AVX2 AVX512BW AVX512BW
ScaleRowDown34_0_Box_16_C - - - -
ScaleRowDown34_0_Box_C NEON AVX2 AVX2 AVX2
ScaleRowDown34_16_C - - - -
ScaleRowDown34_1_Box_16_C - - - -
ScaleRowDown34_1_Box_C NEON AVX2 AVX2 AVX2
ScaleRowDown34_C NEON AVX2 AVX2 AVX2
ScaleRowDown38_16_C - - - -
ScaleRowDown38_2_Box_16_C - - - -
ScaleRowDown38_2_Box_C NEON AVX2 AVX2 AVX2
ScaleRowDown38_3_Box_16_C - - - -
ScaleRowDown38_3_Box_C NEON AVX2 AVX2 AVX2
ScaleRowDown38_C NEON AVX2 AVX2 AVX2
ScaleRowDown4Box_16_C - - - -
ScaleRowDown4Box_C NEON AVX2 AVX2 AVX2
ScaleRowDown4_16_C - - - -
ScaleRowDown4_C NEON AVX2 AVX2 AVX2
ScaleRowUp2_Bilinear_16_Any_C NEON AVX2 AVX2 AVX2
ScaleRowUp2_Bilinear_16_C NEON AVX2 AVX2 AVX2
ScaleRowUp2_Bilinear_Any_C NEON AVX2 AVX2 AVX2
ScaleRowUp2_Bilinear_C NEON AVX2 AVX2 AVX2
ScaleRowUp2_Linear_16_Any_C NEON AVX2 AVX2 AVX2
ScaleRowUp2_Linear_16_C NEON AVX2 AVX2 AVX2
ScaleRowUp2_Linear_Any_C NEON AVX2 AVX2 AVX2
ScaleRowUp2_Linear_C NEON AVX2 AVX2 AVX2
ScaleUVAddCols1_16_C - - - -
ScaleUVAddCols1_C - - - -
ScaleUVAddCols2_16_C - - - -
ScaleUVAddCols2_C - - - -
ScaleUVAddCols3_16_C - - - -
ScaleUVAddCols3_C SME AVX2 AVX512VBMI AVX512VBMI
ScaleUVCols64_16_C - - - -
ScaleUVCols64_C - - - -
ScaleUVColsUp2_16_C - - - -
ScaleUVColsUp2_C - - - -
ScaleUVCols_16_C - - - -
ScaleUVCols_C - - - -
ScaleUVFilterCols64_16_C - - - -
ScaleUVFilterCols64_C - - - -
ScaleUVFilterCols_16_C - - - -
ScaleUVFilterCols_C - - - -
ScaleUVRowDown2Box_16_C - - - -
ScaleUVRowDown2Box_C SME AVX2 AVX2 AVX2
ScaleUVRowDown2Linear_16_C - - - -
ScaleUVRowDown2Linear_C SME AVX2 AVX2 AVX2
ScaleUVRowDown2_16_C - - - -
ScaleUVRowDown2_C SME AVX2 AVX2 AVX2
ScaleUVRowDownEvenBox_16_C - - - -
ScaleUVRowDownEvenBox_C - - - -
ScaleUVRowDownEven_16_C - - - -
ScaleUVRowDownEven_C NEON AVX2 AVX2 AVX2
ScaleUVRowUp2_Bilinear_16_Any_C NEON AVX2 AVX2 AVX2
ScaleUVRowUp2_Bilinear_16_C NEON AVX2 AVX2 AVX2
ScaleUVRowUp2_Bilinear_Any_C NEON AVX2 AVX2 AVX2
ScaleUVRowUp2_Bilinear_C NEON AVX2 AVX2 AVX2
ScaleUVRowUp2_Linear_16_Any_C NEON AVX2 AVX2 AVX2
ScaleUVRowUp2_Linear_16_C NEON AVX2 AVX2 AVX2
ScaleUVRowUp2_Linear_Any_C NEON AVX2 AVX2 AVX2
ScaleUVRowUp2_Linear_C NEON AVX2 AVX2 AVX2
------------------------------------------------------------------------
SIMD (92 scale *_C) 44 48% 48 52% 51 55% 51 55%
------------------------------------------------------------------------
Test: libyuv_unittest
Bug: none
Change-Id: I2b4a832f2a7fe1fe93487b453240cd15c1f5f940
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8676400
Commit-Queue: Frank Barchard <fbarchard@google.com>
Reviewed-by: richard winterton <rrwinterton@gmail.com>
|
||
|
Frank Barchard
|
63e54f7374 |
[libyuv] Add NEON implementations for all remaining C row functions
Implement 32-bit and 64-bit Arm NEON versions (in row_neon.cc and row_neon64.cc) for the 12 remaining row functions that previously fell back to C on Arm: - AR30ToAB30Row_NEON (and AR30ToAB30Row_Any_NEON) - AR30ToABGRRow_NEON (and AR30ToABGRRow_Any_NEON) - AR30ToARGBRow_NEON (and AR30ToARGBRow_Any_NEON) - ARGBAffineRow_NEON - ARGBColorTableRow_NEON - ARGBCopyAlphaRow_NEON (and ARGBCopyAlphaRow_Any_NEON) - ARGBLumaColorTableRow_NEON - ARGBPolynomialRow_NEON (and ARGBPolynomialRow_Any_NEON) - ARGBUnattenuateRow_NEON (and ARGBUnattenuateRow_Any_NEON) - CopyRow_16_NEON (and CopyRow_16_Any_NEON) - MirrorRow_16_NEON (and MirrorRow_16_Any_NEON) - RGBColorTableRow_NEON Add HAS_*_NEON macros, prototypes, _Any wrappers, NEON dispatches in convert.cc, convert_argb.cc, and planar_functions.cc, and enable unit tests in planar_test.cc. Arm row function SIMD coverage is now 164/164 (100%). ARM64 (Apple M6) Benchmark (1280 width, 10000 iterations): ---------------------------------------------------------------- Function Before C (ms) After NEON (ms) Improvement ---------------------------------------------------------------- AR30ToAB30Row 1.20 0.70 1.72x AR30ToABGRRow 1.88 1.03 1.83x AR30ToARGBRow 1.88 0.87 2.17x ARGBAffineRow 7.96 2.31 3.44x ARGBColorTableRow 8.17 7.43 1.10x ARGBCopyAlphaRow 1.37 0.53 2.58x ARGBLumaColorTableRow 8.18 5.24 1.56x ARGBPolynomialRow 8.73 7.55 1.16x ARGBUnattenuateRow 6.00 2.05 2.93x CopyRow_16 0.18 0.18 1.00x MirrorRow_16 0.34 0.20 1.73x RGBColorTableRow 6.15 5.58 1.10x ---------------------------------------------------------------- Maximum SIMD ISA supported per C row function: ---------------------------------------------------------------- C Row Function arm32 arm64 x86 riscv ---------------------------------------------------------------- AB64ToARGBRow_C NEON NEON AVX2 RVV ABGRToAR30Row_C - NEON AVX512BW - AR30ToAB30Row_C NEON NEON AVX512BW - AR30ToABGRRow_C NEON NEON AVX512BW - AR30ToARGBRow_C NEON NEON AVX512BW - AR64ShuffleRow_C NEON NEON AVX512BW RVV AR64ToAB64Row_C NEON NEON AVX512BW RVV AR64ToARGBRow_C NEON NEON AVX2 RVV ARGB1555ToARGBRow_C NEON SVE2 AVX2 - ARGB1555ToUVMatrixRow_C NEON NEON AVX2 - ARGB1555ToUVRow_C NEON NEON - - ARGB1555ToYMatrixRow_C NEON NEON AVX2 - ARGB1555ToYRow_C NEON NEON - - ARGB4444ToARGBRow_C NEON NEON AVX2 - ARGB4444ToUVMatrixRow_C NEON NEON AVX2 - ARGB4444ToUVRow_C NEON NEON - - ARGB4444ToYMatrixRow_C NEON NEON AVX2 - ARGB4444ToYRow_C NEON NEON - - ARGBAddRow_C NEON NEON AVX2 - ARGBAffineRow_C NEON NEON SSE2 - ARGBAlphaBlendRow_C NEON SME AVX512BW RVV ARGBAttenuateRow_C NEON NEON AVX512BW RVV ARGBBlendRow_C NEON NEON AVX512BW RVV ARGBColorMatrixRow_C NEON I8MM SSSE3 - ARGBColorTableRow_C NEON NEON X86 - ARGBCopyAlphaRow_C NEON NEON AVX2 - ARGBCopyYToAlphaRow_C NEON NEON AVX512BW RVV ARGBExtractAlphaRow_C NEON NEON AVX512BW RVV ARGBGrayRow_C NEON DOTPROD AVX512VNNI - ARGBLumaColorTableRow_C NEON NEON SSSE3 - ARGBMirrorRow_C NEON NEON AVX512BW RVV ARGBMultiplyRow_C NEON SME AVX512BW RVV ARGBOrRow_C NEON SME AVX512BW RVV ARGBPolynomialRow_C NEON NEON AVX2 - ARGBQuantizeRow_C NEON NEON SSE2 - ARGBSepiaRow_C NEON DOTPROD SSSE3 - ARGBSetRow_C NEON NEON X86 RVV ARGBShadeRow_C NEON NEON SSE2 - ARGBShuffleRow_C NEON NEON AVX512BW RVV ARGBSubtractRow_C NEON NEON AVX2 - ARGBToAB64Row_C NEON NEON AVX2 RVV ARGBToABGRRow_C NEON NEON AVX512BW RVV ARGBToAR30Row_C - NEON AVX512BW - ARGBToAR64Row_C NEON NEON AVX2 RVV ARGBToARGB1555Row_C NEON NEON AVX2 - ARGBToARGB4444Row_C NEON NEON AVX2 - ARGBToBGRARow_C NEON NEON AVX512BW RVV ARGBToRAWRow_C NEON SVE2 AVX512BW RVV ARGBToRGB24Row_C NEON SVE2 AVX512BW RVV ARGBToRGB565DitherRow_C NEON SVE2 AVX2 - ARGBToRGB565Row_C NEON SVE2 AVX2 - ARGBToRGBARow_C NEON NEON AVX512BW RVV ARGBToUV444MatrixRow_C NEON I8MM AVX512VNNI RVV ARGBToUVMatrixRow_C NEON SME AVX512VNNI RVV ARGBToYMatrixRow_C NEON SME AVX512VNNI RVV ARGBUnattenuateRow_C NEON NEON AVX2 - AYUVToUVRow_C NEON SVE2 AVX512VBMI - AYUVToVURow_C NEON SVE2 AVX512VBMI - AYUVToYRow_C NEON NEON AVX512VBMI - BlendPlaneRow_C NEON NEON AVX512BW RVV ByteToFloatRow_C NEON NEON - - ComputeCumulativeSumRow_C - NEON AVX2 - Convert16To8Row_C NEON SME AVX512BW RVV Convert8To16Row_C NEON SME AVX512BW RVV Convert8To8Row_C NEON SME AVX2 - CopyRow_C NEON SME AVX512BW RVV CopyRow_16_C NEON NEON - - CumulativeSumToAverageRow_C - NEON AVX2 - DetileRow_C NEON NEON SSE2 - DetileRow_16_C NEON NEON AVX - DetileSplitUVRow_C NEON NEON SSSE3 - DivideRow_16_C NEON SVE2 AVX2 - GaussRow_C NEON NEON - - GaussRow_F32_C - NEON - - HalfFloatRow_C NEON SVE2 AVX2 - HalfMergeUVRow_C NEON SVE2 AVX512BW - HalfRow_16To8_C NEON SME AVX512BW RVV HalfWidthRow_16To8_C NEON SME AVX512BW RVV I210AlphaToARGBRow_C - SME AVX2 - I210ToAR30Row_C - SME AVX2 - I210ToARGBRow_C - SME AVX2 - I212ToAR30Row_C - SME AVX2 - I212ToARGBRow_C - SME AVX2 - I400ToARGBRow_C NEON SME AVX512BW RVV I410AlphaToARGBRow_C - SME AVX2 - I410ToAR30Row_C - SME AVX2 - I410ToARGBRow_C - SME AVX2 - I422AlphaToARGBRow_C NEON SME AVX512BW RVV I422ToAR30Row_C - SME AVX512BW RVV I422ToARGB1555Row_C NEON SME AVX2 - I422ToARGB4444Row_C NEON SME AVX2 - I422ToARGBRow_C NEON SME AVX512BW RVV I422ToRGB24Row_C NEON SME AVX512VBMI RVV I422ToRGB565Row_C NEON SME AVX2 - I422ToRGBARow_C NEON SME AVX2 RVV I422ToUYVYRow_C NEON NEON AVX2 - I422ToYUY2Row_C NEON NEON AVX2 - I444AlphaToARGBRow_C NEON SME AVX2 RVV I444ToARGBRow_C NEON SME AVX2 RVV I444ToRGB24Row_C NEON SME AVX2 RVV InterpolateRow_C NEON SME AVX512BW RVV InterpolateRow_16_C NEON SME AVX512BW - J400ToARGBRow_C NEON NEON AVX512BW RVV MergeAR64Row_C NEON NEON AVX2 - MergeARGB16To8Row_C NEON NEON AVX2 - MergeARGBRow_C NEON NEON AVX2 RVV MergeRGBRow_C NEON NEON AVX512VBMI RVV MergeUVRow_C NEON SME AVX512BW RVV MergeUVRow_16_C NEON SME AVX2 - MergeXR30Row_C NEON NEON AVX512BW - MergeXR64Row_C NEON NEON AVX2 - MergeXRGB16To8Row_C NEON NEON AVX2 - MergeXRGBRow_C NEON NEON AVX2 RVV MirrorRow_C NEON NEON AVX512BW - MirrorRow_16_C NEON NEON AVX512BW - MirrorSplitUVRow_C NEON NEON AVX512BW - MirrorUVRow_C NEON NEON AVX512BW RVV MultiplyRow_16_C NEON SME AVX512BW RVV NV12ToARGBRow_C NEON SME AVX512BW RVV NV12ToRGB24Row_C NEON SME AVX512VBMI RVV NV12ToRGB565Row_C NEON NEON AVX2 - NV21ToARGBRow_C NEON SME AVX512BW RVV NV21ToRGB24Row_C NEON SME AVX512VBMI RVV NV21ToYUV24Row_C NEON NEON AVX2 - P210ToAR30Row_C - SME AVX2 - P210ToARGBRow_C - SME AVX2 - P410ToAR30Row_C - SME AVX2 - P410ToARGBRow_C - SME AVX2 - RAWToARGBRow_C NEON SVE2 AVX512BW RVV RAWToRGB24Row_C NEON SVE2 AVX512VBMI RVV RAWToRGBARow_C NEON SVE2 AVX512VBMI RVV RGB24MirrorRow_C NEON NEON AVX512VBMI - RGB24ToARGBRow_C NEON SVE2 AVX512BW RVV RGB565ToARGBRow_C NEON NEON AVX2 - RGB565ToUVMatrixRow_C NEON NEON AVX2 - RGB565ToUVRow_C NEON NEON - - RGB565ToYMatrixRow_C NEON NEON AVX2 - RGB565ToYRow_C NEON NEON - - RGBAToARGBRow_C NEON NEON AVX512BW RVV RGBColorTableRow_C NEON NEON X86 - RGBToUV444MatrixRow_C NEON NEON AVX512BW RVV RGBToUVMatrixRow_C NEON NEON AVX512BW RVV RGBToYMatrixRow_C NEON DOTPROD AVX512BW RVV SetRow_C NEON NEON ERMS - SobelRow_C NEON NEON SSE2 - SobelToPlaneRow_C NEON NEON SSE2 - SobelXRow_C NEON NEON SSE2 - SobelXYRow_C NEON NEON SSE2 - SobelYRow_C NEON NEON SSE2 - SplitARGBRow_C NEON NEON AVX2 RVV SplitRGBRow_C NEON NEON AVX2 RVV SplitUVRow_C NEON SVE2 AVX512BW RVV SplitUVRow_16_C NEON NEON AVX2 - SplitXRGBRow_C NEON NEON AVX2 RVV SwapUVRow_C NEON NEON AVX512BW RVV UYVYToARGBRow_C NEON SME AVX2 - UYVYToUV422Row_C NEON NEON AVX2 - UYVYToUVRow_C NEON SVE2 AVX2 - UYVYToYRow_C NEON SVE2 AVX2 - YUY2ToARGBRow_C NEON SME AVX2 - YUY2ToNVUVRow_C NEON NEON AVX2 - YUY2ToUV422Row_C NEON NEON AVX2 - YUY2ToUVRow_C NEON SVE2 AVX2 - YUY2ToYRow_C NEON SVE2 AVX2 - ---------------------------------------------------------------- SIMD (164/164 100%) 146 89% 164 100% 154 94% 64 39% ---------------------------------------------------------------- Test: libyuv_unittest --gtest_filter=*AR30*:*Affine*:*ColorTable*:*CopyAlpha*:*Polynomial*:*Unattenuate*:*MirrorPlane_16* Bug: None Change-Id: Ib8b68beb77b47de6b996d41fd080397c61539ffb Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8666404 Reviewed-by: richard winterton <rrwinterton@gmail.com> Reviewed-by: Dale Curtis <dalecurtis@chromium.org> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
Frank Barchard
|
a84bf45a21 |
[libyuv] Fix MSan uninitialized errors in AVX512 alpha and scale rows
Clang MemorySanitizer builds define LIBYUV_ENABLE_ROWWIN and compile row_win.cc and scale_win.cc instead of inline assembly. Two AVX-512 intrinsic patterns in those files are not modeled by LLVM MSan: 1. _mm512_mask_cvtepi32_storeu_epi8 and _mm512_mask_cvtepi64_storeu_epi8 lower to @llvm.x86.avx512.mask.pmov.db.mem.512 and @llvm.x86.avx512.mask.pmov.qb.mem.512, which LLVM MSan handles as unknown intrinsics without unpoisoning destination shadow memory. Replace them in ARGBExtractAlphaRow_AVX512BW, ScaleAddCols1_AVX512BW, and ScaleAddCols2_AVX512BW with _mm512_cvtepi32_epi8 / _mm512_cvtepi64_epi8 followed by _mm_mask_storeu_epi8, which lowers to @llvm.masked.store and still compiles to a single vpmovdb/vpmovqb instruction. 2. ARGBCopyYToAlphaRow_AVX512BW loaded dst and blended the alpha channel using _mm512_ternarylogic_epi32, which LLVM MSan does not model bitwise and flags when any destination byte (such as the old alpha channel) is uninitialized. Replace the load + vpternlogd + store in both row_win.cc and row_gcc.cc with byte-masked stores (_mm512_mask_storeu_epi8 / vmovdqu8 with mask 0x8888888888888888) that write only the alpha bytes without reading dst. Expand ARGBCopyAlpha, ARGBExtractAlpha, and ARGBCopyYToAlpha unit tests into _Any, _Unaligned, _Invert, and _Opt variants and poison destination buffers under MSan to catch unmodeled masked stores. Document MSan AVX-512 intrinsic guidelines in GEMINI.md. Test: out/msan/libyuv_unittest --gtest_filter=*Alpha*:*ScalePlaneBox_Any* Bug: 578615529 Change-Id: I54ac4a5844aca6cfe3aa3445fa04b8221c610b35 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8672001 Commit-Queue: Frank Barchard <fbarchard@google.com> Reviewed-by: Dale Curtis <dalecurtis@chromium.org> |
||
|
George Steed
|
2e65a61dff |
AArch64: Avoid lane-indexed LD4 dependency in 32 bit transpose
The original main loop builds v0-v3 with four lane-indexed LD4 load
instructions. Since the first LD4 only updates lane 0, it retains a
false dependency on the registers produced by the previous loop
iteration, leading to a large performance penalty on larger out-of-order
micro-architectures.
Replace the first LD4 with two scalar-register LDPs to zero the top part
of the four registers, breaking the loop-carried dependency.
Performance versus the existing Neon code, higher is better:
Cortex-A55: 0.993x
Cortex-A510: 1.072x
Cortex-A720: 0.999x
Cortex-X4: 1.052x
C1-Nano: 2.093x
C1-Pro: 2.011x
C1-Ultra: 2.128x
Change-Id: Ib5d71e8150d7b11048ddf0fb91e107f9c9323a3f
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8669762
Reviewed-by: Frank Barchard <fbarchard@google.com>
Reviewed-by: Justin Green <greenjustin@google.com>
|
||
|
Frank Barchard
|
4bc0419865 |
[libyuv] Remove unused prototypes and HAS_ macros
Maximum SIMD ISA supported per C row function: ------------------------------------------------------------------------ C Row Function arm x86 riscv wasm loongson hexagon ------------------------------------------------------------------------ AB64ToARGBRow_C NEON AVX2 RVV - - HVX ABGRToAR30Row_C NEON AVX512BW - - - - AR30ToAB30Row_C - AVX512BW - - - - AR30ToABGRRow_C - AVX512BW - - - - AR30ToARGBRow_C - AVX512BW - - - - AR64ShuffleRow_C NEON AVX512BW RVV WASMSIMD - - AR64ToAB64Row_C NEON AVX512BW RVV WASMSIMD - HVX AR64ToARGBRow_C NEON AVX2 RVV - - HVX ARGB1555ToARGBRow_C SVE2 AVX2 - WASMSIMD LASX - ARGB1555ToUVMatrixRow_C NEON AVX2 - WASMSIMD - - ARGB1555ToUVRow_C NEON - - - LASX - ARGB1555ToYMatrixRow_C NEON AVX2 - WASMSIMD - - ARGB1555ToYRow_C NEON - - - LASX - ARGB4444ToARGBRow_C NEON AVX2 - WASMSIMD LASX - ARGB4444ToUVMatrixRow_C NEON AVX2 - WASMSIMD - - ARGB4444ToUVRow_C NEON - - - - - ARGB4444ToYMatrixRow_C NEON AVX2 - WASMSIMD - - ARGB4444ToYRow_C NEON - - - - - ARGBAddRow_C NEON AVX2 - - LASX HVX ARGBAffineRow_C - SSE2 - - - - ARGBAlphaBlendRow_C SME AVX512BW RVV WASMSIMD LASX HVX ARGBAttenuateRow_C NEON AVX512BW RVV - LASX HVX ARGBBlendRow_C NEON AVX512BW RVV WASMSIMD LASX HVX ARGBColorMatrixRow_C I8MM SSSE3 - - LASX - ARGBColorTableRow_C - X86 - - - - ARGBCopyAlphaRow_C - AVX2 - - - HVX ARGBCopyYToAlphaRow_C NEON AVX512BW RVV WASMSIMD - HVX ARGBExtractAlphaRow_C NEON AVX512BW RVV WASMSIMD LASX HVX ARGBGrayRow_C DOTPROD AVX512VNNI - - LASX - ARGBLumaColorTableRow_C - SSSE3 - - - - ARGBMirrorRow_C NEON AVX512BW RVV WASMSIMD LASX HVX ARGBMultiplyRow_C SME AVX512BW RVV - LASX HVX ARGBOrRow_C SME AVX512BW RVV WASMSIMD LASX HVX ARGBPolynomialRow_C - AVX2 - - - - ARGBQuantizeRow_C NEON SSE2 - - LASX - ARGBSepiaRow_C DOTPROD SSSE3 - - LASX - ARGBSetRow_C NEON X86 RVV - LASX HVX ARGBShadeRow_C NEON SSE2 - - LASX - ARGBShuffleRow_C NEON AVX512BW RVV WASMSIMD LASX - ARGBSubtractRow_C NEON AVX2 - - LASX HVX ARGBToAB64Row_C NEON AVX2 RVV - - HVX ARGBToABGRRow_C NEON AVX512BW RVV WASMSIMD LASX HVX ARGBToAR30Row_C NEON AVX512BW - - - - ARGBToAR64Row_C NEON AVX2 RVV - - HVX ARGBToARGB1555Row_C NEON AVX2 - WASMSIMD LASX - ARGBToARGB4444Row_C NEON AVX2 - WASMSIMD LASX - ARGBToBGRARow_C NEON AVX512BW RVV WASMSIMD LASX HVX ARGBToRAWRow_C SVE2 AVX512BW RVV - LASX - ARGBToRGB24Row_C SVE2 AVX512BW RVV - LASX - ARGBToRGB565DitherRow_C SVE2 AVX2 - WASMSIMD LASX - ARGBToRGB565Row_C SVE2 AVX2 - WASMSIMD LASX - ARGBToRGBARow_C NEON AVX512BW RVV WASMSIMD LASX HVX ARGBToUV444MatrixRow_C I8MM AVX512VNNI RVV WASMSIMD - HVX ARGBToUVMatrixRow_C SME AVX512VNNI RVV WASMSIMD - HVX ARGBToYMatrixRow_C SME AVX512VNNI RVV WASMSIMD LASX HVX ARGBUnattenuateRow_C - AVX2 - - - - AYUVToUVRow_C SVE2 AVX512VBMI - - - - AYUVToVURow_C SVE2 AVX512VBMI - - - - AYUVToYRow_C NEON AVX512VBMI - - - - BlendPlaneRow_C NEON AVX512BW RVV WASMSIMD - HVX ByteToFloatRow_C NEON - - - - - ComputeCumulativeSumRow_C NEON AVX2 - - - - Convert16To8Row_C SME AVX512BW RVV - - HVX Convert8To16Row_C SME AVX512BW RVV WASMSIMD - HVX Convert8To8Row_C SME AVX2 - - - HVX CopyRow_C SME AVX512BW RVV - LASX HVX CopyRow_16_C - - - - - HVX CumulativeSumToAverageRow_C NEON AVX2 - - - - DetileRow_C NEON SSE2 - - - - DetileRow_16_C NEON AVX - - - - DetileSplitUVRow_C NEON SSSE3 - - - - DivideRow_16_C SVE2 AVX2 - - - HVX GaussRow_C NEON - - - - - GaussRow_F32_C NEON - - - - - HalfFloatRow_C SVE2 AVX2 - - LASX - HalfMergeUVRow_C SVE2 AVX512BW - - - - HalfRow_16To8_C SME AVX512BW RVV - - HVX HalfWidthRow_16To8_C SME AVX512BW RVV - - - I210AlphaToARGBRow_C SME AVX2 - - - - I210ToAR30Row_C SME AVX2 - - - - I210ToARGBRow_C SME AVX2 - - - - I212ToAR30Row_C SME AVX2 - - - - I212ToARGBRow_C SME AVX2 - - - - I400ToARGBRow_C SME AVX512BW RVV - LASX HVX I410AlphaToARGBRow_C SME AVX2 - - - - I410ToAR30Row_C SME AVX2 - - - - I410ToARGBRow_C SME AVX2 - - - - I422AlphaToARGBRow_C SME AVX512BW RVV - LASX HVX I422ToAR30Row_C SME AVX512BW RVV WASMSIMD - - I422ToARGB1555Row_C SME AVX2 - WASMSIMD LASX - I422ToARGB4444Row_C SME AVX2 - WASMSIMD LASX - I422ToARGBRow_C SME AVX512BW RVV WASMSIMD LASX HVX I422ToRGB24Row_C SME AVX512VBMI RVV WASMSIMD LASX - I422ToRGB565Row_C SME AVX2 - WASMSIMD LASX - I422ToRGBARow_C SME AVX2 RVV - LASX HVX I422ToUYVYRow_C NEON AVX2 - - LASX - I422ToYUY2Row_C NEON AVX2 - - LASX - I444AlphaToARGBRow_C SME AVX2 RVV - - HVX I444ToARGBRow_C SME AVX2 RVV - LASX HVX I444ToRGB24Row_C SME AVX2 RVV - - - InterpolateRow_C SME AVX512BW RVV WASMSIMD LASX HVX InterpolateRow_16_C SME AVX512BW - WASMSIMD - HVX J400ToARGBRow_C NEON AVX512BW RVV WASMSIMD LASX HVX MergeAR64Row_C NEON AVX2 - - - - MergeARGB16To8Row_C NEON AVX2 - - - - MergeARGBRow_C NEON AVX2 RVV - - HVX MergeRGBRow_C NEON AVX512VBMI RVV - - - MergeUVRow_C SME AVX512BW RVV WASMSIMD LASX HVX MergeUVRow_16_C SME AVX2 - - - HVX MergeXR30Row_C NEON AVX512BW - - - - MergeXR64Row_C NEON AVX2 - - - - MergeXRGB16To8Row_C NEON AVX2 - - - - MergeXRGBRow_C NEON AVX2 RVV - - HVX MirrorRow_C NEON AVX512BW - WASMSIMD LASX HVX MirrorRow_16_C - AVX512BW - - - HVX MirrorSplitUVRow_C NEON AVX512BW - WASMSIMD LASX HVX MirrorUVRow_C NEON AVX512BW RVV WASMSIMD LASX HVX MultiplyRow_16_C SME AVX512BW RVV WASMSIMD - HVX NV12ToARGBRow_C SME AVX512BW RVV WASMSIMD LASX HVX NV12ToRGB24Row_C SME AVX512VBMI RVV - - - NV12ToRGB565Row_C NEON AVX2 - WASMSIMD LASX - NV21ToARGBRow_C SME AVX512BW RVV WASMSIMD LASX HVX NV21ToRGB24Row_C SME AVX512VBMI RVV - - - NV21ToYUV24Row_C NEON AVX2 - - - - P210ToAR30Row_C SME AVX2 - - - - P210ToARGBRow_C SME AVX2 - - - - P410ToAR30Row_C SME AVX2 - - - - P410ToARGBRow_C SME AVX2 - - - - RAWToARGBRow_C SVE2 AVX512BW RVV WASMSIMD LASX - RAWToRGB24Row_C SVE2 AVX512VBMI RVV WASMSIMD LASX - RAWToRGBARow_C SVE2 AVX512VBMI RVV - - - RGB24MirrorRow_C NEON AVX512VBMI - WASMSIMD - - RGB24ToARGBRow_C SVE2 AVX512BW RVV WASMSIMD LASX - RGB565ToARGBRow_C NEON AVX2 - WASMSIMD LASX - RGB565ToUVMatrixRow_C NEON AVX2 - WASMSIMD - - RGB565ToUVRow_C NEON - - - LASX - RGB565ToYMatrixRow_C NEON AVX2 - WASMSIMD - - RGB565ToYRow_C NEON - - - LASX - RGBAToARGBRow_C NEON AVX512BW RVV WASMSIMD LASX HVX RGBColorTableRow_C - X86 - - - - RGBToUV444MatrixRow_C NEON AVX512BW RVV WASMSIMD - - RGBToUVMatrixRow_C NEON AVX512BW RVV WASMSIMD - - RGBToYMatrixRow_C DOTPROD AVX512BW RVV WASMSIMD LASX - SetRow_C NEON ERMS - - LASX HVX SobelRow_C NEON SSE2 - - LASX - SobelToPlaneRow_C NEON SSE2 - - LASX - SobelXRow_C NEON SSE2 - - - - SobelXYRow_C NEON SSE2 - - LASX - SobelYRow_C NEON SSE2 - - - - SplitARGBRow_C NEON AVX2 RVV - - HVX SplitRGBRow_C NEON AVX2 RVV - - - SplitUVRow_C SVE2 AVX512BW RVV - LASX HVX SplitUVRow_16_C NEON AVX2 - - - HVX SplitXRGBRow_C NEON AVX2 RVV - - HVX SwapUVRow_C NEON AVX512BW RVV WASMSIMD - HVX UYVYToARGBRow_C SME AVX2 - - LASX - UYVYToUV422Row_C NEON AVX2 - - LASX HVX UYVYToUVRow_C SVE2 AVX2 - - LASX HVX UYVYToYRow_C SVE2 AVX2 - - LASX HVX YUY2ToARGBRow_C SME AVX2 - - LASX - YUY2ToNVUVRow_C NEON AVX2 - - - - YUY2ToUV422Row_C NEON AVX2 - - LASX HVX YUY2ToUVRow_C SVE2 AVX2 - - LASX HVX YUY2ToYRow_C SVE2 AVX2 - - LASX HVX ------------------------------------------------------------------------ SIMD (164/164 100%) 152 93% 154 94% 64 39% 56 34% 73 45% 64 39% ------------------------------------------------------------------------ Bug: None Change-Id: I969a3c5060173aa46b4d1f085837e284fbed1a85 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8665861 Reviewed-by: Mirko Bonadei <mbonadei@chromium.org> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
Frank Barchard
|
205022c08c |
[libyuv] Add AVX-512 VNNI row functions for ARGB to YUV and Gray
Add AVX-512 VNNI (VPDPBUSD / _mm512_dpbusd_epi32) row kernels for 64-bit x86 targets alongside existing AVX-512BW implementations in both row_gcc.cc (inline assembly) and row_win.cc (C++ intrinsics): - ARGBToYMatrixRow_AVX512VNNI and ARGBToYMatrixRow_Any_AVX512VNNI: Pre-computes the 32-bit signed-pixel bias once per row and replaces vpmaddubsw + vpmaddwd + vpaddw with a single vpdpbusd per 16 pixels. - ARGBToUV444MatrixRow_AVX512VNNI and ARGBToUV444MatrixRow_Any_AVX512VNNI: Loads 64 ARGB pixels once instead of twice and computes 32-bit U and V dot products directly via vpdpbusd with signed UV matrix coefficients. - ARGBToUVMatrixRow_AVX512VNNI: Computes 2x2 box-filtered ARGB to U and V conversion using vpdpbusd with k-masked tail handling for arbitrary widths. - ARGBGrayRow_AVX512VNNI: Computes BT.601 grayscale luminance using a single vpdpbusd per 16 pixels, broadcasts Y to RGB via vpshufb, and preserves alpha via vpternlogd with k-masked tail handling. - Wire runtime dispatch via TestCpuFlag(kCpuHasAVX512VNNI) across convert.cc, convert_from_argb.cc, and planar_functions.cc. Add AVX512VNNIVBMI variants of the Y, UV444 and UV kernels. These use both AVX-512 VNNI (vpdpbusd) and AVX-512 VBMI (vpermt2b) and are only selected when the CPU has both features; there are no VBMI versions without VNNI. The AVX512VNNI kernels remain for CPUs with VNNI but no VBMI (Cascade Lake, Cooper Lake). - The 32-bit sums are always in [0, 65535], so Y, U and V are byte 1 of each int32. Two vpermt2b gather 64 results from 4 accumulators, replacing vpackusdw/vpackssdw + vpsrlw + vpackuswb + vpermd. - U and V accumulators start at ~kAddUV so they hold ~(kAddUV - dot). The inverse is folded into the vpternlogd that merges the two 256-bit halves. kAddUV is loaded from ArgbConstants rather than hard coded as 0x8000. - Y folds the source loads into vpxorq. - row_win.cc hides the vpermt2b indices from clang, which otherwise constant folds them into a longer vpshufb + vpor + vpermd sequence. Performance (1280x720, 5000 iterations, AVX512BW -> AVX512VNNI): uArch / Benchmark AVX512BW AVX512VNNI Speedup --------------------------- -------- ---------- ------- Intel Ice Lake ARGBToI420 941 ms 862 ms +9.2% Intel Ice Lake ARGBToI422 1059 ms 947 ms +11.8% Intel Ice Lake ARGBToI444 1140 ms 1038 ms +9.8% Intel Ice Lake ARGBGray 357 ms 273 ms +30.8% Intel Sapphire Rapids GrayTo 317 ms 218 ms +45.4% Intel Emerald Rapids GrayTo 310 ms 213 ms +45.5% Intel Granite Rapids GrayTo 349 ms 241 ms +44.8% AMD Zen 4 ARGBToI420 648 ms 517 ms +25.3% AMD Zen 4 ARGBToI444 695 ms 526 ms +32.1% AMD Zen 4 ARGBGray 403 ms 227 ms +77.5% AMD Zen 5 ARGBToI422 460 ms 403 ms +14.1% AMD Zen 5 ARGBToI444 382 ms 321 ms +19.0% Performance (row_gcc, AVX512VNNI -> AVX512VNNIVBMI): uArch / Benchmark AVX512VNNI VNNIVBMI Speedup ---------------------------- ---------- -------- ------- AMD Zen 6 ARGBToI420 1280x720 327 ms 284 ms +15.1% AMD Zen 6 ARGBToI422 1280x720 372 ms 330 ms +12.7% AMD Zen 6 ARGBToI444 1280x720 357 ms 334 ms +6.9% AMD Zen 6 ARGBToI400 256x24 94 ms 76 ms +23.7% AMD Zen 5 ARGBToI422 1280x720 406 ms 350 ms +16.0% AMD Zen 5 ARGBToI444 1280x720 330 ms 301 ms +9.6% AMD Zen 5 ARGBToI400 256x24 100 ms 90 ms +11.1% AMD Zen 4 ARGBToI420 1280x720 631 ms 540 ms +16.9% AMD Zen 4 ARGBToI422 1280x720 993 ms 825 ms +20.4% AMD Zen 4 ARGBToI400 256x24 245 ms 218 ms +12.4% Intel GNR ARGBToI422 256x24 654 ms 616 ms +6.2% Intel SPR ARGBToI422 256x24 653 ms 613 ms +6.5% Intel GNR, EMR, SPR and Ice Lake are within +/-3% at 1280x720. R=rrwinterton@gmail.com Bug: none Test: out/Release/libyuv_unittest --gtest_filter=*ARGBToI4*:*ARGBGray* Test: LIBYUV_DISABLE_AVX512VBMI=1 out/Release/libyuv_unittest Test: sde64 -gnr / -clx with is_msan=true (row_win.cc) Change-Id: Ia7a37b39dd2948fe4055a8a7694dd1ff8ac3e55c Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8647811 Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Frank Barchard
|
38e21134f7 |
[libyuv] Add ARGBOr and ARGBAlphaBlend with multi-platform SIMD
Add two new planar/row compositing functions for 32-bit ARGB/BGRA
scanlines:
- ARGBOr / ARGBOrRow_*: Applies a bitwise OR with a 32-bit ARGB mask
to every pixel (e.g., 0xFF000000 to force opaque alpha for BGRx to
BGRA conversion). Implemented for C, SSE2, AVX2, AVX-512BW, NEON
(32/64-bit), SVE2, SME, RVV, WASMSIMD, LSX, LASX, HVX, and Windows
intrinsics (row_win.cc).
- ARGBAlphaBlend / ARGBAlphaBlendRow_*: Implements straight-alpha
(unattenuated) Porter-Duff source-over blending matching PDFium's
CompositeRowBgra2Bgra:
dst_a = back_a + src_a - (back_a * src_a) / 255
ratio = (src_a * 255) / dst_a
dst_c = (back_c * (255 - ratio) + src_c * ratio) / 255
Implemented with bit-exact division by 255 across all 65,536
(back_a, src_a) pairs for C, SSSE3, AVX2, AVX-512BW, NEON
(32/64-bit), SVE2, SME, RVV, WASMSIMD, LSX, LASX, HVX, and Windows
intrinsics (row_win.cc).
ARGBOr Performance (2048x64 BGRx->BGRA scanlines, us per call):
uArch Before Highway libyuv vs Hwy
------------------------ ------ ------- ------- ------
Qualcomm Hexagon v81 HVX 259.63 260.97 15.78 16.54x
SiFive X280 (RVV) 1107.57 72.49 50.70 1.43x
SiFive X390 (RVV) 1108.82 70.42 52.26 1.35x
ARGBAlphaBlend Performance (2048x64 BGRA scanlines, us per call):
uArch Before Highway libyuv vs Hwy
------------------------ ------ ------- ------- ------
Intel Lion Cove (AVX2) 429.79 141.97 41.89 3.39x
Intel Cougar Cove (AVX2) 516.83 181.98 51.33 3.55x
Intel Golden Cove (AVX2) 560.31 206.28 67.08 3.08x
AMD Zen 3 (AVX2) 1025.73 251.09 98.99 2.54x
Wasm SIMD128 (V8) 1014.14 964.83 385.80 2.50x
Apple M6 Super (NEON) 221.90 118.42 46.35 2.55x
Apple M5 Super (NEON) 241.90 130.07 49.60 2.62x
ARM C1-Ultra (SME2) 384.63 112.52 63.62 1.77x
Qualcomm Oryon V3-L (SME) 388.13 154.87 87.65 1.77x
Qualcomm Oryon-L (NEON) 318.28 149.36 57.23 2.61x
ARM Cortex-X4 (SVE2) 675.47 213.56 102.53 2.08x
ARM Cortex-A720 (NEON) 795.85 586.55 235.81 2.49x
ARM Cortex-A520 (NEON) 2360.95 2122.99 787.65 2.70x
ARM Cortex-A35 (ARMv7) 4907.31 5895.04 1474.51 4.00x
SiFive X280 (RVV) 4145.06 1096.42 596.82 1.84x
SiFive X390 (RVV) 4144.57 1084.93 597.62 1.82x
Qualcomm Hexagon v81 HVX 4912.03 5786.25 952.19 6.08x
Coral V4 RVV (1024 px) 50276 c 48992 c 4299 c 11.40x
R=wtc@google.com
Bug: none
Change-Id: If661239826a033e89d4b5c1adc0bfdf048a7a726
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8647895
Reviewed-by: richard winterton <rrwinterton@gmail.com>
|
||
|
Frank Barchard
|
47ba8e2160 |
Handle SsimWindowRow remainder in SIMD for AVX512BW, SVE2, SME, RVV
Because each 8x8 SSIM window spans two 4x4 tiles horizontally, num_windows = (width - 5) >> 2, so aligned widths that are multiples of 64 (e.g. 64, 128, 640, 1280, 1920) always have remainder 15 (num_windows & 15 == 15). - Update SsimWindowRow_AVX512BW (compare_gcc.cc and compare_win.cc) to handle any remainder (1..15 windows) via k-masking (load_mask for the (rem + 1) * 4 tile words and win_mask for the rem window scores) and set win_simd_size = 1. - Add SsimWindowRow_SVE2, SsimWindowRow_SME, and SsimWindowRow_RVV with win_simd_size = 1 using predicated/vl-controlled loops so no C code (Ssim4x4Row_C or SsimWindowRow_C) is executed on AVX-512BW, SVE2, SME, or RVV CPUs for any image dimensions. - Remove LIBYUV_TARGET_AVX512BW __attribute__((target(...))) from compare_gcc.cc and GCC_HAS_AVX512 from cpu_support.h, using %%zmm0 directly in inline assembly. SSIM Performance (Before vs After): uArch / Target Before After Speedup -------------------------- --------- --------- ------- Coral RVV Verilator (64x16) 5302 cyc 2219 cyc 2.39x Coral RVV WindowRow (15w) 1854 cyc 308 cyc 6.02x Apple M6 SME (64x36 10kx) 16.41 ms 4.69 ms 3.50x Apple M6 SME (720p 1000x) 350.76 ms 66.09 ms 5.31x R=rrwinterton@gmail.com Bug: libyuv:352521931 Change-Id: Ic7d55184c60611af4a57d113cc4ee3d211debadf Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8621918 Commit-Queue: Frank Barchard <fbarchard@google.com> Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Frank Barchard
|
64ddbbcdde |
[libyuv] Optimize SsimWindowRow with SIMD and use compare_win on x86
- Route 32-bit x86 (__i386__ / _M_IX86) to compare_win.cc intrinsics and restrict compare_gcc.cc to 64-bit x86 (__x86_64__), avoiding x87 float rounding and enabling SSE2, AVX2, and AVX-512 on 32-bit and 64-bit MinGW, GCC, Clang, and MSan (LIBYUV_ENABLE_ROWWIN). - Enable AVX-512BW on GCC >= 6 (GCC_HAS_AVX512) and add LIBYUV_TARGET_AVX512BW attributes to compare_gcc.cc and compare_win.cc. - Refactor SsimWindowRow_C and Ssim8x8_C to use int32_t and float, and add SsimWindowRow_SSE2, SsimWindowRow_AVX2, and SsimWindowRow_AVX512BW in both compare_win.cc and compare_gcc.cc. - Update TestSsimAnyWidth, Ssim, and BenchmarkSsim_Opt in compare_test.cc to use EXPECT_NEAR and deterministic test patterns (identical, different, similar). - Update util/compare.cc and linux.mk to compute I420Ssim. Performance (BenchmarkSsim_Opt, 1280x720): uArch Before After Improvement Intel Lion Cove (x64 AVX2) 120.2 us 68.1 us 1.76x Intel Skymont (x64 SSE2) 249.6 us 146.8 us 1.70x AMD Zen 3 (x64 AVX2) 314.9 us 150.2 us 2.10x Intel Lion Cove (MinGW686) 203.7 us 77.9 us 2.61x Intel Skymont (MinGW686) 338.8 us 167.5 us 2.02x AMD Zen 3 (i686 AVX2) 738.4 us 206.5 us 3.58x Quality (util/compare on 640x360 I420 test images): Image Comparison MSE PSNR SSIM Identical (orig vs orig) 0.00 128.00 dB 1.000000 Similar (light +/-3 noise) 4.57 41.53 dB 0.956556 Degraded (16x16 blocky) 98.18 28.21 dB 0.858893 Very different (inverted) 11160.29 7.65 dB 0.000181 Co-authored-by: Martin Storsjö <martin@martin.st> Bug: b/493910265 Test: out/Release/libyuv_unittest --gtest_filter=*Ssim* Change-Id: I8603dab3f752af32d07ae9039502ca5da5a05425 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8618641 Reviewed-by: richard winterton <rrwinterton@gmail.com> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
Frank Barchard
|
f249eaa72f |
[libyuv] Fix top 20 bugs and performance issues
1. Fix multi-bit TestCpuFlag(A | B) checks in AYUVToNV12, AYUVToNV21, RAWToRGBA, NV12ToRGB24Matrix, NV21ToRGB24Matrix, MergeRGBPlane, and MergeXR30Plane to compare == (A | B) so AVX-512 VBMI/BW kernels are not dispatched when AVX512VBMI or AVX512VL is absent. 2. Fix 2-byte SVE2 heap buffer overflow on odd widths in I422ToARGB1555Row_SVE_SC and I422ToARGB4444Row_SVE_SC using trn1/trn2 + st2b under .b predicates. 3. Fix ARGBPolynomialRow_SSE2 to use packssdw before packuswb, and add missing vzeroupper to SplitRGBRow_AVX2. 4. Fix CalcFrameSsim to return 0.0 instead of 0.0 / 0 (NaN) when width <= 8 || height <= 8 so I420Ssim stays finite on small frames. 5. Fix negative-height vertical inversion in ConvertFromI420, SplitARGBPlane (when dst_a is NULL), I422Rotate, I010Rotate, and I210Rotate. 6. Fix horizontal flip (negative x/dx) in ScaleFilterCols_NEON (signed sbfx #16, #16 offset extraction) and ScaleAddCols1_16_C (shift x >>= 16 before indexing src_ptr). 7. Fix ConvertToARGB odd crop_x chroma alignment in NV12/NV21, src_u offset in I420/YV12/J420/H420/U420, and FOURCC_U422 calling U422ToARGB instead of H422ToARGB. 8. Remove dead 14-instruction block in ScaleRowUp2_Bilinear_16_SSE2. 9. Fix UVScale_16 for src_width == 1 (P010ToP410/P210ToP410), enable and fix column order in TransposeWx4_32_SVE2, fix ByteToFloatRow NEON Any BPP (3 -> 4), fix AB64ToARGBRow_Any_*, ScaleColsUp2_SSE2, ScaleUVRowDown2Box_SME, and UYVYToARGBRow_SME macro guards, fix SVE2 loop tail branches (b.lt/b.ge), and remove duplicate HAS_* macros. 10. Fix ConvertFP16ToFP32Column_NEON loop condition and unsigned b.lo underflow on widths < 8 or not multiples of 8. 11. Fix UYVYToNV12 IS_ALIGNED(awidth, ...) checks and WASM SIMD IS_ALIGNED(width, 16) checks in I420ToRGB24Matrix / I422ToRGB24Matrix. 12. Wire SplitUVRow_SVE2, SplitUVRow_LSX, and SplitUVRow_LASX in NV24ToARGB. 13. Fix MJpegDecoder heap buffer overflows in LoadFrame (reallocate databuf_[i] when scanlines_size changes), DecodeToBuffers, and DecodeToCallback (clamp scanlines_to_copy to lines_left and align skip to max_v_samp_factor). 14. Replace width & (vl - 1) tail calculations with post-loop remainder across 18 SVE2 functions in row_sve.h for non-power-of-2 SVE VL, and remove duplicate label 2: in UYVYToARGBRow_SVE_SC. 15. Handle width & 15 tail pixels in DetileToYUY2_C and validate IS_POWEROFTWO(tile_height) in DetileSplitUVPlane and DetileToYUY2. 16. Avoid signed 32-bit overflow in DivideRow_16_C and C16TO8, fix ClampFloatToZero_SSE2 pointer increment (lea 0x4) and loop counter (sub $0x1), and initialize xmm3 in ARGBCopyYToAlphaRow_SSE2. 17. Check max leaf 7 subleaf and max extended CPUID leaf in GetCpuFlags before querying subleaf 7:1 and extended leaf 0x80000021. 18. Free plane_u on error in I422ToNV21, and fix 4x/2x scratch row over-allocation in ScaleARGBBilinearDown and ScaleUVBilinearDown. 19. Add ARGBSetRow_RVV and ARGBShuffleRow_RVV in row_rvv.cc. 20. Fix NEON mask (15 -> 7) in ScaleUVRowUp2_Linear_Any_NEON / _16_Any_NEON, use SIMD InterpolateRow + SplitUVRow in NV16ToI420, wire SSSE3/SSE2 in ARGBToI400Matrix / ARGBToYUY2Matrix / ARGBToUYVYMatrix and RGB24Mirror, and split HalfFloatRow multiply into (src * scale) * 2^-112 to avoid 50x Intel denormal stalls. Bug: b/574714081 Test: libyuv_unittest Change-Id: Iec3ae5c58d52bee3ef13cf9b765ac84b733aa576 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8601931 Reviewed-by: richard winterton <rrwinterton@gmail.com> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
Frank Barchard
|
378bc91324 |
[libyuv] Multi-platform ScaleUVAddCols3 for ScaleUVBox at 3x
Based on Philipp Hancke's AVX2 column kernel for ScaleUVBox at 3x (https://crrev.com/c/8427347). Add ScaleUVAddCols3 for exact 3x horizontal downscaling (dx == 0x30000 and boxheight <= 85) in ScaleUVBox, plus ScaleAddRow_AVX512BW and ScaleAddRow_WASMSIMD dispatch, and port 3x/box scaling kernels in scale_win.cc to 32/64-bit C++ intrinsics matching scale_gcc.cc: - ScaleUVAddCols3_C (portable C) - ScaleUVAddCols3_AVX2 and ScaleUVAddCols3_Any_AVX2 (scale_gcc.cc and scale_win.cc) - ScaleUVAddCols3_AVX512VBMI (scale_gcc.cc and scale_win.cc) - ScaleAddRow_SSE2, ScaleAddRow_AVX2, ScaleAddRow_AVX512BW, ScaleAddCols1_AVX512BW, ScaleAddCols2_AVX512BW, ScaleRowDown38_SSSE3, ScaleRowDown38_2_Box_SSSE3, and ScaleRowDown38_3_Box_SSSE3 in scale_win.cc for 32/64-bit Visual C and LIBYUV_ENABLE_ROWWIN - ScaleUVAddCols3_NEON (32-bit ARM and 64-bit AArch64) and ScaleUVAddCols3_Any_NEON - ScaleUVAddCols3_SME (ARMv9 Streaming SVE with whilelt predicate tail) - ScaleUVAddCols3_RVV (RISC-V Vector vlseg3e32.v inline asm) - ScaleUVAddCols3_LSX and ScaleUVAddCols3_Any_LSX (LoongArch LSX) - ScaleAddRow_WASMSIMD, ScaleAddRow_Any_WASMSIMD, ScaleUVAddCols3_WASMSIMD, and ScaleUVAddCols3_Any_WASMSIMD - ScaleUVAddCols3_HVX (Hexagon HVX vrdelta + vmpy_VuhRuh) Co-authored-by: Philipp Hancke <philipp.hancke@googlemail.com> Bug: libyuv:564129034 R=rrwinterton@gmail.com Change-Id: Ib9e14f3efcb097842c3d0e9fc9b473b6e2f81871 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8596436 Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Frank Barchard
|
700f153cb8 |
[libyuv] Add Qualcomm Hexagon HVX SIMD support
Implement Qualcomm Hexagon Vector Extensions (HVX) 128-byte SIMD optimizations and runtime CPU detection for Hexagon DSP targets: - CPU detection: Add kCpuHasHexagon and kCpuHasHVX flags, LIBYUV_DISABLE_HVX environment variable support, and Hexagon unit_test mmap/munmap shims. - Rotate/Transpose (rotate_hvx.cc): Implement TransposeWxH_HVX, TransposeUVWxH_HVX, TransposeWxH_16_HVX, and TransposeWxH_32_HVX using power-of-2 vdeal/vshuff butterfly networks. - Compare (compare_hvx.cc): Implement HammingDistance_HVX, SumSquareError_HVX, HashDjb2_HVX, and Ssim4x4Row_HVX. - Scale (scale_hvx.cc): Implement 16 scale row kernels including ScaleAddRow_HVX, ScaleRowDown2*_HVX (8-bit and 16-bit), ScaleRowDown4*_HVX, ScaleARGBRowDown2*_HVX, and ScaleUVRowDown2*_HVX. - Row functions (row_hvx.cc): Implement 58 row kernels covering copy/set/mirror, split/merge, 8/16-bit conversions, YUV-to-ARGB, ARGB-to-YUV, ARGB channel swizzles, blend, attenuate, and interpolate using hardware-predicated partial loads/stores. R=rrwinterton@gmail.com Bug: None Test: libyuv_unittest on Hexagon v81 simulator and Snapdragon 8 Elite Change-Id: Ib6713de51f6246d95e58c95e629ff03f18e3b91e Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8604586 Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Frank Barchard
|
2404f9730e |
[libyuv] Rewrite LoongArch LSX/LASX kernels in compact inline asm
Reduce loop unrolling across LoongArch LSX (128-bit) and LASX (256-bit) compare, row, rotate, and scale kernels to match SSE2/SSSE3 and AVX2 (1-2 vectors per iteration), rewrite intrinsics/macro implementations into GCC/Clang inline assembly, update all _Any wrapper masks and IS_ALIGNED dispatch checks, and port all remaining LSX Convert (+19) and Rotate (+2) kernels to LASX for full parity: ISA | Compare | Convert | Rotate | Scale -----------------|---------|---------|---------|-------- MIPS/Loong (Any) | 1 | 68 | 4 | 20 LSX | 1 | 68 | 4 | 20 LASX | 1 | 68 | 4 | 0 Removes 1,935 macro/helper lines from loongson_intrinsics.h (-99.3%) and reduces total LoongArch compiled .text size by 9,799 bytes (-25.1% overall, -37.4% row_lsx.cc, -52.3% scale_lsx.cc, -31.2% rotate_lsx.cc, -4.9% row_lasx.cc while adding +25 LASX functions): File | Before | After | Change -------------------|----------|----------|-------- scale_lsx.cc.o | 6,055 B | 2,888 B | -52.3% row_lsx.cc.o | 17,252 B | 10,792 B | -37.4% rotate_lsx.cc.o | 1,488 B | 1,024 B | -31.2% row_lasx.cc.o | 13,740 B | 13,072 B | -4.9% rotate_lasx.cc.o | 544 B | 1,504 B | +176.5% TOTAL .text | 39,079 B | 29,280 B | -25.1% R=wtc@google.com, yuanhecai@loongson.cn Test: qemu-loongarch64 -cpu la464 libyuv_unittest (LASX and LIBYUV_DISABLE_LASX=1) Change-Id: I11c1143ad2aad0399f10b2c3fc9bc4c02c740c59 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8564659 Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Frank Barchard
|
ec29aca502 |
[libyuv] Optimize CalcFrameSsim with 2-row 4x4 tile buffer and SIMD
Refactor CalcFrameSsim to decompose overlapping 8x8 SSIM windows
(stepped by 4 pixels horizontally and vertically) into non-overlapping
4x4 pixel tiles across a 2-row rotating buffer (row0, row1), reducing
memory reads and pixel accumulation work by 4x:
- Ssim4x4Row_* computes 4 uint32_t stats {sum_a, sum_b, sum_sq_ab,
sum_axb} per 4x4 pixel tile across a 4-row strip (also sized to fit
future 10-bit/12-bit formats like I010 without uint32_t overflow).
- SsimWindowRow_C combines top and bottom 4x4 tile strips across
adjacent horizontal pairs and returns the exact double SSIM sum,
producing bit-identical results to Ssim8x8_C.
- CalcFrameSsim rotates row0 and row1 (with thread-local buffers and
continuity tracking under _OPENMP) and falls back to Ssim8x8_C only
if buffer allocation fails.
- Implement Ssim4x4Row across all SIMD architectures:
- x86/x64: Ssim4x4Row_SSE2, Ssim4x4Row_AVX2, and Ssim4x4Row_AVX512BW
(with k-masked tail handling for 4..60 pixels) in both
compare_gcc.cc and compare_win.cc.
- ARM: Ssim4x4Row_NEON (compare_neon.cc, compare_neon64.cc),
Ssim4x4Row_NEON_DotProd (compare_neon64.cc), Ssim4x4Row_SVE2
(row_sve.h, row_sve.cc), and Ssim4x4Row_SME (row_sme.cc).
- RISC-V: Ssim4x4Row_RVV (row_rvv.cc).
- LoongArch: Ssim4x4Row_LSX (row_lsx.cc) and Ssim4x4Row_LASX
(row_lasx.cc).
- WebAssembly: Ssim4x4Row_WASMSIMD (compare_wasm.cc).
- Enable BenchmarkSsim_Opt and add BenchmarkSsim_Unaligned and
TestSsimAnyWidth in unit_test/compare_test.cc.
BenchmarkSsim_Opt (1280x720, single core, us/frame):
uArch Before (C) After (us) Improvement
-------------------------------- ---------- ----------- -----------
Intel Lion Cove (AVX2) 356.33 119.09 2.99x
AMD Zen 6 (AVX512BW) 455.86 124.67 3.66x
AMD Zen 5 (AVX512BW) 518.91 128.29 4.04x
Intel Emerald Rapids (AVX512BW) 601.58 180.41 3.33x
Intel Cougar Cove (AVX2) 608.11 183.34 3.32x
Intel Granite Rapids (AVX512BW) 596.57 187.04 3.19x
Intel Golden Cove (AVX2) 534.58 187.93 2.84x
Intel Sapphire Rapids (AVX512BW) 634.36 189.92 3.34x
AMD Zen 4 (AVX512BW) 713.06 223.65 3.19x
Intel Skymont (SSE2) 445.99 250.52 1.78x
Intel Ice Lake (AVX512BW) 845.16 267.23 3.16x
AMD Zen 3 (AVX2) 741.02 267.38 2.77x
AMD Zen 2 (AVX2) 907.04 301.02 3.01x
Intel Skylake-X (AVX512BW) 1097.86 301.63 3.64x
Intel Darkmont (SSE2) 772.18 383.39 2.01x
Intel Gracemont (SSE2) 961.80 421.83 2.28x
Intel Haswell (AVX2) 1159.74 479.14 2.42x
Intel Ivy Bridge (SSE2) 1296.99 597.32 2.17x
Apple M6 Super (SME) 404.66 90.09 4.49x
Apple M6 Perf (DotProd) 444.72 104.59 4.25x
Qualcomm Oryon-L (DotProd) 490.76 108.79 4.51x
Qualcomm Oryon V3-L (SVE2) 663.13 150.75 4.40x
ARM Cortex-X4 (DotProd) 620.21 177.29 3.50x
ARM Neoverse-V2 (SVE2) 682.93 205.18 3.33x
ARM Cortex-X3 (DotProd) 737.38 226.24 3.26x
ARM C1-Ultra (DotProd) 596.71 229.10 2.60x
ARM Cortex-X2 (DotProd) 820.94 247.94 3.31x
Qualcomm Oryon-M (DotProd) 1146.09 257.89 4.44x
Qualcomm Oryon V3-M (SME) 1068.78 276.23 3.87x
ARM Cortex-A720 (DotProd) 993.59 292.62 3.40x
ARM Cortex-X1 (DotProd) 912.19 326.44 2.79x
ARM Cortex-A715 (DotProd) 1135.67 340.12 3.34x
ARM Cortex-A710 (DotProd) 1222.04 342.51 3.57x
ARM Neoverse-N1 (DotProd) 1406.72 376.46 3.74x
ARM Cortex-A725 (SVE2) 1104.13 376.86 2.93x
ARM C1-Pro (SVE2) 1073.69 413.97 2.59x
ARM Cortex-A76 (DotProd) 1491.02 423.58 3.52x
ARM Cortex-A78 (DotProd) 1503.73 485.00 3.10x
AmpereOne (DotProd) 1518.61 518.40 2.93x
Apple M6 Eff (DotProd) 4051.10 572.93 7.07x
ARM Cortex-A75 (NEON) 2055.04 812.09 2.53x
ARM Cortex-A520 (DotProd) 3307.98 963.86 3.43x
ARM Cortex-A73 (NEON) 3120.09 1180.17 2.64x
Qualcomm Kryo Perf (NEON) 4921.16 1459.77 3.37x
ARM Cortex-A510 (DotProd) 4645.10 1641.06 2.83x
ARM Cortex-A55 (DotProd) 4571.02 1672.09 2.73x
Qualcomm Kryo Eff (NEON) 6000.96 1796.17 3.34x
ARM Cortex-A55r0 (NEON) 10486.61 4358.61 2.41x
ARM Cortex-A53 (NEON) 10121.34 5700.99 1.78x
R=rrwinterton@gmail.com
Bug: None
Test: out/Release/libyuv_unittest --gtest_filter=*Ssim*
Change-Id: Ibb95ae74424ecdda073ab7ab10ab1cf610b121b6
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8593040
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
|
||
|
Frank Barchard
|
0b9e5c922d |
Avoid signed int overflow in scale functions using uint32_t
Use uint32_t for the 16.16 fixed-point row coordinate (uy) and max_y in the vertical loops of ScalePlane*, ScaleUV*, and the Up2 linear helpers. With src_height up to 32768, (src_height << 16) and y += dy overflow signed int (e.g. for dst_height == 2 the post-read step reaches 5/4 * (src_height << 16), which exceeds INT_MAX for any src_height >= 26215, and for kFilterNone the final increment after the last row reaches (src_height << 16) + dy / 2). uint32_t cannot overflow for these ranges (uy <= 0xFFFF0000) and stays in a single 32-bit register on 32-bit targets (x86, Arm, WASM), unlike int64_t. Undo the interim +/-32767 height cap from https://crrev.com/c/8524280 so ScalePlane, ScalePlane_16, UVScale, and UVScale_16 accept |src_height| <= 32768 again. ScaleUVBilinearUp2 and ScaleUVBilinearUp2_16 take ptrdiff_t strides so 2 * dst_stride does not overflow int for images over 2 GB. ARGBScale, ARGBScaleClip, and UVScale_16 reject |src_width| > 32768 and |src_height| > 32768, and ScalePlane, ScalePlane_16, and ScaleSlope avoid FixedDiv(32768, 1) overflow when dst_width == 1 or dst_height == 1. Bug: chromium:531172025, chromium:546841026, libyuv:573815252 Test: libyuv_unittest --gtest_filter=*ScalePlaneOversizeHeight*:*ScalePlaneBoxMaxSrcHeight*:*ScaleLargeImageOver2GB*:*ARGBScale* Change-Id: Ica2e05d7dfe94f81c6631c587f1ff46c2ad44735 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8528398 Commit-Queue: Frank Barchard <fbarchard@google.com> Reviewed-by: Wan-Teh Chang <wtc@google.com> |
||
|
Frank Barchard
|
58d817ed81 |
Fix remaining inline asm input/output constraints and clobbers
Mark modified inline asm operands as read-write (+r / +c) and add missing volatile/memory clobbers so compilers and LTO do not assume modified registers remain unchanged across asm blocks: - HashDjb2_NEON (compare_neon64.cc): [src] "+r"(src) for post-increment ldr - FixedDiv1_X86 (scale_gcc.cc): "+c"(div) for sub $0x1, %1 - P410ToARGBRow_SVE_SC & P410ToAR30Row_SVE_SC (row_sve.h): [vl] "+r"(vl) for tail cntw %x[vl] - SwapUVRow_RVV & ARGBMultiplyRow_RVV (row_rvv.cc): use asm volatile and "memory" clobber instead of VLA memory operands on mutated pointers R=rrwinterton@gmail.com Bug: b/571179207 Test: libyuv_unittest Change-Id: I3c2475d1e5bdab13e74643a028eebc350edcc1bb Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8565646 Reviewed-by: richard winterton <rrwinterton@gmail.com> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
Frank Barchard
|
b9d572cc63 |
Add 25 native k-masked AVX-512 row kernels
R=rrwinterton@gmail.com Bug: None Change-Id: I2ee9b2c2d939f63f49d2f951f2d552ca6863d78b Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8564927 Commit-Queue: Frank Barchard <fbarchard@google.com> Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Frank Barchard
|
63b56324de |
[libyuv] Replace AVX-512 _Any wrappers with k-mask tail handling
Replace stack-buffer memcpy _Any wrappers with native AVX-512 k-mask
tail handling across 8 AVX-512 row kernels (unlocking composite
RGBToUVMatrixRow_AVX512BW as well) and remove IS_ALIGNED dispatch
constraints:
- MirrorSplitUVRow_AVX512BW: Masked 16-bit load (vmovdqu16 %{k1%}%{z%}),
dynamic reverse index (vpsubw + vpermw via kMirrorUVPermute_AVX512),
and vpmovwb %{k1%} U/V stores; unlocks unaligned SplitRotateUV180.
- ARGBToUVMatrixRow_AVX512BW: Fix MASK=63 vs 32-pixel loop bug;
zero-load tail dwords per row, duplicate odd final pixel via knotw +
vpshufd, and store masked U/V bytes.
- I422ToARGBRow_AVX512BW & I422ToAR30Row_AVX512BW: Add masked Y/U/V
loads (READYUV422_MASK_AVX512BW) and masked ARGB/AR30 stores.
- CopyRow_AVX512BW: Add 64-byte full vector step + bzhi masked tail.
- BlendPlaneRow_AVX512BW, ARGBShuffleRow_AVX512BW, RGBToARGBRow_AVX512BW
(RAWToARGBRow_AVX512BW / RGB24ToARGBRow_AVX512BW): Masked loads and
stores in both row_gcc.cc inline assembly and row_win.cc intrinsics.
Reduces x86-64 binary .text size by 2,592 bytes with 0 bytes added to
.rodata.
Performance on Intel Sapphire Rapids (1275x1650, 1000 iterations):
Function (Test) | Before | After | Speedup
-------------------------|---------|---------|--------
NV12Rotate180_Opt | 429 ms | 260 ms | 1.65x
RGB24ToI420_Opt | 615 ms | 531 ms | 1.16x
I420ToAR30_Opt | 425 ms | 383 ms | 1.11x
ARGBToI420_Opt | 427 ms | 411 ms | 1.04x
I420ToARGB_Opt | 423 ms | 411 ms | 1.03x
RGB24ToARGB_Opt | 493 ms | 490 ms | 1.01x
ARGBToBGRA_Opt | 544 ms | 542 ms | 1.00x
CopyPlane_Opt | 246 ms | 245 ms | 1.00x
BlendPlane_Opt | 259 ms | 258 ms | 1.00x
RAWToARGB_Opt | 486 ms | 486 ms | 1.00x
R=rrwinterton@gmail.com
Test: LIBYUV_WIDTH=1275 LIBYUV_HEIGHT=1650 LIBYUV_REPEAT=1000 out/Release/libyuv_unittest
Change-Id: Ib5e5ae52076352649108b20f8df2b12d81860fb1
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8563977
Reviewed-by: richard winterton <rrwinterton@gmail.com>
|
||
|
Wan-Teh Chang
|
1e2e3a3599 |
Avoid integer overflow if src_height == +/-32768
The ScalePlaneBox(), ScalePlaneBox_16(), and ScaleUVBox() functions all contain the following expression: const int max_y = (src_height << 16); where src_height is a positive int. This expression overflows if src_height >= 32768. Tracing the callers, we make sure ScalePlane(), ScalePlane_16(), and UVScale() reject src_height < -32767 || src_height > 32767 as invalid input. (These functions negate a negative src_height before calling a function that executes expression src_height << 16.) Although UVScale_16() do not execute the expression src_height << 16, make the same change to UVScale_16() for consistency with UVScale(). Bug: chromium:531172025 Change-Id: Id5b10f11f46f593945cf6bf1435b6027a6629815 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8524280 Reviewed-by: Frank Barchard <fbarchard@google.com> Commit-Queue: Wan-Teh Chang <wtc@google.com> |
||
|
Frank Barchard
|
16af162cf9 |
[libyuv] Optimize RotatePlane_16 SIMD and compact 32-bit transposes
Implement SIMD 16-bit plane rotation (TransposePlane_16, RotatePlane90_16, RotatePlane180_16, RotatePlane270_16) used by I010Rotate, I210Rotate, and I410Rotate, keeping tile sizes compact with full-register writes and zero cross-lane shuffles: - AVX2: 8x16 tile in ymm0..ymm15 with _Any tail (low-end x86 SIMD) - AVX-512BW: 8x32 tile in zmm0..zmm15 with k1/k2 right/bottom masks - NEON (ARM64/ARM32): 8x16 tile in v0..v23 / q0..q11 with _Any tail - SME: 8xVL tile using ZA16 horizontal/vertical array slices - RVV: 8xVL tile using widening unpacks with hardware vl tail - WASM SIMD: 8x8 tile in v128 with _Any tail Also update 32-bit ARGBRotate x86 transposes to compact vertical in-lane tiles with full-register writes and zero cross-lane shuffles: - TransposeWxH_32_AVX2: 4x8 tile in ymm0..ymm7 with vpmaskmovd tails - TransposeWxH_32_AVX512BW: 4x16 tile in zmm0..zmm7 with k1/k2 tails Add RotatePlane16_SplitMergeUV_Opt and RotatePlane16_SplitMergeUV_Odd unit tests comparing 16-bit rotation against SplitUVPlane + RotatePlane + MergeUVPlane across 0/90/180/270 degrees. Performance on 1275x1650 (1000 iterations, before -> after): CPU 90 degree 180 degree Coral RVV (Verilator) 7306 -> 1415c (5.16x) 2274 -> 719c (3.16x) AMD Rome (Zen 2) 821 -> 215ms (3.82x) 259 -> 181ms (1.44x) AMD Genoa (Zen 4) 483 -> 129ms (3.73x) 149 -> 111ms (1.34x) AMD Turin (Zen 5) 321 -> 101ms (3.18x) 109 -> 78ms (1.40x) AMD Milan (Zen 3) 505 -> 189ms (2.67x) 199 -> 118ms (1.69x) Intel Skylake-X 1076 -> 447ms (2.41x) 353 -> 317ms (1.11x) WASM SIMD (d8) 415 -> 203ms (2.04x) 370 -> 106ms (3.49x) Intel Emerald Rapids 735 -> 422ms (1.74x) 238 -> 233ms (1.02x) Intel Haswell 1120 -> 648ms (1.73x) 438 -> 393ms (1.11x) Intel Ice Lake 774 -> 474ms (1.63x) 279 -> 290ms (0.96x) ARM Neoverse-N1 864 -> 544ms (1.59x) 404 -> 266ms (1.52x) Ampere Siryn 862 -> 576ms (1.50x) 568 -> 329ms (1.72x) Google Axion (Maple) 554 -> 379ms (1.46x) 199 -> 142ms (1.41x) Intel Sapphire Rapids 1126 -> 772ms (1.46x) 301 -> 288ms (1.04x) Intel Granite Rapids 730 -> 565ms (1.29x) 280 -> 280ms (1.00x) NVIDIA Grace 462 -> 365ms (1.26x) 192 -> 126ms (1.52x) ARM Cortex-A53 28290 -> 24386ms (1.16x) 7178 -> 5075ms (1.41x) Google Axion 2 (Tamar) 523 -> 453ms (1.15x) 217 -> 112ms (1.93x) Test: libyuv_unittest --gtest_filter=*Rotate* Bug: b/42280942 Change-Id: I91ba6cac22185b5d294fcb98344fdb8a23cc94ff Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8545294 Reviewed-by: richard winterton <rrwinterton@gmail.com> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
Frank Barchard
|
eeb06a6451 |
[libyuv] Optimize ARGBRotate for Intel SSE2, AVX2, and AVX-512
Optimize 32-bit ARGBRotate (0, 90, 180, 270) across Intel x86/x64
architectures (SSE2, AVX2, AVX512BW), with matching kernels for ARM
(NEON, SVE2, SME), WASM, LoongArch (LSX, LASX), and RISC-V (RVV):
- Replace column-by-column ScaleARGBRowDownEven gather loops in
ARGBTranspose (90/270) with full-plane horizontal tile transposes:
- AVX512BW (TransposeWxH_32_AVX512BW / TransposeNx4_32_AVX512BW):
Transpose 16x16 blocks across zmm0..zmm31 using 4-stage
vpunpckldq/hdq, vpunpcklqdq/hqdq, and vshufi32x4 butterflies with
k-masked right and bottom tails, eliminating Any wrappers and scalar
C edge fallbacks.
- AVX2 (TransposeWxH_32_AVX2 / TransposeNx4_32_AVX2):
Transpose 8x8 blocks across ymm0..ymm15 using vpunpck* and
vperm2i128 with vpmaskmovd masked tails.
- SSE2 (Transpose4x4_32_SSE2 / Transpose4x4_32_Any_SSE2):
Transpose 4x4 blocks in xmm registers.
- Implement ARGBRotate180 directly via ARGBMirror with negative height,
avoiding intermediate row buffer allocations and extra copies.
- Implement ARGBRotate0 via ARGBCopy.
Speedup vs Before (1275x1650, US Letter 150 DPI, 1000 iterations):
CPU | 0 deg | 90 deg | 180 deg | 270 deg
---------------------|---------|---------|---------|--------
Intel Ice Lake | 1.02x | 1.34x | 1.18x | 1.33x
Intel Skylake-X | 1.02x | 1.20x | 1.07x | 1.13x
Intel Emerald Rapids | 1.01x | 1.07x | 1.00x | 1.07x
Intel AVX-512 vs C | 1.00x | 2.67x | 2.41x | 2.63x
R=rrwinterton@gmail.com
Bug: b/42280915
Test: LIBYUV_WIDTH=1275 LIBYUV_HEIGHT=1650 LIBYUV_REPEAT=1000 ./out/Release/libyuv_unittest --gtest_filter=*ARGBRotate*
Change-Id: Ib308a225c17e46eab083a9da4033598fcf0c303c
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8538986
Reviewed-by: richard winterton <rrwinterton@gmail.com>
|
||
|
Frank Barchard
|
4ce4edd50e |
Fix missing inline asm register clobbers
Change-Id: Ic573c8d7d8921a1dd00e4481b854a8168d0c94f7 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8543110 Reviewed-by: richard winterton <rrwinterton@gmail.com> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
Wan-Teh Chang
|
9456b06227 |
Remove the Abs() function in source/scale_uv.cc
The Abs() function is used at two places in source/scale_uv.cc as follows: src_width = Abs(src_width); However, before we reach those two places, we have rejected src_width <= 0 as invalid input. Therefore the two Abs(src_width) calls have no effects. Bug: None Change-Id: I04c48ad5d1270b9b93abb18f2b41b90ec818d5ea Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8531148 Reviewed-by: Frank Barchard <fbarchard@google.com> Commit-Queue: Wan-Teh Chang <wtc@google.com> |
||
|
Frank Barchard
|
1ddb18fb6b |
[libyuv] AVX512, NEON, WASM and RVV alpha copy and mirror rows
1275x1650 (US Letter 150 DPI, 1000 iterations) speedup: CPU ARGBExtractAlpha ARGBCopyYToAlpha ARGBMirror RGB24Mirror ----------------------------------------------------------------------- Intel Granite Rapids 1.00x 1.00x 1.00x 1.04x Intel Emerald Rapids 1.01x 0.97x 1.03x 1.07x Intel Sapphire Rapids 0.97x 1.01x 1.08x 1.26x Intel Ice Lake 0.99x 1.02x 0.97x 1.08x Intel Skylake-X 1.04x 1.03x 1.01x NA Intel Haswell NA NA NA NA AMD Turin (Zen 5) 1.01x 1.10x 1.25x 1.31x AMD Genoa (Zen 4) 0.97x 1.04x 1.09x 1.42x AMD Milan (Zen 3) NA NA NA NA AMD Rome (Zen 2) NA NA NA NA NVIDIA Grace (V2) 1.57x 2.51x NA NA Google Axion 2 1.12x 3.17x NA NA Ampere Siryn 1.06x 1.70x NA NA ARM Neoverse-N1 1.49x 2.01x NA NA Google Axion (Maple) 1.47x 2.03x NA NA ARM Cortex-A53 1.06x 1.69x NA NA Add AVX512BW ARGBExtractAlphaRow, ARGBCopyYToAlphaRow and ARGBMirrorRow, and AVX512VBMI RGB24MirrorRow, in row_gcc.cc and as intrinsics in row_win.cc. They are x64 only. The remainder is done with a k-mask (bzhi) zero-masked load and a masked store, so they handle any width without an Any wrapper and never access memory outside the row. RGB24MirrorRow_AVX512VBMI loads and stores 3 full zmm vectors per 64 pixels using 3 vpermi2b. The one output byte that comes from the third input vector is merged with a masked vpbroadcastb. The 1 to 63 pixel tail rebases the same tables by 192 - 3 * r; indices >= 128 are taken from the third vector with vpmovb2m and a masked vpermb. Add ARGBCopyYToAlphaRow_NEON for Arm 32 bit (vld4 + vst4) and AArch64 (zip + sli), and rewrite AArch64 ARGBExtractAlphaRow_NEON with ldp + uzp2. Both prefetch source and destination, which is 2.5x faster on Neoverse-N1 and 1.2x to 1.5x faster on Cortex-A53 for 1280x720. Add WASM SIMD ARGBExtractAlphaRow and ARGBCopyYToAlphaRow, using a swizzle table for CopyYToAlpha. Rework the RVV versions to use unit stride loads with vnsrl and vzext.vf4. R=rrwinterton@gmail.com Bug: None Change-Id: I436f27d70a3db0a2b176c91eb789a9491133cfcb Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8523531 Reviewed-by: richard winterton <rrwinterton@gmail.com> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
Frank Barchard
|
98697ab033 |
[libyuv] Add WASM SIMD128 compare, rotate, scale and row functions
d8 geomean speedup vs C compiled with -O2 -msimd128: File AMD Zen 3 ARM Neoverse-V2 row_wasm.cc 5.67x 3.24x scale_wasm.cc 5.47x 3.17x compare_wasm.cc 4.58x 5.77x rotate_wasm.cc 6.83x 6.03x Bug: None Change-Id: Ia06c805dc4029f94842c40e7301b2933beb8c1e5 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8524106 Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Frank Barchard
|
5323bab29e |
[libyuv] Clobber "vl" in RVV inline assembly only for GCC
Define RVV_VL_CLOBBER to expand to "vl", on GCC and empty on Clang in source/row_rvv.cc and source/scale_rvv.cc, and replace "vl", "vtype", in the RVV inline assembly clobber lists with RVV_VL_CLOBBER. R=masterwilliams@google.com, rrwinterton@gmail.com Test: ninja -C out/riscv64 libyuv_unittest Bug: b/443984359 Change-Id: Ie65bde737bd0714c3e409a2b8ab5fcd98b6a7a32 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8516707 Reviewed-by: Wan-Teh Chang <wtc@google.com> |
||
|
Frank Barchard
|
eb8eda9c79 |
[libyuv] Add WASM SIMD128 and Relaxed SIMD for RAWToRGB24Row
Implement RAWToRGB24Row_WASMSIMD and RAWToRGB24Row_Any_WASMSIMD in source/row_wasm.cc for RGB24ToRAW and RAWToRGB24 conversions: - Process 16 pixels (48 bytes) per loop iteration using three 128-bit loads, three byte-wise shuffles, and four 2-way blends. - Use wasm_i8x16_relaxed_swizzle when __wasm_relaxed_simd__ is enabled (compiling to a single vpshufb on x86 without bounds-checking overhead) and fall back to wasm_i8x16_swizzle for baseline SIMD128. - Add kCpuHasWASM (0x100) and kCpuHasWASMSIMD (0x101) CPU feature flags and enable -msimd128 in BUILD.bazel, BUILD.gn, and CMakeLists.txt for WebAssembly targets. Performance (RAWToRGB24_Opt, 1280x720, V8 d8): CPU WASM C SIMD128 Relaxed SIMD Spd Rlx Spd ------------------------------------------------------------------- AMD Zen 4 595.0 us 137.0 us 84.0 us 4.34x 7.08x AMD Zen 3 708.0 us 148.0 us 107.0 us 4.78x 6.62x AMD Zen 5 375.0 us 107.0 us 67.0 us 3.50x 5.60x ARM Neoverse-V2 638.0 us 118.0 us 116.0 us 5.41x 5.50x ARM Neoverse-N1 1046.0 us 294.0 us 294.0 us 3.56x 3.56x Intel Ice Lake 707.0 us 209.0 us 220.0 us 3.38x 3.21x Intel Haswell 1110.0 us 344.0 us 360.0 us 3.23x 3.08x Intel Emerald Rapids 519.0 us 207.0 us 208.0 us 2.51x 2.50x Intel Sapphire Rapids 551.0 us 220.0 us 255.0 us 2.50x 2.16x ARM Cortex-A53 14440.0 us 8660.0us 6920.0us 1.67x 2.09x Intel Granite Rapids 453.0 us 225.0 us 221.0 us 2.01x 2.05x Bug: 42271072 Test: libyuv_test --gunit_filter=*RAWToRGB24*:*RGB24ToRAW*:*Cpu* Change-Id: Id902668263a2ab70aa300eeeea18851490c917b5 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8515581 Reviewed-by: richard winterton <rrwinterton@gmail.com> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
Frank Barchard
|
2b7ecc0aad |
RAWToRGB24 for NEON, x86, and LoongArch
R=rrwinterton@gmail.com Test: libyuv_unittest --gtest_filter=*RAWToRGB24*:*RGB24ToRAW* Bug: None Change-Id: I4454f3785ba5c39dc776ba1f5151cd9d3e68055b Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8509286 Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Frank Barchard
|
42eb3a552f |
[libyuv] Add ScaleRowDown2Box_AVX512BW with tail masking
Add AVX-512BW implementations of ScaleRowDown2_AVX512BW, ScaleRowDown2Linear_AVX512BW, ScaleRowDown2Box_AVX512BW, and ScaleRowDown2Box_Odd_AVX512BW in scale_gcc.cc. Use 512-bit zmm vectors (64 source bytes -> 32 destination pixels per iteration) with vpmovwb narrowing stores, and handle remaining 1..31 destination pixels in-place using hardware mask register k1 (vmovdqu16 zero-masked loads for 2-byte source pixel pairs and vpmovwb masked narrowing store), eliminating the ScaleRowDown2*_Any_AVX2 fallback on AVX-512BW CPUs. Hook up the new row functions in ScalePlaneDown2 (source/scale.cc) and I420Blend (source/planar_functions.cc). I420ScaleDownBy2_Box (LIBYUV_WIDTH=638 LIBYUV_HEIGHT=360 LIBYUV_REPEAT=2000): CPU SSSE3 AVX2 AVX512 512/AVX2 AMD Turin (Zen 5) 57 62 36 1.72x Intel Granite Rapids 76 76 45 1.69x AMD Genoa (Zen 4) 85 84 50 1.68x Intel Sapphire Rapids 68 73 44 1.66x Intel Emerald Rapids 69 66 48 1.38x Intel Ice Lake 128 128 96 1.33x Intel Skylake-X 153 150 121 1.24x AMD Milan (Zen 3) 95 95 96 - Intel Haswell 193 172 162 - R=rrwinterton@gmail.com Bug: libyuv:960 Test: libyuv_unittest --gtest_filter=*ScalePlaneDown2*:*I420Blend* Change-Id: Ifbfcc4042c50d28022b12cb807dad56e66e7f301 TAG=agy CONV=5f01827f-2aa8-4718-b8c1-7f282730918e Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8504320 Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Philipp Hancke
|
aa6cedb39c |
Add a box filter to ScaleUV
kFilterBox only had 1/2 and 1/4 fast paths. Everything else fell through to ScaleUVSimple and point sampled, so NV12 chroma was unfiltered where I420 was not. 13.6 dB against 50.2 dB at 3x. ScaleUVBox mirrors ScalePlaneBox. Row accumulation reuses ScaleAddRow and its SIMD variants, since summing is position wise and an interleaved row is just src_cols * 2 samples. Only the column pass, which sums U and V at a stride of 2, is new. Output is now byte identical to the I420 path. Tests compare against ScalePlane rather than C against SIMD, which is what missed this. Bug: libyuv:564129034 Change-Id: I26221f5435dbfc845ad2fc13aa347ef56ccf7bc6 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8426668 Reviewed-by: richard winterton <rrwinterton@gmail.com> Reviewed-by: Frank Barchard <fbarchard@google.com> |
||
|
Frank Barchard
|
67b3c0dad8 |
Compare AVX512BW and AVX2, intrinsics for Visual C and ROWWIN
- Add HammingDistance_AVX512BW, SumSquareError_AVX512BW and
HashDjb2_AVX512BW for 32 and 64 bit, as inline assembly in
compare_gcc.cc and intrinsics in compare_win.cc.
- AVX512BW processes 64 bytes per loop with full vectors, and the
remainder of 1 to 63 bytes with one masked load, so compare.cc passes
any count and does not use C for the remainder. HashDjb2 loads the
remainder into the high bytes, ending at src + count, so the same
33 ^ (63 - i) multipliers apply, and multiplies the hash by 33 ^ n.
- Add SumSquareError_AVX2 and HashDjb2_AVX2 to compare_gcc.cc and
HammingDistance_AVX2 to compare_win.cc, so both files have the same
functions. Enable AVX2 in compare_gcc.cc for gcc as well as clang.
- HashDjb2 AVX2 and AVX512BW split the 32 bit multipliers 33 ^ i into
16 bit halves and use vpmaddwd, instead of vpmulld which is 2 uops
with 10 cycle latency. Bytes are zero extended with vpmovzxbw from
memory. The hash = hash * 33 ^ 64 + sum dependency uses scalar imul
instead of vpmulld. AVX2 does 64 bytes per horizontal add.
- Port compare_win.cc from 32 bit Visual C inline assembly to
intrinsics and enable it for Visual C 32/64 bit and
LIBYUV_ENABLE_ROWWIN, which MemorySanitizer also enables. Visual C 64
bit and MemorySanitizer previously used C.
- Fix HammingDistance_SSSE3 and HammingDistance_AVX2 to allow unaligned
pointers. They used movdqa and crashed on unaligned src_a.
- Add TestCompareAnyCount to test every count from 0 to 321, aligned
and unaligned, ending at the end of a page.
Functions available:
Function compare_gcc compare_win
32/64 bit 32/64 bit
HammingDistance_SSE42 asm intrinsics
HammingDistance_SSSE3 asm -
HammingDistance_AVX2 asm intrinsics
HammingDistance_AVX512BW asm intrinsics
SumSquareError_SSE2 asm intrinsics
SumSquareError_AVX2 asm intrinsics
SumSquareError_AVX512BW asm intrinsics
HashDjb2_SSE41 asm intrinsics
HashDjb2_AVX2 asm intrinsics
HashDjb2_AVX512BW asm intrinsics
Performance, 1280x720, LIBYUV_REPEAT=1000, best of 3, ms.
Before is main (SSE, and AVX2 Hamming only, in compare_gcc).
BenchmarkDjb2_Opt main gcc gcc win win
AVX2 AVX512 AVX2 AVX512
Intel Haswell 179 77 - 75 -
AMD Milan (Zen 3) 101 35 - 35 -
Intel Skylake-X 214 63 55 67 59
Intel Ice Lake 191 51 47 52 49
Intel Sapphire Rapids 175 39 39 38 39
Intel Granite Rapids 170 40 38 40 38
AMD Genoa (Zen 4) 91 34 37 34 38
AMD Turin (Zen 5) 80 29 22 28 23
C 502 to 669
BenchmarkSumSquareError main gcc gcc win win
_Opt AVX2 AVX512 AVX2 AVX512
Intel Haswell 79 37 - 38 -
AMD Milan (Zen 3) 47 24 - 25 -
Intel Skylake-X 78 32 26 34 24
Intel Ice Lake 86 43 23 43 22
Intel Sapphire Rapids 48 24 21 24 22
Intel Granite Rapids 47 23 21 23 21
AMD Genoa (Zen 4) 50 25 21 25 21
AMD Turin (Zen 5) 57 28 13 28 14
C 154 to 425
Testing:
LIBYUV_WIDTH=1279 LIBYUV_HEIGHT=719 out/Release/libyuv_unittest \
--gtest_filter='*Compare*:*Djb2*:*Hamming*:*SumSquare*:*Psnr*:*Ssim*'
With out/Release, out/Release_x86, out/rowwin, out/rowwin_x86 and
out/msan, with LIBYUV_DISABLE_AVX512BW and LIBYUV_DISABLE_AVX2, and
LIBYUV_WIDTH of 1, 31 to 33, 63 to 65, 95, 127, 129 and 200. Under
sde64 -skx for AVX512BW with out/Release and out/rowwin.
Built out/android_x86.
Bug: None
Change-Id: Ief197d546daca3db02ee1503d4649f4c755eb189
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8504717
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
|
||
|
Frank Barchard
|
bad060c3a5 |
Rotate intrinsics for Visual C, ROWWIN and 32 bit x86
- Port rotate_gcc.cc to intrinsics in rotate_win.cc: TransposeWx8_SSSE3,
TransposeUVWx8_SSE2, TransposeWx16_AVX2, TransposeUVWx16_AVX2,
Transpose4x4_32_SSE2, Transpose4x4_32_AVX2, and for 64 bit,
TransposeWx16_AVX512BW and TransposeUVWx16_AVX512BW.
- Remove the 32 bit Visual C inline assembly.
- Enable rotate_win.cc for Visual C 32/64 bit, LIBYUV_ENABLE_ROWWIN,
which MemorySanitizer also enables, and all 32 bit x86 compilers.
Visual C 64 bit previously used C.
- rotate_gcc.cc is now 64 bit only. 32 bit gcc and clang only had an
8x8 TransposeWx8_SSSE3 and Transpose4x4_32, because the UV and AVX2
inline assembly uses xmm8..xmm15, so 32 bit NV12 rotate used C.
Remove the 32 bit TransposeWx8_SSSE3 inline assembly.
- Add a target("sse2") attribute to the SSE2 intrinsics so gcc and
clang build them when the baseline lacks SSE2, such as
-march=pentium3.
- Disable the generic x86 scale_row.h and compare_row.h HAS_ macros for
32 bit LIBYUV_ENABLE_ROWWIN, as is done for 64 bit. scale_gcc.cc and
compare_gcc.cc are disabled for ROWWIN and scale_win.cc and
compare_win.cc require _MSC_VER, so out/rowwin_x86 failed to link with
undefined symbols such as ScaleARGBCols_SSE2 and HashDjb2_SSE41.
- TransposeWx8_SSSE3 and TransposeUVWx8_SSE2 share a 16x8 helper.
- TransposeWx8_SSSE3 requires a multiple of 16 on all platforms.
- Byte transposes use a shared helper with dst_a / dst_b outputs so
2, 3 and 4 channel variants can reuse the same shuffles.
Functions available:
Function rotate_gcc rotate_win rotate_win
64 bit 64 bit 32 bit
TransposeWx8_SSSE3 asm intrinsics intrinsics
TransposeUVWx8_SSE2 asm intrinsics intrinsics
TransposeWx16_AVX2 asm intrinsics intrinsics
TransposeUVWx16_AVX2 asm intrinsics intrinsics
Transpose4x4_32_SSE2 asm intrinsics intrinsics
Transpose4x4_32_AVX2 asm intrinsics intrinsics
TransposeWx16_AVX512BW asm intrinsics -
TransposeUVWx16_AVX512BW asm intrinsics -
32 bit x86 Linux clang on AMD Milan (Zen 3), ms
(LIBYUV_WIDTH=1280 LIBYUV_HEIGHT=720 LIBYUV_REPEAT=2000):
C Before After SSE2 After AVX2
I420Rotate90_Opt 944 321 322 220
NV12Rotate90_Opt 1412 935 315 232
Testing:
for w in 1 2 3 7 8 9 15 16 17 23 24 31 32 33 64 65 1279 1280; do
for h in 1 7 8 15 16 17 24 31 32 33 719; do
LIBYUV_WIDTH=$w LIBYUV_HEIGHT=$h out/rowwin/libyuv_unittest \
--gtest_filter='*Rotate*:*Transpose*'
done
done
Repeated with LIBYUV_DISABLE_AVX2=1, LIBYUV_DISABLE_AVX512BW=1, with
out/Release and out/Release_x86, under sde64 -skx for AVX512BW, and
with out/msan (is_msan = true) and out/rowwin_x86
(target_cpu = "x86", libyuv_enable_rowwin = true). Built out/android_x86
(target_os = "android", target_cpu = "x86"). Compiled rotate_win.cc
with g++ and clang++ -m32 -march=pentium3.
Bug: None
Change-Id: Ic70c4f0b7eb46b1b8522a1eba851dac5f506ec3e
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8499394
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
|
||
|
Frank Barchard
|
20b0a91bc1 |
Transpose 16x16 for AVX2
- TransposeWx16_AVX2 and TransposeUVWx16_AVX2 using 2 lanes of 8 ymm registers, with Any for tail. - Move TransposeUVWx16_C to rotate_common.cc and pass the C function to TUVANY, so the LSX 16 row Any version transposes all 16 rows. I420Rotate90_Opt (LIBYUV_WIDTH=1279 LIBYUV_HEIGHT=719 LIBYUV_REPEAT=1000): CPU SSSE3 AVX2 AVX512 AVX2 x 512 x Intel Haswell 1812 916 945 1.98x 1.92x AMD Milan (Zen 3) 177 118 118 1.50x 1.50x Intel Granite Rapids 245 186 186 1.32x 1.32x Intel Emerald Rapids 246 193 176 1.27x 1.40x Intel Sapphire Rapids 278 233 225 1.19x 1.24x Intel Ice Lake 308 213 221 1.45x 1.39x Intel Skylake-X 365 267 246 1.37x 1.48x AMD Turin (Zen 5) 106 75 75 1.41x 1.41x AMD Genoa (Zen 4) 149 98 93 1.52x 1.60x AMD Venice (Zen 6) 106 74 73 1.43x 1.45x NV12Rotate90_Opt (LIBYUV_WIDTH=1279 LIBYUV_HEIGHT=719 LIBYUV_REPEAT=1000): CPU SSSE3 AVX2 AVX512 AVX2 x 512 x Intel Haswell 1771 963 935 1.84x 1.89x AMD Milan (Zen 3) 173 118 118 1.47x 1.47x Intel Granite Rapids 275 189 186 1.46x 1.48x Intel Emerald Rapids 256 188 179 1.36x 1.43x Intel Sapphire Rapids 286 232 218 1.23x 1.31x Intel Ice Lake 309 208 218 1.49x 1.42x Intel Skylake-X 380 269 251 1.41x 1.51x AMD Turin (Zen 5) 112 74 73 1.51x 1.53x AMD Genoa (Zen 4) 146 97 93 1.51x 1.57x AMD Venice (Zen 6) 114 73 72 1.56x 1.58x Bug: None Change-Id: Ia89219e3336f39a817aa662e1452824d781664a3 TAG=agy CONV=bea53b37-b2bf-46fa-9e44-a708b438fabc Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8489456 Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Frank Barchard
|
98fa761478 |
Rotate for AVX512
- TransposeWx16_AVX512BW and TransposeUVWx16_AVX512BW - tail masking instead of 'Any' to odd source width I420Rotate90_Opt (LIBYUV_WIDTH=1279 LIBYUV_HEIGHT=719 LIBYUV_REPEAT=1000): CPU Before After Speedup Intel Emerald Rapids 249 ms 182 ms 1.37x Intel Ice Lake 311 ms 234 ms 1.33x Intel Granite Rapids 246 ms 186 ms 1.32x Intel Sapphire Rapids 274 ms 226 ms 1.21x NV12Rotate90_Opt (LIBYUV_WIDTH=1279 LIBYUV_HEIGHT=719 LIBYUV_REPEAT=1000): CPU Before After Speedup Intel Granite Rapids 263 ms 181 ms 1.45x Intel Emerald Rapids 257 ms 180 ms 1.43x Intel Ice Lake 311 ms 233 ms 1.33x Intel Sapphire Rapids 282 ms 222 ms 1.27x Bug: None Change-Id: Ibe0df79275099dc06bf4218484a24732cd7fb9b5 TAG=agy CONV=bea53b37-b2bf-46fa-9e44-a708b438fabc Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8497726 Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Frank Barchard
|
eee4597984 |
Fix _M_X86 to _M_IX86
Fix _M_X86 to _M_IX86 for Visual C 32 bit Visual C defines _M_IX86 for 32 bit x86, not _M_X86. row.h and row_win.cc tested _M_X86, so a real 32 bit Visual C build got neither USE_ROW_WIN nor USE_ROW_GCC, row_win.cc compiled to nothing, and the HAS_*_AVX2 defines for all x86 platforms were skipped. clang-cl and Android x86 are unaffected since clang defines __i386__. Testing: ninja -C out/Release libyuv_unittest && out/Release/libyuv_unittest ninja -C out/Release_x86 libyuv_unittest && out/Release_x86/libyuv_unittest ninja -C out/rowwin libyuv_unittest && out/rowwin/libyuv_unittest blaze test --config=windows_x86 //third_party/libyuv:libyuv_test blaze build --config=android_x86 //third_party/libyuv:libyuv Bug: None Change-Id: I0f90c03855f2981040ab4f250cd2facc9ef47ac0 TAG=agy CONV=bea53b37-b2bf-46fa-9e44-a708b438fabc Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8499834 Commit-Queue: Frank Barchard <fbarchard@google.com> Reviewed-by: Mirko Bonadei <mbonadei@chromium.org> |
||
|
Frank Barchard
|
18835bfb35 |
ARGBToRGB24 AVX512 with tail masking
Bug: None Change-Id: I7590c66940b5354ca0fdd8ff120392434d2785d7 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8490011 Reviewed-by: richard winterton <rrwinterton@gmail.com> Commit-Queue: Frank Barchard <fbarchard@google.com> |
||
|
Frank Barchard
|
b25fa89920 |
InterpolateRow_AVX512BW with tail masking
I420ScaleTo569x480_Bilinear (LIBYUV_WIDTH=569 LIBYUV_REPEAT=5000): CPU AVX2 AVX512 Speedup Intel Emerald Rapids 124 ms 63 ms 1.97x Intel Sapphire Rapids 134 ms 69 ms 1.94x Intel Skylake-X 219 ms 129 ms 1.70x Intel Granite Rapids 124 ms 76 ms 1.63x Intel Ice Lake 164 ms 101 ms 1.62x Bug: None Change-Id: I7a10446eb4dbcebae68380c2a186d4e3badec551 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8483371 Commit-Queue: Frank Barchard <fbarchard@google.com> Reviewed-by: richard winterton <rrwinterton@gmail.com> |
||
|
Frank Barchard
|
39445fa162 |
[libyuv] Fix RISC-V CMake/GN/Bazel builds and C compiler driver linking
- Use -fno-tree-vectorize and -fno-tree-slp-vectorize instead of Clang-only -fno-vectorize and -fno-slp-vectorize in CMakeLists.txt, riscv-clang.cmake, BUILD.gn, and BUILD.bazel for GCC and Clang compatibility. - Split row_rvv.cc and scale_rvv.cc into a dedicated libyuv_rvv target in BUILD.gn and define LIBYUV_RVV so callers in row.h and scale_row.h can dispatch to RVV kernels when built without global -march=rv64gcv. - Guard the x86 AVX512-BF16 vdpphps inline assembly probe in util/cpuid.c with __i386__ / __x86_64__. - Replace new/delete and new[]/delete[] with calloc/malloc/free in source/mjpeg_decoder.cc and util/yuvconvert.cc to avoid C++ standard library dependencies, and link libm for yuvconstants and libyuv_unittest in CMakeLists.txt. Test: cmake -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ .. Bug: None Change-Id: I701c0625f2730e864806556213a5e013c931e2d6 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8464751 Reviewed-by: richard winterton <rrwinterton@gmail.com> Commit-Queue: Frank Barchard <fbarchard@google.com> |