Because each 8x8 SSIM window spans two 4x4 tiles horizontally,
num_windows = (width - 5) >> 2, so aligned widths that are multiples
of 64 (e.g. 64, 128, 640, 1280, 1920) always have remainder 15
(num_windows & 15 == 15).
- Update SsimWindowRow_AVX512BW (compare_gcc.cc and compare_win.cc)
to handle any remainder (1..15 windows) via k-masking (load_mask
for the (rem + 1) * 4 tile words and win_mask for the rem window
scores) and set win_simd_size = 1.
- Add SsimWindowRow_SVE2, SsimWindowRow_SME, and SsimWindowRow_RVV
with win_simd_size = 1 using predicated/vl-controlled loops so no
C code (Ssim4x4Row_C or SsimWindowRow_C) is executed on AVX-512BW,
SVE2, SME, or RVV CPUs for any image dimensions.
- Remove LIBYUV_TARGET_AVX512BW __attribute__((target(...))) from
compare_gcc.cc and GCC_HAS_AVX512 from cpu_support.h, using %%zmm0
directly in inline assembly.
SSIM Performance (Before vs After):
uArch / Target Before After Speedup
-------------------------- --------- --------- -------
Coral RVV Verilator (64x16) 5302 cyc 2219 cyc 2.39x
Coral RVV WindowRow (15w) 1854 cyc 308 cyc 6.02x
Apple M6 SME (64x36 10kx) 16.41 ms 4.69 ms 3.50x
Apple M6 SME (720p 1000x) 350.76 ms 66.09 ms 5.31x
R=rrwinterton@gmail.com
Bug: libyuv:352521931
Change-Id: Ic7d55184c60611af4a57d113cc4ee3d211debadf
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8621918
Commit-Queue: Frank Barchard <fbarchard@google.com>
Reviewed-by: richard winterton <rrwinterton@gmail.com>
- Route 32-bit x86 (__i386__ / _M_IX86) to compare_win.cc intrinsics
and restrict compare_gcc.cc to 64-bit x86 (__x86_64__), avoiding
x87 float rounding and enabling SSE2, AVX2, and AVX-512 on 32-bit
and 64-bit MinGW, GCC, Clang, and MSan (LIBYUV_ENABLE_ROWWIN).
- Enable AVX-512BW on GCC >= 6 (GCC_HAS_AVX512) and add
LIBYUV_TARGET_AVX512BW attributes to compare_gcc.cc and
compare_win.cc.
- Refactor SsimWindowRow_C and Ssim8x8_C to use int32_t and float,
and add SsimWindowRow_SSE2, SsimWindowRow_AVX2, and
SsimWindowRow_AVX512BW in both compare_win.cc and compare_gcc.cc.
- Update TestSsimAnyWidth, Ssim, and BenchmarkSsim_Opt in
compare_test.cc to use EXPECT_NEAR and deterministic test patterns
(identical, different, similar).
- Update util/compare.cc and linux.mk to compute I420Ssim.
Performance (BenchmarkSsim_Opt, 1280x720):
uArch Before After Improvement
Intel Lion Cove (x64 AVX2) 120.2 us 68.1 us 1.76x
Intel Skymont (x64 SSE2) 249.6 us 146.8 us 1.70x
AMD Zen 3 (x64 AVX2) 314.9 us 150.2 us 2.10x
Intel Lion Cove (MinGW686) 203.7 us 77.9 us 2.61x
Intel Skymont (MinGW686) 338.8 us 167.5 us 2.02x
AMD Zen 3 (i686 AVX2) 738.4 us 206.5 us 3.58x
Quality (util/compare on 640x360 I420 test images):
Image Comparison MSE PSNR SSIM
Identical (orig vs orig) 0.00 128.00 dB 1.000000
Similar (light +/-3 noise) 4.57 41.53 dB 0.956556
Degraded (16x16 blocky) 98.18 28.21 dB 0.858893
Very different (inverted) 11160.29 7.65 dB 0.000181
Co-authored-by: Martin Storsjö <martin@martin.st>
Bug: b/493910265
Test: out/Release/libyuv_unittest --gtest_filter=*Ssim*
Change-Id: I8603dab3f752af32d07ae9039502ca5da5a05425
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8618641
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
1. Fix multi-bit TestCpuFlag(A | B) checks in AYUVToNV12, AYUVToNV21,
RAWToRGBA, NV12ToRGB24Matrix, NV21ToRGB24Matrix, MergeRGBPlane, and
MergeXR30Plane to compare == (A | B) so AVX-512 VBMI/BW kernels are
not dispatched when AVX512VBMI or AVX512VL is absent.
2. Fix 2-byte SVE2 heap buffer overflow on odd widths in
I422ToARGB1555Row_SVE_SC and I422ToARGB4444Row_SVE_SC using
trn1/trn2 + st2b under .b predicates.
3. Fix ARGBPolynomialRow_SSE2 to use packssdw before packuswb, and add
missing vzeroupper to SplitRGBRow_AVX2.
4. Fix CalcFrameSsim to return 0.0 instead of 0.0 / 0 (NaN) when
width <= 8 || height <= 8 so I420Ssim stays finite on small frames.
5. Fix negative-height vertical inversion in ConvertFromI420,
SplitARGBPlane (when dst_a is NULL), I422Rotate, I010Rotate, and
I210Rotate.
6. Fix horizontal flip (negative x/dx) in ScaleFilterCols_NEON (signed
sbfx #16, #16 offset extraction) and ScaleAddCols1_16_C (shift
x >>= 16 before indexing src_ptr).
7. Fix ConvertToARGB odd crop_x chroma alignment in NV12/NV21, src_u
offset in I420/YV12/J420/H420/U420, and FOURCC_U422 calling
U422ToARGB instead of H422ToARGB.
8. Remove dead 14-instruction block in ScaleRowUp2_Bilinear_16_SSE2.
9. Fix UVScale_16 for src_width == 1 (P010ToP410/P210ToP410), enable
and fix column order in TransposeWx4_32_SVE2, fix ByteToFloatRow
NEON Any BPP (3 -> 4), fix AB64ToARGBRow_Any_*, ScaleColsUp2_SSE2,
ScaleUVRowDown2Box_SME, and UYVYToARGBRow_SME macro guards, fix SVE2
loop tail branches (b.lt/b.ge), and remove duplicate HAS_* macros.
10. Fix ConvertFP16ToFP32Column_NEON loop condition and unsigned b.lo
underflow on widths < 8 or not multiples of 8.
11. Fix UYVYToNV12 IS_ALIGNED(awidth, ...) checks and WASM SIMD
IS_ALIGNED(width, 16) checks in I420ToRGB24Matrix /
I422ToRGB24Matrix.
12. Wire SplitUVRow_SVE2, SplitUVRow_LSX, and SplitUVRow_LASX in
NV24ToARGB.
13. Fix MJpegDecoder heap buffer overflows in LoadFrame (reallocate
databuf_[i] when scanlines_size changes), DecodeToBuffers, and
DecodeToCallback (clamp scanlines_to_copy to lines_left and align
skip to max_v_samp_factor).
14. Replace width & (vl - 1) tail calculations with post-loop remainder
across 18 SVE2 functions in row_sve.h for non-power-of-2 SVE VL,
and remove duplicate label 2: in UYVYToARGBRow_SVE_SC.
15. Handle width & 15 tail pixels in DetileToYUY2_C and validate
IS_POWEROFTWO(tile_height) in DetileSplitUVPlane and DetileToYUY2.
16. Avoid signed 32-bit overflow in DivideRow_16_C and C16TO8, fix
ClampFloatToZero_SSE2 pointer increment (lea 0x4) and loop counter
(sub $0x1), and initialize xmm3 in ARGBCopyYToAlphaRow_SSE2.
17. Check max leaf 7 subleaf and max extended CPUID leaf in GetCpuFlags
before querying subleaf 7:1 and extended leaf 0x80000021.
18. Free plane_u on error in I422ToNV21, and fix 4x/2x scratch row
over-allocation in ScaleARGBBilinearDown and ScaleUVBilinearDown.
19. Add ARGBSetRow_RVV and ARGBShuffleRow_RVV in row_rvv.cc.
20. Fix NEON mask (15 -> 7) in ScaleUVRowUp2_Linear_Any_NEON /
_16_Any_NEON, use SIMD InterpolateRow + SplitUVRow in NV16ToI420,
wire SSSE3/SSE2 in ARGBToI400Matrix / ARGBToYUY2Matrix /
ARGBToUYVYMatrix and RGB24Mirror, and split HalfFloatRow multiply
into (src * scale) * 2^-112 to avoid 50x Intel denormal stalls.
Bug: b/574714081
Test: libyuv_unittest
Change-Id: Iec3ae5c58d52bee3ef13cf9b765ac84b733aa576
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8601931
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
Based on Philipp Hancke's AVX2 column kernel for ScaleUVBox at 3x
(https://crrev.com/c/8427347).
Add ScaleUVAddCols3 for exact 3x horizontal downscaling (dx == 0x30000
and boxheight <= 85) in ScaleUVBox, plus ScaleAddRow_AVX512BW and
ScaleAddRow_WASMSIMD dispatch, and port 3x/box scaling kernels in
scale_win.cc to 32/64-bit C++ intrinsics matching scale_gcc.cc:
- ScaleUVAddCols3_C (portable C)
- ScaleUVAddCols3_AVX2 and ScaleUVAddCols3_Any_AVX2 (scale_gcc.cc and
scale_win.cc)
- ScaleUVAddCols3_AVX512VBMI (scale_gcc.cc and scale_win.cc)
- ScaleAddRow_SSE2, ScaleAddRow_AVX2, ScaleAddRow_AVX512BW,
ScaleAddCols1_AVX512BW, ScaleAddCols2_AVX512BW, ScaleRowDown38_SSSE3,
ScaleRowDown38_2_Box_SSSE3, and ScaleRowDown38_3_Box_SSSE3 in
scale_win.cc for 32/64-bit Visual C and LIBYUV_ENABLE_ROWWIN
- ScaleUVAddCols3_NEON (32-bit ARM and 64-bit AArch64) and
ScaleUVAddCols3_Any_NEON
- ScaleUVAddCols3_SME (ARMv9 Streaming SVE with whilelt predicate tail)
- ScaleUVAddCols3_RVV (RISC-V Vector vlseg3e32.v inline asm)
- ScaleUVAddCols3_LSX and ScaleUVAddCols3_Any_LSX (LoongArch LSX)
- ScaleAddRow_WASMSIMD, ScaleAddRow_Any_WASMSIMD,
ScaleUVAddCols3_WASMSIMD, and ScaleUVAddCols3_Any_WASMSIMD
- ScaleUVAddCols3_HVX (Hexagon HVX vrdelta + vmpy_VuhRuh)
Co-authored-by: Philipp Hancke <philipp.hancke@googlemail.com>
Bug: libyuv:564129034
R=rrwinterton@gmail.com
Change-Id: Ib9e14f3efcb097842c3d0e9fc9b473b6e2f81871
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8596436
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Use uint32_t for the 16.16 fixed-point row coordinate (uy) and max_y
in the vertical loops of ScalePlane*, ScaleUV*, and the Up2 linear
helpers. With src_height up to 32768, (src_height << 16) and y += dy
overflow signed int (e.g. for dst_height == 2 the post-read step
reaches 5/4 * (src_height << 16), which exceeds INT_MAX for any
src_height >= 26215, and for kFilterNone the final increment after
the last row reaches (src_height << 16) + dy / 2). uint32_t cannot
overflow for these ranges (uy <= 0xFFFF0000) and stays in a single
32-bit register on 32-bit targets (x86, Arm, WASM), unlike int64_t.
Undo the interim +/-32767 height cap from https://crrev.com/c/8524280
so ScalePlane, ScalePlane_16, UVScale, and UVScale_16 accept
|src_height| <= 32768 again.
ScaleUVBilinearUp2 and ScaleUVBilinearUp2_16 take ptrdiff_t strides
so 2 * dst_stride does not overflow int for images over 2 GB.
ARGBScale, ARGBScaleClip, and UVScale_16 reject |src_width| > 32768
and |src_height| > 32768, and ScalePlane, ScalePlane_16, and
ScaleSlope avoid FixedDiv(32768, 1) overflow when dst_width == 1 or
dst_height == 1.
Bug: chromium:531172025, chromium:546841026, libyuv:573815252
Test: libyuv_unittest --gtest_filter=*ScalePlaneOversizeHeight*:*ScalePlaneBoxMaxSrcHeight*:*ScaleLargeImageOver2GB*:*ARGBScale*
Change-Id: Ica2e05d7dfe94f81c6631c587f1ff46c2ad44735
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8528398
Commit-Queue: Frank Barchard <fbarchard@google.com>
Reviewed-by: Wan-Teh Chang <wtc@google.com>
The ScalePlaneBox(), ScalePlaneBox_16(), and ScaleUVBox() functions all
contain the following expression:
const int max_y = (src_height << 16);
where src_height is a positive int. This expression overflows if
src_height >= 32768.
Tracing the callers, we make sure ScalePlane(), ScalePlane_16(), and
UVScale() reject src_height < -32767 || src_height > 32767 as invalid
input. (These functions negate a negative src_height before calling a
function that executes expression src_height << 16.)
Although UVScale_16() do not execute the expression src_height << 16,
make the same change to UVScale_16() for consistency with UVScale().
Bug: chromium:531172025
Change-Id: Id5b10f11f46f593945cf6bf1435b6027a6629815
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8524280
Reviewed-by: Frank Barchard <fbarchard@google.com>
Commit-Queue: Wan-Teh Chang <wtc@google.com>
The Abs() function is used at two places in source/scale_uv.cc as
follows:
src_width = Abs(src_width);
However, before we reach those two places, we have rejected
src_width <= 0 as invalid input. Therefore the two Abs(src_width) calls
have no effects.
Bug: None
Change-Id: I04c48ad5d1270b9b93abb18f2b41b90ec818d5ea
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8531148
Reviewed-by: Frank Barchard <fbarchard@google.com>
Commit-Queue: Wan-Teh Chang <wtc@google.com>
1275x1650 (US Letter 150 DPI, 1000 iterations) speedup:
CPU ARGBExtractAlpha ARGBCopyYToAlpha ARGBMirror RGB24Mirror
-----------------------------------------------------------------------
Intel Granite Rapids 1.00x 1.00x 1.00x 1.04x
Intel Emerald Rapids 1.01x 0.97x 1.03x 1.07x
Intel Sapphire Rapids 0.97x 1.01x 1.08x 1.26x
Intel Ice Lake 0.99x 1.02x 0.97x 1.08x
Intel Skylake-X 1.04x 1.03x 1.01x NA
Intel Haswell NA NA NA NA
AMD Turin (Zen 5) 1.01x 1.10x 1.25x 1.31x
AMD Genoa (Zen 4) 0.97x 1.04x 1.09x 1.42x
AMD Milan (Zen 3) NA NA NA NA
AMD Rome (Zen 2) NA NA NA NA
NVIDIA Grace (V2) 1.57x 2.51x NA NA
Google Axion 2 1.12x 3.17x NA NA
Ampere Siryn 1.06x 1.70x NA NA
ARM Neoverse-N1 1.49x 2.01x NA NA
Google Axion (Maple) 1.47x 2.03x NA NA
ARM Cortex-A53 1.06x 1.69x NA NA
Add AVX512BW ARGBExtractAlphaRow, ARGBCopyYToAlphaRow and
ARGBMirrorRow, and AVX512VBMI RGB24MirrorRow, in row_gcc.cc and as
intrinsics in row_win.cc. They are x64 only. The remainder is done
with a k-mask (bzhi) zero-masked load and a masked store, so they
handle any width without an Any wrapper and never access memory
outside the row.
RGB24MirrorRow_AVX512VBMI loads and stores 3 full zmm vectors per
64 pixels using 3 vpermi2b. The one output byte that comes from the
third input vector is merged with a masked vpbroadcastb. The 1 to 63
pixel tail rebases the same tables by 192 - 3 * r; indices >= 128 are
taken from the third vector with vpmovb2m and a masked vpermb.
Add ARGBCopyYToAlphaRow_NEON for Arm 32 bit (vld4 + vst4) and
AArch64 (zip + sli), and rewrite AArch64 ARGBExtractAlphaRow_NEON
with ldp + uzp2. Both prefetch source and destination, which is
2.5x faster on Neoverse-N1 and 1.2x to 1.5x faster on Cortex-A53
for 1280x720. Add WASM SIMD ARGBExtractAlphaRow and
ARGBCopyYToAlphaRow, using a swizzle table for CopyYToAlpha. Rework
the RVV versions to use unit stride loads with vnsrl and vzext.vf4.
R=rrwinterton@gmail.com
Bug: None
Change-Id: I436f27d70a3db0a2b176c91eb789a9491133cfcb
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8523531
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
Define RVV_VL_CLOBBER to expand to "vl", on GCC and empty on Clang in
source/row_rvv.cc and source/scale_rvv.cc, and replace "vl", "vtype",
in the RVV inline assembly clobber lists with RVV_VL_CLOBBER.
R=masterwilliams@google.com, rrwinterton@gmail.com
Test: ninja -C out/riscv64 libyuv_unittest
Bug: b/443984359
Change-Id: Ie65bde737bd0714c3e409a2b8ab5fcd98b6a7a32
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8516707
Reviewed-by: Wan-Teh Chang <wtc@google.com>
Implement RAWToRGB24Row_WASMSIMD and RAWToRGB24Row_Any_WASMSIMD in
source/row_wasm.cc for RGB24ToRAW and RAWToRGB24 conversions:
- Process 16 pixels (48 bytes) per loop iteration using three 128-bit
loads, three byte-wise shuffles, and four 2-way blends.
- Use wasm_i8x16_relaxed_swizzle when __wasm_relaxed_simd__ is enabled
(compiling to a single vpshufb on x86 without bounds-checking
overhead) and fall back to wasm_i8x16_swizzle for baseline SIMD128.
- Add kCpuHasWASM (0x100) and kCpuHasWASMSIMD (0x101) CPU feature
flags and enable -msimd128 in BUILD.bazel, BUILD.gn, and
CMakeLists.txt for WebAssembly targets.
Performance (RAWToRGB24_Opt, 1280x720, V8 d8):
CPU WASM C SIMD128 Relaxed SIMD Spd Rlx Spd
-------------------------------------------------------------------
AMD Zen 4 595.0 us 137.0 us 84.0 us 4.34x 7.08x
AMD Zen 3 708.0 us 148.0 us 107.0 us 4.78x 6.62x
AMD Zen 5 375.0 us 107.0 us 67.0 us 3.50x 5.60x
ARM Neoverse-V2 638.0 us 118.0 us 116.0 us 5.41x 5.50x
ARM Neoverse-N1 1046.0 us 294.0 us 294.0 us 3.56x 3.56x
Intel Ice Lake 707.0 us 209.0 us 220.0 us 3.38x 3.21x
Intel Haswell 1110.0 us 344.0 us 360.0 us 3.23x 3.08x
Intel Emerald Rapids 519.0 us 207.0 us 208.0 us 2.51x 2.50x
Intel Sapphire Rapids 551.0 us 220.0 us 255.0 us 2.50x 2.16x
ARM Cortex-A53 14440.0 us 8660.0us 6920.0us 1.67x 2.09x
Intel Granite Rapids 453.0 us 225.0 us 221.0 us 2.01x 2.05x
Bug: 42271072
Test: libyuv_test --gunit_filter=*RAWToRGB24*:*RGB24ToRAW*:*Cpu*
Change-Id: Id902668263a2ab70aa300eeeea18851490c917b5
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8515581
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
kFilterBox only had 1/2 and 1/4 fast paths. Everything else fell
through to ScaleUVSimple and point sampled, so NV12 chroma was
unfiltered where I420 was not. 13.6 dB against 50.2 dB at 3x.
ScaleUVBox mirrors ScalePlaneBox. Row accumulation reuses ScaleAddRow
and its SIMD variants, since summing is position wise and an
interleaved row is just src_cols * 2 samples. Only the column pass,
which sums U and V at a stride of 2, is new.
Output is now byte identical to the I420 path. Tests compare against
ScalePlane rather than C against SIMD, which is what missed this.
Bug: libyuv:564129034
Change-Id: I26221f5435dbfc845ad2fc13aa347ef56ccf7bc6
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8426668
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Reviewed-by: Frank Barchard <fbarchard@google.com>
- Add HammingDistance_AVX512BW, SumSquareError_AVX512BW and
HashDjb2_AVX512BW for 32 and 64 bit, as inline assembly in
compare_gcc.cc and intrinsics in compare_win.cc.
- AVX512BW processes 64 bytes per loop with full vectors, and the
remainder of 1 to 63 bytes with one masked load, so compare.cc passes
any count and does not use C for the remainder. HashDjb2 loads the
remainder into the high bytes, ending at src + count, so the same
33 ^ (63 - i) multipliers apply, and multiplies the hash by 33 ^ n.
- Add SumSquareError_AVX2 and HashDjb2_AVX2 to compare_gcc.cc and
HammingDistance_AVX2 to compare_win.cc, so both files have the same
functions. Enable AVX2 in compare_gcc.cc for gcc as well as clang.
- HashDjb2 AVX2 and AVX512BW split the 32 bit multipliers 33 ^ i into
16 bit halves and use vpmaddwd, instead of vpmulld which is 2 uops
with 10 cycle latency. Bytes are zero extended with vpmovzxbw from
memory. The hash = hash * 33 ^ 64 + sum dependency uses scalar imul
instead of vpmulld. AVX2 does 64 bytes per horizontal add.
- Port compare_win.cc from 32 bit Visual C inline assembly to
intrinsics and enable it for Visual C 32/64 bit and
LIBYUV_ENABLE_ROWWIN, which MemorySanitizer also enables. Visual C 64
bit and MemorySanitizer previously used C.
- Fix HammingDistance_SSSE3 and HammingDistance_AVX2 to allow unaligned
pointers. They used movdqa and crashed on unaligned src_a.
- Add TestCompareAnyCount to test every count from 0 to 321, aligned
and unaligned, ending at the end of a page.
Functions available:
Function compare_gcc compare_win
32/64 bit 32/64 bit
HammingDistance_SSE42 asm intrinsics
HammingDistance_SSSE3 asm -
HammingDistance_AVX2 asm intrinsics
HammingDistance_AVX512BW asm intrinsics
SumSquareError_SSE2 asm intrinsics
SumSquareError_AVX2 asm intrinsics
SumSquareError_AVX512BW asm intrinsics
HashDjb2_SSE41 asm intrinsics
HashDjb2_AVX2 asm intrinsics
HashDjb2_AVX512BW asm intrinsics
Performance, 1280x720, LIBYUV_REPEAT=1000, best of 3, ms.
Before is main (SSE, and AVX2 Hamming only, in compare_gcc).
BenchmarkDjb2_Opt main gcc gcc win win
AVX2 AVX512 AVX2 AVX512
Intel Haswell 179 77 - 75 -
AMD Milan (Zen 3) 101 35 - 35 -
Intel Skylake-X 214 63 55 67 59
Intel Ice Lake 191 51 47 52 49
Intel Sapphire Rapids 175 39 39 38 39
Intel Granite Rapids 170 40 38 40 38
AMD Genoa (Zen 4) 91 34 37 34 38
AMD Turin (Zen 5) 80 29 22 28 23
C 502 to 669
BenchmarkSumSquareError main gcc gcc win win
_Opt AVX2 AVX512 AVX2 AVX512
Intel Haswell 79 37 - 38 -
AMD Milan (Zen 3) 47 24 - 25 -
Intel Skylake-X 78 32 26 34 24
Intel Ice Lake 86 43 23 43 22
Intel Sapphire Rapids 48 24 21 24 22
Intel Granite Rapids 47 23 21 23 21
AMD Genoa (Zen 4) 50 25 21 25 21
AMD Turin (Zen 5) 57 28 13 28 14
C 154 to 425
Testing:
LIBYUV_WIDTH=1279 LIBYUV_HEIGHT=719 out/Release/libyuv_unittest \
--gtest_filter='*Compare*:*Djb2*:*Hamming*:*SumSquare*:*Psnr*:*Ssim*'
With out/Release, out/Release_x86, out/rowwin, out/rowwin_x86 and
out/msan, with LIBYUV_DISABLE_AVX512BW and LIBYUV_DISABLE_AVX2, and
LIBYUV_WIDTH of 1, 31 to 33, 63 to 65, 95, 127, 129 and 200. Under
sde64 -skx for AVX512BW with out/Release and out/rowwin.
Built out/android_x86.
Bug: None
Change-Id: Ief197d546daca3db02ee1503d4649f4c755eb189
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8504717
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
- Port rotate_gcc.cc to intrinsics in rotate_win.cc: TransposeWx8_SSSE3,
TransposeUVWx8_SSE2, TransposeWx16_AVX2, TransposeUVWx16_AVX2,
Transpose4x4_32_SSE2, Transpose4x4_32_AVX2, and for 64 bit,
TransposeWx16_AVX512BW and TransposeUVWx16_AVX512BW.
- Remove the 32 bit Visual C inline assembly.
- Enable rotate_win.cc for Visual C 32/64 bit, LIBYUV_ENABLE_ROWWIN,
which MemorySanitizer also enables, and all 32 bit x86 compilers.
Visual C 64 bit previously used C.
- rotate_gcc.cc is now 64 bit only. 32 bit gcc and clang only had an
8x8 TransposeWx8_SSSE3 and Transpose4x4_32, because the UV and AVX2
inline assembly uses xmm8..xmm15, so 32 bit NV12 rotate used C.
Remove the 32 bit TransposeWx8_SSSE3 inline assembly.
- Add a target("sse2") attribute to the SSE2 intrinsics so gcc and
clang build them when the baseline lacks SSE2, such as
-march=pentium3.
- Disable the generic x86 scale_row.h and compare_row.h HAS_ macros for
32 bit LIBYUV_ENABLE_ROWWIN, as is done for 64 bit. scale_gcc.cc and
compare_gcc.cc are disabled for ROWWIN and scale_win.cc and
compare_win.cc require _MSC_VER, so out/rowwin_x86 failed to link with
undefined symbols such as ScaleARGBCols_SSE2 and HashDjb2_SSE41.
- TransposeWx8_SSSE3 and TransposeUVWx8_SSE2 share a 16x8 helper.
- TransposeWx8_SSSE3 requires a multiple of 16 on all platforms.
- Byte transposes use a shared helper with dst_a / dst_b outputs so
2, 3 and 4 channel variants can reuse the same shuffles.
Functions available:
Function rotate_gcc rotate_win rotate_win
64 bit 64 bit 32 bit
TransposeWx8_SSSE3 asm intrinsics intrinsics
TransposeUVWx8_SSE2 asm intrinsics intrinsics
TransposeWx16_AVX2 asm intrinsics intrinsics
TransposeUVWx16_AVX2 asm intrinsics intrinsics
Transpose4x4_32_SSE2 asm intrinsics intrinsics
Transpose4x4_32_AVX2 asm intrinsics intrinsics
TransposeWx16_AVX512BW asm intrinsics -
TransposeUVWx16_AVX512BW asm intrinsics -
32 bit x86 Linux clang on AMD Milan (Zen 3), ms
(LIBYUV_WIDTH=1280 LIBYUV_HEIGHT=720 LIBYUV_REPEAT=2000):
C Before After SSE2 After AVX2
I420Rotate90_Opt 944 321 322 220
NV12Rotate90_Opt 1412 935 315 232
Testing:
for w in 1 2 3 7 8 9 15 16 17 23 24 31 32 33 64 65 1279 1280; do
for h in 1 7 8 15 16 17 24 31 32 33 719; do
LIBYUV_WIDTH=$w LIBYUV_HEIGHT=$h out/rowwin/libyuv_unittest \
--gtest_filter='*Rotate*:*Transpose*'
done
done
Repeated with LIBYUV_DISABLE_AVX2=1, LIBYUV_DISABLE_AVX512BW=1, with
out/Release and out/Release_x86, under sde64 -skx for AVX512BW, and
with out/msan (is_msan = true) and out/rowwin_x86
(target_cpu = "x86", libyuv_enable_rowwin = true). Built out/android_x86
(target_os = "android", target_cpu = "x86"). Compiled rotate_win.cc
with g++ and clang++ -m32 -march=pentium3.
Bug: None
Change-Id: Ic70c4f0b7eb46b1b8522a1eba851dac5f506ec3e
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8499394
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
- TransposeWx16_AVX512BW and TransposeUVWx16_AVX512BW
- tail masking instead of 'Any' to odd source width
I420Rotate90_Opt (LIBYUV_WIDTH=1279 LIBYUV_HEIGHT=719 LIBYUV_REPEAT=1000):
CPU Before After Speedup
Intel Emerald Rapids 249 ms 182 ms 1.37x
Intel Ice Lake 311 ms 234 ms 1.33x
Intel Granite Rapids 246 ms 186 ms 1.32x
Intel Sapphire Rapids 274 ms 226 ms 1.21x
NV12Rotate90_Opt (LIBYUV_WIDTH=1279 LIBYUV_HEIGHT=719 LIBYUV_REPEAT=1000):
CPU Before After Speedup
Intel Granite Rapids 263 ms 181 ms 1.45x
Intel Emerald Rapids 257 ms 180 ms 1.43x
Intel Ice Lake 311 ms 233 ms 1.33x
Intel Sapphire Rapids 282 ms 222 ms 1.27x
Bug: None
Change-Id: Ibe0df79275099dc06bf4218484a24732cd7fb9b5
TAG=agy
CONV=bea53b37-b2bf-46fa-9e44-a708b438fabc
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8497726
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Fix _M_X86 to _M_IX86 for Visual C 32 bit
Visual C defines _M_IX86 for 32 bit x86, not _M_X86.
row.h and row_win.cc tested _M_X86, so a real 32 bit Visual C build
got neither USE_ROW_WIN nor USE_ROW_GCC, row_win.cc compiled to nothing,
and the HAS_*_AVX2 defines for all x86 platforms were skipped.
clang-cl and Android x86 are unaffected since clang defines __i386__.
Testing:
ninja -C out/Release libyuv_unittest && out/Release/libyuv_unittest
ninja -C out/Release_x86 libyuv_unittest && out/Release_x86/libyuv_unittest
ninja -C out/rowwin libyuv_unittest && out/rowwin/libyuv_unittest
blaze test --config=windows_x86 //third_party/libyuv:libyuv_test
blaze build --config=android_x86 //third_party/libyuv:libyuv
Bug: None
Change-Id: I0f90c03855f2981040ab4f250cd2facc9ef47ac0
TAG=agy
CONV=bea53b37-b2bf-46fa-9e44-a708b438fabc
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8499834
Commit-Queue: Frank Barchard <fbarchard@google.com>
Reviewed-by: Mirko Bonadei <mbonadei@chromium.org>
I420ScaleTo569x480_Bilinear (LIBYUV_WIDTH=569 LIBYUV_REPEAT=5000):
CPU AVX2 AVX512 Speedup
Intel Emerald Rapids 124 ms 63 ms 1.97x
Intel Sapphire Rapids 134 ms 69 ms 1.94x
Intel Skylake-X 219 ms 129 ms 1.70x
Intel Granite Rapids 124 ms 76 ms 1.63x
Intel Ice Lake 164 ms 101 ms 1.62x
Bug: None
Change-Id: I7a10446eb4dbcebae68380c2a186d4e3badec551
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8483371
Commit-Queue: Frank Barchard <fbarchard@google.com>
Reviewed-by: richard winterton <rrwinterton@gmail.com>
- Use -fno-tree-vectorize and -fno-tree-slp-vectorize instead of
Clang-only -fno-vectorize and -fno-slp-vectorize in CMakeLists.txt,
riscv-clang.cmake, BUILD.gn, and BUILD.bazel for GCC and Clang
compatibility.
- Split row_rvv.cc and scale_rvv.cc into a dedicated libyuv_rvv target
in BUILD.gn and define LIBYUV_RVV so callers in row.h and scale_row.h
can dispatch to RVV kernels when built without global -march=rv64gcv.
- Guard the x86 AVX512-BF16 vdpphps inline assembly probe in
util/cpuid.c with __i386__ / __x86_64__.
- Replace new/delete and new[]/delete[] with calloc/malloc/free in
source/mjpeg_decoder.cc and util/yuvconvert.cc to avoid C++ standard
library dependencies, and link libm for yuvconstants and
libyuv_unittest in CMakeLists.txt.
Test: cmake -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ ..
Bug: None
Change-Id: I701c0625f2730e864806556213a5e013c931e2d6
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8464751
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
Enable LIBYUV_ENABLE_ROWWIN instead of LIBYUV_DISABLE_X86 under
MemorySanitizer so that C++ SIMD intrinsics in row_win.cc and their
corresponding _Any wrappers in row_any.cc are compiled and tested
under MSan.
Rename MEMSET to MSAN_MEMSET in row_any.cc and scale_any.cc, and
initialize vout buffers across all ANY* macros in row_any.cc. Use
calloc for align_buffer_64 and align_buffer_page_end under
LIBYUV_MSAN, and zero-initialize stack pixel buffers in color_test.cc.
Test: out/msan/libyuv_unittest
Bug: None
Change-Id: I0b9fc69af159531699bb2b2db8b48f39831289ee
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8456982
Reviewed-by: richard winterton <rrwinterton@gmail.com>
==================================================
Processor / Architecture Before After Speedup
--------------------------------------------------
Intel Granite Rapids 838 us 222 us 3.77x
Intel Emerald Rapids 823 us 237 us 3.47x
Intel Sapphire Rapids 884 us 241 us 3.67x
AMD Turin (Zen 5) 684 us 119 us 5.75x
AMD Genoa (Zen 4) 858 us 233 us 3.68x
Intel Skylake-X 1421 us 319 us 4.45x
Intel Ice Lake 1105 us 284 us 3.89x
AMD Milan (Zen 3) 980 us 978 us 1.00x
Intel Haswell 1356 us 1387 us 0.98x
==================================================
Bug: None
Change-Id: Id5473082cc2537234db9c53d53c38f2c9698d9d2
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8467771
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
Add ComputeCumulativeSumRow_AVX2, CumulativeSumToAverageRow_AVX2,
ComputeCumulativeSumRow_NEON, and CumulativeSumToAverageRow_NEON to
accelerate ARGBBlur and ARGBComputeCumulativeSum on x86_64 and aarch64.
Update ARGBBlurSmall_Opt to benchmark 320x180 with radius 1 and use a
circular cumsum buffer of height radius * 2 + 2 in TestBlur.
ARGBBlurSmall_Opt (320x180, r=1, 10000 iterations):
CPU Was C/SSE2 Now AVX2/NEON Speedup
-------------------- -------------- --------------- ---------------
Intel Granite Rapids 29 ms (SSE2) 24 ms (AVX2) 1.21x
Intel Emerald Rapids 30 ms (SSE2) 23 ms (AVX2) 1.30x
Intel Sapphire Rapid 32 ms (SSE2) 28 ms (AVX2) 1.14x
Intel Ice Lake 43 ms (SSE2) 34 ms (AVX2) 1.26x
Intel Skylake-X 54 ms (SSE2) 45 ms (AVX2) 1.20x
Intel Haswell 59 ms (SSE2) 54 ms (AVX2) 1.09x
AMD Zen 5 32 ms (SSE2) 19 ms (AVX2) 1.68x
AMD Zen 4 36 ms (SSE2) 26 ms (AVX2) 1.38x
AMD Zen 3 42 ms (SSE2) 28 ms (AVX2) 1.50x
ARM Neoverse-V2 256 ms (C) 40 ms (NEON) 6.40x
ARGBBlur_Opt (1280x720, r=55, 1000 iterations):
CPU Was C/SSE2 Now AVX2/NEON Speedup
-------------------- -------------- --------------- ---------------
Intel Granite Rapids 1150 ms (SSE2) 984 ms (AVX2) 1.17x
Intel Emerald Rapids 1081 ms (SSE2) 953 ms (AVX2) 1.13x
Intel Sapphire Rapid 1124 ms (SSE2) 973 ms (AVX2) 1.16x
Intel Ice Lake 1450 ms (SSE2) 1167 ms (AVX2) 1.24x
Intel Skylake-X 1935 ms (SSE2) 1630 ms (AVX2) 1.19x
Intel Haswell 2211 ms (SSE2) 1982 ms (AVX2) 1.12x
AMD Zen 5 1089 ms (SSE2) 694 ms (AVX2) 1.57x
AMD Zen 4 1045 ms (SSE2) 929 ms (AVX2) 1.12x
AMD Zen 3 1215 ms (SSE2) 967 ms (AVX2) 1.26x
ARM Neoverse-V2 6529 ms (C) 1544 ms (NEON) 4.23x
Bug: b/456384776
Change-Id: I39c463c791826d3adb05f0b0845b0b1f56f0e4da
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8464496
Commit-Queue: Frank Barchard <fbarchard@google.com>
Reviewed-by: richard winterton <rrwinterton@gmail.com>
I422ToRGB24Matrix (299x299, repeat=10000)
====================================================================
CPU Before After Improvement
--------------------------------------------------------------------
Intel Skylake-X 275 ms 213 ms -22.5% (1.29x)
Intel Ice Lake 225 ms 177 ms -21.3% (1.27x)
Intel Sapphire Rapids 181 ms 144 ms -20.4% (1.26x)
Intel Emerald Rapids 167 ms 136 ms -18.6% (1.23x)
Intel Granite Rapids 170 ms 140 ms -17.6% (1.21x)
AMD Zen 4 (Genoa) 192 ms 136 ms -29.2% (1.41x)
AMD Zen 5 (Turin) 102 ms 68 ms -33.3% (1.50x)
AMD Zen 6 (Venice) 102 ms 60 ms -41.2% (1.70x)
====================================================================
I422ToRGB24Matrix (298x299, repeat=10000)
====================================================================
CPU Before After Improvement
--------------------------------------------------------------------
Intel Skylake-X 271 ms 184 ms -32.1% (1.47x)
Intel Ice Lake 235 ms 152 ms -35.3% (1.55x)
Intel Sapphire Rapids 199 ms 128 ms -35.7% (1.55x)
Intel Emerald Rapids 170 ms 121 ms -28.8% (1.40x)
Intel Granite Rapids 177 ms 125 ms -29.4% (1.42x)
AMD Zen 4 (Genoa) 185 ms 119 ms -35.7% (1.55x)
AMD Zen 5 (Turin) 102 ms 59 ms -42.2% (1.73x)
AMD Zen 6 (Venice) 100 ms 51 ms -49.0% (1.96x)
====================================================================
Bug: None
Change-Id: Id75e3d1df6d26bc28a252672816a7c438cb664a4
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8452502
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
I422ToRGB24Matrix (299x299, repeat=10000)
====================================================================
CPU Before After Improvement
--------------------------------------------------------------------
Intel Ice Lake 220 ms 181 ms -17.7% (1.22x)
Intel Sapphire Rapids 181 ms 145 ms -19.9% (1.25x)
Intel Emerald Rapids 169 ms 148 ms -12.4% (1.14x)
Intel Granite Rapids 191 ms 141 ms -26.2% (1.35x)
AMD Zen 4 (Genoa) 171 ms 136 ms -20.5% (1.26x)
AMD Zen 5 (Turin) 94 ms 72 ms -23.4% (1.31x)
AMD Zen 6 (Venice) 95 ms 66 ms -30.5% (1.44x)
====================================================================
Bug: None
Change-Id: I4cd657410cc5253b5e8fd2caa78d17e2b3812393
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8445804
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
P010 and P210 were auxiliary FourCCs and P410 did not exist, so
ConvertToI420, ConvertToARGB, and ConvertFromI420 could not consume or
produce them. Callers had to pick I010/I210 helpers and pack UV by hand.
NV12Scale existed, but NV16/NV24 and P010/P210/P410 had no Scale APIs.
Promote P010 and P210 to primary FourCCs, add FOURCC_P410, and add the
I4xx/Px10 converters those entry points need: I410ToP410, I412ToP412,
P210ToI210, P410ToI410, P010/P210/P410ToI420, I420ToP010/P210/P410,
I422ToP210, I444ToP410, and P410ToARGB/AR30Matrix.
Add NV16Scale (same path as NV12Scale: ScalePlane + UVScale).
Add P010Scale, P210Scale, and P410Scale: 10-bit samples stay in the MSBs, so Y/UV convert to 10-bit LSB, ScalePlane_16, then back to MSB.
UVScale_16 cannot general-scale, so UV is split, scaled as planes, and merged.
Packed-plane layout, odd-width strides, ConvertTo/From round-trips, and
the existing scale factor/size/swap tests cover the new APIs.
Bug: 537818862
Change-Id: I1ae5e6cad5e6d83acf1d0df69f94c23688bbe0e8
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8346569
Reviewed-by: Frank Barchard <fbarchard@google.com>
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Add matrix and BT.601 wrappers that convert ARGB to 16-bit biplanar
YUV in P010 (4:2:0), P210 (4:2:2), and P410 (4:4:4) layouts.
Reuse optimized ARGB matrix row functions for 8-bit Y and chroma,
merge UV with MergeUVRow, then expand with Convert8To16Row (bits=10)
and MultiplyRow_16 (scale=64) so 10-bit samples are stored in the
high bits of each uint16.
Reject width > INT_MAX / 2 so UV width (halfwidth * 2 or width * 2)
does not overflow Convert8To16Row.
Bug: 537818862
Change-Id: I7a776fb3d6f52ce46a0f6825903bfdb9c6dcfbe2
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8314189
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Reviewed-by: Frank Barchard <fbarchard@google.com>
- avoids intrinsics API changes
- build options for rvv not needed for inline instructions
- declare vl and vtype clobbers instead of cc
- tune for SiFive X280 using 16/32-bit loads and stores instead of
vlseg/vsseg where more efficient
- tune LMUL for SpacemiT K3 (256-bit vectors) with max m4 (m2 where
constrained by register pressure or vlseg/vsseg field rules)
- YUVTORGB use vmulhu
NV12ToARGB performance estimates:
CPU VLEN C Intrinsics Inline ASM Speedup (Inline)
------------------------------------------------------------------------
SiFive X390 1024-bit 360 ms 29 ms 9.5 ms 37.89x (3.05x)
SiFive X280 512-bit 980 ms 115 ms 38.0 ms 25.79x (3.03x)
SpacemiT K3 256-bit 420 ms 68 ms 28.0 ms 15.00x (2.43x)
Google Coral 256-bit 1120 ms 190 ms 58.0 ms 19.31x (3.28x)
T-Head C920 128-bit 510 ms 88 ms 39.0 ms 13.08x (2.26x)
T-Head C908 128-bit 860 ms 145 ms 62.0 ms 13.87x (2.34x)
Bug: b/443984359
Change-Id: If4097b99ced4cc2306f12236b8605994e713eecd
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8426529
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
- AVX2 uses vpmulhrsw to compute ((128 * a * b) + 16384) >> 15.
- RVV uses e8, m2 (widening to e16, m4) with vwmulu.vv, vwaddu.wx 128,
and vnsrl.wi by 8.
CPU C Intrinsics Inline ASM Speedup (Inline)
------------------------------------------------------------------------
SiFive X390 gen 2 (est.) 2150 ms 29 ms 27 ms 79.63x (1.07x)
SpacemiT K3 (est.) 980 ms 41 ms 38 ms 25.79x (1.08x)
T-Head C908 (est.) 2060 ms 100 ms 94 ms 21.91x (1.06x)
Google Coral (est.) 2680 ms 94 ms 88 ms 30.45x (1.07x)
Test: libyuv_unittest --gtest_filter=*ARGBMultiply*
Bug: None
Change-Id: Iff65794623a6f29e5cb6bda7bd3084c8a5f676d1
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8422278
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
Apple processors (such as A18/A19) may report
hw.optional.arm.FEAT_SME = 1 even when the XNU kernel on iOS disables
EL0 (user-mode) SME execution (hw.optional.arm.sme_max_svl_b == 0),
which causes SMSTART or RDSVL to fault with EXC_BAD_INSTRUCTION.
Gate SME/SME2 flags on have_feature("hw.optional.arm.sme_max_svl_b")
being nonzero. See https://github.com/pytorch/cpuinfo/issues/432 and
https://github.com/pytorch/cpuinfo/pull/433.
Also add SME2 detection test in cpu_test.cc and LIBYUV_DISABLE_SME2
environment variable support in unit_test.cc.
Test: libyuv_unittest --gtest_filter=*TestLinuxAArch64*
Bug: 8339302
Change-Id: If3e4d425ebaa89bf27965295ebdb43b844797ca1
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8428226
Reviewed-by: Wan-Teh Chang <wtc@google.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
Add SwapUVRow_RVV using RVV intrinsics by default and unit-stride
e16, m4 inline assembly (vle16.v, vsrl.vi, vsll.vi, vor.vv, vse16.v)
under #ifdef RVV_ASM.
CPU C Intrinsics Inline ASM Speedup (Inline)
--------------------------------------------------------------------
SiFive X280 (est.) 980 ms 115 ms 38 ms 25.79x (3.03x)
SpacemiT K3 (est.) 420 ms 68 ms 28 ms 15.00x (2.43x)
T-Head C908 (est.) 860 ms 145 ms 62 ms 13.87x (2.34x)
Google Coral (est.) 1120 ms 190 ms 58 ms 19.31x (3.28x)
Original author: Matteo Forzan <matteo.forzan95@gmail.com>
R=wtc@google.comCC=matteo.forzan95@gmail.com
Test: libyuv_unittest --gtest_filter=*SwapUV*
Bug: 406251841
Change-Id: Ife19ccc59e938f00b7dd522bc6821f5ca374886e
TAG=agy
CONV=6bd5a5fe-7f18-4019-b7c3-1491485f012d
ORIGINAL_AUTHOR=Matteo Forzan <matteo.forzan95@gmail.com>
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8339302
Reviewed-by: David Gao <davidgao@google.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
The existing SVE2 path only computes the average for a pair of input
values instead of groups of four as done in the Neon implementation.
The average of two values must be retained for the final odd element if
present. Existing code just handles this by falling back to the C
implementation, so mirror that for SVE2 as well.
Reduction in time taken relative to the existing Neon code, the SVE2
code remains faster even with the correct averaging:
Cortex-A510: -12.66%
Lumex C1-Nano: -6.38%
Cortex-A720: -9.73%
Lumex C1-Pro: +0.99%
Cortex-X4: -3.05%
Lumex C1-Ultra: -18.98%
Bug: 546962730
Change-Id: I7ab348e64a9d079b81d070cea5fd575243b037d2
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8252281
Reviewed-by: Wan-Teh Chang <wtc@google.com>
Reviewed-by: Justin Green <greenjustin@google.com>
Commit-Queue: Wan-Teh Chang <wtc@google.com>
Add SwapUVRow_AVX2 implementation in row_win.cc for MSVC and enable
HAS_SWAPUVROW_AVX2 under USE_ROW_WIN.
Share kShuffleUVToVU shuffle table between SSSE3 and AVX2 in row_gcc.cc.
Make NV12ToNV21 the primary implementation and alias NV21ToNV12 to it.
Add NV12ToNV21 test in convert_test.cc and benchmark dispatch for
SSSE3 and AVX2 in planar_test.cc.
CPU Before After Speedup
----------------------------------------------------------------
Intel Granite Rapids 282 ms 125 ms 2.26x
Intel Emerald Rapids 264 ms 108 ms 2.44x
Intel Sapphire Rapids 276 ms 122 ms 2.26x
Intel Ice Lake 310 ms 122 ms 2.54x
Intel Skylake-X 573 ms 157 ms 3.65x
Intel Haswell 620 ms 162 ms 3.83x
AMD Zen 4 (Genoa) 266 ms 52 ms 5.12x
AMD Zen 6 (Venice) 229 ms 43 ms 5.33x
ARM Neoverse-N1 140 ms 71 ms 1.97x
NVIDIA Grace (Neoverse-V2) 62 ms 69 ms 0.90x
R=wtc@google.com
Test: libyuv_unittest --gtest_filter=*SwapUV*:*NV12ToNV21*:*NV21ToNV12*
Bug: 406251841
Change-Id: I9bed5522aedadea13cdf69c27e3cbe633e31dd5b
TAG=agy
CONV=bb50ac62-0c55-4969-8fff-c71ab3d9a79f
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8356968
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
- instead of 64 bit load, use vlseg2 with 32 bit
Test build with:
-fsanitize=alignment -fsanitize-trap=alignment
Test: libyuv_unittest --gtest_filter=*UVScale*:*ScalePlane*:*ARGBScale*
Bug: 553156658
Change-Id: Ib6febe49f085f5b3deb33adb6b6f82f7d836c3fa
TAG=agy
CONV=b29f62f9-15a4-4f38-8414-86c8330e61fb
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8315435
Reviewed-by: Wan-Teh Chang <wtc@google.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
- Also remove redundant neon check for I8MM kernels
======================================================================
CPU Before After Speedup
----------------------------------------------------------------------
Intel Granite Rapids (AVX10.1) 251 ms 141 ms 1.78x (+43.8%)
Intel Emerald Rapids (AVX512) 199 ms 108 ms 1.84x (+45.7%)
Intel Sapphire Rapids (AVX512) 230 ms 112 ms 2.05x (+51.3%)
AMD Turin Zen 5 (AVX512) 156 ms 99 ms 1.58x (+36.5%)
AMD Venice Zen 6 (AVX512) 204 ms 106 ms 1.92x (+48.0%)
Intel Skylake-X (AVX512) 320 ms 156 ms 2.05x (+51.2%)
AMD Genoa Zen 4 (AVX512) 236 ms 346 ms 0.68x (-46.6%)
Intel Ice Lake (AVX512) 242 ms 316 ms 0.77x (-30.6%)
AMD Milan Zen 3 (AVX2) 252 ms 230 ms 1.10x (+8.7%)
Intel Haswell (AVX2) 369 ms 456 ms 0.81x (-23.6%)
NVIDIA Grace (SVE2) 336 ms 337 ms 1.00x (-0.3%)
ARM Neoverse-N1 (NEON) 1081 ms 1105 ms 0.98x (-2.2%)
======================================================================
Bug: 537818862
Change-Id: I3ddeddb040fc52a30dabe7ae2c71d16dda021568
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8343467
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
