- Route 32-bit x86 (__i386__ / _M_IX86) to compare_win.cc intrinsics
and restrict compare_gcc.cc to 64-bit x86 (__x86_64__), avoiding
x87 float rounding and enabling SSE2, AVX2, and AVX-512 on 32-bit
and 64-bit MinGW, GCC, Clang, and MSan (LIBYUV_ENABLE_ROWWIN).
- Enable AVX-512BW on GCC >= 6 (GCC_HAS_AVX512) and add
LIBYUV_TARGET_AVX512BW attributes to compare_gcc.cc and
compare_win.cc.
- Refactor SsimWindowRow_C and Ssim8x8_C to use int32_t and float,
and add SsimWindowRow_SSE2, SsimWindowRow_AVX2, and
SsimWindowRow_AVX512BW in both compare_win.cc and compare_gcc.cc.
- Update TestSsimAnyWidth, Ssim, and BenchmarkSsim_Opt in
compare_test.cc to use EXPECT_NEAR and deterministic test patterns
(identical, different, similar).
- Update util/compare.cc and linux.mk to compute I420Ssim.
Performance (BenchmarkSsim_Opt, 1280x720):
uArch Before After Improvement
Intel Lion Cove (x64 AVX2) 120.2 us 68.1 us 1.76x
Intel Skymont (x64 SSE2) 249.6 us 146.8 us 1.70x
AMD Zen 3 (x64 AVX2) 314.9 us 150.2 us 2.10x
Intel Lion Cove (MinGW686) 203.7 us 77.9 us 2.61x
Intel Skymont (MinGW686) 338.8 us 167.5 us 2.02x
AMD Zen 3 (i686 AVX2) 738.4 us 206.5 us 3.58x
Quality (util/compare on 640x360 I420 test images):
Image Comparison MSE PSNR SSIM
Identical (orig vs orig) 0.00 128.00 dB 1.000000
Similar (light +/-3 noise) 4.57 41.53 dB 0.956556
Degraded (16x16 blocky) 98.18 28.21 dB 0.858893
Very different (inverted) 11160.29 7.65 dB 0.000181
Co-authored-by: Martin Storsjö <martin@martin.st>
Bug: b/493910265
Test: out/Release/libyuv_unittest --gtest_filter=*Ssim*
Change-Id: I8603dab3f752af32d07ae9039502ca5da5a05425
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8618641
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
Float computations can get different rounding depending on
the implementation. In particular, if the C code is built with
a compiler that uses x87 for float math, it is common that
there is a small variance in the results.
This fixes test failures on GCC i686 mingw targets, where it
previously failed like this:
unit_test/compare_test.cc:725: Failure
Expected equality of these values:
c_ssim
Which is: -0.076010095487114751
opt_ssim
Which is: -0.076010095487114751
width 17 offset 1
[...]
Change-Id: Id109058a89da78e38a8a7224a071a5c2075c66d6
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8615658
Reviewed-by: Frank Barchard <fbarchard@google.com>
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Based on Philipp Hancke's AVX2 column kernel for ScaleUVBox at 3x
(https://crrev.com/c/8427347).
Add ScaleUVAddCols3 for exact 3x horizontal downscaling (dx == 0x30000
and boxheight <= 85) in ScaleUVBox, plus ScaleAddRow_AVX512BW and
ScaleAddRow_WASMSIMD dispatch, and port 3x/box scaling kernels in
scale_win.cc to 32/64-bit C++ intrinsics matching scale_gcc.cc:
- ScaleUVAddCols3_C (portable C)
- ScaleUVAddCols3_AVX2 and ScaleUVAddCols3_Any_AVX2 (scale_gcc.cc and
scale_win.cc)
- ScaleUVAddCols3_AVX512VBMI (scale_gcc.cc and scale_win.cc)
- ScaleAddRow_SSE2, ScaleAddRow_AVX2, ScaleAddRow_AVX512BW,
ScaleAddCols1_AVX512BW, ScaleAddCols2_AVX512BW, ScaleRowDown38_SSSE3,
ScaleRowDown38_2_Box_SSSE3, and ScaleRowDown38_3_Box_SSSE3 in
scale_win.cc for 32/64-bit Visual C and LIBYUV_ENABLE_ROWWIN
- ScaleUVAddCols3_NEON (32-bit ARM and 64-bit AArch64) and
ScaleUVAddCols3_Any_NEON
- ScaleUVAddCols3_SME (ARMv9 Streaming SVE with whilelt predicate tail)
- ScaleUVAddCols3_RVV (RISC-V Vector vlseg3e32.v inline asm)
- ScaleUVAddCols3_LSX and ScaleUVAddCols3_Any_LSX (LoongArch LSX)
- ScaleAddRow_WASMSIMD, ScaleAddRow_Any_WASMSIMD,
ScaleUVAddCols3_WASMSIMD, and ScaleUVAddCols3_Any_WASMSIMD
- ScaleUVAddCols3_HVX (Hexagon HVX vrdelta + vmpy_VuhRuh)
Co-authored-by: Philipp Hancke <philipp.hancke@googlemail.com>
Bug: libyuv:564129034
R=rrwinterton@gmail.com
Change-Id: Ib9e14f3efcb097842c3d0e9fc9b473b6e2f81871
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8596436
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Use uint32_t for the 16.16 fixed-point row coordinate (uy) and max_y
in the vertical loops of ScalePlane*, ScaleUV*, and the Up2 linear
helpers. With src_height up to 32768, (src_height << 16) and y += dy
overflow signed int (e.g. for dst_height == 2 the post-read step
reaches 5/4 * (src_height << 16), which exceeds INT_MAX for any
src_height >= 26215, and for kFilterNone the final increment after
the last row reaches (src_height << 16) + dy / 2). uint32_t cannot
overflow for these ranges (uy <= 0xFFFF0000) and stays in a single
32-bit register on 32-bit targets (x86, Arm, WASM), unlike int64_t.
Undo the interim +/-32767 height cap from https://crrev.com/c/8524280
so ScalePlane, ScalePlane_16, UVScale, and UVScale_16 accept
|src_height| <= 32768 again.
ScaleUVBilinearUp2 and ScaleUVBilinearUp2_16 take ptrdiff_t strides
so 2 * dst_stride does not overflow int for images over 2 GB.
ARGBScale, ARGBScaleClip, and UVScale_16 reject |src_width| > 32768
and |src_height| > 32768, and ScalePlane, ScalePlane_16, and
ScaleSlope avoid FixedDiv(32768, 1) overflow when dst_width == 1 or
dst_height == 1.
Bug: chromium:531172025, chromium:546841026, libyuv:573815252
Test: libyuv_unittest --gtest_filter=*ScalePlaneOversizeHeight*:*ScalePlaneBoxMaxSrcHeight*:*ScaleLargeImageOver2GB*:*ARGBScale*
Change-Id: Ica2e05d7dfe94f81c6631c587f1ff46c2ad44735
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8528398
Commit-Queue: Frank Barchard <fbarchard@google.com>
Reviewed-by: Wan-Teh Chang <wtc@google.com>
Implement RAWToRGB24Row_WASMSIMD and RAWToRGB24Row_Any_WASMSIMD in
source/row_wasm.cc for RGB24ToRAW and RAWToRGB24 conversions:
- Process 16 pixels (48 bytes) per loop iteration using three 128-bit
loads, three byte-wise shuffles, and four 2-way blends.
- Use wasm_i8x16_relaxed_swizzle when __wasm_relaxed_simd__ is enabled
(compiling to a single vpshufb on x86 without bounds-checking
overhead) and fall back to wasm_i8x16_swizzle for baseline SIMD128.
- Add kCpuHasWASM (0x100) and kCpuHasWASMSIMD (0x101) CPU feature
flags and enable -msimd128 in BUILD.bazel, BUILD.gn, and
CMakeLists.txt for WebAssembly targets.
Performance (RAWToRGB24_Opt, 1280x720, V8 d8):
CPU WASM C SIMD128 Relaxed SIMD Spd Rlx Spd
-------------------------------------------------------------------
AMD Zen 4 595.0 us 137.0 us 84.0 us 4.34x 7.08x
AMD Zen 3 708.0 us 148.0 us 107.0 us 4.78x 6.62x
AMD Zen 5 375.0 us 107.0 us 67.0 us 3.50x 5.60x
ARM Neoverse-V2 638.0 us 118.0 us 116.0 us 5.41x 5.50x
ARM Neoverse-N1 1046.0 us 294.0 us 294.0 us 3.56x 3.56x
Intel Ice Lake 707.0 us 209.0 us 220.0 us 3.38x 3.21x
Intel Haswell 1110.0 us 344.0 us 360.0 us 3.23x 3.08x
Intel Emerald Rapids 519.0 us 207.0 us 208.0 us 2.51x 2.50x
Intel Sapphire Rapids 551.0 us 220.0 us 255.0 us 2.50x 2.16x
ARM Cortex-A53 14440.0 us 8660.0us 6920.0us 1.67x 2.09x
Intel Granite Rapids 453.0 us 225.0 us 221.0 us 2.01x 2.05x
Bug: 42271072
Test: libyuv_test --gunit_filter=*RAWToRGB24*:*RGB24ToRAW*:*Cpu*
Change-Id: Id902668263a2ab70aa300eeeea18851490c917b5
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8515581
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
kFilterBox only had 1/2 and 1/4 fast paths. Everything else fell
through to ScaleUVSimple and point sampled, so NV12 chroma was
unfiltered where I420 was not. 13.6 dB against 50.2 dB at 3x.
ScaleUVBox mirrors ScalePlaneBox. Row accumulation reuses ScaleAddRow
and its SIMD variants, since summing is position wise and an
interleaved row is just src_cols * 2 samples. Only the column pass,
which sums U and V at a stride of 2, is new.
Output is now byte identical to the I420 path. Tests compare against
ScalePlane rather than C against SIMD, which is what missed this.
Bug: libyuv:564129034
Change-Id: I26221f5435dbfc845ad2fc13aa347ef56ccf7bc6
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8426668
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Reviewed-by: Frank Barchard <fbarchard@google.com>
- Add HammingDistance_AVX512BW, SumSquareError_AVX512BW and
HashDjb2_AVX512BW for 32 and 64 bit, as inline assembly in
compare_gcc.cc and intrinsics in compare_win.cc.
- AVX512BW processes 64 bytes per loop with full vectors, and the
remainder of 1 to 63 bytes with one masked load, so compare.cc passes
any count and does not use C for the remainder. HashDjb2 loads the
remainder into the high bytes, ending at src + count, so the same
33 ^ (63 - i) multipliers apply, and multiplies the hash by 33 ^ n.
- Add SumSquareError_AVX2 and HashDjb2_AVX2 to compare_gcc.cc and
HammingDistance_AVX2 to compare_win.cc, so both files have the same
functions. Enable AVX2 in compare_gcc.cc for gcc as well as clang.
- HashDjb2 AVX2 and AVX512BW split the 32 bit multipliers 33 ^ i into
16 bit halves and use vpmaddwd, instead of vpmulld which is 2 uops
with 10 cycle latency. Bytes are zero extended with vpmovzxbw from
memory. The hash = hash * 33 ^ 64 + sum dependency uses scalar imul
instead of vpmulld. AVX2 does 64 bytes per horizontal add.
- Port compare_win.cc from 32 bit Visual C inline assembly to
intrinsics and enable it for Visual C 32/64 bit and
LIBYUV_ENABLE_ROWWIN, which MemorySanitizer also enables. Visual C 64
bit and MemorySanitizer previously used C.
- Fix HammingDistance_SSSE3 and HammingDistance_AVX2 to allow unaligned
pointers. They used movdqa and crashed on unaligned src_a.
- Add TestCompareAnyCount to test every count from 0 to 321, aligned
and unaligned, ending at the end of a page.
Functions available:
Function compare_gcc compare_win
32/64 bit 32/64 bit
HammingDistance_SSE42 asm intrinsics
HammingDistance_SSSE3 asm -
HammingDistance_AVX2 asm intrinsics
HammingDistance_AVX512BW asm intrinsics
SumSquareError_SSE2 asm intrinsics
SumSquareError_AVX2 asm intrinsics
SumSquareError_AVX512BW asm intrinsics
HashDjb2_SSE41 asm intrinsics
HashDjb2_AVX2 asm intrinsics
HashDjb2_AVX512BW asm intrinsics
Performance, 1280x720, LIBYUV_REPEAT=1000, best of 3, ms.
Before is main (SSE, and AVX2 Hamming only, in compare_gcc).
BenchmarkDjb2_Opt main gcc gcc win win
AVX2 AVX512 AVX2 AVX512
Intel Haswell 179 77 - 75 -
AMD Milan (Zen 3) 101 35 - 35 -
Intel Skylake-X 214 63 55 67 59
Intel Ice Lake 191 51 47 52 49
Intel Sapphire Rapids 175 39 39 38 39
Intel Granite Rapids 170 40 38 40 38
AMD Genoa (Zen 4) 91 34 37 34 38
AMD Turin (Zen 5) 80 29 22 28 23
C 502 to 669
BenchmarkSumSquareError main gcc gcc win win
_Opt AVX2 AVX512 AVX2 AVX512
Intel Haswell 79 37 - 38 -
AMD Milan (Zen 3) 47 24 - 25 -
Intel Skylake-X 78 32 26 34 24
Intel Ice Lake 86 43 23 43 22
Intel Sapphire Rapids 48 24 21 24 22
Intel Granite Rapids 47 23 21 23 21
AMD Genoa (Zen 4) 50 25 21 25 21
AMD Turin (Zen 5) 57 28 13 28 14
C 154 to 425
Testing:
LIBYUV_WIDTH=1279 LIBYUV_HEIGHT=719 out/Release/libyuv_unittest \
--gtest_filter='*Compare*:*Djb2*:*Hamming*:*SumSquare*:*Psnr*:*Ssim*'
With out/Release, out/Release_x86, out/rowwin, out/rowwin_x86 and
out/msan, with LIBYUV_DISABLE_AVX512BW and LIBYUV_DISABLE_AVX2, and
LIBYUV_WIDTH of 1, 31 to 33, 63 to 65, 95, 127, 129 and 200. Under
sde64 -skx for AVX512BW with out/Release and out/rowwin.
Built out/android_x86.
Bug: None
Change-Id: Ief197d546daca3db02ee1503d4649f4c755eb189
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8504717
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
Enable LIBYUV_ENABLE_ROWWIN instead of LIBYUV_DISABLE_X86 under
MemorySanitizer so that C++ SIMD intrinsics in row_win.cc and their
corresponding _Any wrappers in row_any.cc are compiled and tested
under MSan.
Rename MEMSET to MSAN_MEMSET in row_any.cc and scale_any.cc, and
initialize vout buffers across all ANY* macros in row_any.cc. Use
calloc for align_buffer_64 and align_buffer_page_end under
LIBYUV_MSAN, and zero-initialize stack pixel buffers in color_test.cc.
Test: out/msan/libyuv_unittest
Bug: None
Change-Id: I0b9fc69af159531699bb2b2db8b48f39831289ee
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8456982
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Add ComputeCumulativeSumRow_AVX2, CumulativeSumToAverageRow_AVX2,
ComputeCumulativeSumRow_NEON, and CumulativeSumToAverageRow_NEON to
accelerate ARGBBlur and ARGBComputeCumulativeSum on x86_64 and aarch64.
Update ARGBBlurSmall_Opt to benchmark 320x180 with radius 1 and use a
circular cumsum buffer of height radius * 2 + 2 in TestBlur.
ARGBBlurSmall_Opt (320x180, r=1, 10000 iterations):
CPU Was C/SSE2 Now AVX2/NEON Speedup
-------------------- -------------- --------------- ---------------
Intel Granite Rapids 29 ms (SSE2) 24 ms (AVX2) 1.21x
Intel Emerald Rapids 30 ms (SSE2) 23 ms (AVX2) 1.30x
Intel Sapphire Rapid 32 ms (SSE2) 28 ms (AVX2) 1.14x
Intel Ice Lake 43 ms (SSE2) 34 ms (AVX2) 1.26x
Intel Skylake-X 54 ms (SSE2) 45 ms (AVX2) 1.20x
Intel Haswell 59 ms (SSE2) 54 ms (AVX2) 1.09x
AMD Zen 5 32 ms (SSE2) 19 ms (AVX2) 1.68x
AMD Zen 4 36 ms (SSE2) 26 ms (AVX2) 1.38x
AMD Zen 3 42 ms (SSE2) 28 ms (AVX2) 1.50x
ARM Neoverse-V2 256 ms (C) 40 ms (NEON) 6.40x
ARGBBlur_Opt (1280x720, r=55, 1000 iterations):
CPU Was C/SSE2 Now AVX2/NEON Speedup
-------------------- -------------- --------------- ---------------
Intel Granite Rapids 1150 ms (SSE2) 984 ms (AVX2) 1.17x
Intel Emerald Rapids 1081 ms (SSE2) 953 ms (AVX2) 1.13x
Intel Sapphire Rapid 1124 ms (SSE2) 973 ms (AVX2) 1.16x
Intel Ice Lake 1450 ms (SSE2) 1167 ms (AVX2) 1.24x
Intel Skylake-X 1935 ms (SSE2) 1630 ms (AVX2) 1.19x
Intel Haswell 2211 ms (SSE2) 1982 ms (AVX2) 1.12x
AMD Zen 5 1089 ms (SSE2) 694 ms (AVX2) 1.57x
AMD Zen 4 1045 ms (SSE2) 929 ms (AVX2) 1.12x
AMD Zen 3 1215 ms (SSE2) 967 ms (AVX2) 1.26x
ARM Neoverse-V2 6529 ms (C) 1544 ms (NEON) 4.23x
Bug: b/456384776
Change-Id: I39c463c791826d3adb05f0b0845b0b1f56f0e4da
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8464496
Commit-Queue: Frank Barchard <fbarchard@google.com>
Reviewed-by: richard winterton <rrwinterton@gmail.com>
I422ToRGB24Matrix (299x299, repeat=10000)
====================================================================
CPU Before After Improvement
--------------------------------------------------------------------
Intel Skylake-X 275 ms 213 ms -22.5% (1.29x)
Intel Ice Lake 225 ms 177 ms -21.3% (1.27x)
Intel Sapphire Rapids 181 ms 144 ms -20.4% (1.26x)
Intel Emerald Rapids 167 ms 136 ms -18.6% (1.23x)
Intel Granite Rapids 170 ms 140 ms -17.6% (1.21x)
AMD Zen 4 (Genoa) 192 ms 136 ms -29.2% (1.41x)
AMD Zen 5 (Turin) 102 ms 68 ms -33.3% (1.50x)
AMD Zen 6 (Venice) 102 ms 60 ms -41.2% (1.70x)
====================================================================
I422ToRGB24Matrix (298x299, repeat=10000)
====================================================================
CPU Before After Improvement
--------------------------------------------------------------------
Intel Skylake-X 271 ms 184 ms -32.1% (1.47x)
Intel Ice Lake 235 ms 152 ms -35.3% (1.55x)
Intel Sapphire Rapids 199 ms 128 ms -35.7% (1.55x)
Intel Emerald Rapids 170 ms 121 ms -28.8% (1.40x)
Intel Granite Rapids 177 ms 125 ms -29.4% (1.42x)
AMD Zen 4 (Genoa) 185 ms 119 ms -35.7% (1.55x)
AMD Zen 5 (Turin) 102 ms 59 ms -42.2% (1.73x)
AMD Zen 6 (Venice) 100 ms 51 ms -49.0% (1.96x)
====================================================================
Bug: None
Change-Id: Id75e3d1df6d26bc28a252672816a7c438cb664a4
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8452502
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
P010 and P210 were auxiliary FourCCs and P410 did not exist, so
ConvertToI420, ConvertToARGB, and ConvertFromI420 could not consume or
produce them. Callers had to pick I010/I210 helpers and pack UV by hand.
NV12Scale existed, but NV16/NV24 and P010/P210/P410 had no Scale APIs.
Promote P010 and P210 to primary FourCCs, add FOURCC_P410, and add the
I4xx/Px10 converters those entry points need: I410ToP410, I412ToP412,
P210ToI210, P410ToI410, P010/P210/P410ToI420, I420ToP010/P210/P410,
I422ToP210, I444ToP410, and P410ToARGB/AR30Matrix.
Add NV16Scale (same path as NV12Scale: ScalePlane + UVScale).
Add P010Scale, P210Scale, and P410Scale: 10-bit samples stay in the MSBs, so Y/UV convert to 10-bit LSB, ScalePlane_16, then back to MSB.
UVScale_16 cannot general-scale, so UV is split, scaled as planes, and merged.
Packed-plane layout, odd-width strides, ConvertTo/From round-trips, and
the existing scale factor/size/swap tests cover the new APIs.
Bug: 537818862
Change-Id: I1ae5e6cad5e6d83acf1d0df69f94c23688bbe0e8
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8346569
Reviewed-by: Frank Barchard <fbarchard@google.com>
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Add matrix and BT.601 wrappers that convert ARGB to 16-bit biplanar
YUV in P010 (4:2:0), P210 (4:2:2), and P410 (4:4:4) layouts.
Reuse optimized ARGB matrix row functions for 8-bit Y and chroma,
merge UV with MergeUVRow, then expand with Convert8To16Row (bits=10)
and MultiplyRow_16 (scale=64) so 10-bit samples are stored in the
high bits of each uint16.
Reject width > INT_MAX / 2 so UV width (halfwidth * 2 or width * 2)
does not overflow Convert8To16Row.
Bug: 537818862
Change-Id: I7a776fb3d6f52ce46a0f6825903bfdb9c6dcfbe2
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8314189
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Reviewed-by: Frank Barchard <fbarchard@google.com>
Apple processors (such as A18/A19) may report
hw.optional.arm.FEAT_SME = 1 even when the XNU kernel on iOS disables
EL0 (user-mode) SME execution (hw.optional.arm.sme_max_svl_b == 0),
which causes SMSTART or RDSVL to fault with EXC_BAD_INSTRUCTION.
Gate SME/SME2 flags on have_feature("hw.optional.arm.sme_max_svl_b")
being nonzero. See https://github.com/pytorch/cpuinfo/issues/432 and
https://github.com/pytorch/cpuinfo/pull/433.
Also add SME2 detection test in cpu_test.cc and LIBYUV_DISABLE_SME2
environment variable support in unit_test.cc.
Test: libyuv_unittest --gtest_filter=*TestLinuxAArch64*
Bug: 8339302
Change-Id: If3e4d425ebaa89bf27965295ebdb43b844797ca1
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8428226
Reviewed-by: Wan-Teh Chang <wtc@google.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
Add SwapUVRow_RVV using RVV intrinsics by default and unit-stride
e16, m4 inline assembly (vle16.v, vsrl.vi, vsll.vi, vor.vv, vse16.v)
under #ifdef RVV_ASM.
CPU C Intrinsics Inline ASM Speedup (Inline)
--------------------------------------------------------------------
SiFive X280 (est.) 980 ms 115 ms 38 ms 25.79x (3.03x)
SpacemiT K3 (est.) 420 ms 68 ms 28 ms 15.00x (2.43x)
T-Head C908 (est.) 860 ms 145 ms 62 ms 13.87x (2.34x)
Google Coral (est.) 1120 ms 190 ms 58 ms 19.31x (3.28x)
Original author: Matteo Forzan <matteo.forzan95@gmail.com>
R=wtc@google.comCC=matteo.forzan95@gmail.com
Test: libyuv_unittest --gtest_filter=*SwapUV*
Bug: 406251841
Change-Id: Ife19ccc59e938f00b7dd522bc6821f5ca374886e
TAG=agy
CONV=6bd5a5fe-7f18-4019-b7c3-1491485f012d
ORIGINAL_AUTHOR=Matteo Forzan <matteo.forzan95@gmail.com>
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8339302
Reviewed-by: David Gao <davidgao@google.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
Add SwapUVRow_AVX2 implementation in row_win.cc for MSVC and enable
HAS_SWAPUVROW_AVX2 under USE_ROW_WIN.
Share kShuffleUVToVU shuffle table between SSSE3 and AVX2 in row_gcc.cc.
Make NV12ToNV21 the primary implementation and alias NV21ToNV12 to it.
Add NV12ToNV21 test in convert_test.cc and benchmark dispatch for
SSSE3 and AVX2 in planar_test.cc.
CPU Before After Speedup
----------------------------------------------------------------
Intel Granite Rapids 282 ms 125 ms 2.26x
Intel Emerald Rapids 264 ms 108 ms 2.44x
Intel Sapphire Rapids 276 ms 122 ms 2.26x
Intel Ice Lake 310 ms 122 ms 2.54x
Intel Skylake-X 573 ms 157 ms 3.65x
Intel Haswell 620 ms 162 ms 3.83x
AMD Zen 4 (Genoa) 266 ms 52 ms 5.12x
AMD Zen 6 (Venice) 229 ms 43 ms 5.33x
ARM Neoverse-N1 140 ms 71 ms 1.97x
NVIDIA Grace (Neoverse-V2) 62 ms 69 ms 0.90x
R=wtc@google.com
Test: libyuv_unittest --gtest_filter=*SwapUV*:*NV12ToNV21*:*NV21ToNV12*
Bug: 406251841
Change-Id: I9bed5522aedadea13cdf69c27e3cbe633e31dd5b
TAG=agy
CONV=bb50ac62-0c55-4969-8fff-c71ab3d9a79f
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8356968
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
- Also remove redundant neon check for I8MM kernels
======================================================================
CPU Before After Speedup
----------------------------------------------------------------------
Intel Granite Rapids (AVX10.1) 251 ms 141 ms 1.78x (+43.8%)
Intel Emerald Rapids (AVX512) 199 ms 108 ms 1.84x (+45.7%)
Intel Sapphire Rapids (AVX512) 230 ms 112 ms 2.05x (+51.3%)
AMD Turin Zen 5 (AVX512) 156 ms 99 ms 1.58x (+36.5%)
AMD Venice Zen 6 (AVX512) 204 ms 106 ms 1.92x (+48.0%)
Intel Skylake-X (AVX512) 320 ms 156 ms 2.05x (+51.2%)
AMD Genoa Zen 4 (AVX512) 236 ms 346 ms 0.68x (-46.6%)
Intel Ice Lake (AVX512) 242 ms 316 ms 0.77x (-30.6%)
AMD Milan Zen 3 (AVX2) 252 ms 230 ms 1.10x (+8.7%)
Intel Haswell (AVX2) 369 ms 456 ms 0.81x (-23.6%)
NVIDIA Grace (SVE2) 336 ms 337 ms 1.00x (-0.3%)
ARM Neoverse-N1 (NEON) 1081 ms 1105 ms 0.98x (-2.2%)
======================================================================
Bug: 537818862
Change-Id: I3ddeddb040fc52a30dabe7ae2c71d16dda021568
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8343467
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
- NV16/24 FOURCC and document
- Rotate take uint8* for ubsan
Bug: 537818862
Change-Id: I2bc7109abaffe2464e031e33c22296c007077d82
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8339427
Commit-Queue: Frank Barchard <fbarchard@google.com>
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Add matrix and BT.601 wrapper APIs for converting ARGB to NV16 and
NV24. Reuse the optimized ARGB Y and UV row functions, then merge
the planar chroma rows into the interleaved destination plane.
NV16 performs horizontal 2:1 chroma subsampling on every source row.
NV24 preserves full-resolution 4:4:4 chroma. Both paths support odd
widths, negative heights, and the existing matrix constants.
Add tests that compare scalar and optimized paths at an odd width and
height through the matrix API and its BT.601 wrapper.
Bug: 537818862
Change-Id: Iddefc714a800cdec4dd1394ac7a52bf7b51fd350
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8314188
Reviewed-by: Frank Barchard <fbarchard@google.com>
Reviewed-by: richard winterton <rrwinterton@gmail.com>
- Add avx512bw, neon 32 bit and RVV versions
- Adapt SSE2 to psrlw instead of pmulhuw
CPU Before After Improvement
---------------------------------------------------------------
Granite Rapids (AVX10) 77 ms 75 ms +2.6% (1.03x)
Emerald Rapids (AVX512) 75 ms 67 ms +10.7% (1.12x)
Sapphire Rapids (AVX512) 91 ms 85 ms +6.6% (1.07x)
AMD Venice Zen 6 (AVX512) 30 ms 30 ms 0.0% (1.00x)
AMD Turin Zen 5 (AVX512) 32 ms 32 ms 0.0% (1.00x)
AMD Genoa Zen 4 (AVX512) 35 ms 36 ms -2.9% (0.97x)
Skylake-X (AVX512) 119 ms 119 ms 0.0% (1.00x)
Ice Lake (AVX512) 93 ms 104 ms -11.8% (0.89x)
AMD Milan Zen 3 (AVX2) 43 ms 44 ms -2.3% (0.98x)
Haswell (AVX2) 129 ms 138 ms -7.0% (0.93x)
NVIDIA Grace (SVE2) 42 ms 43 ms -2.4% (0.98x)
Neoverse-N1 (NEON) 87 ms 87 ms 0.0% (1.00x)
---------------------------------------------------------------
Bug: None
Change-Id: Ia8f310a70c8b1229927fcdf117fe22e131d70ea9
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8330912
Commit-Queue: Frank Barchard <fbarchard@google.com>
Reviewed-by: richard winterton <rrwinterton@gmail.com>
1. source/convert_from_argb.cc should include libyuv/convert.h because
it calls ARGBToI420Matrix().
2. Include <string.h> for memset().
Change-Id: I3f4841fe604639c7357148ef4788fe0415f0b81b
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8255654
Commit-Queue: Wan-Teh Chang <wtc@google.com>
Reviewed-by: Frank Barchard <fbarchard@google.com>
Reduces include/libyuv/basic_types.h to only define LIBYUV_API.
Directly includes <stdint.h> and <stddef.h> where needed in public
headers.
Replaces LIBYUV_BOOL, LIBYUV_FALSE, and LIBYUV_TRUE with C++ bool,
true, and false in C++ code (MJpegDecoder, unit tests) and int (0, 1)
for C API functions.
Updates documentation and unit tests.
Test: ninja -C out/Release libyuv_unittest && ./out/Release/libyuv_unittest --gtest_filter=*
Change-Id: I2d58e872f0f976263bcda546dcc3b1b5f7e53d58
TAG=agy
CONV=2730bc56-93ff-48ed-b615-dcde1b834394
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8280662
Commit-Queue: Wan-Teh Chang <wtc@google.com>
Reviewed-by: Wan-Teh Chang <wtc@google.com>
CPU / Arch Before (Kernel) After (Kernel) Delta
----------------------------------------------------------------------
Haswell (x86_64) 263 ms (SSSE3) 263 ms (AVX2) 0.0%
Milan (Zen 3) 115 ms (SSSE3) 138 ms (AVX2) -20.0%
SapphRap (x86_64) 173 ms (SSSE3) 178 ms (AVX512VBMI) -2.9%
GNR (Granite R) 191 ms (SSSE3) 189 ms (AVX512VBMI) +1.0%
Turin (Zen 5) 69 ms (SSSE3) 55 ms (AVX512VBMI) +20.3%
Icelake (x86_64) 204 ms (SSSE3) 186 ms (AVX512VBMI) +8.8%
Cascade Lake 237 ms (SSSE3) 229 ms (AVX512BW) +3.4%
Neoverse (aarch64)* 103 ms (SVE2) 103 ms (SVE2) 0.0%
Bug: None
Change-Id: Ib218632e16b3308a214e9ffcbc745d6c41116c6e
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8280536
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
ScalePlane's same-width vertical fast path used the endpoint-oriented
FixedDiv1 step for all upsampling. That stepping is appropriate for
bilinear interpolation, but not for point sampling, where source rows
should cover equally sized destination cells.
Use the same centered FixedDiv stepping as the generic point sampler.
Allow the point path to select the final source row while passing a
zero interpolation stride so row kernels cannot form a pointer beyond
it. Add 8-bit and 16-bit point regression coverage and verify that
bilinear upsampling remains unchanged.
Bug: 537818862
Change-Id: I7bee21b85913b31c146ecf7fe9f5f08608a8a17d
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8264449
Reviewed-by: Wan-Teh Chang <wtc@google.com>
Commit-Queue: Wan-Teh Chang <wtc@google.com>
Reviewed-by: Frank Barchard <fbarchard@google.com>
In P010ToARGBMatrixFilterOverflow, P210ToARGBMatrixFilterOverflow,
P010ToAR30MatrixFilterOverflow, and P210ToAR30MatrixFilterOverflow,
the tests set src_stride_uv to width (1039) and allocated
width * src_uv_height elements. For odd widths, the interleaved UV
row requires (width + 1) & ~1 elements (1040). Using width for the
stride and buffer size caused the second row to read past the allocated
buffer and access uninitialized memory under MSan.
Fix this by computing src_stride_uv = (width + 1) & ~1 and sizing the
allocated buffer to src_stride_uv * src_uv_height.
In TestI400LargeSize, the test attempts large allocations totaling
over 10 GB (~2 GB for source and ~8.5 GB for destination). Under
MemorySanitizer (MSan), large allocations can exceed shadow memory
virtual address limits or assert. Skip this test under MSan by checking
__SANITIZE_MEMORY__ and __has_feature(memory_sanitizer).
Test: libyuv_unittest --gtest_filter=*FilterOverflow*:*I400Large*
Bug: 549112862
Change-Id: I679b7c67d93822e83f066b1895924589fc7242e7
TAG=agy
CONV=dd635180-175b-4e35-b7f0-2e38105d3506
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8269585
Commit-Queue: Frank Barchard <fbarchard@google.com>
Reviewed-by: Wan-Teh Chang <wtc@google.com>
HalfWidthRow_16To8 downsamples 2x2 16-bit pixels to 8-bit.
For width output pixels, row 0 and row 1 each require width * 2 uint16_t
elements (4 * width bytes).
With two rows separated by src_uv_stride, total source buffer requirement
is 8 * width bytes (4 * width uint16_t elements).
The previous buffer allocation only allocated kPixels * 4 bytes
(2 * kPixels uint16_t elements).
Because align_buffer_page_end places the buffer at the end of a page,
reading the second row caused an out-of-bounds access past the page
boundary, failing the test:
[ RUN ] LibYUVPlanarTest.HalfWidthRow_16To8_Opt
[ FAILED ] LibYUVPlanarTest.HalfWidthRow_16To8_Opt
[----------] Global test environment tear-down
[==========] 3603 tests from 8 test suites ran.
[ PASSED ] 3602 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] LibYUVPlanarTest.HalfWidthRow_16To8_Opt
1 FAILED TEST.
Use align_buffer_page_end_16 to allocate 4 * width uint16_t elements
(8 * width bytes) and set src_uv_stride to 2 * width elements
in HalfWidthRow_16To8_Opt and HalfWidthRow_16To8_Any.
Bug: None
Test: LibYUVPlanarTest.HalfWidthRow_16To8_*
Change-Id: Ieba81ec211a34cc315e60fc26aad64fb41a88641
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8252627
Commit-Queue: Frank Barchard <fbarchard@google.com>
Reviewed-by: Wan-Teh Chang <wtc@google.com>
In ScaleARGBBilinearDown and ScaleARGBBilinearUp, adding dy to y can
overflow signed 32-bit int when scaling down large height images (e.g.,
src_height = 32768 down to 2). Check if max_y - dy < y before adding dy
to y to prevent signed integer overflow.
R=wtc
Fixed: 540206290, 539110937
Test: LibYUVScaleTest.ARGBScale_LargeHeightOverflow
Change-Id: If353d002714481ce0018653798047066663f1fb9
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8175358
Reviewed-by: Frank Barchard <fbarchard@google.com>
Reviewed-by: Wan-Teh Chang <wtc@google.com>
Commit-Queue: Dale Curtis <dalecurtis@chromium.org>
Update JNV12ToABGR, JNV21ToABGR, JNV12ToRAW, and JNV21ToRAW test macros
in convert_argb_test.cc to use the public NV12ToABGRMatrix,
NV21ToABGRMatrix, NV12ToRAWMatrix, and NV21ToRAWMatrix wrapper macros.
Test: libyuv_unittest --gtest_filter=*JNV*
libyuv:537503370
Change-Id: I53c2496311cb30f621593f8a9d0011e1106ab501
TAG=agy
CONV=5daf08a8-11bd-4d47-8c71-c5b7a882a788
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8133049
Commit-Queue: Frank Barchard <fbarchard@google.com>
Reviewed-by: richard winterton <rrwinterton@gmail.com>
RAW format in libyuv is Red-first ([R, G, B] in memory), whereas RGB24
is Blue-first ([B, G, R] in memory).
RGBToYMatrixRow and RGBToUVMatrixRow internally expand 3-byte input
into 4-byte [byte0, byte1, byte2, 255]. For RAW, this buffer becomes
[R, G, B, 255] (matching ABGR byte order). RAWToI420 previously passed
&kArgbI601Constants (which expects [B, G, R, 255]) instead of
&kAbgrI601Constants, causing Red and Blue channels to be swapped.
Pass &kAbgrI601Constants in RAWToI420, fix parameter names in
source/convert.cc, and add a unit test.
Test: libyuv_unittest --gtest_filter=*RAWToI420*
Bug: libyuv:537503370
TAG=agy
CONV=2c54c4b5-f515-421b-80b3-079caa9f948b
Change-Id: Ic72470813eaf15e80046c2c01cf804cb827039ab
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8133048
Reviewed-by: richard winterton <rrwinterton@gmail.com>
This CL addresses two security findings related to integer overflows:
1. Input validation in ScalePlane, ScalePlane_16, and ScalePlane_12:
Added checks to reject invalid dimensions (e.g. width <= 0, height
== 0) and dimensions larger than 32768 (or smaller than -32768 for
height). This prevents FixedDiv signed integer overflows that can
lead to division by zero/overflow crashes (SIGFPE on x86) or
incorrect step calculations.
2. Stride overflow in ARGBAffineRow_C:
Casted pointer arithmetic operands to ptrdiff_t before multiplication
(y * stride and x * 4) to ensure 64-bit calculations, preventing
signed 32-bit integer overflow when calculating source pixel offsets.
Added unit tests to verify the input validation in ScalePlane functions.
Test: libyuv_unittest --gtest_filter=*InvalidInputs*
Test: libyuv_unittest --gtest_filter=*Scale*
Test: libyuv_unittest --gtest_filter=*TestAffine*
Bug: None
TAG=agy
CONV=0e990960-611b-4f38-94ec-24e79b66242e
R=wtc@google.com
Change-Id: I252af47a98e45dff8bb5f06308c3739c6eead741
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/7886217
Reviewed-by: Wan-Teh Chang <wtc@google.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
Audit all occurrences of "stride *" in the libyuv source tree. Ensure
that these multiplications are performed in the ptrdiff_t type.
For functions not declared in a public header (such as static
functions), prefer to declare the stride parameters (typically named
src_stride and dst_stride) and related stride local variables as
ptrdiff_t. If this is not possible, add ptrdiff_t casts to the stride
parameters in multiplications. If intptr_t or int64_t casts were used,
change them to ptrdiff_t casts.
Bug: chromium:516986556
Change-Id: I6cd8a8eb00cbb5380db828bf83e4d89ff95891f3
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/7882967
Commit-Queue: Wan-Teh Chang <wtc@google.com>
Reviewed-by: Frank Barchard <fbarchard@google.com>
