Clang MemorySanitizer builds define LIBYUV_ENABLE_ROWWIN and compile
row_win.cc and scale_win.cc instead of inline assembly. Two AVX-512
intrinsic patterns in those files are not modeled by LLVM MSan:
1. _mm512_mask_cvtepi32_storeu_epi8 and _mm512_mask_cvtepi64_storeu_epi8
lower to @llvm.x86.avx512.mask.pmov.db.mem.512 and
@llvm.x86.avx512.mask.pmov.qb.mem.512, which LLVM MSan handles as
unknown intrinsics without unpoisoning destination shadow memory.
Replace them in ARGBExtractAlphaRow_AVX512BW, ScaleAddCols1_AVX512BW,
and ScaleAddCols2_AVX512BW with _mm512_cvtepi32_epi8 /
_mm512_cvtepi64_epi8 followed by _mm_mask_storeu_epi8, which lowers
to @llvm.masked.store and still compiles to a single vpmovdb/vpmovqb
instruction.
2. ARGBCopyYToAlphaRow_AVX512BW loaded dst and blended the alpha channel
using _mm512_ternarylogic_epi32, which LLVM MSan does not model
bitwise and flags when any destination byte (such as the old alpha
channel) is uninitialized. Replace the load + vpternlogd + store in
both row_win.cc and row_gcc.cc with byte-masked stores
(_mm512_mask_storeu_epi8 / vmovdqu8 with mask 0x8888888888888888)
that write only the alpha bytes without reading dst.
Expand ARGBCopyAlpha, ARGBExtractAlpha, and ARGBCopyYToAlpha unit tests
into _Any, _Unaligned, _Invert, and _Opt variants and poison destination
buffers under MSan to catch unmodeled masked stores. Document MSan
AVX-512 intrinsic guidelines in GEMINI.md.
Test: out/msan/libyuv_unittest --gtest_filter=*Alpha*:*ScalePlaneBox_Any*
Bug: 578615529
Change-Id: I54ac4a5844aca6cfe3aa3445fa04b8221c610b35
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8672001
Commit-Queue: Frank Barchard <fbarchard@google.com>
Reviewed-by: Dale Curtis <dalecurtis@chromium.org>
Add ComputeCumulativeSumRow_AVX2, CumulativeSumToAverageRow_AVX2,
ComputeCumulativeSumRow_NEON, and CumulativeSumToAverageRow_NEON to
accelerate ARGBBlur and ARGBComputeCumulativeSum on x86_64 and aarch64.
Update ARGBBlurSmall_Opt to benchmark 320x180 with radius 1 and use a
circular cumsum buffer of height radius * 2 + 2 in TestBlur.
ARGBBlurSmall_Opt (320x180, r=1, 10000 iterations):
CPU Was C/SSE2 Now AVX2/NEON Speedup
-------------------- -------------- --------------- ---------------
Intel Granite Rapids 29 ms (SSE2) 24 ms (AVX2) 1.21x
Intel Emerald Rapids 30 ms (SSE2) 23 ms (AVX2) 1.30x
Intel Sapphire Rapid 32 ms (SSE2) 28 ms (AVX2) 1.14x
Intel Ice Lake 43 ms (SSE2) 34 ms (AVX2) 1.26x
Intel Skylake-X 54 ms (SSE2) 45 ms (AVX2) 1.20x
Intel Haswell 59 ms (SSE2) 54 ms (AVX2) 1.09x
AMD Zen 5 32 ms (SSE2) 19 ms (AVX2) 1.68x
AMD Zen 4 36 ms (SSE2) 26 ms (AVX2) 1.38x
AMD Zen 3 42 ms (SSE2) 28 ms (AVX2) 1.50x
ARM Neoverse-V2 256 ms (C) 40 ms (NEON) 6.40x
ARGBBlur_Opt (1280x720, r=55, 1000 iterations):
CPU Was C/SSE2 Now AVX2/NEON Speedup
-------------------- -------------- --------------- ---------------
Intel Granite Rapids 1150 ms (SSE2) 984 ms (AVX2) 1.17x
Intel Emerald Rapids 1081 ms (SSE2) 953 ms (AVX2) 1.13x
Intel Sapphire Rapid 1124 ms (SSE2) 973 ms (AVX2) 1.16x
Intel Ice Lake 1450 ms (SSE2) 1167 ms (AVX2) 1.24x
Intel Skylake-X 1935 ms (SSE2) 1630 ms (AVX2) 1.19x
Intel Haswell 2211 ms (SSE2) 1982 ms (AVX2) 1.12x
AMD Zen 5 1089 ms (SSE2) 694 ms (AVX2) 1.57x
AMD Zen 4 1045 ms (SSE2) 929 ms (AVX2) 1.12x
AMD Zen 3 1215 ms (SSE2) 967 ms (AVX2) 1.26x
ARM Neoverse-V2 6529 ms (C) 1544 ms (NEON) 4.23x
Bug: b/456384776
Change-Id: I39c463c791826d3adb05f0b0845b0b1f56f0e4da
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8464496
Commit-Queue: Frank Barchard <fbarchard@google.com>
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Add SwapUVRow_RVV using RVV intrinsics by default and unit-stride
e16, m4 inline assembly (vle16.v, vsrl.vi, vsll.vi, vor.vv, vse16.v)
under #ifdef RVV_ASM.
CPU C Intrinsics Inline ASM Speedup (Inline)
--------------------------------------------------------------------
SiFive X280 (est.) 980 ms 115 ms 38 ms 25.79x (3.03x)
SpacemiT K3 (est.) 420 ms 68 ms 28 ms 15.00x (2.43x)
T-Head C908 (est.) 860 ms 145 ms 62 ms 13.87x (2.34x)
Google Coral (est.) 1120 ms 190 ms 58 ms 19.31x (3.28x)
Original author: Matteo Forzan <matteo.forzan95@gmail.com>
R=wtc@google.comCC=matteo.forzan95@gmail.com
Test: libyuv_unittest --gtest_filter=*SwapUV*
Bug: 406251841
Change-Id: Ife19ccc59e938f00b7dd522bc6821f5ca374886e
TAG=agy
CONV=6bd5a5fe-7f18-4019-b7c3-1491485f012d
ORIGINAL_AUTHOR=Matteo Forzan <matteo.forzan95@gmail.com>
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8339302
Reviewed-by: David Gao <davidgao@google.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
Add SwapUVRow_AVX2 implementation in row_win.cc for MSVC and enable
HAS_SWAPUVROW_AVX2 under USE_ROW_WIN.
Share kShuffleUVToVU shuffle table between SSSE3 and AVX2 in row_gcc.cc.
Make NV12ToNV21 the primary implementation and alias NV21ToNV12 to it.
Add NV12ToNV21 test in convert_test.cc and benchmark dispatch for
SSSE3 and AVX2 in planar_test.cc.
CPU Before After Speedup
----------------------------------------------------------------
Intel Granite Rapids 282 ms 125 ms 2.26x
Intel Emerald Rapids 264 ms 108 ms 2.44x
Intel Sapphire Rapids 276 ms 122 ms 2.26x
Intel Ice Lake 310 ms 122 ms 2.54x
Intel Skylake-X 573 ms 157 ms 3.65x
Intel Haswell 620 ms 162 ms 3.83x
AMD Zen 4 (Genoa) 266 ms 52 ms 5.12x
AMD Zen 6 (Venice) 229 ms 43 ms 5.33x
ARM Neoverse-N1 140 ms 71 ms 1.97x
NVIDIA Grace (Neoverse-V2) 62 ms 69 ms 0.90x
R=wtc@google.com
Test: libyuv_unittest --gtest_filter=*SwapUV*:*NV12ToNV21*:*NV21ToNV12*
Bug: 406251841
Change-Id: I9bed5522aedadea13cdf69c27e3cbe633e31dd5b
TAG=agy
CONV=bb50ac62-0c55-4969-8fff-c71ab3d9a79f
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8356968
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
- Also remove redundant neon check for I8MM kernels
======================================================================
CPU Before After Speedup
----------------------------------------------------------------------
Intel Granite Rapids (AVX10.1) 251 ms 141 ms 1.78x (+43.8%)
Intel Emerald Rapids (AVX512) 199 ms 108 ms 1.84x (+45.7%)
Intel Sapphire Rapids (AVX512) 230 ms 112 ms 2.05x (+51.3%)
AMD Turin Zen 5 (AVX512) 156 ms 99 ms 1.58x (+36.5%)
AMD Venice Zen 6 (AVX512) 204 ms 106 ms 1.92x (+48.0%)
Intel Skylake-X (AVX512) 320 ms 156 ms 2.05x (+51.2%)
AMD Genoa Zen 4 (AVX512) 236 ms 346 ms 0.68x (-46.6%)
Intel Ice Lake (AVX512) 242 ms 316 ms 0.77x (-30.6%)
AMD Milan Zen 3 (AVX2) 252 ms 230 ms 1.10x (+8.7%)
Intel Haswell (AVX2) 369 ms 456 ms 0.81x (-23.6%)
NVIDIA Grace (SVE2) 336 ms 337 ms 1.00x (-0.3%)
ARM Neoverse-N1 (NEON) 1081 ms 1105 ms 0.98x (-2.2%)
======================================================================
Bug: 537818862
Change-Id: I3ddeddb040fc52a30dabe7ae2c71d16dda021568
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8343467
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
- NV16/24 FOURCC and document
- Rotate take uint8* for ubsan
Bug: 537818862
Change-Id: I2bc7109abaffe2464e031e33c22296c007077d82
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8339427
Commit-Queue: Frank Barchard <fbarchard@google.com>
Reviewed-by: richard winterton <rrwinterton@gmail.com>
- Add avx512bw, neon 32 bit and RVV versions
- Adapt SSE2 to psrlw instead of pmulhuw
CPU Before After Improvement
---------------------------------------------------------------
Granite Rapids (AVX10) 77 ms 75 ms +2.6% (1.03x)
Emerald Rapids (AVX512) 75 ms 67 ms +10.7% (1.12x)
Sapphire Rapids (AVX512) 91 ms 85 ms +6.6% (1.07x)
AMD Venice Zen 6 (AVX512) 30 ms 30 ms 0.0% (1.00x)
AMD Turin Zen 5 (AVX512) 32 ms 32 ms 0.0% (1.00x)
AMD Genoa Zen 4 (AVX512) 35 ms 36 ms -2.9% (0.97x)
Skylake-X (AVX512) 119 ms 119 ms 0.0% (1.00x)
Ice Lake (AVX512) 93 ms 104 ms -11.8% (0.89x)
AMD Milan Zen 3 (AVX2) 43 ms 44 ms -2.3% (0.98x)
Haswell (AVX2) 129 ms 138 ms -7.0% (0.93x)
NVIDIA Grace (SVE2) 42 ms 43 ms -2.4% (0.98x)
Neoverse-N1 (NEON) 87 ms 87 ms 0.0% (1.00x)
---------------------------------------------------------------
Bug: None
Change-Id: Ia8f310a70c8b1229927fcdf117fe22e131d70ea9
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8330912
Commit-Queue: Frank Barchard <fbarchard@google.com>
Reviewed-by: richard winterton <rrwinterton@gmail.com>
HalfWidthRow_16To8 downsamples 2x2 16-bit pixels to 8-bit.
For width output pixels, row 0 and row 1 each require width * 2 uint16_t
elements (4 * width bytes).
With two rows separated by src_uv_stride, total source buffer requirement
is 8 * width bytes (4 * width uint16_t elements).
The previous buffer allocation only allocated kPixels * 4 bytes
(2 * kPixels uint16_t elements).
Because align_buffer_page_end places the buffer at the end of a page,
reading the second row caused an out-of-bounds access past the page
boundary, failing the test:
[ RUN ] LibYUVPlanarTest.HalfWidthRow_16To8_Opt
[ FAILED ] LibYUVPlanarTest.HalfWidthRow_16To8_Opt
[----------] Global test environment tear-down
[==========] 3603 tests from 8 test suites ran.
[ PASSED ] 3602 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] LibYUVPlanarTest.HalfWidthRow_16To8_Opt
1 FAILED TEST.
Use align_buffer_page_end_16 to allocate 4 * width uint16_t elements
(8 * width bytes) and set src_uv_stride to 2 * width elements
in HalfWidthRow_16To8_Opt and HalfWidthRow_16To8_Any.
Bug: None
Test: LibYUVPlanarTest.HalfWidthRow_16To8_*
Change-Id: Ieba81ec211a34cc315e60fc26aad64fb41a88641
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8252627
Commit-Queue: Frank Barchard <fbarchard@google.com>
Reviewed-by: Wan-Teh Chang <wtc@google.com>
planar_test.cc was
Error: selected processor does not support `vmrs r3,fpscr' in ARM mode
Error: selected processor does not support `vmsr fpscr,r3' in ARM mode
Bug: None
Change-Id: I2ee0e7191c372277901c94e29d9ed91bbac71af2
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/7063737
Reviewed-by: Justin Green <greenjustin@google.com>
Commit-Queue: Frank Barchard <fbarchard@chromium.org>
The declarations of ARGBAffineRow_C and ARGBAffineRow_SSE2 and the code
to support those declarations are duplicated in planar_functions.h. They
are already in row.h, so we can simply remove them.
Change-Id: I9b522fdd201ca530f1268bf4200cd2e18b806ba5
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/6434733
Reviewed-by: Frank Barchard <fbarchard@chromium.org>
Commit-Queue: Wan-Teh Chang <wtc@google.com>
The existing tests reuse the intermediate buffers between the reference
and optimized implementations. In particular the existing tests appear
to pass even if the optimized implementation is completely empty, so
long as it does not modify the desintation buffers since these are
already filled with correct values from the reference code.
To avoid this, allocate separate buffers for optimized and reference
implementations to store intermediate data between function calls.
Additionally remove unused buffers from HalfMergeUVPlane_Opt tests.
Change-Id: I7e9ea21fc193e7be21cc24e2be0d7a122e068f6e
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/6074941
Reviewed-by: Frank Barchard <fbarchard@chromium.org>
- Remove special case Scale of 1 which used fp16 cvt but requires cpuid
- Port aarch64 to aarch32
- Use C for aarch32 with small (denormal) scale value
Bug: 377693555
Change-Id: I38e207e79ac54907ed6e65118b8109288fddb207
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/6043392
Reviewed-by: Wan-Teh Chang <wtc@google.com>
The existing behaviour does not round correctly in all cases, so adjust
it to match the existing Neon implementation.
Update the tests to require bit-exactness and disable other
implementations that do not round correctly.
Change-Id: Ie790fb4b4805b555d74d689d83802e1dd4f33df5
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/5869115
Reviewed-by: Frank Barchard <fbarchard@chromium.org>
Declare functions as static. Declare functions in a header. Include the
header that declares the functions. Delete undeclared and unused
functions ScaleFilterRows_NEON() and ScaleRowUp2_16_NEON(). Delete
unused function ScaleY() in psnr_main.cc.
Change-Id: I182ec30611df83c61ffd01bbab595cd61fb5f1e5
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/5778601
Commit-Queue: Wan-Teh Chang <wtc@google.com>
Reviewed-by: Frank Barchard <fbarchard@chromium.org>
- ARM Planar test use regular asm volatile syntax
- x86 row functions remove volatile from asm
Bug: 347111119, 347112532
Change-Id: I535b3dfa1a7a19824503bd95584a63b047b0e9a1
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/5637058
Reviewed-by: Justin Green <greenjustin@google.com>
- Makes ARM and Intel match and fixes some off by 1 cases
- Add ARGBToUV444MatrixRow_NEON
- Add ConvertFP16ToFP32Column_NEON
- scale_rvv fix intinsic build error
- disable row_win version of ARGBAttenuate/Unattenuate
Bug: libyuv:936, libyuv:956
Change-Id: Ied99aaad3a11a8eb69212b628c58f86ec0723c38
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/4617013
Reviewed-by: Wan-Teh Chang <wtc@google.com>
Commit-Queue: Frank Barchard <fbarchard@chromium.org>
* Run on SiFive internal FPGA:
TestARGBExtractAlpha(~3.2x vs scalar)
TestARGBCopyYToAlpha(~1.6x vs scalar)
Change-Id: I36525c67e8ac3f71ea9d1a58c7dc15a4009d9da1
Signed-off-by: Bruce Lai <bruce.lai@sifive.com>
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/4617955
Commit-Queue: Frank Barchard <fbarchard@chromium.org>
Reviewed-by: Frank Barchard <fbarchard@chromium.org>
- Convert MergeUVRow_AVX512BW to assembly
- Enable MergeUVRow_AVX512BW for Windows with clangcl
- MergeUVRow_AVX2 use vpmovzxbw and vpsllw
- MergeUVRow_16_AVX2 use vpmovzxbw and vpsllw with different shift for U and V
AMD Zen 4 640x360 100000 iterations
Was
AVX512 MergeUVPlane_Opt (884 ms)
AVX2 MergeUVPlane_Opt (945 ms)
AVX2 MergeUVPlane_16_Opt (2167 ms)
Now
AVX512 MergeUVPlane_Opt (865 ms)
AVX2 MergeUVPlane_Opt (943 ms)
SSE2 MergeUVPlane_Opt (973 ms)
AVX2 MergeUVPlane_16_Opt (2102 ms)
Bug: None
Change-Id: I658ada2a75d44c3f93be8bd3ed96f83d5fa2ab8d
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/4271230
Reviewed-by: Fritz Koenig <frkoenig@chromium.org>
Commit-Queue: Frank Barchard <fbarchard@chromium.org>
Reviewed-by: richard winterton <rrwinterton@gmail.com>
- Add I210ToI420 to convert 10 bit 4:2:2 YUV to 4:2:0 8 bit
- Add NEON InterpolateRow_16 for fast 10 bit scaling
- When scaling up, set step to interpolate toward height - 1 to avoid buffer overread
- When scaling down, center the 2 rows used for source to achieve filtering.
- CopyPlane check for 0 size and return
Bug: libyuv:931, b/228605787, b/233233302, b/233634772, b/234558395, b/234340482
Change-Id: I63e8580710a57812b683c2fe40583ac5a179c4f1
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/3687552
Reviewed-by: Mirko Bonadei <mbonadei@chromium.org>
Reviewed-by: richard winterton <rrwinterton@gmail.com>
If a width, height, and src/dst strides passed in are all 0, height is updated to 1 which means some CPU optimized functions may try to copy data when the dst rect is not valid.
Bug: b:234340482
Change-Id: I63be1c6ba05d669d67f5079d812acbec09c8f6c9
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/3689909
Reviewed-by: Frank Barchard <fbarchard@chromium.org>
This function reads 2 byte values and writes the 2nd byte to the destination.
It turns out this is useful for P010ToNV12 as well, so adding the planar function allows a high level to call this.
And adds UYVY support for something YUY2 already had. Which is writing the 1st byte.
Bug: b/233233302, b/233634772
Change-Id: I10a9454cb4f5b2c4ac5532fa86feddf78284d8b8
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/3659055
Commit-Queue: Frank Barchard <fbarchard@chromium.org>
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Reviewed-by: Frank Barchard <fbarchard@chromium.org>
Was
[----------] 59 tests from LibYUVScaleTest (223179 ms total)
Now
[----------] 13 tests from LibYUVScaleTest (15926 ms total)
Bug: b/224814071, b/228518489
Change-Id: Ifcb9c86793e94f32fd7cd2dd112dc3e6df77e283
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/3583609
Reviewed-by: Mirko Bonadei <mbonadei@chromium.org>
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@chromium.org>
Add support for MM21 to NV12 and I420 conversion, and add SIMD
optimizations for arm, aarch64, SSE2, and SSSE3 machines.
Bug: libyuv:915, b/215425056
Change-Id: Iecb0c33287f35766a6169d4adf3b7397f1ba8b5d
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/3433269
Reviewed-by: Frank Barchard <fbarchard@chromium.org>
Commit-Queue: Justin Green <greenjustin@google.com>
- ubsan complains on unaligned tests when an int16 or int32 is stored unaligned in C.
Although current Intel, ARM, Mips and PPC can do unaligned load/store, its not guaranteed
and could crash a CPU that doesnt support it.
- unaligned tests use offset of 2 or 4, which ubsan accepts.
- unittest fills in random buffer with 2 bytes at a time instead of a short.
- row common functions for int16 types use 2 shorts instead of 1 int.
Bug: libyuv:908, b/203243873
Change-Id: Idf13fa901647d7b0975f1947291caa781999a9bc
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/3229782
Commit-Queue: Frank Barchard <fbarchard@chromium.org>
Reviewed-by: Mirko Bonadei <mbonadei@chromium.org>
- reenable Intel SIMD unaffected by BIT_EXACT
- add bit exact version of ARGBAttenuate, which uses ARM version of formula.
- add bit exact version of ARGBUnatenuate, which mimics the AVX code.
Apply clang format to cleanup code.
Bug: libyuv:908, b/202888439
Change-Id: Ie842b1b3956b48f4190858e61c02998caedc2897
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/3224702
Commit-Queue: Frank Barchard <fbarchard@chromium.org>
Reviewed-by: richard winterton <rrwinterton@gmail.com>
