16 Commits

Author SHA1 Message Date
Frank Barchard
16af162cf9 [libyuv] Optimize RotatePlane_16 SIMD and compact 32-bit transposes
Implement SIMD 16-bit plane rotation (TransposePlane_16,
RotatePlane90_16, RotatePlane180_16, RotatePlane270_16) used by
I010Rotate, I210Rotate, and I410Rotate, keeping tile sizes compact
with full-register writes and zero cross-lane shuffles:
- AVX2: 8x16 tile in ymm0..ymm15 with _Any tail (low-end x86 SIMD)
- AVX-512BW: 8x32 tile in zmm0..zmm15 with k1/k2 right/bottom masks
- NEON (ARM64/ARM32): 8x16 tile in v0..v23 / q0..q11 with _Any tail
- SME: 8xVL tile using ZA16 horizontal/vertical array slices
- RVV: 8xVL tile using widening unpacks with hardware vl tail
- WASM SIMD: 8x8 tile in v128 with _Any tail

Also update 32-bit ARGBRotate x86 transposes to compact vertical
in-lane tiles with full-register writes and zero cross-lane shuffles:
- TransposeWxH_32_AVX2: 4x8 tile in ymm0..ymm7 with vpmaskmovd tails
- TransposeWxH_32_AVX512BW: 4x16 tile in zmm0..zmm7 with k1/k2 tails

Add RotatePlane16_SplitMergeUV_Opt and RotatePlane16_SplitMergeUV_Odd
unit tests comparing 16-bit rotation against SplitUVPlane +
RotatePlane + MergeUVPlane across 0/90/180/270 degrees.

Performance on 1275x1650 (1000 iterations, before -> after):
CPU                     90 degree               180 degree
Coral RVV (Verilator)   7306 -> 1415c (5.16x)   2274 -> 719c (3.16x)
AMD Rome (Zen 2)        821 -> 215ms (3.82x)    259 -> 181ms (1.44x)
AMD Genoa (Zen 4)       483 -> 129ms (3.73x)    149 -> 111ms (1.34x)
AMD Turin (Zen 5)       321 -> 101ms (3.18x)    109 -> 78ms (1.40x)
AMD Milan (Zen 3)       505 -> 189ms (2.67x)    199 -> 118ms (1.69x)
Intel Skylake-X         1076 -> 447ms (2.41x)   353 -> 317ms (1.11x)
WASM SIMD (d8)          415 -> 203ms (2.04x)    370 -> 106ms (3.49x)
Intel Emerald Rapids    735 -> 422ms (1.74x)    238 -> 233ms (1.02x)
Intel Haswell           1120 -> 648ms (1.73x)   438 -> 393ms (1.11x)
Intel Ice Lake          774 -> 474ms (1.63x)    279 -> 290ms (0.96x)
ARM Neoverse-N1         864 -> 544ms (1.59x)    404 -> 266ms (1.52x)
Ampere Siryn            862 -> 576ms (1.50x)    568 -> 329ms (1.72x)
Google Axion (Maple)    554 -> 379ms (1.46x)    199 -> 142ms (1.41x)
Intel Sapphire Rapids   1126 -> 772ms (1.46x)   301 -> 288ms (1.04x)
Intel Granite Rapids    730 -> 565ms (1.29x)    280 -> 280ms (1.00x)
NVIDIA Grace            462 -> 365ms (1.26x)    192 -> 126ms (1.52x)
ARM Cortex-A53          28290 -> 24386ms (1.16x) 7178 -> 5075ms (1.41x)
Google Axion 2 (Tamar)  523 -> 453ms (1.15x)    217 -> 112ms (1.93x)

Test: libyuv_unittest --gtest_filter=*Rotate*
Bug: b/42280942
Change-Id: I91ba6cac22185b5d294fcb98344fdb8a23cc94ff
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8545294
Reviewed-by: richard winterton <rrwinterton@gmail.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
2026-10-12 10:57:34 -07:00
Frank Barchard
eeb06a6451 [libyuv] Optimize ARGBRotate for Intel SSE2, AVX2, and AVX-512
Optimize 32-bit ARGBRotate (0, 90, 180, 270) across Intel x86/x64
architectures (SSE2, AVX2, AVX512BW), with matching kernels for ARM
(NEON, SVE2, SME), WASM, LoongArch (LSX, LASX), and RISC-V (RVV):

- Replace column-by-column ScaleARGBRowDownEven gather loops in
  ARGBTranspose (90/270) with full-plane horizontal tile transposes:
  - AVX512BW (TransposeWxH_32_AVX512BW / TransposeNx4_32_AVX512BW):
    Transpose 16x16 blocks across zmm0..zmm31 using 4-stage
    vpunpckldq/hdq, vpunpcklqdq/hqdq, and vshufi32x4 butterflies with
    k-masked right and bottom tails, eliminating Any wrappers and scalar
    C edge fallbacks.
  - AVX2 (TransposeWxH_32_AVX2 / TransposeNx4_32_AVX2):
    Transpose 8x8 blocks across ymm0..ymm15 using vpunpck* and
    vperm2i128 with vpmaskmovd masked tails.
  - SSE2 (Transpose4x4_32_SSE2 / Transpose4x4_32_Any_SSE2):
    Transpose 4x4 blocks in xmm registers.
- Implement ARGBRotate180 directly via ARGBMirror with negative height,
  avoiding intermediate row buffer allocations and extra copies.
- Implement ARGBRotate0 via ARGBCopy.

Speedup vs Before (1275x1650, US Letter 150 DPI, 1000 iterations):
CPU                  | 0 deg   | 90 deg  | 180 deg | 270 deg
---------------------|---------|---------|---------|--------
Intel Ice Lake       | 1.02x   | 1.34x   | 1.18x   | 1.33x
Intel Skylake-X      | 1.02x   | 1.20x   | 1.07x   | 1.13x
Intel Emerald Rapids | 1.01x   | 1.07x   | 1.00x   | 1.07x
Intel AVX-512 vs C   | 1.00x   | 2.67x   | 2.41x   | 2.63x

R=rrwinterton@gmail.com

Bug: b/42280915
Test: LIBYUV_WIDTH=1275 LIBYUV_HEIGHT=1650 LIBYUV_REPEAT=1000 ./out/Release/libyuv_unittest --gtest_filter=*ARGBRotate*
Change-Id: Ib308a225c17e46eab083a9da4033598fcf0c303c
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8538986
Reviewed-by: richard winterton <rrwinterton@gmail.com>
2026-10-08 16:20:46 -07:00
Frank Barchard
20b0a91bc1 Transpose 16x16 for AVX2
- TransposeWx16_AVX2 and TransposeUVWx16_AVX2 using 2 lanes of 8 ymm
  registers, with Any for tail.
- Move TransposeUVWx16_C to rotate_common.cc and pass the C function to
  TUVANY, so the LSX 16 row Any version transposes all 16 rows.

I420Rotate90_Opt (LIBYUV_WIDTH=1279 LIBYUV_HEIGHT=719 LIBYUV_REPEAT=1000):
CPU                      SSSE3   AVX2  AVX512  AVX2 x   512 x
Intel Haswell             1812    916     945   1.98x   1.92x
AMD Milan (Zen 3)          177    118     118   1.50x   1.50x
Intel Granite Rapids       245    186     186   1.32x   1.32x
Intel Emerald Rapids       246    193     176   1.27x   1.40x
Intel Sapphire Rapids      278    233     225   1.19x   1.24x
Intel Ice Lake             308    213     221   1.45x   1.39x
Intel Skylake-X            365    267     246   1.37x   1.48x
AMD Turin (Zen 5)          106     75      75   1.41x   1.41x
AMD Genoa (Zen 4)          149     98      93   1.52x   1.60x
AMD Venice (Zen 6)         106     74      73   1.43x   1.45x

NV12Rotate90_Opt (LIBYUV_WIDTH=1279 LIBYUV_HEIGHT=719 LIBYUV_REPEAT=1000):
CPU                      SSSE3   AVX2  AVX512  AVX2 x   512 x
Intel Haswell             1771    963     935   1.84x   1.89x
AMD Milan (Zen 3)          173    118     118   1.47x   1.47x
Intel Granite Rapids       275    189     186   1.46x   1.48x
Intel Emerald Rapids       256    188     179   1.36x   1.43x
Intel Sapphire Rapids      286    232     218   1.23x   1.31x
Intel Ice Lake             309    208     218   1.49x   1.42x
Intel Skylake-X            380    269     251   1.41x   1.51x
AMD Turin (Zen 5)          112     74      73   1.51x   1.53x
AMD Genoa (Zen 4)          146     97      93   1.51x   1.57x
AMD Venice (Zen 6)         114     73      72   1.56x   1.58x

Bug: None
Change-Id: Ia89219e3336f39a817aa662e1452824d781664a3
TAG=agy
CONV=bea53b37-b2bf-46fa-9e44-a708b438fabc
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8489456
Reviewed-by: richard winterton <rrwinterton@gmail.com>
2026-10-01 17:31:33 -07:00
Frank Barchard
2fcc1272dd Convert8To16 takes bits instead of scale.
- NV16/24 FOURCC and document
- Rotate take uint8* for ubsan

Bug: 537818862
Change-Id: I2bc7109abaffe2464e031e33c22296c007077d82
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8339427
Commit-Queue: Frank Barchard <fbarchard@google.com>
Reviewed-by: richard winterton <rrwinterton@gmail.com>
2026-09-01 16:07:23 -07:00
Wan-Teh Chang
904f562d86 Remove redundant #include <stddef.h>
"libyuv/basic_types.h" includes <stddef.h>. So it is not necessary to
include both <stddef.h> and "libyuv/basic_types.h".

Change-Id: I5a461258a3c6820d1007ac635838f910237f367f
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/7884381
Reviewed-by: Frank Barchard <fbarchard@google.com>
2026-05-28 17:10:22 -07:00
Wan-Teh Chang
ebe6fef903 Fix integer overflow in multiplications of stride
Audit all occurrences of "stride *" in the libyuv source tree. Ensure
that these multiplications are performed in the ptrdiff_t type.

For functions not declared in a public header (such as static
functions), prefer to declare the stride parameters (typically named
src_stride and dst_stride) and related stride local variables as
ptrdiff_t. If this is not possible, add ptrdiff_t casts to the stride
parameters in multiplications. If intptr_t or int64_t casts were used,
change them to ptrdiff_t casts.

Bug: chromium:516986556
Change-Id: I6cd8a8eb00cbb5380db828bf83e4d89ff95891f3
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/7882967
Commit-Queue: Wan-Teh Chang <wtc@google.com>
Reviewed-by: Frank Barchard <fbarchard@google.com>
2026-05-28 14:12:37 -07:00
George Steed
4f7fd808b7 [AArch64] Use full vectors in TransposeWx{8 => 16}_NEON
The existing Neon code only makes use of 64-bit vectors throughout which
limits the performance on larger cores. To avoid this, swap the Neon
code from a Wx8 implementation to a Wx16 implementation and process
blocks of 16 full vectors at a time.

The original code also handled widths that were not exact multiples of
16, however this should already be handled by the "any" kernel so it is
removed.

Finally, avoid duplicating the TransposeWx16_C fallback kernel
definition in all architectures that need it, and just put it once in
rotate_common.cc instead.

Observed speedups for TransposePlane across a range of
micro-architectures:

 Cortex-A53: -40.0%
 Cortex-A55: -20.7%
 Cortex-A57: -43.9%
Cortex-A510: -43.5%
Cortex-A520: -43.9%
Cortex-A720: -31.1%
  Cortex-X2: -38.3%
  Cortex-X4: -43.6%

Change-Id: Ic7c4d5f24eb27091d743ddc00cd95ef178b6984e
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/5545459
Reviewed-by: Frank Barchard <fbarchard@chromium.org>
2024-05-21 07:46:42 +00:00
Frank Barchard
650be7496f Fix warnings for missing prototypes
- Add static to internal scale and rotate functions
- Remove unittest that tested an internal scale function
- Remove unused private functions
- Include missing scale_argb.h header
- Bump version and apply clang format

Bug: libyuv:830
Change-Id: I45bab0423b86334f9707f935aedd0c6efc442dd4
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/4658956
Reviewed-by: Mirko Bonadei <mbonadei@chromium.org>
2023-06-30 17:46:56 +00:00
Frank Barchard
2bdc210be9 MergeUV_AVX512BW for I420ToNV12
On Skylake Xeon 640x360 100000 iterations
AVX512   MergeUVPlane_Opt (1196 ms)
AVX2     MergeUVPlane_Opt (1565 ms)
SSE2     MergeUVPlane_Opt (1780 ms)
Pixel 7  MergeUVPlane_Opt (1177 ms)

Bug: None
Change-Id: If47d4fa957cf27781bba5fd6a2f0bf554101a5c6
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/4242247
Commit-Queue: Frank Barchard <fbarchard@chromium.org>
Reviewed-by: richard winterton <rrwinterton@gmail.com>
2023-02-13 20:14:57 +00:00
Frank Barchard
6e4b0acb4b I422Rotate take stride for temporary buffers
- Minor variable name changes first/last to top/bottom
- Comments explaining rotate temporary buffers usage
- Add asserts for scale parameter
- Use NULL and stddef.h instead of 0
- Use void * for allocation in row.h
- Add () around size parameter in macros


Bug: libyuv:926, libyuv:949
Change-Id: Ib55417570926ccada0a0f8abd1753dc12e5b162e
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/4136762
Reviewed-by: Wan-Teh Chang <wtc@google.com>
Commit-Queue: Frank Barchard <fbarchard@chromium.org>
2023-01-04 23:11:52 +00:00
Sergio Garcia Murillo
f8626a7224 Add 10 bit rotate methods.
This initial implementation is based on current unoptimized code in webrtc using just plain for loops.

Bug: libyuv:949
Change-Id: Ic87ee49c3a0b62edbaaa4255c263c1f7be4ea02b
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/4110782
Reviewed-by: Frank Barchard <fbarchard@chromium.org>
Commit-Queue: Frank Barchard <fbarchard@chromium.org>
2023-01-04 21:10:01 +00:00
Frank Barchard
7e389884a1 Switch to C99 types
Append _t to all sized types.
uint64 becomes uint64_t etc

Bug: libyuv:774
Test: try bots build on all platforms
Change-Id: Ide273d7f8012313d6610415d514a956d6f3a8cac
Reviewed-on: https://chromium-review.googlesource.com/879922
Reviewed-by: Miguel Casas <mcasas@chromium.org>
2018-01-23 19:16:05 +00:00
Frank Barchard
73a603e120 clang-format 5.0 applied to libyuv
BUG=None
TEST=try bots and lint test

Change-Id: I1ab462adf2d309117862c5eb4b244a61ae202951
Reviewed-on: https://chromium-review.googlesource.com/450658
Commit-Queue: Frank Barchard <fbarchard@google.com>
Reviewed-by: Henrik Kjellander <kjellander@chromium.org>
2017-03-08 18:50:12 +00:00
Frank Barchard
e62309f259 clang-format libyuv
BUG=libyuv:654
R=kjellander@chromium.org

Review URL: https://codereview.chromium.org/2469353005 .
2016-11-07 17:37:23 -08:00
Frank Barchard
3a3a89ccd4 rotate include and proto cleanup
R=harryjin@google.com
BUG=libyuv:468

Review URL: https://webrtc-codereview.appspot.com/55679005.
2015-07-22 18:09:04 -07:00
Frank Barchard
3d190ee9f1 break rotate into files by cpu in preparation for optimization.
R=bcornell@google.com
BUG=libyuv:464

Review URL: https://webrtc-codereview.appspot.com/51289004.
2015-07-14 10:23:10 -07:00