This CL addresses two security findings related to integer overflows:
1. Input validation in ScalePlane, ScalePlane_16, and ScalePlane_12:
Added checks to reject invalid dimensions (e.g. width <= 0, height
== 0) and dimensions larger than 32768 (or smaller than -32768 for
height). This prevents FixedDiv signed integer overflows that can
lead to division by zero/overflow crashes (SIGFPE on x86) or
incorrect step calculations.
2. Stride overflow in ARGBAffineRow_C:
Casted pointer arithmetic operands to ptrdiff_t before multiplication
(y * stride and x * 4) to ensure 64-bit calculations, preventing
signed 32-bit integer overflow when calculating source pixel offsets.
Added unit tests to verify the input validation in ScalePlane functions.
Test: libyuv_unittest --gtest_filter=*InvalidInputs*
Test: libyuv_unittest --gtest_filter=*Scale*
Test: libyuv_unittest --gtest_filter=*TestAffine*
Bug: None
TAG=agy
CONV=0e990960-611b-4f38-94ec-24e79b66242e
R=wtc@google.com
Change-Id: I252af47a98e45dff8bb5f06308c3739c6eead741
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/7886217
Reviewed-by: Wan-Teh Chang <wtc@google.com>
Commit-Queue: Frank Barchard <fbarchard@google.com>
ef08f21f6d