Clang MemorySanitizer builds define LIBYUV_ENABLE_ROWWIN and compile
row_win.cc and scale_win.cc instead of inline assembly. Two AVX-512
intrinsic patterns in those files are not modeled by LLVM MSan:
1. _mm512_mask_cvtepi32_storeu_epi8 and _mm512_mask_cvtepi64_storeu_epi8
lower to @llvm.x86.avx512.mask.pmov.db.mem.512 and
@llvm.x86.avx512.mask.pmov.qb.mem.512, which LLVM MSan handles as
unknown intrinsics without unpoisoning destination shadow memory.
Replace them in ARGBExtractAlphaRow_AVX512BW, ScaleAddCols1_AVX512BW,
and ScaleAddCols2_AVX512BW with _mm512_cvtepi32_epi8 /
_mm512_cvtepi64_epi8 followed by _mm_mask_storeu_epi8, which lowers
to @llvm.masked.store and still compiles to a single vpmovdb/vpmovqb
instruction.
2. ARGBCopyYToAlphaRow_AVX512BW loaded dst and blended the alpha channel
using _mm512_ternarylogic_epi32, which LLVM MSan does not model
bitwise and flags when any destination byte (such as the old alpha
channel) is uninitialized. Replace the load + vpternlogd + store in
both row_win.cc and row_gcc.cc with byte-masked stores
(_mm512_mask_storeu_epi8 / vmovdqu8 with mask 0x8888888888888888)
that write only the alpha bytes without reading dst.
Expand ARGBCopyAlpha, ARGBExtractAlpha, and ARGBCopyYToAlpha unit tests
into _Any, _Unaligned, _Invert, and _Opt variants and poison destination
buffers under MSan to catch unmodeled masked stores. Document MSan
AVX-512 intrinsic guidelines in GEMINI.md.
Test: out/msan/libyuv_unittest --gtest_filter=*Alpha*:*ScalePlaneBox_Any*
Bug: 578615529
Change-Id: I54ac4a5844aca6cfe3aa3445fa04b8221c610b35
Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/8672001
Commit-Queue: Frank Barchard <fbarchard@google.com>
Reviewed-by: Dale Curtis <dalecurtis@chromium.org>
a84bf45a21