This is an automated email from the git hooks/post-receive script.
git pushed a commit to branch fix-release-build
in repository efl.
View the commit online.
commit 98163f38fd63e84bc20ede4cb704c52ecbbb3565
Author: [email protected] <[email protected]>
AuthorDate: Mon Aug 3 21:17:25 2026 -0600
evas: add AVX2 blend mask+colour and blend_rel span kernels
Ports _op_blend_mas_c_dp_sse3 and _op_blend_mas_can_dp_sse3 (and their
6 aliases) to op_blend_mask_color_avx2.c, reusing mul_sym_avx2 and
interp4_256_avx2. Both SM_AS slots are bit-exact against C and SSE3 -
the differential test at CPU_SSE3 shows no divergence for this group
(only the SM_N pixel+colour group's mul4_sym diverges), so no
exemption from the strict gate was needed. Registers all 8 slots,
including the 2 slots SSE3 leaves disabled behind FIXME comments
naming compiler bugs on a 32-bit Core i5 750 (Nehalem) and a 64-bit
Core i5 2500 (Sandy Bridge) - neither CPU supports AVX2, so those bug
reports do not apply here, the same call made for the pixel and
colour kernels. The two extra slots have no SSE3 counterpart to
compare against, so evas_avx2_vs_sse3_ops reports them as avx2-only;
their only gate is the (stricter) vs-C test, which passes.
There is no bit-exact/no-whitelist-exception language needed for this
group in the differential-test comments: since SSE3 itself matches C
exactly here, there is no SSE3-inherited exemption to apply in the
first place - unlike the pixel+colour group's SC/SC_AN slots.
Needed a mechanical fix versus the SSE3 template: SSE3's "all mask
bytes zero -> skip" fast path used `continue`, valid inside
LOOP_ALIGNED_U1_A48's loop but not inside
LOOP_ALIGNED_U1_A4_A8_A16's pre-alignment `if(...) A4OP` call site
(not a loop). Restructured as if/else with unconditional
pointer/length advancement instead.
Test results: evas_avx2_ops (vs C) PASS, 40 pairs (32 + 8 new), 0
outside tolerance. evas_avx2_vs_sse3_ops PASS, bit exact, 29 pairs
(23 + 6 new), 11 avx2-only slots (9 + 2 new).
Measured with expedite (buffer engine), median of 3 runs per tier at 20
loops per test, on an -O2 build. AVX2 was selected by leaving
EVAS_CPU_NO_AVX2 unset; the SSE3 baseline was selected by setting it.
Note EVAS_CPU_NO_AVX2 is tested by presence, not value - setting it to 0
still disables AVX2, it must be left unset to enable AVX2.
Tests 3/4/17: geometric mean +30% (test 3 +15%, test 4 +10%, test 17
+78%), pixel-identical output.
Also ports the 6 real blend_rel span kernels (plus their aliases)
across all five op_blend_*_avx2.c groups, registers them via
init_blend_rel_*_span_funcs_avx2(), and wires them into
evas_common_op_blend_rel_init_avx2() (previously an empty stub):
- pixel: 2 real kernels, rest aliased (pas_dp -> p_dp; DP_AN slots
alias the plain DP kernels, same as SSE3's own DP_AN->DP alias
chain; pan_dpan is NULL, matching SSE3).
- colour: 1 real kernel, caa_dp aliased to it, DP_AN aliased to the
(already-aliased) non-rel DP_AN kernels.
- pixel+colour: 1 real kernel; all 8 other DP slots are pure aliases
to it, matching SSE3's alias chain exactly.
- pixel+mask: 1 real kernel. SSE3's rel kernel here is #if-0'd out
with a "most likely don't perform the correct operation" FIXME and
its slot is #defined NULL, so there was no working SSE3 kernel to
port from or match; ported instead from the plain-C reference,
whose rel formula has no 0/255 mask fast path (unlike the base
kernel), so it vectorises directly with the same
mul_sym_avx2/mul_256_avx2 helpers already proven bit-exact vs C for
the group's base kernel.
- mask+colour: 1 real kernel, 2 aliases to it, DP_AN aliased to the
non-rel DP_AN kernels.
No _rel kernel was left NULL or dropped - all six vectorised cleanly.
Adding blend_rel exposed that the differential test's hand-maintained
whole-slot whitelist for tolerated C-divergence no longer scaled: it
added 15 more C-divergent SM_N slots (6 SC, 6 SC_AN, 3 of 6 SC_AA),
and a naive extension of the old three-condition whitelist would have
wrongly tolerated the 3 SC_AA slots that are actually exact against
C. Replaced with classify_diff(), a derived, per-pixel rule: when
SIMD_TIER==CPU_AVX2 and not SIMD_REQUIRE_EXACT, a C/AVX2 pixel
difference is tolerated only if the SSE3 slot for the same table
entry also differs from C on that exact pixel AND AVX2's result
matches SSE3's exactly - i.e. inherited rounding, not a new bug -
checked by running the SSE3 kernel over the same buffers alongside C
and AVX2. SIMD_REQUIRE_EXACT (the AVX2-vs-SSE3 gate) and the
evas_avx2_vs_sse3_ops target are untouched by this. The NEON build has
no SSE3 tier to consult, so classify_diff() falls back to the file's
original blanket "tolerate up to 1 LSB" rule there - verified by
reading and code review only, since this x86 host has no cpu_neon
build target to compile against.
Verification (build-opt): evas_avx2_ops PASS, blend 40 pairs
(unchanged), blend_rel 41 pairs (0 -> 41), 0 pixels outside the gate's
tolerance. evas_avx2_vs_sse3_ops PASS, bit-exact, blend_rel 35 pairs.
Gate-catches-new-bug proof: XORed 1 LSB into one AVX2 A8OP lane in the
colour kernel's rel path - both targets FAIL (exit 1, 2620
unexpected/differing pixels); reverted - both back to clean PASS.
Measured with expedite (buffer engine), median of 3 runs per tier at 20
loops per test, on an -O2 build. AVX2 was selected by leaving
EVAS_CPU_NO_AVX2 unset; the SSE3 baseline was selected by setting it.
Note EVAS_CPU_NO_AVX2 is tested by presence, not value - setting it to 0
still disables AVX2, it must be left unset to enable AVX2.
Tests 3/4/17 all faster under AVX2 (+12%, +15%, +75%), geometric mean
+30%.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
---
.../common/evas_op_blend/op_blend_color_avx2.c | 94 +++++-
.../evas_op_blend/op_blend_mask_color_avx2.c | 341 +++++++++++++++++++++
.../common/evas_op_blend/op_blend_master_avx2.c | 7 +
.../common/evas_op_blend/op_blend_pixel_avx2.c | 151 +++++++++
.../evas_op_blend/op_blend_pixel_color_avx2.c | 127 ++++++++
.../evas_op_blend/op_blend_pixel_mask_avx2.c | 112 +++++++
src/tests/evas/evas_test_simd_ops.c | 146 ++++++---
7 files changed, 920 insertions(+), 58 deletions(-)
diff --git a/src/lib/evas/common/evas_op_blend/op_blend_color_avx2.c b/src/lib/evas/common/evas_op_blend/op_blend_color_avx2.c
index f9a47539b0..00932a0b5f 100644
--- a/src/lib/evas/common/evas_op_blend/op_blend_color_avx2.c
+++ b/src/lib/evas/common/evas_op_blend/op_blend_color_avx2.c
@@ -1,17 +1,6 @@
/* blend color -> dst */
-/* What the differential test does and does not prove for this file: the four
- * slots registered below are backed by a single kernel function (plus its
- * aliases), at one arithmetic shape - constant colour blended over dest, no
- * source pointer, no mask. A bit-exact pass here validates the porting
- * pattern (mul_256_avx2 as a lane-local port of the SSE3 helper) and the
- * LOOP_ALIGNED_U1_A4_A8_A16 alignment handling for colour-only blends. It
- * does NOT validate any other arithmetic shape - pixel/mask blends, the
- * relative-blend variants, etc. Later kernel groups that copy this file's
- * pattern each need their own differential-test run against their own C
- * reference; none of that verification can be inherited from this result.
- *
- * The A4OP block is the SSE3 kernel's A4OP body verbatim (__m128i, _sse3
+/* The A4OP block is the SSE3 kernel's A4OP body verbatim (__m128i, _sse3
* helpers): the 4-wide stage exists so AVX2 executes the same instructions
* SSE3 does at 4-7 pixels. mul_256_sse3 is exact against plain C at every
* width, so this doesn't change this kernel's bit-exactness against C.
@@ -92,4 +81,85 @@ init_blend_color_span_funcs_avx2(void)
op_blend_span_funcs[SP_N][SM_N][SC_AA][DP_AN][CPU_AVX2] = _op_blend_caa_dpan_avx2;
}
+/*-----*/
+
+/* blend_rel color -> dst */
+
+static void
+_op_blend_rel_c_dp_avx2(DATA32 *s EINA_UNUSED, DATA8 *m EINA_UNUSED, DATA32 c, DATA32 *d, int l) {
+
+ int alpha = 256 - (c >> 24);
+
+ const __m256i c_packed = _mm256_set1_epi32(c);
+ const __m256i alpha_packed = _mm256_set1_epi32(alpha);
+ const __m128i c_packed128 = _mm_set_epi32(c, c, c, c);
+ const __m128i alpha_packed128 = _mm_set_epi32(alpha, alpha, alpha, alpha);
+
+ LOOP_ALIGNED_U1_A4_A8_A16(d, l,
+ { /* UOP */
+
+ *d = MUL_SYM(*d >> 24, c) + MUL_256(alpha, *d);
+ d++; l--;
+ },
+ { /* A4OP - verbatim SSE3 A4OP body, see file header */
+
+ __m128i d0 = _mm_load_si128((__m128i *)d);
+
+ __m128i mul0 = mul_256_sse3(alpha_packed128, d0);
+ __m128i sym0 = mul_sym_sse3(_mm_srli_epi32(d0, 24), c_packed128);
+
+ d0 = _mm_add_epi32(mul0, sym0);
+
+ _mm_store_si128((__m128i *)d, d0);
+
+ d += 4; l -= 4;
+ },
+ { /* A8OP */
+
+ __m256i d0 = _mm256_load_si256((__m256i *)d);
+
+ __m256i mul0 = mul_256_avx2(alpha_packed, d0);
+ __m256i sym0 = mul_sym_avx2(_mm256_srli_epi32(d0, 24), c_packed);
+
+ d0 = _mm256_add_epi32(mul0, sym0);
+
+ _mm256_store_si256((__m256i *)d, d0);
+
+ d += 8; l -= 8;
+ },
+ { /* A16OP */
+
+ __m256i d0 = _mm256_load_si256((__m256i *)d);
+ __m256i d1 = _mm256_load_si256((__m256i *)(d+8));
+
+ __m256i mul0 = mul_256_avx2(alpha_packed, d0);
+ __m256i mul1 = mul_256_avx2(alpha_packed, d1);
+
+ __m256i sym0 = mul_sym_avx2(_mm256_srli_epi32(d0, 24), c_packed);
+ __m256i sym1 = mul_sym_avx2(_mm256_srli_epi32(d1, 24), c_packed);
+
+ d0 = _mm256_add_epi32(mul0, sym0);
+ d1 = _mm256_add_epi32(mul1, sym1);
+
+ _mm256_store_si256((__m256i *)d, d0);
+ _mm256_store_si256((__m256i *)(d+8), d1);
+
+ d += 16; l -= 16;
+ })
+}
+
+#define _op_blend_rel_caa_dp_avx2 _op_blend_rel_c_dp_avx2
+#define _op_blend_rel_c_dpan_avx2 _op_blend_c_dpan_avx2
+#define _op_blend_rel_caa_dpan_avx2 _op_blend_caa_dpan_avx2
+
+static void
+init_blend_rel_color_span_funcs_avx2(void)
+{
+ op_blend_rel_span_funcs[SP_N][SM_N][SC][DP][CPU_AVX2] = _op_blend_rel_c_dp_avx2;
+ op_blend_rel_span_funcs[SP_N][SM_N][SC_AA][DP][CPU_AVX2] = _op_blend_rel_caa_dp_avx2;
+
+ op_blend_rel_span_funcs[SP_N][SM_N][SC][DP_AN][CPU_AVX2] = _op_blend_rel_c_dpan_avx2;
+ op_blend_rel_span_funcs[SP_N][SM_N][SC_AA][DP_AN][CPU_AVX2] = _op_blend_rel_caa_dpan_avx2;
+}
+
#endif
diff --git a/src/lib/evas/common/evas_op_blend/op_blend_mask_color_avx2.c b/src/lib/evas/common/evas_op_blend/op_blend_mask_color_avx2.c
new file mode 100644
index 0000000000..f21dd9266b
--- /dev/null
+++ b/src/lib/evas/common/evas_op_blend/op_blend_mask_color_avx2.c
@@ -0,0 +1,341 @@
+/* blend mask x color -> dst */
+
+/* Port of op_blend_mask_color_sse3.c's two span kernels
+ * (_op_blend_mas_c_dp_sse3, _op_blend_mas_can_dp_sse3). mul_sym_sse3/
+ * mul_sym_avx2 and interp4_256_sse3/interp4_256_avx2 agree with their C
+ * counterparts exactly for every SM_AS slot, so every slot registered below
+ * must be bit-exact against BOTH C and SSE3 - there is no rounding-gap
+ * excuse available here the way there is for the SC/SC_AN pixel+colour
+ * kernels in op_blend_pixel_color_avx2.c.
+ *
+ * As in that file, the A4OP stage runs the SSE3 kernel's 4-wide body
+ * verbatim (mul_sym_sse3/interp4_256_sse3, __m128i) so a 4-7 pixel span
+ * takes the identical instructions on both tiers; A8OP/A16OP use the AVX2
+ * counterparts mul_sym_avx2/interp4_256_avx2 from evas_blend_ops.h.
+ *
+ * MASK WIDENING: m is DATA8* (1 byte/pixel). A8OP widens 8 mask bytes with
+ * _mm256_cvtepu8_epi32(_mm_loadl_epi64(...)); A16OP does the same for m and
+ * m+8 - byte i always lands in 32-bit lane i, no lane-crossing shuffle
+ * involved. */
+
+#ifdef BUILD_AVX2
+
+static void
+_op_blend_mas_c_dp_avx2(DATA32 *s EINA_UNUSED, DATA8 *m, DATA32 c, DATA32 *d, int l) {
+
+ const __m256i c_packed = _mm256_set1_epi32(c);
+ const __m128i c_packed128 = _mm_set_epi32(c, c, c, c);
+
+ LOOP_ALIGNED_U1_A4_A8_A16(d, l,
+ { /* UOP - byte-identical to SSE3's UOP */
+
+ DATA32 a = *m;
+ DATA32 mc = MUL_SYM(a, c);
+ a = 256 - (mc >> 24);
+ *d = mc + MUL_256(a, *d);
+ m++; d++; l--;
+ },
+ { /* A4OP - verbatim SSE3 A4OP body (continue -> if/else: the
+ pre-alignment stage of LOOP_ALIGNED_U1_A4_A8_A16 invokes A4OP
+ outside any loop, where `continue` is illegal) */
+
+ if ((m[3] | m[2] | m[1] | m[0]) != 0) {
+ __m128i m0 = _mm_set_epi32(m[3], m[2], m[1], m[0]);
+ __m128i d0 = _mm_load_si128((__m128i *)d);
+
+ __m128i mc0 = mul_sym_sse3(m0, c_packed128);
+ __m128i a0 = sub4_alpha_sse3(mc0);
+ __m128i mul0 = mul_256_sse3(a0, d0);
+
+ mul0 = _mm_add_epi32(mul0, mc0);
+
+ _mm_store_si128((__m128i *)d, mul0);
+ }
+
+ m += 4; d += 4; l -= 4;
+ },
+ { /* A8OP */
+
+ if ((m[7] | m[6] | m[5] | m[4] | m[3] | m[2] | m[1] | m[0]) != 0) {
+ __m256i m0 = _mm256_cvtepu8_epi32(_mm_loadl_epi64((__m128i *)m));
+ __m256i d0 = _mm256_load_si256((__m256i *)d);
+
+ __m256i mc0 = mul_sym_avx2(m0, c_packed);
+ __m256i a0 = sub4_alpha_avx2(mc0);
+ __m256i mul0 = mul_256_avx2(a0, d0);
+
+ mul0 = _mm256_add_epi32(mul0, mc0);
+
+ _mm256_store_si256((__m256i *)d, mul0);
+ }
+
+ m += 8; d += 8; l -= 8;
+ },
+ { /* A16OP */
+
+ if ((m[7] | m[6] | m[5] | m[4] | m[3] | m[2] | m[1] | m[0] |
+ m[15] | m[14] | m[13] | m[12] | m[11] | m[10] | m[9] | m[8]) != 0) {
+ __m256i m0 = _mm256_cvtepu8_epi32(_mm_loadl_epi64((__m128i *)m));
+ __m256i d0 = _mm256_load_si256((__m256i *)d);
+
+ __m256i m1 = _mm256_cvtepu8_epi32(_mm_loadl_epi64((__m128i *)(m+8)));
+ __m256i d1 = _mm256_load_si256((__m256i *)(d+8));
+
+ __m256i mc0 = mul_sym_avx2(m0, c_packed);
+ __m256i a0 = sub4_alpha_avx2(mc0);
+ __m256i mul0 = mul_256_avx2(a0, d0);
+
+ mul0 = _mm256_add_epi32(mc0, mul0);
+
+ __m256i mc1 = mul_sym_avx2(m1, c_packed);
+ __m256i a1 = sub4_alpha_avx2(mc1);
+ __m256i mul1 = mul_256_avx2(a1, d1);
+
+ mul1 = _mm256_add_epi32(mc1, mul1);
+
+ _mm256_store_si256((__m256i *)d, mul0);
+ _mm256_store_si256((__m256i *)(d+8), mul1);
+ }
+
+ m += 16; d += 16; l -= 16;
+ })
+}
+
+static void
+_op_blend_mas_can_dp_avx2(DATA32 *s EINA_UNUSED, DATA8 *m, DATA32 c, DATA32 *d, int l) {
+
+ DATA32 alpha;
+
+ const __m256i _one_ = _mm256_set1_epi32(1);
+ const __m256i c_packed = _mm256_set1_epi32(c);
+ const __m128i _one128_ = _mm_set_epi32(1, 1, 1, 1);
+ const __m128i c_packed128 = _mm_set_epi32(c, c, c, c);
+
+ LOOP_ALIGNED_U1_A4_A8_A16(d, l,
+ { /* UOP - byte-identical to SSE3's UOP */
+
+ alpha = *m;
+ switch(alpha)
+ {
+ case 0:
+ break;
+ case 255:
+ *d = c;
+ break;
+ default:
+ alpha++;
+ *d = INTERP_256(alpha, c, *d);
+ break;
+ }
+ m++; d++; l--;
+ },
+ { /* A4OP - verbatim SSE3 A4OP body (continue -> if/else, see above) */
+
+ if ((m[3] | m[2] | m[1] | m[0]) != 0) {
+ __m128i m0 = _mm_set_epi32(m[3], m[2], m[1], m[0]);
+ __m128i d0 = _mm_load_si128((__m128i *)d);
+
+ __m128i zm0 = _mm_cmpeq_epi32(m0, _mm_setzero_si128());
+
+ m0 = _mm_add_epi32(one128, m0);
+
+ __m128i r0 = interp4_256_sse3(m0, c_packed128, d0);
+
+ r0 = _mm_and_si128(~zm0, r0);
+ d0 = _mm_and_si128(zm0, d0);
+
+ d0 = _mm_add_epi32(r0, d0);
+
+ _mm_store_si128((__m128i *)d, d0);
+ }
+
+ m += 4; d += 4; l -= 4;
+ },
+ { /* A8OP */
+
+ if ((m[7] | m[6] | m[5] | m[4] | m[3] | m[2] | m[1] | m[0]) != 0) {
+ __m256i m0 = _mm256_cvtepu8_epi32(_mm_loadl_epi64((__m128i *)m));
+ __m256i d0 = _mm256_load_si256((__m256i *)d);
+
+ __m256i zm0 = _mm256_cmpeq_epi32(m0, _mm256_setzero_si256());
+
+ m0 = _mm256_add_epi32(one, m0);
+
+ __m256i r0 = interp4_256_avx2(m0, c_packed, d0);
+
+ r0 = _mm256_andnot_si256(zm0, r0);
+ d0 = _mm256_and_si256(zm0, d0);
+
+ d0 = _mm256_add_epi32(r0, d0);
+
+ _mm256_store_si256((__m256i *)d, d0);
+ }
+
+ m += 8; d += 8; l -= 8;
+ },
+ { /* A16OP */
+
+ if ((m[7] | m[6] | m[5] | m[4] | m[3] | m[2] | m[1] | m[0] |
+ m[15] | m[14] | m[13] | m[12] | m[11] | m[10] | m[9] | m[8]) != 0) {
+ __m256i m0 = _mm256_cvtepu8_epi32(_mm_loadl_epi64((__m128i *)m));
+ __m256i d0 = _mm256_load_si256((__m256i *)d);
+
+ __m256i m1 = _mm256_cvtepu8_epi32(_mm_loadl_epi64((__m128i *)(m+8)));
+ __m256i d1 = _mm256_load_si256((__m256i *)(d+8));
+
+ __m256i zm0 = _mm256_cmpeq_epi32(m0, _mm256_setzero_si256());
+ __m256i zm1 = _mm256_cmpeq_epi32(m1, _mm256_setzero_si256());
+
+ m0 = _mm256_add_epi32(one, m0);
+ m1 = _mm256_add_epi32(one, m1);
+
+ __m256i r0 = interp4_256_avx2(m0, c_packed, d0);
+ __m256i r1 = interp4_256_avx2(m1, c_packed, d1);
+
+ r0 = _mm256_andnot_si256(zm0, r0);
+ d0 = _mm256_and_si256(zm0, d0);
+
+ r1 = _mm256_andnot_si256(zm1, r1);
+ d1 = _mm256_and_si256(zm1, d1);
+
+ d0 = _mm256_add_epi32(d0, r0);
+ d1 = _mm256_add_epi32(d1, r1);
+
+ _mm256_store_si256((__m256i *)d, d0);
+ _mm256_store_si256((__m256i *)(d+8), d1);
+ }
+
+ m += 16; d += 16; l -= 16;
+ })
+}
+
+#define _op_blend_mas_cn_dp_avx2 _op_blend_mas_can_dp_avx2
+#define _op_blend_mas_caa_dp_avx2 _op_blend_mas_c_dp_avx2
+
+#define _op_blend_mas_c_dpan_avx2 _op_blend_mas_c_dp_avx2
+#define _op_blend_mas_cn_dpan_avx2 _op_blend_mas_cn_dp_avx2
+#define _op_blend_mas_can_dpan_avx2 _op_blend_mas_can_dp_avx2
+#define _op_blend_mas_caa_dpan_avx2 _op_blend_mas_caa_dp_avx2
+
+static void
+init_blend_mask_color_span_funcs_avx2(void)
+{
+ /* SSE3 leaves these slots disabled (FIXMEs naming pre-Haswell CPUs that
+ * cannot reach an AVX2 path); AVX2 registers them - bit-exact vs C. */
+ op_blend_span_funcs[SP_N][SM_AS][SC][DP][CPU_AVX2] = _op_blend_mas_c_dp_avx2;
+ op_blend_span_funcs[SP_N][SM_AS][SC_N][DP][CPU_AVX2] = _op_blend_mas_cn_dp_avx2;
+ op_blend_span_funcs[SP_N][SM_AS][SC_AN][DP][CPU_AVX2] = _op_blend_mas_can_dp_avx2;
+ op_blend_span_funcs[SP_N][SM_AS][SC_AA][DP][CPU_AVX2] = _op_blend_mas_caa_dp_avx2;
+
+ op_blend_span_funcs[SP_N][SM_AS][SC][DP_AN][CPU_AVX2] = _op_blend_mas_c_dpan_avx2;
+ op_blend_span_funcs[SP_N][SM_AS][SC_N][DP_AN][CPU_AVX2] = _op_blend_mas_cn_dpan_avx2;
+ op_blend_span_funcs[SP_N][SM_AS][SC_AN][DP_AN][CPU_AVX2] = _op_blend_mas_can_dpan_avx2;
+ op_blend_span_funcs[SP_N][SM_AS][SC_AA][DP_AN][CPU_AVX2] = _op_blend_mas_caa_dpan_avx2;
+}
+
+/*-----*/
+
+/* blend_rel mask x color -> dst */
+
+static void
+_op_blend_rel_mas_c_dp_avx2(DATA32 *s EINA_UNUSED, DATA8 *m, DATA32 c, DATA32 *d, int l) {
+
+ const __m256i c_packed = _mm256_set1_epi32(c);
+ const __m128i c_packed128 = _mm_set_epi32(c, c, c, c);
+
+ LOOP_ALIGNED_U1_A4_A8_A16(d, l,
+ { /* UOP - byte-identical to SSE3's UOP */
+
+ DATA32 mc = MUL_SYM(*m, c);
+ int alpha = 256 - (mc >> 24);
+ *d = MUL_SYM(*d >> 24, mc) + MUL_256(alpha, *d);
+ d++; m++; l--;
+ },
+ { /* A4OP - verbatim SSE3 A4OP body, see file header */
+
+ __m128i m0 = _mm_set_epi32(m[3], m[2], m[1], m[0]);
+ __m128i d0 = _mm_load_si128((__m128i *) d);
+
+ __m128i mc0 = mul_sym_sse3(m0, c_packed128);
+ __m128i a0 = sub4_alpha_sse3(mc0);
+
+ __m128i d0_sym = mul_sym_sse3(_mm_srli_epi32(d0, 24), mc0);
+ d0 = mul_256_sse3(a0, d0);
+
+ d0 = _mm_add_epi32(d0, d0_sym);
+
+ _mm_store_si128((__m128i *)d, d0);
+
+ d += 4; m += 4; l -= 4;
+ },
+ { /* A8OP */
+
+ __m256i m0 = _mm256_cvtepu8_epi32(_mm_loadl_epi64((__m128i *)m));
+ __m256i d0 = _mm256_load_si256((__m256i *)d);
+
+ __m256i mc0 = mul_sym_avx2(m0, c_packed);
+ __m256i a0 = sub4_alpha_avx2(mc0);
+
+ __m256i d0_sym = mul_sym_avx2(_mm256_srli_epi32(d0, 24), mc0);
+ d0 = mul_256_avx2(a0, d0);
+
+ d0 = _mm256_add_epi32(d0, d0_sym);
+
+ _mm256_store_si256((__m256i *)d, d0);
+
+ d += 8; m += 8; l -= 8;
+ },
+ { /* A16OP */
+
+ __m256i m0 = _mm256_cvtepu8_epi32(_mm_loadl_epi64((__m128i *)m));
+ __m256i d0 = _mm256_load_si256((__m256i *)d);
+
+ __m256i m1 = _mm256_cvtepu8_epi32(_mm_loadl_epi64((__m128i *)(m+8)));
+ __m256i d1 = _mm256_load_si256((__m256i *)(d+8));
+
+ __m256i mc0 = mul_sym_avx2(m0, c_packed);
+ __m256i mc1 = mul_sym_avx2(m1, c_packed);
+
+ __m256i a0 = sub4_alpha_avx2(mc0);
+ __m256i a1 = sub4_alpha_avx2(mc1);
+
+ __m256i d0_sym = mul_sym_avx2(_mm256_srli_epi32(d0, 24), mc0);
+ __m256i d1_sym = mul_sym_avx2(_mm256_srli_epi32(d1, 24), mc1);
+
+ d0 = mul_256_avx2(a0, d0);
+ d1 = mul_256_avx2(a1, d1);
+
+ d0 = _mm256_add_epi32(d0, d0_sym);
+ d1 = _mm256_add_epi32(d1, d1_sym);
+
+ _mm256_store_si256((__m256i *)d, d0);
+ _mm256_store_si256((__m256i *)(d+8), d1);
+
+ d += 16; m += 16; l -= 16;
+ })
+}
+
+#define _op_blend_rel_mas_cn_dp_avx2 _op_blend_rel_mas_c_dp_avx2
+#define _op_blend_rel_mas_can_dp_avx2 _op_blend_rel_mas_c_dp_avx2
+#define _op_blend_rel_mas_caa_dp_avx2 _op_blend_rel_mas_c_dp_avx2
+
+#define _op_blend_rel_mas_c_dpan_avx2 _op_blend_mas_c_dpan_avx2
+#define _op_blend_rel_mas_cn_dpan_avx2 _op_blend_mas_cn_dpan_avx2
+#define _op_blend_rel_mas_can_dpan_avx2 _op_blend_mas_can_dpan_avx2
+#define _op_blend_rel_mas_caa_dpan_avx2 _op_blend_mas_caa_dpan_avx2
+
+static void
+init_blend_rel_mask_color_span_funcs_avx2(void)
+{
+ op_blend_rel_span_funcs[SP_N][SM_AS][SC][DP][CPU_AVX2] = _op_blend_rel_mas_c_dp_avx2;
+ op_blend_rel_span_funcs[SP_N][SM_AS][SC_N][DP][CPU_AVX2] = _op_blend_rel_mas_can_dp_avx2;
+ op_blend_rel_span_funcs[SP_N][SM_AS][SC_AN][DP][CPU_AVX2] = _op_blend_rel_mas_can_dp_avx2;
+ op_blend_rel_span_funcs[SP_N][SM_AS][SC_AA][DP][CPU_AVX2] = _op_blend_rel_mas_caa_dp_avx2;
+
+ op_blend_rel_span_funcs[SP_N][SM_AS][SC][DP_AN][CPU_AVX2] = _op_blend_rel_mas_c_dpan_avx2;
+ op_blend_rel_span_funcs[SP_N][SM_AS][SC_N][DP_AN][CPU_AVX2] = _op_blend_rel_mas_cn_dpan_avx2;
+ op_blend_rel_span_funcs[SP_N][SM_AS][SC_AN][DP_AN][CPU_AVX2] = _op_blend_rel_mas_can_dpan_avx2;
+ op_blend_rel_span_funcs[SP_N][SM_AS][SC_AA][DP_AN][CPU_AVX2] = _op_blend_rel_mas_caa_dpan_avx2;
+}
+
+#endif
diff --git a/src/lib/evas/common/evas_op_blend/op_blend_master_avx2.c b/src/lib/evas/common/evas_op_blend/op_blend_master_avx2.c
index a27368a55d..ba88560a6a 100644
--- a/src/lib/evas/common/evas_op_blend/op_blend_master_avx2.c
+++ b/src/lib/evas/common/evas_op_blend/op_blend_master_avx2.c
@@ -48,6 +48,7 @@ extern RGBA_Gfx_Func op_blend_rel_span_funcs[SP_LAST][SM_LAST][SC_LAST][DP_L
# include "op_blend_color_avx2.c"
# include "op_blend_pixel_color_avx2.c"
# include "op_blend_pixel_mask_avx2.c"
+# include "op_blend_mask_color_avx2.c"
void
evas_common_op_blend_init_avx2(void)
@@ -76,6 +77,7 @@ evas_common_op_blend_init_avx2(void)
init_blend_color_span_funcs_avx2();
init_blend_pixel_color_span_funcs_avx2();
init_blend_pixel_mask_span_funcs_avx2();
+ init_blend_mask_color_span_funcs_avx2();
#endif
}
@@ -83,5 +85,10 @@ void
evas_common_op_blend_rel_init_avx2(void)
{
#ifdef BUILD_AVX2
+ init_blend_rel_pixel_span_funcs_avx2();
+ init_blend_rel_color_span_funcs_avx2();
+ init_blend_rel_pixel_color_span_funcs_avx2();
+ init_blend_rel_pixel_mask_span_funcs_avx2();
+ init_blend_rel_mask_color_span_funcs_avx2();
#endif
}
diff --git a/src/lib/evas/common/evas_op_blend/op_blend_pixel_avx2.c b/src/lib/evas/common/evas_op_blend/op_blend_pixel_avx2.c
index fd3d267822..636c673487 100644
--- a/src/lib/evas/common/evas_op_blend/op_blend_pixel_avx2.c
+++ b/src/lib/evas/common/evas_op_blend/op_blend_pixel_avx2.c
@@ -96,4 +96,155 @@ init_blend_pixel_span_funcs_avx2(void)
op_blend_span_funcs[SP_AS][SM_N][SC_N][DP_AN][CPU_AVX2] = _op_blend_pas_dp_avx2;
}
+/*-----*/
+
+/* blend_rel pixel -> dst */
+
+static void
+_op_blend_rel_p_dp_avx2(DATA32 *s, DATA8 *m EINA_UNUSED, DATA32 c, DATA32 *d, int l) {
+
+ const __m128i _ones128_ = _mm_set_epi32(1, 1, 1, 1);
+ const __m256i _ones_ = _mm256_set1_epi32(1);
+
+ LOOP_ALIGNED_U1_A4_A8_A16(d, l,
+ { /* UOP */
+
+ int alpha = 256 - (*s >> 24);
+ c = 1 + (*d >> 24);
+ *d = MUL_256(c, *s) + MUL_256(alpha, *d);
+ d++; s++; l--;
+ },
+ { /*A4OP - verbatim SSE3 A4OP body, see file header */
+
+ __m128i s0 = _mm_lddqu_si128((__m128i *)s);
+ __m128i d0 = _mm_load_si128((__m128i *)d);
+
+ __m128i c0 = _mm_add_epi32(_mm_srli_epi32(d0, 24), ones128);
+ __m128i a0 = sub4_alpha_sse3(s0);
+
+ d0 = _mm_add_epi32(mul_256_sse3(c0, s0), mul_256_sse3(a0, d0));
+
+ _mm_store_si128((__m128i *)d, d0);
+
+ d += 4; s += 4; l -= 4;
+ },
+ { /* A8OP */
+
+ __m256i s0 = _mm256_loadu_si256((__m256i *)s);
+ __m256i d0 = _mm256_load_si256((__m256i *)d);
+
+ __m256i c0 = _mm256_add_epi32(_mm256_srli_epi32(d0, 24), ones);
+ __m256i a0 = sub4_alpha_avx2(s0);
+
+ d0 = _mm256_add_epi32(mul_256_avx2(c0, s0), mul_256_avx2(a0, d0));
+
+ _mm256_store_si256((__m256i *)d, d0);
+
+ d += 8; s += 8; l -= 8;
+ },
+ { /* A16OP */
+
+ __m256i s0 = _mm256_loadu_si256((__m256i *)s);
+ __m256i d0 = _mm256_load_si256((__m256i *)d);
+
+ __m256i s1 = _mm256_loadu_si256((__m256i *)(s+8));
+ __m256i d1 = _mm256_load_si256((__m256i *)(d+8));
+
+ __m256i c0 = _mm256_add_epi32(_mm256_srli_epi32(d0, 24), ones);
+ __m256i c1 = _mm256_add_epi32(_mm256_srli_epi32(d1, 24), ones);
+
+ __m256i a0 = sub4_alpha_avx2(s0);
+ __m256i a1 = sub4_alpha_avx2(s1);
+
+ d0 = _mm256_add_epi32(mul_256_avx2(c0, s0), mul_256_avx2(a0, d0));
+ d1 = _mm256_add_epi32(mul_256_avx2(c1, s1), mul_256_avx2(a1, d1));
+
+ _mm256_store_si256((__m256i *)d, d0);
+ _mm256_store_si256((__m256i *)(d+8), d1);
+
+ d += 16; s += 16; l -= 16;
+ })
+}
+
+static void
+_op_blend_rel_pan_dp_avx2(DATA32 *s, DATA8 *m EINA_UNUSED, DATA32 c, DATA32 *d, int l) {
+
+ const __m128i _ones128_ = _mm_set_epi32(1, 1, 1, 1);
+ const __m256i _ones_ = _mm256_set1_epi32(1);
+
+ LOOP_ALIGNED_U1_A4_A8_A16(d, l,
+ { /* UOP */
+
+ c = 1 + (*d >> 24);
+ *d++ = MUL_256(c, *s);
+ s++; l--;
+ },
+ { /* A4OP - verbatim SSE3 A4OP body, see file header */
+
+ __m128i s0 = _mm_lddqu_si128((__m128i *)s);
+ __m128i d0 = _mm_load_si128((__m128i *)d);
+
+ __m128i c0 = _mm_add_epi32(_mm_srli_epi32(d0, 24), ones128);
+ d0 = mul_256_sse3(c0, s0);
+
+ _mm_store_si128((__m128i *)d, d0);
+
+ d += 4; s += 4; l -= 4;
+ },
+ { /* A8OP */
+
+ __m256i s0 = _mm256_loadu_si256((__m256i *)s);
+ __m256i d0 = _mm256_load_si256((__m256i *)d);
+
+ __m256i c0 = _mm256_add_epi32(_mm256_srli_epi32(d0, 24), ones);
+ d0 = mul_256_avx2(c0, s0);
+
+ _mm256_store_si256((__m256i *)d, d0);
+
+ d += 8; s += 8; l -= 8;
+ },
+ { /* A16OP */
+
+ __m256i s0 = _mm256_loadu_si256((__m256i *)s);
+ __m256i d0 = _mm256_load_si256((__m256i *)d);
+
+ __m256i s1 = _mm256_loadu_si256((__m256i *)(s+8));
+ __m256i d1 = _mm256_load_si256((__m256i *)(d+8));
+
+ __m256i c0 = _mm256_add_epi32(_mm256_srli_epi32(d0, 24), ones);
+ __m256i c1 = _mm256_add_epi32(_mm256_srli_epi32(d1, 24), ones);
+
+ d0 = mul_256_avx2(c0, s0);
+ d1 = mul_256_avx2(c1, s1);
+
+ _mm256_store_si256((__m256i *)d, d0);
+ _mm256_store_si256((__m256i *)(d+8), d1);
+
+ d += 16; s += 16; l -= 16;
+ })
+}
+
+#define _op_blend_rel_pas_dp_avx2 _op_blend_rel_p_dp_avx2
+
+/* SSE3's DP_AN rel aliases point at its (non-rel) DP_AN kernels, which are
+ * themselves aliases of the DP kernels (_op_blend_p_dpan_sse3 ==
+ * _op_blend_p_dp_sse3, etc. - see op_blend_pixel_sse3.c). AVX2 registers
+ * those DP_AN slots directly from the DP kernels rather than defining
+ * separate _dpan_avx2 symbols, so alias straight to the same DP kernels here. */
+#define _op_blend_rel_p_dpan_avx2 _op_blend_p_dp_avx2
+#define _op_blend_rel_pan_dpan_avx2 NULL
+#define _op_blend_rel_pas_dpan_avx2 _op_blend_pas_dp_avx2
+
+static void
+init_blend_rel_pixel_span_funcs_avx2(void)
+{
+ op_blend_rel_span_funcs[SP][SM_N][SC_N][DP][CPU_AVX2] = _op_blend_rel_p_dp_avx2;
+ op_blend_rel_span_funcs[SP_AS][SM_N][SC_N][DP][CPU_AVX2] = _op_blend_rel_pas_dp_avx2;
+ op_blend_rel_span_funcs[SP_AN][SM_N][SC_N][DP][CPU_AVX2] = _op_blend_rel_pan_dp_avx2;
+
+ op_blend_rel_span_funcs[SP][SM_N][SC_N][DP_AN][CPU_AVX2] = _op_blend_rel_p_dpan_avx2;
+ op_blend_rel_span_funcs[SP_AS][SM_N][SC_N][DP_AN][CPU_AVX2] = _op_blend_rel_pas_dpan_avx2;
+ op_blend_rel_span_funcs[SP_AN][SM_N][SC_N][DP_AN][CPU_AVX2] = _op_blend_rel_pan_dpan_avx2;
+}
+
#endif
diff --git a/src/lib/evas/common/evas_op_blend/op_blend_pixel_color_avx2.c b/src/lib/evas/common/evas_op_blend/op_blend_pixel_color_avx2.c
index 723731dc70..cd2a7c77b7 100644
--- a/src/lib/evas/common/evas_op_blend/op_blend_pixel_color_avx2.c
+++ b/src/lib/evas/common/evas_op_blend/op_blend_pixel_color_avx2.c
@@ -459,4 +459,131 @@ init_blend_pixel_color_span_funcs_avx2(void)
op_blend_span_funcs[SP_AN][SM_N][SC_AA][DP_AN][CPU_AVX2] = _op_blend_pan_caa_dpan_avx2;
}
+/*-----*/
+
+/* blend_rel pixel x color -> dst */
+
+static void
+_op_blend_rel_p_c_dp_avx2(DATA32 *s, DATA8 *m EINA_UNUSED, DATA32 c, DATA32 *d, int l) {
+
+ int alpha;
+
+ const __m256i c_packed = _mm256_set1_epi32(c);
+ const __m128i c_packed128 = _mm_set_epi32(c, c, c, c);
+
+ LOOP_ALIGNED_U1_A4_A8_A16(d, l,
+ { /* UOP */
+
+ DATA32 sc = MUL4_SYM(c, *s);
+ alpha = 256 - (sc >> 24);
+ *d = MUL_SYM(*d >> 24, sc) + MUL_256(alpha, *d);
+ d++; s++; l--;
+ },
+ { /* A4OP - verbatim SSE3 A4OP body, see file header */
+
+ __m128i s0 = _mm_lddqu_si128((__m128i *)s);
+ __m128i d0 = _mm_load_si128((__m128i *)d);
+
+ __m128i sc0 = mul4_sym_sse3(c_packed128, s0);
+ __m128i a0 = sub4_alpha_sse3(sc0);
+
+ __m128i l0 = mul_sym_sse3(_mm_srli_epi32(d0, 24), sc0);
+ __m128i r0 = mul_256_sse3(a0, d0);
+
+ r0 = _mm_add_epi32(l0, r0);
+
+ _mm_store_si128((__m128i *)d, r0);
+
+ d += 4; s += 4; l -= 4;
+ },
+ { /* A8OP */
+
+ __m256i s0 = _mm256_loadu_si256((__m256i *)s);
+ __m256i d0 = _mm256_load_si256((__m256i *)d);
+
+ __m256i sc0 = mul4_sym_avx2(c_packed, s0);
+ __m256i a0 = sub4_alpha_avx2(sc0);
+
+ __m256i l0 = mul_sym_avx2(_mm256_srli_epi32(d0, 24), sc0);
+ __m256i r0 = mul_256_avx2(a0, d0);
+
+ r0 = _mm256_add_epi32(l0, r0);
+
+ _mm256_store_si256((__m256i *)d, r0);
+
+ d += 8; s += 8; l -= 8;
+ },
+ { /* A16OP */
+
+ __m256i s0 = _mm256_loadu_si256((__m256i *)s);
+ __m256i d0 = _mm256_load_si256((__m256i *)d);
+
+ __m256i s1 = _mm256_loadu_si256((__m256i *)(s+8));
+ __m256i d1 = _mm256_load_si256((__m256i *)(d+8));
+
+ __m256i sc0 = mul4_sym_avx2(c_packed, s0);
+ __m256i sc1 = mul4_sym_avx2(c_packed, s1);
+
+ __m256i a0 = sub4_alpha_avx2(sc0);
+ __m256i a1 = sub4_alpha_avx2(sc1);
+
+ __m256i l0 = mul_sym_avx2(_mm256_srli_epi32(d0, 24), sc0);
+ __m256i r0 = mul_256_avx2(a0, d0);
+
+ __m256i l1 = mul_sym_avx2(_mm256_srli_epi32(d1, 24), sc1);
+ __m256i r1 = mul_256_avx2(a1, d1);
+
+ r0 = _mm256_add_epi32(l0, r0);
+ r1 = _mm256_add_epi32(l1, r1);
+
+ _mm256_store_si256((__m256i *)d, r0);
+ _mm256_store_si256((__m256i *)(d+8), r1);
+
+ d += 16; s += 16; l -= 16;
+ })
+}
+
+#define _op_blend_rel_pas_c_dp_avx2 _op_blend_rel_p_c_dp_avx2
+#define _op_blend_rel_pan_c_dp_avx2 _op_blend_rel_p_c_dp_avx2
+#define _op_blend_rel_p_can_dp_avx2 _op_blend_rel_p_c_dp_avx2
+#define _op_blend_rel_pas_can_dp_avx2 _op_blend_rel_p_c_dp_avx2
+#define _op_blend_rel_pan_can_dp_avx2 _op_blend_rel_p_c_dp_avx2
+#define _op_blend_rel_p_caa_dp_avx2 _op_blend_rel_p_c_dp_avx2
+#define _op_blend_rel_pas_caa_dp_avx2 _op_blend_rel_p_c_dp_avx2
+#define _op_blend_rel_pan_caa_dp_avx2 _op_blend_rel_p_c_dp_avx2
+
+#define _op_blend_rel_p_c_dpan_avx2 _op_blend_p_c_dpan_avx2
+#define _op_blend_rel_pas_c_dpan_avx2 _op_blend_pas_c_dpan_avx2
+#define _op_blend_rel_pan_c_dpan_avx2 _op_blend_pan_c_dpan_avx2
+#define _op_blend_rel_p_can_dpan_avx2 _op_blend_p_can_dpan_avx2
+#define _op_blend_rel_pas_can_dpan_avx2 _op_blend_pas_can_dpan_avx2
+#define _op_blend_rel_pan_can_dpan_avx2 _op_blend_pan_can_dpan_avx2
+#define _op_blend_rel_p_caa_dpan_avx2 _op_blend_p_caa_dpan_avx2
+#define _op_blend_rel_pas_caa_dpan_avx2 _op_blend_pas_caa_dpan_avx2
+#define _op_blend_rel_pan_caa_dpan_avx2 _op_blend_pan_caa_dpan_avx2
+
+static void
+init_blend_rel_pixel_color_span_funcs_avx2(void)
+{
+ op_blend_rel_span_funcs[SP][SM_N][SC][DP][CPU_AVX2] = _op_blend_rel_p_c_dp_avx2;
+ op_blend_rel_span_funcs[SP_AS][SM_N][SC][DP][CPU_AVX2] = _op_blend_rel_pas_c_dp_avx2;
+ op_blend_rel_span_funcs[SP_AN][SM_N][SC][DP][CPU_AVX2] = _op_blend_rel_pan_c_dp_avx2;
+ op_blend_rel_span_funcs[SP][SM_N][SC_AN][DP][CPU_AVX2] = _op_blend_rel_p_can_dp_avx2;
+ op_blend_rel_span_funcs[SP_AS][SM_N][SC_AN][DP][CPU_AVX2] = _op_blend_rel_pas_can_dp_avx2;
+ op_blend_rel_span_funcs[SP_AN][SM_N][SC_AN][DP][CPU_AVX2] = _op_blend_rel_pan_can_dp_avx2;
+ op_blend_rel_span_funcs[SP][SM_N][SC_AA][DP][CPU_AVX2] = _op_blend_rel_p_caa_dp_avx2;
+ op_blend_rel_span_funcs[SP_AS][SM_N][SC_AA][DP][CPU_AVX2] = _op_blend_rel_pas_caa_dp_avx2;
+ op_blend_rel_span_funcs[SP_AN][SM_N][SC_AA][DP][CPU_AVX2] = _op_blend_rel_pan_caa_dp_avx2;
+
+ op_blend_rel_span_funcs[SP][SM_N][SC][DP_AN][CPU_AVX2] = _op_blend_rel_p_c_dpan_avx2;
+ op_blend_rel_span_funcs[SP_AS][SM_N][SC][DP_AN][CPU_AVX2] = _op_blend_rel_pas_c_dpan_avx2;
+ op_blend_rel_span_funcs[SP_AN][SM_N][SC][DP_AN][CPU_AVX2] = _op_blend_rel_pan_c_dpan_avx2;
+ op_blend_rel_span_funcs[SP][SM_N][SC_AN][DP_AN][CPU_AVX2] = _op_blend_rel_p_can_dpan_avx2;
+ op_blend_rel_span_funcs[SP_AS][SM_N][SC_AN][DP_AN][CPU_AVX2] = _op_blend_rel_pas_can_dpan_avx2;
+ op_blend_rel_span_funcs[SP_AN][SM_N][SC_AN][DP_AN][CPU_AVX2] = _op_blend_rel_pan_can_dpan_avx2;
+ op_blend_rel_span_funcs[SP][SM_N][SC_AA][DP_AN][CPU_AVX2] = _op_blend_rel_p_caa_dpan_avx2;
+ op_blend_rel_span_funcs[SP_AS][SM_N][SC_AA][DP_AN][CPU_AVX2] = _op_blend_rel_pas_caa_dpan_avx2;
+ op_blend_rel_span_funcs[SP_AN][SM_N][SC_AA][DP_AN][CPU_AVX2] = _op_blend_rel_pan_caa_dpan_avx2;
+}
+
#endif
diff --git a/src/lib/evas/common/evas_op_blend/op_blend_pixel_mask_avx2.c b/src/lib/evas/common/evas_op_blend/op_blend_pixel_mask_avx2.c
index 641cdd8cfb..03ac257c91 100644
--- a/src/lib/evas/common/evas_op_blend/op_blend_pixel_mask_avx2.c
+++ b/src/lib/evas/common/evas_op_blend/op_blend_pixel_mask_avx2.c
@@ -166,4 +166,116 @@ init_blend_pixel_mask_span_funcs_avx2(void)
op_blend_span_funcs[SP_AN][SM_AS][SC_N][DP_AN][CPU_AVX2] = _op_blend_pan_mas_dpan_avx2;
}
+/*-----*/
+
+/* blend_rel pixel x mask -> dst */
+
+/* Same situation as the base kernel above: the SSE3 _rel_p_mas_dp body is
+ * compiled out (#if 0, same FIXME) and its slot is #defined NULL, so there
+ * is no working SSE3 kernel to match. Ported instead from the plain-C
+ * reference (op_blend_pixel_mask_.c), which for the rel variant is simpler
+ * than the base kernel: no 0/255 mask special-casing at all, just
+ * `c = MUL_SYM(*m, *s); *d = MUL_SYM(*d>>24, c) + MUL_256(256-(c>>24), *d)`
+ * unconditionally - directly vectorisable with mul_sym_avx2/mul_256_avx2,
+ * same helpers already proven bit-exact against C for this group's base
+ * kernel above. */
+
+static void
+_op_blend_rel_p_mas_dp_avx2(DATA32 *s, DATA8 *m, DATA32 c, DATA32 *d, int l) {
+
+ LOOP_ALIGNED_U1_A4_A8_A16(d, l,
+ { /* UOP - byte-identical to the C reference's UOP */
+
+ c = MUL_SYM(*m, *s);
+ int alpha = 256 - (c >> 24);
+ *d = MUL_SYM(*d >> 24, c) + MUL_256(alpha, *d);
+ d++; m++; s++; l--;
+ },
+ { /* A4OP - disabled SSE3 A4OP body verbatim, see file header */
+
+ __m128i s0 = _mm_lddqu_si128((__m128i *)s);
+ __m128i m0 = _mm_set_epi32(m[3], m[2], m[1], m[0]);
+ __m128i d0 = _mm_load_si128((__m128i *)d);
+
+ __m128i c0 = mul_sym_sse3(m0, s0);
+ __m128i a0 = sub4_alpha_sse3(c0);
+
+ __m128i l0 = mul_sym_sse3(_mm_srli_epi32(d0, 24), c0);
+ __m128i r0 = mul_256_sse3(a0, d0);
+
+ d0 = _mm_add_epi32(l0, r0);
+
+ _mm_store_si128((__m128i *)d, d0);
+
+ d += 4; m += 4; s += 4; l -= 4;
+ },
+ { /* A8OP */
+
+ __m256i s0 = _mm256_loadu_si256((__m256i *)s);
+ __m256i m0 = _mm256_cvtepu8_epi32(_mm_loadl_epi64((__m128i *)m));
+ __m256i d0 = _mm256_load_si256((__m256i *)d);
+
+ __m256i c0 = mul_sym_avx2(m0, s0);
+ __m256i a0 = sub4_alpha_avx2(c0);
+
+ __m256i l0 = mul_sym_avx2(_mm256_srli_epi32(d0, 24), c0);
+ __m256i r0 = mul_256_avx2(a0, d0);
+
+ d0 = _mm256_add_epi32(l0, r0);
+
+ _mm256_store_si256((__m256i *)d, d0);
+
+ d += 8; m += 8; s += 8; l -= 8;
+ },
+ { /* A16OP */
+
+ __m256i s0 = _mm256_loadu_si256((__m256i *)s);
+ __m256i m0 = _mm256_cvtepu8_epi32(_mm_loadl_epi64((__m128i *)m));
+ __m256i d0 = _mm256_load_si256((__m256i *)d);
+
+ __m256i s1 = _mm256_loadu_si256((__m256i *)(s+8));
+ __m256i m1 = _mm256_cvtepu8_epi32(_mm_loadl_epi64((__m128i *)(m+8)));
+ __m256i d1 = _mm256_load_si256((__m256i *)(d+8));
+
+ __m256i c0 = mul_sym_avx2(m0, s0);
+ __m256i c1 = mul_sym_avx2(m1, s1);
+
+ __m256i a0 = sub4_alpha_avx2(c0);
+ __m256i a1 = sub4_alpha_avx2(c1);
+
+ __m256i l0 = mul_sym_avx2(_mm256_srli_epi32(d0, 24), c0);
+ __m256i r0 = mul_256_avx2(a0, d0);
+
+ __m256i l1 = mul_sym_avx2(_mm256_srli_epi32(d1, 24), c1);
+ __m256i r1 = mul_256_avx2(a1, d1);
+
+ d0 = _mm256_add_epi32(l0, r0);
+ d1 = _mm256_add_epi32(l1, r1);
+
+ _mm256_store_si256((__m256i *)d, d0);
+ _mm256_store_si256((__m256i *)(d+8), d1);
+
+ d += 16; m += 16; s += 16; l -= 16;
+ })
+}
+
+#define _op_blend_rel_pas_mas_dp_avx2 _op_blend_rel_p_mas_dp_avx2
+#define _op_blend_rel_pan_mas_dp_avx2 _op_blend_rel_p_mas_dp_avx2
+
+#define _op_blend_rel_p_mas_dpan_avx2 _op_blend_p_mas_dpan_avx2
+#define _op_blend_rel_pas_mas_dpan_avx2 _op_blend_pas_mas_dpan_avx2
+#define _op_blend_rel_pan_mas_dpan_avx2 _op_blend_pan_mas_dpan_avx2
+
+static void
+init_blend_rel_pixel_mask_span_funcs_avx2(void)
+{
+ op_blend_rel_span_funcs[SP][SM_AS][SC_N][DP][CPU_AVX2] = _op_blend_rel_p_mas_dp_avx2;
+ op_blend_rel_span_funcs[SP_AS][SM_AS][SC_N][DP][CPU_AVX2] = _op_blend_rel_pas_mas_dp_avx2;
+ op_blend_rel_span_funcs[SP_AN][SM_AS][SC_N][DP][CPU_AVX2] = _op_blend_rel_pan_mas_dp_avx2;
+
+ op_blend_rel_span_funcs[SP][SM_AS][SC_N][DP_AN][CPU_AVX2] = _op_blend_rel_p_mas_dpan_avx2;
+ op_blend_rel_span_funcs[SP_AS][SM_AS][SC_N][DP_AN][CPU_AVX2] = _op_blend_rel_pas_mas_dpan_avx2;
+ op_blend_rel_span_funcs[SP_AN][SM_AS][SC_N][DP_AN][CPU_AVX2] = _op_blend_rel_pan_mas_dpan_avx2;
+}
+
#endif
diff --git a/src/tests/evas/evas_test_simd_ops.c b/src/tests/evas/evas_test_simd_ops.c
index 915b303c3f..cff50342af 100644
--- a/src/tests/evas/evas_test_simd_ops.c
+++ b/src/tests/evas/evas_test_simd_ops.c
@@ -312,38 +312,62 @@ typedef struct
* SIMD_REQUIRE_EXACT above */
} Stats;
-/* Whitelist for the plain-C-reference gate (see SIMD_MODE_NAME above): the
- * only slots allowed to differ from CPU_C are the 12 pixel+colour SC/SC_AN
- * AVX2 kernels, whose AVX2 form deliberately matches SSE3's mul4_sym/
- * mul3_sym rounding rather than the plain-C MUL4_SYM/MUL3_SYM macros (see
- * op_blend_pixel_color_avx2.c). Every other slot - including the SC_AA
- * kernels in the same file, and every kernel in every other blend/copy
- * table - is expected to be exact against C, so a difference there is
- * always a hard failure, never "rounding". Irrelevant (and unused) when
- * SIMD_REQUIRE_EXACT is defined, since that mode disallows any diff at all
- * regardless of slot.
+/* Derived (not hand-maintained) exemption for the plain-C-reference gate (see
+ * SIMD_MODE_NAME above).
*
- * Scoped to SIMD_TIER == CPU_AVX2 only: this whitelist documents a property
- * of the AVX2 port specifically. The NEON build compiles this same file
- * with SIMD_TIER=CPU_NEON and no SIMD_REF_TIER override (so it also runs in
- * plain-C-reference mode); it must keep its pre-existing, unrelated
- * tolerance - up to 1 LSB anywhere, gated only by the total.max_delta check
- * below - rather than suddenly being held to an AVX2-specific slot list it
- * has nothing to do with. Outside CPU_AVX2 this always returns "tolerated"
- * so the per-slot unexpected_diff accounting is a no-op and behaviour is
- * unchanged from before this gate existed. Unused (and compiled out) under
- * SIMD_REQUIRE_EXACT, where every caller is itself compiled out too. */
+ * A hand-written (table, m, c) whitelist - as this file used to have - goes
+ * stale the moment a new kernel group lands: the measured fact is that in
+ * the blend_rel table 15 of 35 SM_N slots diverge from C at
+ * CPU_SSE3, split as SC (6), SC_AN (6) and SC_AA (only 3 of 6) - so a triple
+ * list extended by hand to cover blend_rel would either miss slots or, worse,
+ * wrongly tolerate the 3 SC_AA slots that are actually exact. Instead this
+ * derives the exemption per pixel, from the code under test itself: CPU_C,
+ * CPU_SSE3 and CPU_AVX2 are all live in the same process (same op tables,
+ * different [CPU_*] slots), so all three can be run over the identical input
+ * and compared directly rather than guessed at from a list.
+ *
+ * classify_diff() below is the rule: a pixel where AVX2 disagrees with C is
+ * tolerated only if SSE3 *also* disagrees with C on that exact pixel, and
+ * AVX2's result matches SSE3's exactly - i.e. the divergence is inherited
+ * from SSE3's pre-existing rounding, not a new AVX2-only bug. Any other
+ * disagreement (no SSE3 slot to check against on this tier, or AVX2 differs
+ * from SSE3 too, or SSE3 happens to agree with C while AVX2 doesn't) is a
+ * hard, unexplained divergence and fails.
+ *
+ * Scoped to SIMD_TIER == CPU_AVX2: only that tier has an SSE3 slot in the
+ * same op tables to consult. The NEON build (SIMD_TIER == CPU_NEON) has no
+ * SSE3 tier at all - referencing CPU_SSE3 there would either fail to compile
+ * (BUILD_SSE3 not necessarily on for a NEON build) or dereference a slot
+ * nothing ever populated. It therefore falls back to the older, coarser
+ * "no per-pixel exemption; a difference is only caught if it exceeds the
+ * documented 1-LSB bound" rule below, which is exactly the pre-existing
+ * behaviour this file always had before this per-pixel scheme existed. That
+ * is checked purely by build target - only the AVX2 targets ever define
+ * -DSIMD_TIER=CPU_AVX2, so this is a compile-time, not runtime, distinction
+ * (no live NEON hardware or NEON build was used to verify this branch).
+ *
+ * Irrelevant (and unused) when SIMD_REQUIRE_EXACT is defined: that mode
+ * (the AVX2-vs-SSE3 build) rejects any in-span difference outright and never
+ * calls this. */
#ifndef SIMD_REQUIRE_EXACT
static int
-known_c_divergent_slot(const char *table, int m, int c)
+classify_diff(DATA32 vn, DATA32 vc, DATA32 vr, int have_ref, int delta)
{
-#if (SIMD_TIER != CPU_AVX2)
- (void)table; (void)m; (void)c;
- return 1;
+#if (SIMD_TIER == CPU_AVX2)
+ (void)delta;
+ if (!have_ref) return 0; /* no SSE3 slot for this entry: not tolerated */
+ if (vr == vc) return 0; /* SSE3 itself matches C: AVX2 diverging is new */
+ return (vn == vr); /* tolerated iff AVX2 reproduces SSE3 exactly */
#else
- if (strcmp(table, "blend") != 0) return 0;
- if (m != SM_N) return 0;
- return (c == SC) || (c == SC_AN);
+ /* No SSE3 tier to consult on this build (NEON). Fall back to the
+ * pre-existing blanket rule: tolerate anything up to 1 LSB per channel,
+ * exactly as this file always did before the per-pixel scheme above. The
+ * final total.max_delta > 1 check in main() is what actually enforces
+ * the bound; returning "tolerated" here for delta<=1 just keeps this
+ * path's per-slot UNEXPECTED accounting from firing on that pre-existing,
+ * accepted tolerance. */
+ (void)vn; (void)vc; (void)vr; (void)have_ref;
+ return (delta <= 1);
#endif
}
#endif
@@ -391,13 +415,18 @@ report(Category cat, const char *table, int s, int m, int c, int d, int len,
printf(" ... further reports of this kind suppressed (--max-report to raise)\n");
}
-/* Run one span function pair over one configuration. */
+/* Run one span function pair over one configuration. fr is an optional third
+ * (reference-tier, e.g. SSE3) kernel used only to classify a C/SIMD_TIER
+ * difference as inherited-vs-new - see classify_diff() above. May be NULL
+ * (no such slot on this tier, or SIMD_REQUIRE_EXACT builds that never need
+ * it). */
static void
run_span_case(const char *table, RGBA_Gfx_Func fc, RGBA_Gfx_Func fn,
+ RGBA_Gfx_Func fr,
int s, int m, int c, int d, int len, const int *off,
Pattern pat, Stats *st)
{
- static Buf src, msk, src_o, msk_o, dst_c, dst_n, dst_o;
+ static Buf src, msk, src_o, msk_o, dst_c, dst_n, dst_o, dst_r;
static int inited = 0;
int soff = off[0], doff = off[1], moff = off[2];
DATA32 col;
@@ -407,7 +436,7 @@ run_span_case(const char *table, RGBA_Gfx_Func fc, RGBA_Gfx_Func fn,
{
buf_new(&src); buf_new(&msk);
buf_new(&src_o); buf_new(&msk_o);
- buf_new(&dst_c); buf_new(&dst_n); buf_new(&dst_o);
+ buf_new(&dst_c); buf_new(&dst_n); buf_new(&dst_o); buf_new(&dst_r);
inited = 1;
}
@@ -455,6 +484,16 @@ run_span_case(const char *table, RGBA_Gfx_Func fc, RGBA_Gfx_Func fn,
memcpy(msk.raw, msk_o.raw, BUF_BYTES);
}
+ /* Reference-tier run (e.g. SSE3), purely for classify_diff() below. Not
+ * subject to the clobber/oob checks above - those already cover fc/fn. */
+ if (fr)
+ {
+ memcpy(dst_r.raw, dst_o.raw, BUF_BYTES);
+ fr(src.pix + soff, (DATA8 *)msk.pix + moff, col, dst_r.pix + doff, len);
+ memcpy(src.raw, src_o.raw, BUF_BYTES);
+ memcpy(msk.raw, msk_o.raw, BUF_BYTES);
+ }
+
st->cases++;
/* 1. out-of-span writes, either implementation */
@@ -506,6 +545,13 @@ run_span_case(const char *table, RGBA_Gfx_Func fc, RGBA_Gfx_Func fn,
if (delta > st->max_delta) st->max_delta = delta;
st->span_diff++;
+#ifndef SIMD_REQUIRE_EXACT
+ if (!classify_diff(vn, vc, fr ? dst_r.pix[doff + i] : 0, fr != NULL, delta))
+ st->unexpected_diff++;
+#else
+ st->unexpected_diff++;
+#endif
+
if (verbose || delta > 1)
report(CAT_DIFF, table, s, m, c, d, len, off, pat,
delta > 1 ? "mismatch beyond rounding" : "rounding difference",
@@ -529,6 +575,16 @@ walk_span_table(const char *table,
{
RGBA_Gfx_Func fc = t[s][m][c][d][SIMD_REF_TIER];
RGBA_Gfx_Func fn = t[s][m][c][d][SIMD_TIER];
+#if !defined(SIMD_REQUIRE_EXACT) && (SIMD_TIER == CPU_AVX2)
+ /* Only meaningful when SIMD_REF_TIER is CPU_C (the vs-C build):
+ * classify_diff() wants an independent SSE3 opinion to compare
+ * against. When SIMD_REF_TIER is itself CPU_SSE3 (the
+ * SIMD_REQUIRE_EXACT vs-SSE3 build) this branch is compiled out
+ * anyway. */
+ RGBA_Gfx_Func fr = t[s][m][c][d][CPU_SSE3];
+#else
+ RGBA_Gfx_Func fr = NULL;
+#endif
Stats st;
if (!fn) continue;
@@ -546,7 +602,7 @@ walk_span_table(const char *table,
for (li = 0; li < NLENS; li++)
for (oi = 0; oi < NOFFS; oi++)
for (p = 0; p < PAT_LAST; p++)
- run_span_case(table, fc, fn, s, m, c, d,
+ run_span_case(table, fc, fn, fr, s, m, c, d,
lens[li], offs[oi], (Pattern)p, &st);
total->cases += st.cases;
@@ -559,23 +615,15 @@ walk_span_table(const char *table,
total->src_clobber += st.src_clobber;
if (st.max_delta > total->max_delta) total->max_delta = st.max_delta;
- /* Gate: under SIMD_REQUIRE_EXACT no slot may differ; otherwise
- * only the explicitly whitelisted C-divergent slots may. */
-#ifdef SIMD_REQUIRE_EXACT
- if (st.span_diff) total->unexpected_diff += st.span_diff;
-#else
- if (st.span_diff && !known_c_divergent_slot(table, m, c))
- total->unexpected_diff += st.span_diff;
-#endif
+ /* Gate: st.unexpected_diff was accumulated per pixel inside
+ * run_span_case, via classify_diff() (or unconditionally under
+ * SIMD_REQUIRE_EXACT) - see there. */
+ total->unexpected_diff += st.unexpected_diff;
if (st.span_diff || st.oob_simd || st.oob_c || st.src_clobber)
printf(" %-10s[%-5s][%-5s][%-5s][%-5s]%s diff=%llu oob_simd=%llu oob_c=%llu clobber=%llu maxdelta=%d\n",
table, sp_names[s], sm_names[m], sc_names[c], dp_names[d],
-#ifdef SIMD_REQUIRE_EXACT
- st.span_diff ? " UNEXPECTED" : "",
-#else
- (st.span_diff && !known_c_divergent_slot(table, m, c)) ? " UNEXPECTED" : "",
-#endif
+ st.unexpected_diff ? " UNEXPECTED" : "",
st.span_diff, st.oob_simd, st.oob_c, st.src_clobber,
st.max_delta);
/* Worth naming even though the input is out of contract: map and
@@ -609,6 +657,11 @@ walk_pt_table(const char *table,
{
RGBA_Gfx_Pt_Func fc = t[s][m][c][d][SIMD_REF_TIER];
RGBA_Gfx_Pt_Func fn = t[s][m][c][d][SIMD_TIER];
+#if !defined(SIMD_REQUIRE_EXACT) && (SIMD_TIER == CPU_AVX2)
+ RGBA_Gfx_Pt_Func fr = t[s][m][c][d][CPU_SSE3];
+#else
+ RGBA_Gfx_Pt_Func fr = NULL;
+#endif
unsigned long long diff = 0;
int maxd = 0;
@@ -632,12 +685,13 @@ walk_pt_table(const char *table,
| ((DATA32)MIN(corners[i], da) << 16)
| ((DATA32)MIN(corners[j], da) << 8)
| MIN(corners[k], da);
- DATA32 dc, dn;
+ DATA32 dc, dn, dr = 0;
int delta;
dc = dn = dorig;
fc(sv, mv, col, &dc);
fn(sv, mv, col, &dn);
+ if (fr) { dr = dorig; fr(sv, mv, col, &dr); }
if (dc == dn) continue;
delta = chan_delta(dn, dc);
@@ -648,7 +702,7 @@ walk_pt_table(const char *table,
#ifdef SIMD_REQUIRE_EXACT
total->unexpected_diff++;
#else
- if (!known_c_divergent_slot(table, m, c))
+ if (!classify_diff(dn, dc, dr, fr != NULL, delta))
total->unexpected_diff++;
#endif
if (verbose || delta > 1)
--
To stop receiving notification emails like this one, please contact
the administrator of this repository.
