Richard Biener <rguent...@suse.de> writes:
> On Thu, 19 Oct 2023, Robin Dapp wrote:
>
>> Ugh, I didn't push yet because with a rebased trunk I am
>> seeing different behavior for some riscv testcases.
>>
>> A reduction is not recognized because there is yet another
>> "double use" occurrence in check_reduction_path. I guess it's
>> reasonable to loosen the restriction for conditional operations
>> here as well.
>>
>> The only change to v4 therefore is:
>>
>> diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
>> index ebab1953b9c..64654a55e4c 100644
>> --- a/gcc/tree-vect-loop.cc
>> +++ b/gcc/tree-vect-loop.cc
>> @@ -4085,7 +4094,15 @@ pop:
>> || flow_bb_inside_loop_p (loop, gimple_bb (op_use_stmt))))
>> FOR_EACH_IMM_USE_ON_STMT (use_p, imm_iter)
>> cnt++;
>> - if (cnt != 1)
>> +
>> + bool cond_fn_p = op.code.is_internal_fn ()
>> + && (conditional_internal_fn_code (internal_fn (*code))
>> + != ERROR_MARK);
>> +
>> + /* In case of a COND_OP (mask, op1, op2, op1) reduction we might have
>> + op1 twice (once as definition, once as else) in the same operation.
>> + Allow this. */
>> + if ((!cond_fn_p && cnt != 1) || (opi == 1 && cond_fn_p && cnt != 2))
>>
>> Bootstrapped and regtested again on x86, aarch64 and power10.
>> Testsuite on riscv unchanged.
>
> Hmm, why opi == 1 only? I think
>
> # _1 = PHI <.., _4>
> _3 = .COND_ADD (_1, _2, _1);
> _4 = .COND_ADD (_3, _5, _3);
>
> would be fine as well. I think we want to simply ignore the 'else' value
> of conditional internal functions. I suppose we have unary, binary
> and ternary conditional functions - I miss a internal_fn_else_index,
> but I suppose it's always the last one?
Yeah, it was always the last one before the introduction of .COND_LEN.
I agree internal_fn_else_index would be useful now.
Thanks,
Richard
>
> I think a single use on .COND functions is also OK, even when on the
> 'else' value only? But maybe that's not too important here.
>
> Maybe
>
> gimple *op_use_stmt;
> unsigned cnt = 0;
> FOR_EACH_IMM_USE_STMT (op_use_stmt, imm_iter, op.ops[opi])
> if (.. op_use_stmt is conditional internal function ..)
> {
> for (unsigned j = 0; j < gimple_call_num_args (call) - 1; ++j)
> if (gimple_call_arg (call, j) == op.ops[opi])
> cnt++;
> }
> else if (!is_gimple_debug (op_use_stmt)
> && (*code != ERROR_MARK
> || flow_bb_inside_loop_p (loop, gimple_bb (op_use_stmt))))
> FOR_EACH_IMM_USE_ON_STMT (use_p, imm_iter)
> cnt++;
>
> ?
>
>> Regards
>> Robin
>>
>> Subject: [PATCH v5] ifcvt/vect: Emit COND_OP for conditional scalar
>> reduction.
>>
>> As described in PR111401 we currently emit a COND and a PLUS expression
>> for conditional reductions. This makes it difficult to combine both
>> into a masked reduction statement later.
>> This patch improves that by directly emitting a COND_ADD/COND_OP during
>> ifcvt and adjusting some vectorizer code to handle it.
>>
>> It also makes neutral_op_for_reduction return -0 if HONOR_SIGNED_ZEROS
>> is true.
>>
>> gcc/ChangeLog:
>>
>> PR middle-end/111401
>> * tree-if-conv.cc (convert_scalar_cond_reduction): Emit COND_OP
>> if supported.
>> (predicate_scalar_phi): Add whitespace.
>> * tree-vect-loop.cc (fold_left_reduction_fn): Add IFN_COND_OP.
>> (neutral_op_for_reduction): Return -0 for PLUS.
>> (check_reduction_path): Don't count else operand in COND_OP.
>> (vect_is_simple_reduction): Ditto.
>> (vect_create_epilog_for_reduction): Fix whitespace.
>> (vectorize_fold_left_reduction): Add COND_OP handling.
>> (vectorizable_reduction): Don't count else operand in COND_OP.
>> (vect_transform_reduction): Add COND_OP handling.
>> * tree-vectorizer.h (neutral_op_for_reduction): Add default
>> parameter.
>>
>> gcc/testsuite/ChangeLog:
>>
>> * gcc.dg/vect/vect-cond-reduc-in-order-2-signed-zero.c: New test.
>> * gcc.target/riscv/rvv/autovec/cond/pr111401.c: New test.
>> * gcc.target/riscv/rvv/autovec/reduc/reduc_call-2.c: Adjust.
>> * gcc.target/riscv/rvv/autovec/reduc/reduc_call-4.c: Ditto.
>> ---
>> .../vect-cond-reduc-in-order-2-signed-zero.c | 141 +++++++++++++++
>> .../riscv/rvv/autovec/cond/pr111401.c | 139 +++++++++++++++
>> .../riscv/rvv/autovec/reduc/reduc_call-2.c | 4 +-
>> .../riscv/rvv/autovec/reduc/reduc_call-4.c | 4 +-
>> gcc/tree-if-conv.cc | 49 +++--
>> gcc/tree-vect-loop.cc | 168 ++++++++++++++----
>> gcc/tree-vectorizer.h | 2 +-
>> 7 files changed, 456 insertions(+), 51 deletions(-)
>> create mode 100644
>> gcc/testsuite/gcc.dg/vect/vect-cond-reduc-in-order-2-signed-zero.c
>> create mode 100644
>> gcc/testsuite/gcc.target/riscv/rvv/autovec/cond/pr111401.c
>>
>> diff --git
>> a/gcc/testsuite/gcc.dg/vect/vect-cond-reduc-in-order-2-signed-zero.c
>> b/gcc/testsuite/gcc.dg/vect/vect-cond-reduc-in-order-2-signed-zero.c
>> new file mode 100644
>> index 00000000000..7b46e7d8a2a
>> --- /dev/null
>> +++ b/gcc/testsuite/gcc.dg/vect/vect-cond-reduc-in-order-2-signed-zero.c
>> @@ -0,0 +1,141 @@
>> +/* Make sure a -0 stays -0 when we perform a conditional reduction. */
>> +/* { dg-do run } */
>> +/* { dg-require-effective-target vect_double } */
>> +/* { dg-add-options ieee } */
>> +/* { dg-additional-options "-std=gnu99 -fno-fast-math" } */
>> +
>> +#include "tree-vect.h"
>> +
>> +#include <math.h>
>> +
>> +#define N (VECTOR_BITS * 17)
>> +
>> +double __attribute__ ((noinline, noclone))
>> +reduc_plus_double (double *restrict a, double init, int *cond, int n)
>> +{
>> + double res = init;
>> + for (int i = 0; i < n; i++)
>> + if (cond[i])
>> + res += a[i];
>> + return res;
>> +}
>> +
>> +double __attribute__ ((noinline, noclone, optimize ("0")))
>> +reduc_plus_double_ref (double *restrict a, double init, int *cond, int n)
>> +{
>> + double res = init;
>> + for (int i = 0; i < n; i++)
>> + if (cond[i])
>> + res += a[i];
>> + return res;
>> +}
>> +
>> +double __attribute__ ((noinline, noclone))
>> +reduc_minus_double (double *restrict a, double init, int *cond, int n)
>> +{
>> + double res = init;
>> + for (int i = 0; i < n; i++)
>> + if (cond[i])
>> + res -= a[i];
>> + return res;
>> +}
>> +
>> +double __attribute__ ((noinline, noclone, optimize ("0")))
>> +reduc_minus_double_ref (double *restrict a, double init, int *cond, int n)
>> +{
>> + double res = init;
>> + for (int i = 0; i < n; i++)
>> + if (cond[i])
>> + res -= a[i];
>> + return res;
>> +}
>> +
>> +int __attribute__ ((optimize (1)))
>> +main ()
>> +{
>> + int n = 19;
>> + double a[N];
>> + int cond1[N], cond2[N];
>> +
>> + for (int i = 0; i < N; i++)
>> + {
>> + a[i] = (i * 0.1) * (i & 1 ? 1 : -1);
>> + cond1[i] = 0;
>> + cond2[i] = i & 4 ? 1 : 0;
>> + asm volatile ("" ::: "memory");
>> + }
>> +
>> + double res1 = reduc_plus_double (a, -0.0, cond1, n);
>> + double ref1 = reduc_plus_double_ref (a, -0.0, cond1, n);
>> + double res2 = reduc_minus_double (a, -0.0, cond1, n);
>> + double ref2 = reduc_minus_double_ref (a, -0.0, cond1, n);
>> + double res3 = reduc_plus_double (a, -0.0, cond1, n);
>> + double ref3 = reduc_plus_double_ref (a, -0.0, cond1, n);
>> + double res4 = reduc_minus_double (a, -0.0, cond1, n);
>> + double ref4 = reduc_minus_double_ref (a, -0.0, cond1, n);
>> +
>> + if (res1 != ref1 || signbit (res1) != signbit (ref1))
>> + __builtin_abort ();
>> + if (res2 != ref2 || signbit (res2) != signbit (ref2))
>> + __builtin_abort ();
>> + if (res3 != ref3 || signbit (res3) != signbit (ref3))
>> + __builtin_abort ();
>> + if (res4 != ref4 || signbit (res4) != signbit (ref4))
>> + __builtin_abort ();
>> +
>> + res1 = reduc_plus_double (a, 0.0, cond1, n);
>> + ref1 = reduc_plus_double_ref (a, 0.0, cond1, n);
>> + res2 = reduc_minus_double (a, 0.0, cond1, n);
>> + ref2 = reduc_minus_double_ref (a, 0.0, cond1, n);
>> + res3 = reduc_plus_double (a, 0.0, cond1, n);
>> + ref3 = reduc_plus_double_ref (a, 0.0, cond1, n);
>> + res4 = reduc_minus_double (a, 0.0, cond1, n);
>> + ref4 = reduc_minus_double_ref (a, 0.0, cond1, n);
>> +
>> + if (res1 != ref1 || signbit (res1) != signbit (ref1))
>> + __builtin_abort ();
>> + if (res2 != ref2 || signbit (res2) != signbit (ref2))
>> + __builtin_abort ();
>> + if (res3 != ref3 || signbit (res3) != signbit (ref3))
>> + __builtin_abort ();
>> + if (res4 != ref4 || signbit (res4) != signbit (ref4))
>> + __builtin_abort ();
>> +
>> + res1 = reduc_plus_double (a, -0.0, cond2, n);
>> + ref1 = reduc_plus_double_ref (a, -0.0, cond2, n);
>> + res2 = reduc_minus_double (a, -0.0, cond2, n);
>> + ref2 = reduc_minus_double_ref (a, -0.0, cond2, n);
>> + res3 = reduc_plus_double (a, -0.0, cond2, n);
>> + ref3 = reduc_plus_double_ref (a, -0.0, cond2, n);
>> + res4 = reduc_minus_double (a, -0.0, cond2, n);
>> + ref4 = reduc_minus_double_ref (a, -0.0, cond2, n);
>> +
>> + if (res1 != ref1 || signbit (res1) != signbit (ref1))
>> + __builtin_abort ();
>> + if (res2 != ref2 || signbit (res2) != signbit (ref2))
>> + __builtin_abort ();
>> + if (res3 != ref3 || signbit (res3) != signbit (ref3))
>> + __builtin_abort ();
>> + if (res4 != ref4 || signbit (res4) != signbit (ref4))
>> + __builtin_abort ();
>> +
>> + res1 = reduc_plus_double (a, 0.0, cond2, n);
>> + ref1 = reduc_plus_double_ref (a, 0.0, cond2, n);
>> + res2 = reduc_minus_double (a, 0.0, cond2, n);
>> + ref2 = reduc_minus_double_ref (a, 0.0, cond2, n);
>> + res3 = reduc_plus_double (a, 0.0, cond2, n);
>> + ref3 = reduc_plus_double_ref (a, 0.0, cond2, n);
>> + res4 = reduc_minus_double (a, 0.0, cond2, n);
>> + ref4 = reduc_minus_double_ref (a, 0.0, cond2, n);
>> +
>> + if (res1 != ref1 || signbit (res1) != signbit (ref1))
>> + __builtin_abort ();
>> + if (res2 != ref2 || signbit (res2) != signbit (ref2))
>> + __builtin_abort ();
>> + if (res3 != ref3 || signbit (res3) != signbit (ref3))
>> + __builtin_abort ();
>> + if (res4 != ref4 || signbit (res4) != signbit (ref4))
>> + __builtin_abort ();
>> +
>> + return 0;
>> +}
>> diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/cond/pr111401.c
>> b/gcc/testsuite/gcc.target/riscv/rvv/autovec/cond/pr111401.c
>> new file mode 100644
>> index 00000000000..83dbd61b3f3
>> --- /dev/null
>> +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/cond/pr111401.c
>> @@ -0,0 +1,139 @@
>> +/* { dg-do run { target { riscv_v } } } */
>> +/* { dg-additional-options "-march=rv64gcv -mabi=lp64d --param
>> riscv-autovec-preference=scalable -fdump-tree-vect-details" } */
>> +
>> +double
>> +__attribute__ ((noipa))
>> +foo2 (double *__restrict a, double init, int *__restrict cond, int n)
>> +{
>> + for (int i = 0; i < n; i++)
>> + if (cond[i])
>> + init += a[i];
>> + return init;
>> +}
>> +
>> +double
>> +__attribute__ ((noipa))
>> +foo3 (double *__restrict a, double init, int *__restrict cond, int n)
>> +{
>> + for (int i = 0; i < n; i++)
>> + if (cond[i])
>> + init -= a[i];
>> + return init;
>> +}
>> +
>> +double
>> +__attribute__ ((noipa))
>> +foo4 (double *__restrict a, double init, int *__restrict cond, int n)
>> +{
>> + for (int i = 0; i < n; i++)
>> + if (cond[i])
>> + init *= a[i];
>> + return init;
>> +}
>> +
>> +int
>> +__attribute__ ((noipa))
>> +foo5 (int *__restrict a, int init, int *__restrict cond, int n)
>> +{
>> + for (int i = 0; i < n; i++)
>> + if (cond[i])
>> + init &= a[i];
>> + return init;
>> +}
>> +
>> +int
>> +__attribute__ ((noipa))
>> +foo6 (int *__restrict a, int init, int *__restrict cond, int n)
>> +{
>> + for (int i = 0; i < n; i++)
>> + if (cond[i])
>> + init |= a[i];
>> + return init;
>> +}
>> +
>> +int
>> +__attribute__ ((noipa))
>> +foo7 (int *__restrict a, int init, int *__restrict cond, int n)
>> +{
>> + for (int i = 0; i < n; i++)
>> + if (cond[i])
>> + init ^= a[i];
>> + return init;
>> +}
>> +
>> +#define SZ 125
>> +
>> +int
>> +main ()
>> +{
>> + double res1 = 0, res2 = 0, res3 = 0;
>> + double a1[SZ], a2[SZ], a3[SZ];
>> + int c1[SZ], c2[SZ], c3[SZ];
>> +
>> + int a4[SZ], a5[SZ], a6[SZ];
>> + int res4 = 0, res5 = 0, res6 = 0;
>> + int c4[SZ], c5[SZ], c6[SZ];
>> +
>> + for (int i = 0; i < SZ; i++)
>> + {
>> + a1[i] = i * 3 + (i & 4) - (i & 7);
>> + a2[i] = i * 3 + (i & 4) - (i & 7);
>> + a3[i] = i * 0.05 + (i & 4) - (i & 7);
>> + a4[i] = i * 3 + (i & 4) - (i & 7);
>> + a5[i] = i * 3 + (i & 4) - (i & 7);
>> + a6[i] = i * 3 + (i & 4) - (i & 7);
>> + c1[i] = i & 1;
>> + c2[i] = i & 2;
>> + c3[i] = i & 3;
>> + c4[i] = i & 4;
>> + c5[i] = i & 5;
>> + c6[i] = i & 6;
>> + __asm__ volatile ("" : : : "memory");
>> + }
>> +
>> + double init1 = 2.7, init2 = 8.2, init3 = 0.1;
>> + double ref1 = init1, ref2 = init2, ref3 = init3;
>> +
>> + int init4 = 87, init5 = 11, init6 = -123894344;
>> + int ref4 = init4, ref5 = init5, ref6 = init6;
>> +
>> +#pragma GCC novector
>> + for (int i = 0; i < SZ; i++)
>> + {
>> + if (c1[i])
>> + ref1 += a1[i];
>> + if (c2[i])
>> + ref2 -= a2[i];
>> + if (c3[i])
>> + ref3 *= a3[i];
>> + if (c4[i])
>> + ref4 &= a4[i];
>> + if (c5[i])
>> + ref5 |= a5[i];
>> + if (c6[i])
>> + ref6 ^= a6[i];
>> + }
>> +
>> + res1 = foo2 (a1, init1, c1, SZ);
>> + res2 = foo3 (a2, init2, c2, SZ);
>> + res3 = foo4 (a3, init3, c3, SZ);
>> + res4 = foo5 (a4, init4, c4, SZ);
>> + res5 = foo6 (a5, init5, c5, SZ);
>> + res6 = foo7 (a6, init6, c6, SZ);
>> +
>> + if (res1 != ref1)
>> + __builtin_abort ();
>> + if (res2 != ref2)
>> + __builtin_abort ();
>> + if (res3 != ref3)
>> + __builtin_abort ();
>> + if (res4 != ref4)
>> + __builtin_abort ();
>> + if (res5 != ref5)
>> + __builtin_abort ();
>> + if (res6 != ref6)
>> + __builtin_abort ();
>> +}
>> +
>> +/* { dg-final { scan-tree-dump-times "vectorized 1 loop" 5 "vect" } } */
>> +/* { dg-final { scan-tree-dump-not "VCOND_MASK" "vect" } } */
>> diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/reduc/reduc_call-2.c
>> b/gcc/testsuite/gcc.target/riscv/rvv/autovec/reduc/reduc_call-2.c
>> index cc07a047cd5..7be22d60bf2 100644
>> --- a/gcc/testsuite/gcc.target/riscv/rvv/autovec/reduc/reduc_call-2.c
>> +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/reduc/reduc_call-2.c
>> @@ -3,4 +3,6 @@
>>
>> #include "reduc_call-1.c"
>>
>> -/* { dg-final { scan-assembler-times
>> {vfmacc\.vv\s+v[0-9]+,v[0-9]+,v[0-9]+,v0.t} 1 } } */
>> +/* { dg-final { scan-assembler-times {vfmadd\.vv\s+v[0-9]+,v[0-9]+,v[0-9]+}
>> 1 } } */
>> +/* { dg-final { scan-assembler-times
>> {vfadd\.vv\s+v[0-9]+,v[0-9]+,v[0-9]+,v0.t} 1 } } */
>> +/* { dg-final { scan-assembler-not {vmerge} } } */
>> diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/reduc/reduc_call-4.c
>> b/gcc/testsuite/gcc.target/riscv/rvv/autovec/reduc/reduc_call-4.c
>> index 6d00c404d2a..83beabeff97 100644
>> --- a/gcc/testsuite/gcc.target/riscv/rvv/autovec/reduc/reduc_call-4.c
>> +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/reduc/reduc_call-4.c
>> @@ -3,4 +3,6 @@
>>
>> #include "reduc_call-1.c"
>>
>> -/* { dg-final { scan-assembler {vfmacc\.vv\s+v[0-9]+,v[0-9]+,v[0-9]+,v0.t}
>> } } */
>> +/* { dg-final { scan-assembler {vfmadd\.vv\s+v[0-9]+,v[0-9]+,v[0-9]+} } } */
>> +/* { dg-final { scan-assembler {vfadd\.vv\s+v[0-9]+,v[0-9]+,v[0-9]+,v0.t} }
>> } */
>> +/* { dg-final { scan-assembler-not {vmerge} } } */
>> diff --git a/gcc/tree-if-conv.cc b/gcc/tree-if-conv.cc
>> index c381d14b801..9571351805c 100644
>> --- a/gcc/tree-if-conv.cc
>> +++ b/gcc/tree-if-conv.cc
>> @@ -1856,10 +1856,12 @@ convert_scalar_cond_reduction (gimple *reduc,
>> gimple_stmt_iterator *gsi,
>> gimple *new_assign;
>> tree rhs;
>> tree rhs1 = gimple_assign_rhs1 (reduc);
>> + tree lhs = gimple_assign_lhs (reduc);
>> tree tmp = make_temp_ssa_name (TREE_TYPE (rhs1), NULL, "_ifc_");
>> tree c;
>> enum tree_code reduction_op = gimple_assign_rhs_code (reduc);
>> - tree op_nochange = neutral_op_for_reduction (TREE_TYPE (rhs1),
>> reduction_op, NULL);
>> + tree op_nochange = neutral_op_for_reduction (TREE_TYPE (rhs1),
>> reduction_op,
>> + NULL, false);
>> gimple_seq stmts = NULL;
>>
>> if (dump_file && (dump_flags & TDF_DETAILS))
>> @@ -1868,19 +1870,36 @@ convert_scalar_cond_reduction (gimple *reduc,
>> gimple_stmt_iterator *gsi,
>> print_gimple_stmt (dump_file, reduc, 0, TDF_SLIM);
>> }
>>
>> - /* Build cond expression using COND and constant operand
>> - of reduction rhs. */
>> - c = fold_build_cond_expr (TREE_TYPE (rhs1),
>> - unshare_expr (cond),
>> - swap ? op_nochange : op1,
>> - swap ? op1 : op_nochange);
>> -
>> - /* Create assignment stmt and insert it at GSI. */
>> - new_assign = gimple_build_assign (tmp, c);
>> - gsi_insert_before (gsi, new_assign, GSI_SAME_STMT);
>> - /* Build rhs for unconditional increment/decrement/logic_operation. */
>> - rhs = gimple_build (&stmts, reduction_op,
>> - TREE_TYPE (rhs1), op0, tmp);
>> + /* If possible create a COND_OP instead of a COND_EXPR and an OP_EXPR.
>> + The COND_OP will have a neutral_op else value. */
>> + internal_fn ifn;
>> + ifn = get_conditional_internal_fn (reduction_op);
>> + if (ifn != IFN_LAST
>> + && vectorized_internal_fn_supported_p (ifn, TREE_TYPE (lhs))
>> + && !swap)
>> + {
>> + gcall *cond_call = gimple_build_call_internal (ifn, 4,
>> + unshare_expr (cond),
>> + op0, op1, op0);
>> + gsi_insert_before (gsi, cond_call, GSI_SAME_STMT);
>> + gimple_call_set_lhs (cond_call, tmp);
>> + rhs = tmp;
>> + }
>> + else
>> + {
>> + /* Build cond expression using COND and constant operand
>> + of reduction rhs. */
>> + c = fold_build_cond_expr (TREE_TYPE (rhs1),
>> + unshare_expr (cond),
>> + swap ? op_nochange : op1,
>> + swap ? op1 : op_nochange);
>> + /* Create assignment stmt and insert it at GSI. */
>> + new_assign = gimple_build_assign (tmp, c);
>> + gsi_insert_before (gsi, new_assign, GSI_SAME_STMT);
>> + /* Build rhs for unconditional increment/decrement/logic_operation.
>> */
>> + rhs = gimple_build (&stmts, reduction_op,
>> + TREE_TYPE (rhs1), op0, tmp);
>> + }
>>
>> if (has_nop)
>> {
>> @@ -2292,7 +2311,7 @@ predicate_scalar_phi (gphi *phi, gimple_stmt_iterator
>> *gsi)
>> {
>> /* Convert reduction stmt into vectorizable form. */
>> rhs = convert_scalar_cond_reduction (reduc, gsi, cond, op0, op1,
>> - swap,has_nop, nop_reduc);
>> + swap, has_nop, nop_reduc);
>> redundant_ssa_names.safe_push (std::make_pair (res, rhs));
>> }
>> new_stmt = gimple_build_assign (res, rhs);
>> diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
>> index ebab1953b9c..1c455701c73 100644
>> --- a/gcc/tree-vect-loop.cc
>> +++ b/gcc/tree-vect-loop.cc
>> @@ -3762,7 +3762,10 @@ vect_analyze_loop (class loop *loop, vec_info_shared
>> *shared)
>> static bool
>> fold_left_reduction_fn (code_helper code, internal_fn *reduc_fn)
>> {
>> - if (code == PLUS_EXPR)
>> + /* We support MINUS_EXPR by negating the operand. This also preserves an
>> + initial -0.0 since -0.0 - 0.0 (neutral op for MINUS_EXPR) == -0.0 +
>> + (-0.0) = -0.0. */
>> + if (code == PLUS_EXPR || code == MINUS_EXPR)
>> {
>> *reduc_fn = IFN_FOLD_LEFT_PLUS;
>> return true;
>> @@ -3841,23 +3844,29 @@ reduction_fn_for_scalar_code (code_helper code,
>> internal_fn *reduc_fn)
>> by the introduction of additional X elements, return that X, otherwise
>> return null. CODE is the code of the reduction and SCALAR_TYPE is type
>> of the scalar elements. If the reduction has just a single initial value
>> - then INITIAL_VALUE is that value, otherwise it is null. */
>> + then INITIAL_VALUE is that value, otherwise it is null.
>> + If AS_INITIAL is TRUE the value is supposed to be used as initial value.
>> + In that case no signed zero is returned. */
>>
>> tree
>> neutral_op_for_reduction (tree scalar_type, code_helper code,
>> - tree initial_value)
>> + tree initial_value, bool as_initial)
>> {
>> if (code.is_tree_code ())
>> switch (tree_code (code))
>> {
>> - case WIDEN_SUM_EXPR:
>> case DOT_PROD_EXPR:
>> case SAD_EXPR:
>> - case PLUS_EXPR:
>> case MINUS_EXPR:
>> case BIT_IOR_EXPR:
>> case BIT_XOR_EXPR:
>> return build_zero_cst (scalar_type);
>> + case WIDEN_SUM_EXPR:
>> + case PLUS_EXPR:
>> + if (!as_initial && HONOR_SIGNED_ZEROS (scalar_type))
>> + return build_real (scalar_type, dconstm0);
>> + else
>> + return build_zero_cst (scalar_type);
>>
>> case MULT_EXPR:
>> return build_one_cst (scalar_type);
>> @@ -4085,7 +4094,15 @@ pop:
>> || flow_bb_inside_loop_p (loop, gimple_bb (op_use_stmt))))
>> FOR_EACH_IMM_USE_ON_STMT (use_p, imm_iter)
>> cnt++;
>> - if (cnt != 1)
>> +
>> + bool cond_fn_p = op.code.is_internal_fn ()
>> + && (conditional_internal_fn_code (internal_fn (*code))
>> + != ERROR_MARK);
>> +
>> + /* In case of a COND_OP (mask, op1, op2, op1) reduction we might have
>> + op1 twice (once as definition, once as else) in the same operation.
>> + Allow this. */
>> + if ((!cond_fn_p && cnt != 1) || (opi == 1 && cond_fn_p && cnt != 2))
>> {
>> fail = true;
>> break;
>> @@ -4187,8 +4204,14 @@ vect_is_simple_reduction (loop_vec_info loop_info,
>> stmt_vec_info phi_info,
>> return NULL;
>> }
>>
>> - nphi_def_loop_uses++;
>> - phi_use_stmt = use_stmt;
>> + /* In case of a COND_OP (mask, op1, op2, op1) reduction we might have
>> + op1 twice (once as definition, once as else) in the same operation.
>> + Only count it as one. */
>> + if (use_stmt != phi_use_stmt)
>> + {
>> + nphi_def_loop_uses++;
>> + phi_use_stmt = use_stmt;
>> + }
>> }
>>
>> tree latch_def = PHI_ARG_DEF_FROM_EDGE (phi, loop_latch_edge (loop));
>> @@ -6122,7 +6145,7 @@ vect_create_epilog_for_reduction (loop_vec_info
>> loop_vinfo,
>> gcc_assert (STMT_VINFO_IN_PATTERN_P (orig_stmt_info));
>> gcc_assert (STMT_VINFO_RELATED_STMT (orig_stmt_info) == stmt_info);
>> }
>> -
>> +
>> scalar_dest = gimple_get_lhs (orig_stmt_info->stmt);
>> scalar_type = TREE_TYPE (scalar_dest);
>> scalar_results.truncate (0);
>> @@ -6459,7 +6482,7 @@ vect_create_epilog_for_reduction (loop_vec_info
>> loop_vinfo,
>> if (REDUC_GROUP_FIRST_ELEMENT (stmt_info))
>> initial_value = reduc_info->reduc_initial_values[0];
>> neutral_op = neutral_op_for_reduction (TREE_TYPE (vectype), code,
>> - initial_value);
>> + initial_value, false);
>> }
>> if (neutral_op)
>> vector_identity = gimple_build_vector_from_val (&seq, vectype,
>> @@ -6941,8 +6964,8 @@ vectorize_fold_left_reduction (loop_vec_info
>> loop_vinfo,
>> gimple_stmt_iterator *gsi,
>> gimple **vec_stmt, slp_tree slp_node,
>> gimple *reduc_def_stmt,
>> - tree_code code, internal_fn reduc_fn,
>> - tree ops[3], tree vectype_in,
>> + code_helper code, internal_fn reduc_fn,
>> + tree *ops, int num_ops, tree vectype_in,
>> int reduc_index, vec_loop_masks *masks,
>> vec_loop_lens *lens)
>> {
>> @@ -6958,17 +6981,48 @@ vectorize_fold_left_reduction (loop_vec_info
>> loop_vinfo,
>>
>> gcc_assert (!nested_in_vect_loop_p (loop, stmt_info));
>> gcc_assert (ncopies == 1);
>> - gcc_assert (TREE_CODE_LENGTH (code) == binary_op);
>> +
>> + bool is_cond_op = false;
>> + if (!code.is_tree_code ())
>> + {
>> + code = conditional_internal_fn_code (internal_fn (code));
>> + gcc_assert (code != ERROR_MARK);
>> + is_cond_op = true;
>> + }
>> +
>> + gcc_assert (TREE_CODE_LENGTH (tree_code (code)) == binary_op);
>>
>> if (slp_node)
>> - gcc_assert (known_eq (TYPE_VECTOR_SUBPARTS (vectype_out),
>> - TYPE_VECTOR_SUBPARTS (vectype_in)));
>> + {
>> + if (is_cond_op)
>> + {
>> + if (dump_enabled_p ())
>> + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
>> + "fold-left reduction on SLP not supported.\n");
>> + return false;
>> + }
>>
>> - tree op0 = ops[1 - reduc_index];
>> + gcc_assert (known_eq (TYPE_VECTOR_SUBPARTS (vectype_out),
>> + TYPE_VECTOR_SUBPARTS (vectype_in)));
>> + }
>> +
>> + /* The operands either come from a binary operation or an IFN_COND
>> operation.
>> + The former is a gimple assign with binary rhs and the latter is a
>> + gimple call with four arguments. */
>> + gcc_assert (num_ops == 2 || num_ops == 4);
>> + tree op0, opmask;
>> + if (!is_cond_op)
>> + op0 = ops[1 - reduc_index];
>> + else
>> + {
>> + op0 = ops[2];
>> + opmask = ops[0];
>> + gcc_assert (!slp_node);
>> + }
>>
>> int group_size = 1;
>> stmt_vec_info scalar_dest_def_info;
>> - auto_vec<tree> vec_oprnds0;
>> + auto_vec<tree> vec_oprnds0, vec_opmask;
>> if (slp_node)
>> {
>> auto_vec<vec<tree> > vec_defs (2);
>> @@ -6984,9 +7038,15 @@ vectorize_fold_left_reduction (loop_vec_info
>> loop_vinfo,
>> vect_get_vec_defs_for_operand (loop_vinfo, stmt_info, 1,
>> op0, &vec_oprnds0);
>> scalar_dest_def_info = stmt_info;
>> +
>> + /* For an IFN_COND_OP we also need the vector mask operand. */
>> + if (is_cond_op)
>> + vect_get_vec_defs_for_operand (loop_vinfo, stmt_info, 1,
>> + opmask, &vec_opmask);
>> }
>>
>> - tree scalar_dest = gimple_assign_lhs (scalar_dest_def_info->stmt);
>> + gimple *sdef = scalar_dest_def_info->stmt;
>> + tree scalar_dest = gimple_get_lhs (sdef);
>> tree scalar_type = TREE_TYPE (scalar_dest);
>> tree reduc_var = gimple_phi_result (reduc_def_stmt);
>>
>> @@ -7020,13 +7080,16 @@ vectorize_fold_left_reduction (loop_vec_info
>> loop_vinfo,
>> tree bias = NULL_TREE;
>> if (LOOP_VINFO_FULLY_MASKED_P (loop_vinfo))
>> mask = vect_get_loop_mask (loop_vinfo, gsi, masks, vec_num, vectype_in,
>> i);
>> + else if (is_cond_op)
>> + mask = vec_opmask[0];
>> if (LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo))
>> {
>> len = vect_get_loop_len (loop_vinfo, gsi, lens, vec_num, vectype_in,
>> i, 1);
>> signed char biasval = LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo);
>> bias = build_int_cst (intQI_type_node, biasval);
>> - mask = build_minus_one_cst (truth_type_for (vectype_in));
>> + if (!is_cond_op)
>> + mask = build_minus_one_cst (truth_type_for (vectype_in));
>> }
>>
>> /* Handle MINUS by adding the negative. */
>> @@ -7038,7 +7101,8 @@ vectorize_fold_left_reduction (loop_vec_info
>> loop_vinfo,
>> def0 = negated;
>> }
>>
>> - if (mask && mask_reduc_fn == IFN_LAST)
>> + if (LOOP_VINFO_FULLY_MASKED_P (loop_vinfo)
>> + && mask && mask_reduc_fn == IFN_LAST)
>> def0 = merge_with_identity (gsi, mask, vectype_out, def0,
>> vector_identity);
>>
>> @@ -7069,8 +7133,8 @@ vectorize_fold_left_reduction (loop_vec_info
>> loop_vinfo,
>> }
>> else
>> {
>> - reduc_var = vect_expand_fold_left (gsi, scalar_dest_var, code,
>> - reduc_var, def0);
>> + reduc_var = vect_expand_fold_left (gsi, scalar_dest_var,
>> + tree_code (code), reduc_var, def0);
>> new_stmt = SSA_NAME_DEF_STMT (reduc_var);
>> /* Remove the statement, so that we can use the same code paths
>> as for statements that we've just created. */
>> @@ -7521,8 +7585,13 @@ vectorizable_reduction (loop_vec_info loop_vinfo,
>> if (i == STMT_VINFO_REDUC_IDX (stmt_info))
>> continue;
>>
>> + /* For an IFN_COND_OP we might hit the reduction definition operand
>> + twice (once as definition, once as else). */
>> + if (op.ops[i] == op.ops[STMT_VINFO_REDUC_IDX (stmt_info)])
>> + continue;
>> +
>> /* There should be only one cycle def in the stmt, the one
>> - leading to reduc_def. */
>> + leading to reduc_def. */
>> if (VECTORIZABLE_CYCLE_DEF (dt))
>> return false;
>>
>> @@ -7721,6 +7790,15 @@ vectorizable_reduction (loop_vec_info loop_vinfo,
>> when generating the code inside the loop. */
>>
>> code_helper orig_code = STMT_VINFO_REDUC_CODE (phi_info);
>> +
>> + /* If conversion might have created a conditional operation like
>> + IFN_COND_ADD already. Use the internal code for the following checks.
>> */
>> + if (orig_code.is_internal_fn ())
>> + {
>> + tree_code new_code = conditional_internal_fn_code (internal_fn
>> (orig_code));
>> + orig_code = new_code != ERROR_MARK ? new_code : orig_code;
>> + }
>> +
>> STMT_VINFO_REDUC_CODE (reduc_info) = orig_code;
>>
>> vect_reduction_type reduction_type = STMT_VINFO_REDUC_TYPE (reduc_info);
>> @@ -7759,7 +7837,7 @@ vectorizable_reduction (loop_vec_info loop_vinfo,
>> {
>> if (dump_enabled_p ())
>> dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
>> - "reduction: not commutative/associative");
>> + "reduction: not commutative/associative\n");
>> return false;
>> }
>> }
>> @@ -8143,9 +8221,8 @@ vectorizable_reduction (loop_vec_info loop_vinfo,
>> LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) = false;
>> }
>> else if (reduction_type == FOLD_LEFT_REDUCTION
>> - && reduc_fn == IFN_LAST
>> + && internal_fn_mask_index (reduc_fn) == -1
>> && FLOAT_TYPE_P (vectype_in)
>> - && HONOR_SIGNED_ZEROS (vectype_in)
>> && HONOR_SIGN_DEPENDENT_ROUNDING (vectype_in))
>> {
>> if (dump_enabled_p ())
>> @@ -8294,6 +8371,7 @@ vect_transform_reduction (loop_vec_info loop_vinfo,
>>
>> code_helper code = canonicalize_code (op.code, op.type);
>> internal_fn cond_fn = get_conditional_internal_fn (code, op.type);
>> +
>> vec_loop_masks *masks = &LOOP_VINFO_MASKS (loop_vinfo);
>> vec_loop_lens *lens = &LOOP_VINFO_LENS (loop_vinfo);
>> bool mask_by_cond_expr = use_mask_by_cond_expr_p (code, cond_fn,
>> vectype_in);
>> @@ -8312,17 +8390,29 @@ vect_transform_reduction (loop_vec_info loop_vinfo,
>> if (code == COND_EXPR)
>> gcc_assert (ncopies == 1);
>>
>> + /* A binary COND_OP reduction must have the same definition and else
>> + value. */
>> + bool cond_fn_p = code.is_internal_fn ()
>> + && conditional_internal_fn_code (internal_fn (code)) != ERROR_MARK;
>> + if (cond_fn_p)
>> + {
>> + gcc_assert (code == IFN_COND_ADD || code == IFN_COND_SUB
>> + || code == IFN_COND_MUL || code == IFN_COND_AND
>> + || code == IFN_COND_IOR || code == IFN_COND_XOR);
>> + gcc_assert (op.num_ops == 4 && (op.ops[1] == op.ops[3]));
>> + }
>> +
>> bool masked_loop_p = LOOP_VINFO_FULLY_MASKED_P (loop_vinfo);
>>
>> vect_reduction_type reduction_type = STMT_VINFO_REDUC_TYPE (reduc_info);
>> if (reduction_type == FOLD_LEFT_REDUCTION)
>> {
>> internal_fn reduc_fn = STMT_VINFO_REDUC_FN (reduc_info);
>> - gcc_assert (code.is_tree_code ());
>> + gcc_assert (code.is_tree_code () || cond_fn_p);
>> return vectorize_fold_left_reduction
>> (loop_vinfo, stmt_info, gsi, vec_stmt, slp_node, reduc_def_phi,
>> - tree_code (code), reduc_fn, op.ops, vectype_in, reduc_index, masks,
>> - lens);
>> + code, reduc_fn, op.ops, op.num_ops, vectype_in,
>> + reduc_index, masks, lens);
>> }
>>
>> bool single_defuse_cycle = STMT_VINFO_FORCE_SINGLE_CYCLE (reduc_info);
>> @@ -8335,14 +8425,20 @@ vect_transform_reduction (loop_vec_info loop_vinfo,
>> tree scalar_dest = gimple_get_lhs (stmt_info->stmt);
>> tree vec_dest = vect_create_destination_var (scalar_dest, vectype_out);
>>
>> + /* Get NCOPIES vector definitions for all operands except the reduction
>> + definition. */
>> vect_get_vec_defs (loop_vinfo, stmt_info, slp_node, ncopies,
>> single_defuse_cycle && reduc_index == 0
>> ? NULL_TREE : op.ops[0], &vec_oprnds0,
>> single_defuse_cycle && reduc_index == 1
>> ? NULL_TREE : op.ops[1], &vec_oprnds1,
>> - op.num_ops == 3
>> - && !(single_defuse_cycle && reduc_index == 2)
>> + op.num_ops == 4
>> + || (op.num_ops == 3
>> + && !(single_defuse_cycle && reduc_index == 2))
>> ? op.ops[2] : NULL_TREE, &vec_oprnds2);
>> +
>> + /* For single def-use cycles get one copy of the vectorized reduction
>> + definition. */
>> if (single_defuse_cycle)
>> {
>> gcc_assert (!slp_node);
>> @@ -8382,7 +8478,7 @@ vect_transform_reduction (loop_vec_info loop_vinfo,
>> }
>> else
>> {
>> - if (op.num_ops == 3)
>> + if (op.num_ops >= 3)
>> vop[2] = vec_oprnds2[i];
>>
>> if (masked_loop_p && mask_by_cond_expr)
>> @@ -8395,10 +8491,16 @@ vect_transform_reduction (loop_vec_info loop_vinfo,
>> if (emulated_mixed_dot_prod)
>> new_stmt = vect_emulate_mixed_dot_prod (loop_vinfo, stmt_info, gsi,
>> vec_dest, vop);
>> - else if (code.is_internal_fn ())
>> +
>> + else if (code.is_internal_fn () && !cond_fn_p)
>> new_stmt = gimple_build_call_internal (internal_fn (code),
>> op.num_ops,
>> vop[0], vop[1], vop[2]);
>> + else if (code.is_internal_fn () && cond_fn_p)
>> + new_stmt = gimple_build_call_internal (internal_fn (code),
>> + op.num_ops,
>> + vop[0], vop[1], vop[2],
>> + vop[1]);
>> else
>> new_stmt = gimple_build_assign (vec_dest, tree_code (op.code),
>> vop[0], vop[1], vop[2]);
>> diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
>> index a4043e4a656..254d172231d 100644
>> --- a/gcc/tree-vectorizer.h
>> +++ b/gcc/tree-vectorizer.h
>> @@ -2350,7 +2350,7 @@ extern tree vect_create_addr_base_for_vector_ref
>> (vec_info *,
>> tree);
>>
>> /* In tree-vect-loop.cc. */
>> -extern tree neutral_op_for_reduction (tree, code_helper, tree);
>> +extern tree neutral_op_for_reduction (tree, code_helper, tree, bool = true);
>> extern widest_int vect_iv_limit_for_partial_vectors (loop_vec_info
>> loop_vinfo);
>> bool vect_rgroup_iv_might_wrap_p (loop_vec_info, rgroup_controls *);
>> /* Used in tree-vect-loop-manip.cc */
>>
