On Thu, May 24, 2018 at 11:36 AM Richard Sandiford
<richard.sandif...@linaro.org> wrote:
>
> This patch adds match.pd support for applying normal folds to their
> IFN_COND_* forms. E.g. the rule:
>
> (plus @0 (negate @1)) -> (minus @0 @1)
>
> also allows the fold:
>
> (IFN_COND_ADD @0 @1 (negate @2) @3) -> (IFN_COND_SUB @0 @1 @2 @3)
>
> Actually doing this by direct matches in gimple-match.c would
> probably lead to combinatorial explosion, so instead, the patch
> makes gimple_match_op carry a condition under which the operation
> happens ("cond"), and the value to use when the condition is false
> ("else_value"). Thus in the example above we'd do the following
>
> (a) convert:
>
> cond:NULL_TREE (IFN_COND_ADD @0 @1 @4 @3) else_value:NULL_TREE
>
> to:
>
> cond:@0 (plus @1 @4) else_value:@3
>
> (b) apply gimple_resimplify to (plus @1 @4)
>
> (c) reintroduce cond and else_value when constructing the result.
>
> Nested operations inherit the condition of the outer operation
> (so that we don't introduce extra faults) but have a null else_value.
> If we try to build such an operation, the target gets to choose what
> else_value it can handle efficiently: obvious choices include one of
> the operands or a zero constant. (The alternative would be to have some
> representation for an undefined value, but that seems a bit invasive,
> and isn't likely to be useful here.)
>
> I've made the condition a mandatory part of the gimple_match_op
> constructor so that it doesn't accidentally get dropped.
>
> Tested on aarch64-linux-gnu (with and without SVE), aarch64_be-elf
> and x86_64-linux-gnu. OK to install?
It looks somewhat clever but after looking for a while it doesn't handle
simplifying
(IFN_COND_ADD @0 @1 (IFN_COND_SUB @0 @2 @1 @3) @3)
to
(cond @0 @2 @3)
right? Because while the conditional gimple_match_op is built
by try_conditional_simplification it isn't built when doing
SSA use->def following in the generated matching code?
So it looks like a bit much noise for this very special case?
I suppose you ran into the need of these foldings from looking
at real code - which foldings specifically were appearing here?
Usually code is well optimized before if-conversion/vectorization
so we shouldn't need full-blown handling?
That said, I'm not sure how much work it is to massage
if (gimple *def_stmt = get_def (valueize, op2))
{
if (gassign *def = dyn_cast <gassign *> (def_stmt))
switch (gimple_assign_rhs_code (def))
{
case PLUS_EXPR:
to look like
if (gimple *def_stmt = get_def (valueize, op2))
{
code = ERROR_MARK;
if (!is_cond_ifn_with_cond (curr_gimple_match_op, &code))
if (gassign *def dyn_cast <gassign *> (def_stmt))
code = gimple_assign_rhs_code (def);
switch (code)
{
case PLUS_EXPR:
thus transparently treat the IFN_COND_* as their "code" if the condition
matches that of the context (I'm not sure if we can do anything for
mismatching contexts).
Richard.
> Richard
>
>
> 2018-05-24 Richard Sandiford <richard.sandif...@linaro.org>
>
> gcc/
> * target.def (preferred_else_value): New target hook.
> * doc/tm.texi.in (TARGET_PREFERRED_ELSE_VALUE): New hook.
> * doc/tm.texi: Regenerate.
> * targhooks.h (default_preferred_else_value): Declare.
> * targhooks.c (default_preferred_else_value): New function.
> * internal-fn.h (conditional_internal_fn_code): Declare.
> * internal-fn.c (FOR_EACH_CODE_MAPPING): New macro.
> (get_conditional_internal_fn): Use it.
> (conditional_internal_fn_code): New function.
> * gimple-match.h (gimple_match_cond): New struct.
> (gimple_match_op): Add a cond member function.
> (gimple_match_op::gimple_match_op): Update all forms to take a
> gimple_match_cond.
> * genmatch.c (expr::gen_transform): Use the same condition as res_op
> for the suboperation, but don't specify a particular else_value.
> * tree-ssa-sccvn.c (vn_nary_simplify, vn_reference_lookup_3)
> (visit_nary_op, visit_reference_op_load): Pass
> gimple_match_cond::UNCOND to the gimple_match_op constructor.
> * gimple-match-head.c: Include tree-eh.h
> (convert_conditional_op): New function.
> (maybe_resimplify_conditional_op): Likewise.
> (gimple_resimplify1): Call maybe_resimplify_conditional_op.
> (gimple_resimplify2): Likewise.
> (gimple_resimplify3): Likewise.
> (gimple_resimplify4): Likewise.
> (maybe_push_res_to_seq): Return null for conditional operations.
> (try_conditional_simplification): New function.
> (gimple_simplify): Call it. Pass conditions to the gimple_match_op
> constructor.
> * match.pd: Fold VEC_COND_EXPRs of an IFN_COND_* call to a new
> IFN_COND_* call.
> * config/aarch64/aarch64.c (aarch64_preferred_else_value): New
> function.
> (TARGET_PREFERRED_ELSE_VALUE): Redefine.
>
> gcc/testsuite/
> * gcc.dg/vect/vect-cond-arith-2.c: New test.
> * gcc.target/aarch64/sve/loop_add_6.c: Likewise.
>
> Index: gcc/target.def
> ===================================================================
> --- gcc/target.def 2018-05-01 19:30:30.159632586 +0100
> +++ gcc/target.def 2018-05-24 10:33:30.871095132 +0100
> @@ -2040,6 +2040,25 @@ HOOK_VECTOR_END (vectorize)
> #define HOOK_PREFIX "TARGET_"
>
> DEFHOOK
> +(preferred_else_value,
> + "This hook returns the target's preferred final argument for a call\n\
> +to conditional internal function @var{ifn} (really of type\n\
> +@code{internal_fn}). @var{type} specifies the return type of the\n\
> +function and @var{ops} are the operands to the conditional operation,\n\
> +of which there are @var{nops}.\n\
> +\n\
> +For example, if @var{ifn} is @code{IFN_COND_ADD}, the hook returns\n\
> +a value of type @var{type} that should be used when @samp{@var{ops}[0]}\n\
> +and @samp{@var{ops}[1]} are conditionally added together.\n\
> +\n\
> +This hook is only relevant if the target supports conditional patterns\n\
> +like @code{cond_add@var{m}}. The default implementation returns a zero\n\
> +constant of type @var{type}.",
> + tree,
> + (unsigned ifn, tree type, unsigned nops, tree *ops),
> + default_preferred_else_value)
> +
> +DEFHOOK
> (record_offload_symbol,
> "Used when offloaded functions are seen in the compilation unit and no
> named\n\
> sections are available. It is called once for each symbol that must be\n\
> Index: gcc/doc/tm.texi.in
> ===================================================================
> --- gcc/doc/tm.texi.in 2018-05-01 19:30:28.730694873 +0100
> +++ gcc/doc/tm.texi.in 2018-05-24 10:33:30.869095197 +0100
> @@ -4149,6 +4149,8 @@ address; but often a machine-dependent
>
> @hook TARGET_GOACC_REDUCTION
>
> +@hook TARGET_PREFERRED_ELSE_VALUE
> +
> @node Anchored Addresses
> @section Anchored Addresses
> @cindex anchored addresses
> Index: gcc/doc/tm.texi
> ===================================================================
> --- gcc/doc/tm.texi 2018-05-01 19:30:28.722695224 +0100
> +++ gcc/doc/tm.texi 2018-05-24 10:33:30.868095229 +0100
> @@ -6046,6 +6046,22 @@ expanded sequence has been inserted. Th
> for allocating any storage for reductions when necessary.
> @end deftypefn
>
> +@deftypefn {Target Hook} tree TARGET_PREFERRED_ELSE_VALUE (unsigned
> @var{ifn}, tree @var{type}, unsigned @var{nops}, tree *@var{ops})
> +This hook returns the target's preferred final argument for a call
> +to conditional internal function @var{ifn} (really of type
> +@code{internal_fn}). @var{type} specifies the return type of the
> +function and @var{ops} are the operands to the conditional operation,
> +of which there are @var{nops}.
> +
> +For example, if @var{ifn} is @code{IFN_COND_ADD}, the hook returns
> +a value of type @var{type} that should be used when @samp{@var{ops}[0]}
> +and @samp{@var{ops}[1]} are conditionally added together.
> +
> +This hook is only relevant if the target supports conditional patterns
> +like @code{cond_add@var{m}}. The default implementation returns a zero
> +constant of type @var{type}.
> +@end deftypefn
> +
> @node Anchored Addresses
> @section Anchored Addresses
> @cindex anchored addresses
> Index: gcc/targhooks.h
> ===================================================================
> --- gcc/targhooks.h 2018-05-01 19:30:29.390666052 +0100
> +++ gcc/targhooks.h 2018-05-24 10:33:30.872095099 +0100
> @@ -289,5 +289,6 @@ extern unsigned int default_min_arithmet
> default_excess_precision (enum excess_precision_type ATTRIBUTE_UNUSED);
> extern bool default_stack_clash_protection_final_dynamic_probe (rtx);
> extern void default_select_early_remat_modes (sbitmap);
> +extern tree default_preferred_else_value (unsigned, tree, unsigned, tree *);
>
> #endif /* GCC_TARGHOOKS_H */
> Index: gcc/targhooks.c
> ===================================================================
> --- gcc/targhooks.c 2018-05-01 19:30:29.390666052 +0100
> +++ gcc/targhooks.c 2018-05-24 10:33:30.871095132 +0100
> @@ -2345,4 +2345,12 @@ default_select_early_remat_modes (sbitma
> {
> }
>
> +/* The default implementation of TARGET_PREFERRED_ELSE_VALUE. */
> +
> +tree
> +default_preferred_else_value (unsigned, tree type, unsigned, tree *)
> +{
> + return build_zero_cst (type);
> +}
> +
> #include "gt-targhooks.h"
> Index: gcc/internal-fn.h
> ===================================================================
> --- gcc/internal-fn.h 2018-05-17 11:52:13.507173989 +0100
> +++ gcc/internal-fn.h 2018-05-24 10:33:30.870095164 +0100
> @@ -193,6 +193,7 @@ direct_internal_fn_supported_p (internal
> extern bool set_edom_supported_p (void);
>
> extern internal_fn get_conditional_internal_fn (tree_code);
> +extern tree_code conditional_internal_fn_code (internal_fn);
>
> extern bool internal_load_fn_p (internal_fn);
> extern bool internal_store_fn_p (internal_fn);
> Index: gcc/internal-fn.c
> ===================================================================
> --- gcc/internal-fn.c 2018-05-24 10:12:10.146352152 +0100
> +++ gcc/internal-fn.c 2018-05-24 10:33:30.870095164 +0100
> @@ -3219,6 +3219,21 @@ #define DEF_INTERNAL_FN(CODE, FLAGS, FNS
> 0
> };
>
> +/* Invoke T(CODE, IFN) for each conditional function IFN that maps to a
> + tree code CODE. */
> +#define FOR_EACH_CODE_MAPPING(T) \
> + T (PLUS_EXPR, IFN_COND_ADD) \
> + T (MINUS_EXPR, IFN_COND_SUB) \
> + T (MULT_EXPR, IFN_COND_MUL) \
> + T (TRUNC_DIV_EXPR, IFN_COND_DIV) \
> + T (TRUNC_MOD_EXPR, IFN_COND_MOD) \
> + T (RDIV_EXPR, IFN_COND_RDIV) \
> + T (MIN_EXPR, IFN_COND_MIN) \
> + T (MAX_EXPR, IFN_COND_MAX) \
> + T (BIT_AND_EXPR, IFN_COND_AND) \
> + T (BIT_IOR_EXPR, IFN_COND_IOR) \
> + T (BIT_XOR_EXPR, IFN_COND_XOR)
> +
> /* Return a function that only performs CODE when a certain condition is met
> and that uses a given fallback value otherwise. For example, if CODE is
> a binary operation associated with conditional function FN:
> @@ -3238,31 +3253,30 @@ get_conditional_internal_fn (tree_code c
> {
> switch (code)
> {
> - case PLUS_EXPR:
> - return IFN_COND_ADD;
> - case MINUS_EXPR:
> - return IFN_COND_SUB;
> - case MIN_EXPR:
> - return IFN_COND_MIN;
> - case MAX_EXPR:
> - return IFN_COND_MAX;
> - case TRUNC_DIV_EXPR:
> - return IFN_COND_DIV;
> - case TRUNC_MOD_EXPR:
> - return IFN_COND_MOD;
> - case RDIV_EXPR:
> - return IFN_COND_RDIV;
> - case BIT_AND_EXPR:
> - return IFN_COND_AND;
> - case BIT_IOR_EXPR:
> - return IFN_COND_IOR;
> - case BIT_XOR_EXPR:
> - return IFN_COND_XOR;
> +#define CASE(CODE, IFN) case CODE: return IFN;
> + FOR_EACH_CODE_MAPPING(CASE)
> +#undef CASE
> default:
> return IFN_LAST;
> }
> }
>
> +/* If IFN implements the conditional form of a tree code, return that
> + tree code, otherwise return ERROR_MARK. */
> +
> +tree_code
> +conditional_internal_fn_code (internal_fn ifn)
> +{
> + switch (ifn)
> + {
> +#define CASE(CODE, IFN) case IFN: return CODE;
> + FOR_EACH_CODE_MAPPING(CASE)
> +#undef CASE
> + default:
> + return ERROR_MARK;
> + }
> +}
> +
> /* Return true if IFN is some form of load from memory. */
>
> bool
> Index: gcc/gimple-match.h
> ===================================================================
> --- gcc/gimple-match.h 2018-05-24 09:54:37.509451356 +0100
> +++ gcc/gimple-match.h 2018-05-24 10:33:30.870095164 +0100
> @@ -40,16 +40,57 @@ #define GCC_GIMPLE_MATCH_H
> int rep;
> };
>
> +/* Represents the condition under which an operation should happen,
> + and the value to use otherwise. The condition applies elementwise
> + (as for VEC_COND_EXPR) if the values are vectors. */
> +struct gimple_match_cond
> +{
> + enum uncond { UNCOND };
> +
> + /* Build an unconditional op. */
> + gimple_match_cond (uncond) : cond (NULL_TREE), else_value (NULL_TREE) {}
> + gimple_match_cond (tree, tree);
> +
> + gimple_match_cond any_else () const;
> +
> + /* The condition under which the operation occurs, or NULL_TREE
> + if the operation is unconditional. */
> + tree cond;
> +
> + /* The value to use when the condition is false. This is NULL_TREE if
> + the operation is unconditional or if the value doesn't matter. */
> + tree else_value;
> +};
> +
> +inline
> +gimple_match_cond::gimple_match_cond (tree cond_in, tree else_value_in)
> + : cond (cond_in), else_value (else_value_in)
> +{
> +}
> +
> +/* Return a gimple_match_cond with the same condition but with an
> + arbitrary ELSE_VALUE. */
> +
> +inline gimple_match_cond
> +gimple_match_cond::any_else () const
> +{
> + return gimple_match_cond (cond, NULL_TREE);
> +}
> +
> /* Represents an operation to be simplified, or the result of the
> simplification. */
> struct gimple_match_op
> {
> - gimple_match_op () : type (NULL_TREE), num_ops (0) {}
> - gimple_match_op (code_helper, tree, unsigned int);
> - gimple_match_op (code_helper, tree, tree);
> - gimple_match_op (code_helper, tree, tree, tree);
> - gimple_match_op (code_helper, tree, tree, tree, tree);
> - gimple_match_op (code_helper, tree, tree, tree, tree, tree);
> + gimple_match_op ();
> + gimple_match_op (const gimple_match_cond &, code_helper, tree, unsigned
> int);
> + gimple_match_op (const gimple_match_cond &,
> + code_helper, tree, tree);
> + gimple_match_op (const gimple_match_cond &,
> + code_helper, tree, tree, tree);
> + gimple_match_op (const gimple_match_cond &,
> + code_helper, tree, tree, tree, tree);
> + gimple_match_op (const gimple_match_cond &,
> + code_helper, tree, tree, tree, tree, tree);
>
> void set_op (code_helper, tree, unsigned int);
> void set_op (code_helper, tree, tree);
> @@ -63,6 +104,10 @@ struct gimple_match_op
> /* The maximum value of NUM_OPS. */
> static const unsigned int MAX_NUM_OPS = 4;
>
> + /* The conditions under which the operation is performed, and the value to
> + use as a fallback. */
> + gimple_match_cond cond;
> +
> /* The operation being performed. */
> code_helper code;
>
> @@ -76,39 +121,49 @@ struct gimple_match_op
> tree ops[MAX_NUM_OPS];
> };
>
> -/* Constructor that takes the code, type and number of operands, but leaves
> - the caller to fill in the operands. */
> +inline
> +gimple_match_op::gimple_match_op ()
> + : cond (gimple_match_cond::UNCOND), type (NULL_TREE), num_ops (0)
> +{
> +}
> +
> +/* Constructor that takes the condition, code, type and number of
> + operands, but leaves the caller to fill in the operands. */
>
> inline
> -gimple_match_op::gimple_match_op (code_helper code_in, tree type_in,
> +gimple_match_op::gimple_match_op (const gimple_match_cond &cond_in,
> + code_helper code_in, tree type_in,
> unsigned int num_ops_in)
> - : code (code_in), type (type_in), num_ops (num_ops_in)
> + : cond (cond_in), code (code_in), type (type_in), num_ops (num_ops_in)
> {
> }
>
> /* Constructors for various numbers of operands. */
>
> inline
> -gimple_match_op::gimple_match_op (code_helper code_in, tree type_in,
> +gimple_match_op::gimple_match_op (const gimple_match_cond &cond_in,
> + code_helper code_in, tree type_in,
> tree op0)
> - : code (code_in), type (type_in), num_ops (1)
> + : cond (cond_in), code (code_in), type (type_in), num_ops (1)
> {
> ops[0] = op0;
> }
>
> inline
> -gimple_match_op::gimple_match_op (code_helper code_in, tree type_in,
> +gimple_match_op::gimple_match_op (const gimple_match_cond &cond_in,
> + code_helper code_in, tree type_in,
> tree op0, tree op1)
> - : code (code_in), type (type_in), num_ops (2)
> + : cond (cond_in), code (code_in), type (type_in), num_ops (2)
> {
> ops[0] = op0;
> ops[1] = op1;
> }
>
> inline
> -gimple_match_op::gimple_match_op (code_helper code_in, tree type_in,
> +gimple_match_op::gimple_match_op (const gimple_match_cond &cond_in,
> + code_helper code_in, tree type_in,
> tree op0, tree op1, tree op2)
> - : code (code_in), type (type_in), num_ops (3)
> + : cond (cond_in), code (code_in), type (type_in), num_ops (3)
> {
> ops[0] = op0;
> ops[1] = op1;
> @@ -116,9 +171,10 @@ gimple_match_op::gimple_match_op (code_h
> }
>
> inline
> -gimple_match_op::gimple_match_op (code_helper code_in, tree type_in,
> +gimple_match_op::gimple_match_op (const gimple_match_cond &cond_in,
> + code_helper code_in, tree type_in,
> tree op0, tree op1, tree op2, tree op3)
> - : code (code_in), type (type_in), num_ops (4)
> + : cond (cond_in), code (code_in), type (type_in), num_ops (4)
> {
> ops[0] = op0;
> ops[1] = op1;
> Index: gcc/genmatch.c
> ===================================================================
> --- gcc/genmatch.c 2018-05-24 10:12:10.145352193 +0100
> +++ gcc/genmatch.c 2018-05-24 10:33:30.869095197 +0100
> @@ -2507,8 +2507,8 @@ expr::gen_transform (FILE *f, int indent
> /* ??? Building a stmt can fail for various reasons here, seq being
> NULL or the stmt referencing SSA names occuring in abnormal PHIs.
> So if we fail here we should continue matching other patterns. */
> - fprintf_indent (f, indent, "gimple_match_op tem_op (%s, %s",
> - opr_name, type);
> + fprintf_indent (f, indent, "gimple_match_op tem_op "
> + "(res_op->cond.any_else (), %s, %s", opr_name, type);
> for (unsigned i = 0; i < ops.length (); ++i)
> fprintf (f, ", ops%d[%u]", depth, i);
> fprintf (f, ");\n");
> Index: gcc/tree-ssa-sccvn.c
> ===================================================================
> --- gcc/tree-ssa-sccvn.c 2018-05-24 09:02:28.765328358 +0100
> +++ gcc/tree-ssa-sccvn.c 2018-05-24 10:33:30.872095099 +0100
> @@ -1804,7 +1804,8 @@ vn_nary_simplify (vn_nary_op_t nary)
> {
> if (nary->length > gimple_match_op::MAX_NUM_OPS)
> return NULL_TREE;
> - gimple_match_op op (nary->opcode, nary->type, nary->length);
> + gimple_match_op op (gimple_match_cond::UNCOND, nary->opcode,
> + nary->type, nary->length);
> memcpy (op.ops, nary->op, sizeof (tree) * nary->length);
> return vn_nary_build_or_lookup_1 (&op, false);
> }
> @@ -2031,8 +2032,8 @@ vn_reference_lookup_3 (ao_ref *ref, tree
> else if (INTEGRAL_TYPE_P (vr->type)
> && known_eq (ref->size, 8))
> {
> - gimple_match_op res_op (NOP_EXPR, vr->type,
> - gimple_call_arg (def_stmt, 1));
> + gimple_match_op res_op (gimple_match_cond::UNCOND, NOP_EXPR,
> + vr->type, gimple_call_arg (def_stmt,
> 1));
> val = vn_nary_build_or_lookup (&res_op);
> if (!val
> || (TREE_CODE (val) == SSA_NAME
> @@ -2172,7 +2173,8 @@ vn_reference_lookup_3 (ao_ref *ref, tree
> || known_eq (ref->size, TYPE_PRECISION (vr->type)))
> && multiple_p (ref->size, BITS_PER_UNIT))
> {
> - gimple_match_op op (BIT_FIELD_REF, vr->type,
> + gimple_match_op op (gimple_match_cond::UNCOND,
> + BIT_FIELD_REF, vr->type,
> SSA_VAL (gimple_assign_rhs1 (def_stmt)),
> bitsize_int (ref->size),
> bitsize_int (offset - offset2));
> @@ -3701,7 +3703,8 @@ visit_nary_op (tree lhs, gassign *stmt)
> unsigned rhs_prec = TYPE_PRECISION (TREE_TYPE (rhs1));
> if (lhs_prec == rhs_prec)
> {
> - gimple_match_op match_op (NOP_EXPR, type, ops[0]);
> + gimple_match_op match_op (gimple_match_cond::UNCOND,
> + NOP_EXPR, type, ops[0]);
> result = vn_nary_build_or_lookup (&match_op);
> if (result)
> {
> @@ -3714,7 +3717,8 @@ visit_nary_op (tree lhs, gassign *stmt)
> {
> tree mask = wide_int_to_tree
> (type, wi::mask (rhs_prec, false, lhs_prec));
> - gimple_match_op match_op (BIT_AND_EXPR,
> + gimple_match_op match_op (gimple_match_cond::UNCOND,
> + BIT_AND_EXPR,
> TREE_TYPE (lhs),
> ops[0], mask);
> result = vn_nary_build_or_lookup (&match_op);
> @@ -3838,7 +3842,8 @@ visit_reference_op_load (tree lhs, tree
> of VIEW_CONVERT_EXPR <TREE_TYPE (result)> (result).
> So first simplify and lookup this expression to see if it
> is already available. */
> - gimple_match_op res_op (VIEW_CONVERT_EXPR, TREE_TYPE (op), result);
> + gimple_match_op res_op (gimple_match_cond::UNCOND,
> + VIEW_CONVERT_EXPR, TREE_TYPE (op), result);
> result = vn_nary_build_or_lookup (&res_op);
> }
>
> Index: gcc/gimple-match-head.c
> ===================================================================
> --- gcc/gimple-match-head.c 2018-05-24 09:54:37.509451356 +0100
> +++ gcc/gimple-match-head.c 2018-05-24 10:33:30.870095164 +0100
> @@ -40,6 +40,7 @@ Software Foundation; either version 3, o
> #include "case-cfn-macros.h"
> #include "gimplify.h"
> #include "optabs-tree.h"
> +#include "tree-eh.h"
>
>
> /* Forward declarations of the private auto-generated matchers.
> @@ -68,6 +69,95 @@ constant_for_folding (tree t)
> && TREE_CODE (TREE_OPERAND (t, 0)) == STRING_CST));
> }
>
> +/* Try to convert conditional operation ORIG_OP into an IFN_COND_*
> + operation. Return true on success, storing the new operation in NEW_OP.
> */
> +
> +static bool
> +convert_conditional_op (gimple_match_op *orig_op,
> + gimple_match_op *new_op)
> +{
> + internal_fn ifn;
> + if (orig_op->code.is_tree_code ())
> + ifn = get_conditional_internal_fn ((tree_code) orig_op->code);
> + else
> + return false;
> + if (ifn == IFN_LAST)
> + return false;
> + unsigned int num_ops = orig_op->num_ops;
> + new_op->set_op (as_combined_fn (ifn), orig_op->type, num_ops + 2);
> + new_op->ops[0] = orig_op->cond.cond;
> + for (unsigned int i = 0; i < num_ops; ++i)
> + new_op->ops[i + 1] = orig_op->ops[i];
> + tree else_value = orig_op->cond.else_value;
> + if (!else_value)
> + else_value = targetm.preferred_else_value (ifn, orig_op->type,
> + num_ops, orig_op->ops);
> + new_op->ops[num_ops + 1] = else_value;
> + return true;
> +}
> +
> +/* RES_OP is the result of a simplification. If it is conditional,
> + try to replace it with the equivalent UNCOND form, such as an
> + IFN_COND_* call or a VEC_COND_EXPR. Also try to resimplify the
> + result of the replacement if appropriate, adding any new statements to
> + SEQ and using VALUEIZE as the valueization function. Return true if
> + this resimplification occurred and resulted in at least one change. */
> +
> +static bool
> +maybe_resimplify_conditional_op (gimple_seq *seq, gimple_match_op *res_op,
> + tree (*valueize) (tree))
> +{
> + if (!res_op->cond.cond)
> + return false;
> +
> + if (!res_op->cond.else_value
> + && res_op->code.is_tree_code ())
> + {
> + /* The "else" value doesn't matter. If the "then" value is a
> + gimple value, just use it unconditionally. This isn't a
> + simplification in itself, since there was no operation to
> + build in the first place. */
> + if (gimple_simplified_result_is_gimple_val (res_op))
> + {
> + res_op->cond.cond = NULL_TREE;
> + return false;
> + }
> +
> + /* Likewise if the operation would not trap. */
> + bool honor_trapv = (INTEGRAL_TYPE_P (res_op->type)
> + && TYPE_OVERFLOW_TRAPS (res_op->type));
> + if (!operation_could_trap_p ((tree_code) res_op->code,
> + FLOAT_TYPE_P (res_op->type),
> + honor_trapv, res_op->op_or_null (1)))
> + {
> + res_op->cond.cond = NULL_TREE;
> + return false;
> + }
> + }
> +
> + /* If the "then" value is a gimple value and the "else" value matters,
> + create a VEC_COND_EXPR between them, then see if it can be further
> + simplified. */
> + gimple_match_op new_op;
> + if (res_op->cond.else_value
> + && VECTOR_TYPE_P (res_op->type)
> + && gimple_simplified_result_is_gimple_val (res_op))
> + {
> + new_op.set_op (VEC_COND_EXPR, res_op->type,
> + res_op->cond.cond, res_op->ops[0],
> + res_op->cond.else_value);
> + *res_op = new_op;
> + return gimple_resimplify3 (seq, res_op, valueize);
> + }
> +
> + /* Otherwise try rewriting the operation as an IFN_COND_* call.
> + Again, this isn't a simplification in itself, since it's what
> + RES_OP already described. */
> + if (convert_conditional_op (res_op, &new_op))
> + *res_op = new_op;
> +
> + return false;
> +}
>
> /* Helper that matches and simplifies the toplevel result from
> a gimple_simplify run (where we don't want to build
> @@ -93,6 +183,7 @@ gimple_resimplify1 (gimple_seq *seq, gim
> if (TREE_OVERFLOW_P (tem))
> tem = drop_tree_overflow (tem);
> res_op->set_value (tem);
> + maybe_resimplify_conditional_op (seq, res_op, valueize);
> return true;
> }
> }
> @@ -105,6 +196,9 @@ gimple_resimplify1 (gimple_seq *seq, gim
> return true;
> }
>
> + if (maybe_resimplify_conditional_op (seq, res_op, valueize))
> + return true;
> +
> return false;
> }
>
> @@ -134,6 +228,7 @@ gimple_resimplify2 (gimple_seq *seq, gim
> if (TREE_OVERFLOW_P (tem))
> tem = drop_tree_overflow (tem);
> res_op->set_value (tem);
> + maybe_resimplify_conditional_op (seq, res_op, valueize);
> return true;
> }
> }
> @@ -160,6 +255,9 @@ gimple_resimplify2 (gimple_seq *seq, gim
> return true;
> }
>
> + if (maybe_resimplify_conditional_op (seq, res_op, valueize))
> + return true;
> +
> return canonicalized;
> }
>
> @@ -191,6 +289,7 @@ gimple_resimplify3 (gimple_seq *seq, gim
> if (TREE_OVERFLOW_P (tem))
> tem = drop_tree_overflow (tem);
> res_op->set_value (tem);
> + maybe_resimplify_conditional_op (seq, res_op, valueize);
> return true;
> }
> }
> @@ -214,6 +313,9 @@ gimple_resimplify3 (gimple_seq *seq, gim
> return true;
> }
>
> + if (maybe_resimplify_conditional_op (seq, res_op, valueize))
> + return true;
> +
> return canonicalized;
> }
>
> @@ -239,6 +341,9 @@ gimple_resimplify4 (gimple_seq *seq, gim
> return true;
> }
>
> + if (maybe_resimplify_conditional_op (seq, res_op, valueize))
> + return true;
> +
> return false;
> }
>
> @@ -297,6 +402,12 @@ maybe_push_res_to_seq (gimple_match_op *
> tree *ops = res_op->ops;
> unsigned num_ops = res_op->num_ops;
>
> + /* The caller should have converted conditional operations into an UNCOND
> + form and resimplified as appropriate. The conditional form only
> + survives this far if that conversion failed. */
> + if (res_op->cond.cond)
> + return NULL_TREE;
> +
> if (res_op->code.is_tree_code ())
> {
> if (!res
> @@ -558,6 +669,50 @@ do_valueize (tree op, tree (*valueize)(t
> return op;
> }
>
> +/* If RES_OP is a call to a conditional internal function, try simplifying
> + the associated unconditional operation and using the result to build
> + a new conditional operation. For example, if RES_OP is:
> +
> + IFN_COND_ADD (COND, A, B, ELSE)
> +
> + try simplifying (plus A B) and using the result to build a replacement
> + for the whole IFN_COND_ADD.
> +
> + Return true if this approach led to a simplification, otherwise leave
> + RES_OP unchanged (and so suitable for other simplifications). When
> + returning true, add any new statements to SEQ and use VALUEIZE as the
> + valueization function.
> +
> + RES_OP is known to be a call to IFN. */
> +
> +static bool
> +try_conditional_simplification (internal_fn ifn, gimple_match_op *res_op,
> + gimple_seq *seq, tree (*valueize) (tree))
> +{
> + tree_code code = conditional_internal_fn_code (ifn);
> + if (code == ERROR_MARK)
> + return false;
> +
> + unsigned int num_ops = res_op->num_ops;
> + gimple_match_op cond_op (gimple_match_cond (res_op->ops[0],
> + res_op->ops[num_ops - 1]),
> + code, res_op->type, num_ops - 2);
> + for (unsigned int i = 1; i < num_ops - 1; ++i)
> + cond_op.ops[i - 1] = res_op->ops[i];
> + switch (num_ops - 2)
> + {
> + case 2:
> + if (!gimple_resimplify2 (seq, &cond_op, valueize))
> + return false;
> + break;
> + default:
> + gcc_unreachable ();
> + }
> + *res_op = cond_op;
> + maybe_resimplify_conditional_op (seq, res_op, valueize);
> + return true;
> +}
> +
> /* The main STMT based simplification entry. It is used by the fold_stmt
> and the fold_stmt_to_constant APIs. */
>
> @@ -643,7 +798,7 @@ gimple_simplify (gimple *stmt, gimple_ma
> tree rhs = TREE_OPERAND (rhs1, 1);
> lhs = do_valueize (lhs, top_valueize, valueized);
> rhs = do_valueize (rhs, top_valueize, valueized);
> - gimple_match_op res_op2 (TREE_CODE (rhs1),
> + gimple_match_op res_op2 (res_op->cond, TREE_CODE (rhs1),
> TREE_TYPE (rhs1), lhs, rhs);
> if ((gimple_resimplify2 (seq, &res_op2, valueize)
> || valueized)
> @@ -714,6 +869,10 @@ gimple_simplify (gimple *stmt, gimple_ma
> tree arg = gimple_call_arg (stmt, i);
> res_op->ops[i] = do_valueize (arg, top_valueize, valueized);
> }
> + if (internal_fn_p (cfn)
> + && try_conditional_simplification (as_internal_fn (cfn),
> + res_op, seq, valueize))
> + return true;
> switch (num_args)
> {
> case 1:
> Index: gcc/match.pd
> ===================================================================
> --- gcc/match.pd 2018-05-24 10:12:10.146352152 +0100
> +++ gcc/match.pd 2018-05-24 10:33:30.870095164 +0100
> @@ -4797,3 +4797,12 @@ DEFINE_INT_AND_FLOAT_ROUND_FN (RINT)
> (with { tree op_type = TREE_TYPE (@4); }
> (if (element_precision (type) == element_precision (op_type))
> (view_convert (cond_op (bit_not @0) @2 @3 (view_convert:op_type @1)))))))
> +
> +/* Detect cases in which a VEC_COND_EXPR effectively replaces the
> + "else" value of an IFN_COND_*. */
> +(for cond_op (COND_BINARY)
> + (simplify
> + (vec_cond @0 (view_convert? (cond_op @0 @1 @2 @3)) @4)
> + (with { tree op_type = TREE_TYPE (@3); }
> + (if (element_precision (type) == element_precision (op_type))
> + (view_convert (cond_op @0 @1 @2 (view_convert:op_type @4)))))))
> Index: gcc/config/aarch64/aarch64.c
> ===================================================================
> --- gcc/config/aarch64/aarch64.c 2018-05-24 09:54:37.507451418 +0100
> +++ gcc/config/aarch64/aarch64.c 2018-05-24 10:33:30.867095262 +0100
> @@ -1292,6 +1292,16 @@ aarch64_get_mask_mode (poly_uint64 nunit
> return default_get_mask_mode (nunits, nbytes);
> }
>
> +/* Implement TARGET_PREFERRED_ELSE_VALUE. Prefer to use the first
> + arithmetic operand as the else value if the else value doesn't matter,
> + since that exactly matches the SVE destructive merging form. */
> +
> +static tree
> +aarch64_preferred_else_value (unsigned, tree, unsigned int, tree *ops)
> +{
> + return ops[0];
> +}
> +
> /* Implement TARGET_HARD_REGNO_NREGS. */
>
> static unsigned int
> @@ -17980,6 +17990,9 @@ #define TARGET_VECTORIZE_GET_MASK_MODE a
> #undef TARGET_VECTORIZE_EMPTY_MASK_IS_EXPENSIVE
> #define TARGET_VECTORIZE_EMPTY_MASK_IS_EXPENSIVE \
> aarch64_empty_mask_is_expensive
> +#undef TARGET_PREFERRED_ELSE_VALUE
> +#define TARGET_PREFERRED_ELSE_VALUE \
> + aarch64_preferred_else_value
>
> #undef TARGET_INIT_LIBFUNCS
> #define TARGET_INIT_LIBFUNCS aarch64_init_libfuncs
> Index: gcc/testsuite/gcc.dg/vect/vect-cond-arith-2.c
> ===================================================================
> --- /dev/null 2018-04-20 16:19:46.369131350 +0100
> +++ gcc/testsuite/gcc.dg/vect/vect-cond-arith-2.c 2018-05-24
> 10:33:30.872095099 +0100
> @@ -0,0 +1,45 @@
> +/* { dg-do compile } */
> +/* { dg-additional-options "-fgimple -fdump-tree-optimized -ffast-math" } */
> +
> +double __GIMPLE (startwith("loop"))
> +neg_xi (double *x)
> +{
> + int i;
> + long unsigned int index;
> + long unsigned int offset;
> + double * xi_ptr;
> + double xi;
> + double neg_xi;
> + double res;
> + unsigned int ivtmp;
> +
> + bb_1:
> + goto bb_2;
> +
> + bb_2:
> + res_1 = __PHI (bb_1: 0.0, bb_3: res_2);
> + i_4 = __PHI (bb_1: 0, bb_3: i_5);
> + ivtmp_6 = __PHI (bb_1: 100U, bb_3: ivtmp_7);
> + index = (long unsigned int) i_4;
> + offset = index * 8UL;
> + xi_ptr = x_8(D) + offset;
> + xi = *xi_ptr;
> + neg_xi = -xi;
> + res_2 = neg_xi + res_1;
> + i_5 = i_4 + 1;
> + ivtmp_7 = ivtmp_6 - 1U;
> + if (ivtmp_7 != 0U)
> + goto bb_3;
> + else
> + goto bb_4;
> +
> + bb_3:
> + goto bb_2;
> +
> + bb_4:
> + res_3 = __PHI (bb_2: res_2);
> + return res_3;
> +}
> +
> +/* { dg-final { scan-tree-dump { = \.COND_ADD} "vect" { target {
> vect_double_cond_arith && vect_fully_masked } } } } */
> +/* { dg-final { scan-tree-dump { = \.COND_SUB} "optimized" { target {
> vect_double_cond_arith && vect_fully_masked } } } } */
> Index: gcc/testsuite/gcc.target/aarch64/sve/loop_add_6.c
> ===================================================================
> --- /dev/null 2018-04-20 16:19:46.369131350 +0100
> +++ gcc/testsuite/gcc.target/aarch64/sve/loop_add_6.c 2018-05-24
> 10:33:30.872095099 +0100
> @@ -0,0 +1,46 @@
> +/* { dg-do compile } */
> +/* { dg-options "-O2 -ftree-vectorize -fgimple -ffast-math" } */
> +
> +double __GIMPLE (startwith("loop"))
> +neg_xi (double *x)
> +{
> + int i;
> + long unsigned int index;
> + long unsigned int offset;
> + double * xi_ptr;
> + double xi;
> + double neg_xi;
> + double res;
> + unsigned int ivtmp;
> +
> + bb_1:
> + goto bb_2;
> +
> + bb_2:
> + res_1 = __PHI (bb_1: 0.0, bb_3: res_2);
> + i_4 = __PHI (bb_1: 0, bb_3: i_5);
> + ivtmp_6 = __PHI (bb_1: 100U, bb_3: ivtmp_7);
> + index = (long unsigned int) i_4;
> + offset = index * 8UL;
> + xi_ptr = x_8(D) + offset;
> + xi = *xi_ptr;
> + neg_xi = -xi;
> + res_2 = neg_xi + res_1;
> + i_5 = i_4 + 1;
> + ivtmp_7 = ivtmp_6 - 1U;
> + if (ivtmp_7 != 0U)
> + goto bb_3;
> + else
> + goto bb_4;
> +
> + bb_3:
> + goto bb_2;
> +
> + bb_4:
> + res_3 = __PHI (bb_2: res_2);
> + return res_3;
> +}
> +
> +/* { dg-final { scan-assembler {\tfsub\tz[0-9]+\.d, p[0-7]/m} } } */
> +/* { dg-final { scan-assembler-not {\tsel\t} } } */
> +/* { dg-final { scan-assembler-not {\tmovprfx\t} } } */
