This patch introduces six new vector cbranch optabs
1. vec_cbranch_any and vec_cbranch_all.
2. cond_vec_cbranch_any and cond_vec_cbranch_all.
3. cond_len_vec_cbranch_any and cond_len_vec_cbranch_all.
Today cbranch can be used for both vector and scalar modes. In both these
cases it's intended to compare boolean values, either scalar or vector.
The optab documentation does not however state that it can only handle
comparisons against 0. So many targets have added code for the vector variant
that tries to deal with the case where we branch based on two non-zero
registers.
However this code can't ever be reached because the cbranch expansion only deals
with comparisons against 0 for vectors. This is because for vectors the rest of
the compiler has no way to generate a non-zero comparison. e.g. the vectorizer
will always generate a zero comparison, and the C/C++ front-ends won't allow
vectors to be used in a cbranch as it expects a boolean value. ISAs like SVE
work around this by requiring you to use an SVE PTEST intrinsics which results
in a single scalar boolean value that represents the flag values.
e.g. if (svptest_any (..))
The natural question is why do we not at expand time then rewrite the comparison
to a non-zero comparison if the target supports it.
The reason is we can't safely do so. For an ANY comparison (e.g. != b) this is
trivial, but for an ALL comparison (e.g. == b) we would have to flip both branch
and invert the value being compared. i.e. we have to make it a != b comparison.
But in emit_cmp_and_jump_insns we can't flip the branches anymore because they
have already been lowered into a fall through branch (PC) and a label, ready for
use in an if_then_else RTL expression.
Now why does any of this matter? Well there are three optimizations we want to
be
able to do.
1. Adv. SIMD does not support a vector !=, as in there's no instruction for it.
For both Integer and FP vectors we perform the comparisons as EQ and then
invert the resulting mask. Ideally we'd like to replace this with just a XOR
and the appropriate branch.
2. When on an SVE enabled system we would like to use an SVE compare + branch
for the Adv. SIMD sequence which could happen due to cost modelling. However
we can only do so based on if we know that the values being compared against
are the boolean masks. This means we can't really use combine to do this
because combine would have to match the entire sequence including the
vector comparisons because at RTL we've lost the information that
VECTOR_BOOLEAN_P would have given us. This sequence would be too long for
combine to match due to it having to match the compare + branch sequence
being generated as well. It also becomes a bit messy to match ANY and ALL
sequences.
3. For SVE systems we would like to avoid generating the PTEST operation
whenever possible. Because SVE vector integer comparisons already set flags
we don't need the PTEST on an any or all check. Eliminating this in RTL is
difficult, so the best approach is to not generate the PTEST at all when not
needed.
To handle these three cases the new optabs are added and the current cbranch is
no longer required if the target does not need help in distinguishing between
boolean vector vs data vector operands.
This difference is not important for correctness, but it is for optimization.
So I've chosen not to deprecate the cbranch_optab but make it completely
optional.
I'll try to explain why:
An example is when unrolling is done on Adv. SIMD early break loops.
We generate
vect__1.8_29 = MEM <vector(4) int> [(int *)_25];
vect__1.9_31 = MEM <vector(4) int> [(int *)_25 + 16B];
mask_patt_10.10_32 = vect__1.8_29 == { 124, 124, 124, 124 };
mask_patt_10.10_33 = vect__1.9_31 == { 124, 124, 124, 124 };
vexit_reduc_34 = .VEC_TRUNC_ADD_HIGH (mask_patt_10.10_33, mask_patt_10.10_32);
if (vexit_reduc_34 != { 0, 0, 0, 0 })
goto <bb 4>; [5.50%]
else
goto <bb 18>; [94.50%]
And so the new optabs aren't immediately useful because the comparisons can't
be done by the optab itself.
As such vec_cbranch_any would be called with vexit_reduc_34 and { 0, 0, 0, 0 }
however since this expects to perform the comparison itself we end up with
ldp q30, q31, [x0], 32
cmeq v30.4s, v30.4s, v27.4s
cmeq v31.4s, v31.4s, v27.4s
addhn v31.4h, v31.4s, v30.4s
cmtst v31.4h, v31.4h, v31.4h
fmov x3, d31
cbz x3, .L2
instead of
ldp q30, q31, [x0], 32
cmeq v30.4s, v30.4s, v27.4s
cmeq v31.4s, v31.4s, v27.4s
addhn v31.4h, v31.4s, v30.4s
fmov x3, d31
cbz x3, .L2
because we don't know that the value is already a boolean -1/0 value. Without
this we can't safely not perform the compare.
The conversion is needed because e.g. it's not valid to drop the compare with
zero when the vector just contains data:
v30.8h = [ 0x0001, 0x0002, 0x0003, 0x0004, 0x0005, 0x0006, 0x0007, 0x0008 ]
cmeq v31.8h, v30.8h, #0 // -> v31.8h = [0,0,0,0,0,0,0,0]
umaxp v31.4s, v31.4s, v31.4s // pairwise-OR over 0/FFFF masks -> still
[0,0,0,0]
fmov x7, d31 // x7 = 0
cbnz x7, .L6 // NOT taken (correct: there were no zeros)
vs
umaxp v31.4s, v31.4s, v31.4s // pairwise unsigned max:
// [ max(0x00020001,0x00040003)=0x00040003,
// max(0x00060005,0x00080007)=0x00080007,
... ]
fmov x7, d31 // x7 = 0x0008000700040003 (non-zero)
cbnz x7, .L66 // TAKEN
As such, to avoid the extra compare on boolean vectors, we still need the
cbranch_optab or the new vec_cbranch_* optabs need an extre operand to indicate
what kind of data they hold. Note that this isn't an issue for SVE because
SVE has BImode for booleans.
With these two optabs it's trivial to implement all the optimizations I
described above.
I.e. with them we can now generate
.L2:
ldr q31, [x1, x2]
add v29.4s, v29.4s, v25.4s
add v28.4s, v28.4s, v26.4s
add v31.4s, v31.4s, v30.4s
str q31, [x1, x2]
add x1, x1, 16
cmp x1, 2560
beq .L1
.L6:
ldr q30, [x3, x1]
cmpeq p15.s, p7/z, z30.s, z27.s
b.none .L2
and easily prove it correct.
Bootstrapped Regtested on aarch64-none-linux-gnu,
arm-none-linux-gnueabihf, x86_64-pc-linux-gnu
-m32, -m64 and no issues.
Ok for master?
Thanks,
Tamar
gcc/ChangeLog:
PR target/118974
* optabs.def (vec_cbranch_any_optab, vec_cbranch_all_optab,
cond_vec_cbranch_any_optab, cond_vec_cbranch_all_optab,
cond_len_vec_cbranch_any_optab, cond_len_vec_cbranch_all_optab): New.
* doc/md.texi: Document them.
* optabs.cc (prepare_cmp_insn): Refactor to take optab to check for
instead of hardcoded cbranch and support mask and len.
(emit_cmp_and_jump_insn_1, emit_cmp_and_jump_insns): Implement them.
(emit_conditional_move, emit_conditional_add, gen_cond_trap): Update
after changing function signatures to support new optabs.
---
diff --git a/gcc/doc/md.texi b/gcc/doc/md.texi
index
ae5d709bd47945272e6f45f83840e21c68bb6534..e668048a387e146b072d414168c5ed6db3707609
100644
--- a/gcc/doc/md.texi
+++ b/gcc/doc/md.texi
@@ -7664,8 +7664,65 @@ position of Operand 1 to test. Operand 3 is the
@code{code_label} to jump to.
Conditional branch instruction combined with a compare instruction.
Operand 0 is a comparison operator. Operand 1 and operand 2 are the
first and second operands of the comparison, respectively. Operand 3
+is the @code{code_label} to jump to. This optab is only used for comparisons
of
+VECTOR_BOOLEAN_TYPE_P values and it never called for data-registers. Data
+vector operands should use one of the patterns below instead.
+
+@cindex @code{vec_cbranch_any@var{mode}} instruction pattern
+@item @samp{vec_cbranch_any@var{mode}}
+Conditional branch instruction based on a vector compare that branches
+when at least one of the elementwise comparisons of the two input
+vectors is true.
+Operand 0 is a comparison operator. Operand 1 and operand 2 are the
+first and second operands of the comparison, respectively. Operand 3
is the @code{code_label} to jump to.
+@cindex @code{vec_cbranch_all@var{mode}} instruction pattern
+@item @samp{vec_cbranch_all@var{mode}}
+Conditional branch instruction based on a vector compare that branches
+when all of the elementwise comparisons of the two input vectors is true.
+Operand 0 is a comparison operator. Operand 1 and operand 2 are the
+first and second operands of the comparison, respectively. Operand 3
+is the @code{code_label} to jump to.
+
+@cindex @code{cond_vec_cbranch_any@var{mode}} instruction pattern
+@item @samp{cond_vec_cbranch_any@var{mode}}
+Masked conditional branch instruction based on a vector compare that branches
+when at least one of the elementwise comparisons of the two input
+vectors is true.
+Operand 0 is a comparison operator. Operand 1 is the mask operand.
+Operand 2 and operand 3 are the first and second operands of the comparison,
+respectively. Operand 4 is the else value for the masked operation.
+Operand 5 is the @code{code_label} to jump to.
+
+@cindex @code{cond_vec_cbranch_all@var{mode}} instruction pattern
+@item @samp{cond_vec_cbranch_all@var{mode}}
+Masked conditional branch instruction based on a vector compare that branches
+when all of the elementwise comparisons of the two input vectors is true.
+Operand 0 is a comparison operator. Operand 1 is the mask operand.
+Operand 2 and operand 3 are the first and second operands of the comparison,
+respectively. Operand 4 is the else value for the masked operation.
+Operand 5 is the @code{code_label} to jump to.
+
+@cindex @code{cond_len_vec_cbranch_any@var{mode}} instruction pattern
+@item @samp{cond_len_vec_cbranch_any@var{mode}}
+Len based conditional branch instruction based on a vector compare that
branches
+when at least one of the elementwise comparisons of the two input
+vectors is true.
+Operand 0 is a comparison operator. Operand 1 and operand 2 are the first and
+second operands of the comparison, respectively. Operand 3 is the len operand.
+Operand 4 is the else value for the masked operation. Operand 5 is the
+@code{code_label} to jump to.
+
+@cindex @code{cond_len_vec_cbranch_all@var{mode}} instruction pattern
+@item @samp{cond_len_vec_cbranch_all@var{mode}}
+Len based conditional branch instruction based on a vector compare that
branches
+when all of the elementwise comparisons of the two input vectors is true.
+Operand 0 is a comparison operator. Operand 1 and operand 2 are the first and
+second operands of the comparison, respectively. Operand 3 is the len operand.
+Operand 4 is the else value for the masked operation. Operand 5 is the
+@code{code_label} to jump to.
+
@cindex @code{jump} instruction pattern
@item @samp{jump}
A jump inside a function; an unconditional branch. Operand 0 is the
diff --git a/gcc/optabs.cc b/gcc/optabs.cc
index
0865fc2e19aeb2b3056c8634334d6c1644a3cc96..1072239fef086e4ed959e472f299ed048fd507ad
100644
--- a/gcc/optabs.cc
+++ b/gcc/optabs.cc
@@ -48,6 +48,8 @@ along with GCC; see the file COPYING3. If not see
#include "langhooks.h"
#include "gimple.h"
#include "ssa.h"
+#include "tree-ssa-live.h"
+#include "tree-outof-ssa.h"
static void prepare_float_lib_cmp (rtx, rtx, enum rtx_code, rtx *,
machine_mode *);
@@ -4405,6 +4407,9 @@ can_vec_extract_var_idx_p (machine_mode vec_mode,
machine_mode extr_mode)
*PMODE is the mode of the inputs (in case they are const_int).
+ *OPTAB is the optab to check for OPTAB_DIRECT support. Defaults to
+ cbranch_optab.
+
This function performs all the setup necessary so that the caller only has
to emit a single comparison insn. This setup can involve doing a BLKmode
comparison or emitting a library call to perform the comparison if no insn
@@ -4414,9 +4419,9 @@ can_vec_extract_var_idx_p (machine_mode vec_mode,
machine_mode extr_mode)
comparisons must have already been folded. */
static void
-prepare_cmp_insn (rtx x, rtx y, enum rtx_code comparison, rtx size,
+prepare_cmp_insn (rtx x, rtx y, rtx *mask, enum rtx_code comparison, rtx size,
int unsignedp, enum optab_methods methods,
- rtx *ptest, machine_mode *pmode)
+ rtx *ptest, machine_mode *pmode, optab optab)
{
machine_mode mode = *pmode;
rtx libfunc, test;
@@ -4534,7 +4539,7 @@ prepare_cmp_insn (rtx x, rtx y, enum rtx_code comparison,
rtx size,
FOR_EACH_WIDER_MODE_FROM (cmp_mode, mode)
{
enum insn_code icode;
- icode = optab_handler (cbranch_optab, cmp_mode);
+ icode = optab_handler (optab, cmp_mode);
if (icode != CODE_FOR_nothing
&& insn_operand_matches (icode, 0, test))
{
@@ -4566,8 +4571,8 @@ prepare_cmp_insn (rtx x, rtx y, enum rtx_code comparison,
rtx size,
/* Small trick if UNORDERED isn't implemented by the hardware. */
if (comparison == UNORDERED && rtx_equal_p (x, y))
{
- prepare_cmp_insn (x, y, UNLT, NULL_RTX, unsignedp, OPTAB_WIDEN,
- ptest, pmode);
+ prepare_cmp_insn (x, y, mask, UNLT, NULL_RTX, unsignedp, OPTAB_WIDEN,
+ ptest, pmode, optab);
if (*ptest)
return;
}
@@ -4618,8 +4623,8 @@ prepare_cmp_insn (rtx x, rtx y, enum rtx_code comparison,
rtx size,
}
*pmode = ret_mode;
- prepare_cmp_insn (x, y, comparison, NULL_RTX, unsignedp, methods,
- ptest, pmode);
+ prepare_cmp_insn (x, y, mask, comparison, NULL_RTX, unsignedp, methods,
+ ptest, pmode, optab);
}
return;
@@ -4657,9 +4662,10 @@ prepare_operand (enum insn_code icode, rtx x, int opnum,
machine_mode mode,
we can do the branch. */
static void
-emit_cmp_and_jump_insn_1 (rtx test, machine_mode mode, rtx label,
- direct_optab cmp_optab, profile_probability prob,
- bool test_branch)
+emit_cmp_and_jump_insn_1 (rtx test, rtx cond, rtx inactive, machine_mode mode,
+ rtx label, direct_optab cmp_optab,
+ profile_probability prob, bool test_branch,
+ bool len_op)
{
machine_mode optab_mode;
enum mode_class mclass;
@@ -4672,12 +4678,20 @@ emit_cmp_and_jump_insn_1 (rtx test, machine_mode mode,
rtx label,
gcc_assert (icode != CODE_FOR_nothing);
gcc_assert (test_branch || insn_operand_matches (icode, 0, test));
+ gcc_assert (cond == NULL_RTX || (cond != NULL_RTX && !test_branch));
if (test_branch)
insn = emit_jump_insn (GEN_FCN (icode) (XEXP (test, 0),
XEXP (test, 1), label));
- else
+ else if (cond == NULL_RTX)
insn = emit_jump_insn (GEN_FCN (icode) (test, XEXP (test, 0),
XEXP (test, 1), label));
+ else if (len_op)
+ insn = emit_jump_insn (GEN_FCN (icode) (test, XEXP (test, 0),
+ XEXP (test, 1), cond, inactive,
+ label));
+ else
+ insn = emit_jump_insn (GEN_FCN (icode) (test, cond, XEXP (test, 0),
+ XEXP (test, 1), inactive, label));
if (prob.initialized_p ()
&& profile_status_for_fn (cfun) != PROFILE_ABSENT
@@ -4796,22 +4810,202 @@ emit_cmp_and_jump_insns (rtx x, rtx y, enum rtx_code
comparison, rtx size,
if (unsignedp)
comparison = unsigned_condition (comparison);
- prepare_cmp_insn (op0, op1, comparison, size, unsignedp, OPTAB_LIB_WIDEN,
- &test, &mode);
+ /* cbranch is no longer allowed for vectors, so when using a vector mode
+ check vec_cbranch variants instead. */
+ if (!VECTOR_MODE_P (GET_MODE (op0)))
+ prepare_cmp_insn (op0, op1, NULL, comparison, size, unsignedp,
+ OPTAB_LIB_WIDEN, &test, &mode, cbranch_optab);
/* Check if we're comparing a truth type with 0, and if so check if
the target supports tbranch. */
machine_mode tmode = mode;
direct_optab optab;
- if (op1 == CONST0_RTX (GET_MODE (op1))
- && validate_test_and_branch (val, &test, &tmode,
- &optab) != CODE_FOR_nothing)
+ if (op1 == CONST0_RTX (GET_MODE (op1)))
{
- emit_cmp_and_jump_insn_1 (test, tmode, label, optab, prob, true);
- return;
+ if (!VECTOR_MODE_P (GET_MODE (op1))
+ && validate_test_and_branch (val, &test, &tmode,
+ &optab) != CODE_FOR_nothing)
+ {
+ emit_cmp_and_jump_insn_1 (test, NULL_RTX, NULL_RTX, tmode, label,
+ optab, prob, true, false);
+ return;
+ }
+
+ /* If we are comparing equality with 0, check if VAL is another equality
+ comparison and if the target supports it directly. */
+ gimple *def_stmt = NULL;
+ if (val && TREE_CODE (val) == SSA_NAME
+ && VECTOR_BOOLEAN_TYPE_P (TREE_TYPE (val))
+ && (comparison == NE || comparison == EQ)
+ && (def_stmt = get_gimple_for_ssa_name (val)))
+ {
+ tree masked_op = NULL_TREE;
+ tree len_op = NULL_TREE;
+ tree len_else_op = NULL_TREE;
+ /* First determine if the operation should be masked or unmasked. */
+ if (is_gimple_assign (def_stmt)
+ && gimple_assign_rhs_code (def_stmt) == BIT_AND_EXPR)
+ {
+ /* See if one side if a comparison, if so use the other side as
+ the mask. */
+ gimple *mask_def = NULL;
+ tree rhs1 = gimple_assign_rhs1 (def_stmt);
+ tree rhs2 = gimple_assign_rhs2 (def_stmt);
+ if ((mask_def = get_gimple_for_ssa_name (rhs1))
+ && is_gimple_assign (mask_def)
+ && TREE_CODE_CLASS (gimple_assign_rhs_code (mask_def)))
+ masked_op = rhs2;
+ else if ((mask_def = get_gimple_for_ssa_name (rhs2))
+ && is_gimple_assign (mask_def)
+ && TREE_CODE_CLASS (gimple_assign_rhs_code (mask_def)))
+ masked_op = rhs1;
+
+ if (masked_op)
+ def_stmt = mask_def;
+ }
+ /* Else check to see if we're a LEN target. */
+ else if (is_gimple_call (def_stmt)
+ && gimple_call_internal_p (def_stmt)
+ && gimple_call_internal_fn (def_stmt) == IFN_VCOND_MASK_LEN)
+ {
+ /* Example to consume:
+
+ a = _59 != vect__4.17_75;
+ vcmp = .VCOND_MASK_LEN (a, { -1, ... }, { 0, ... }, _90, 0);
+ if (vcmp != { 0, ... })
+
+ and transform into
+
+ if (cond_len_vec_cbranch_any (a, _90, 0)). */
+ gcall *call = dyn_cast <gcall *> (def_stmt);
+ tree true_branch = gimple_call_arg (call, 1);
+ tree false_branch = gimple_call_arg (call, 2);
+ if (integer_minus_onep (true_branch)
+ && integer_zerop (false_branch))
+ {
+ len_op = gimple_call_arg (call, 3);
+ len_else_op = gimple_call_arg (call, 4);
+
+ def_stmt = SSA_NAME_DEF_STMT (gimple_call_arg (call, 0));
+ }
+ }
+
+ bool cond_op = masked_op || len_op;
+ enum insn_code icode;
+ if (is_gimple_assign (def_stmt)
+ && TREE_CODE_CLASS (gimple_assign_rhs_code (def_stmt))
+ == tcc_comparison)
+ {
+ class expand_operand ops[4];
+ rtx_insn *tmp = NULL;
+ start_sequence ();
+ rtx op0c = expand_normal (gimple_assign_rhs1 (def_stmt));
+ rtx op1c = expand_normal (gimple_assign_rhs2 (def_stmt));
+ machine_mode mode2 = GET_MODE (op0c);
+
+ int nops = cond_op ? 4 : 2;
+ int offset = masked_op ? 1 : 0;
+ create_input_operand (&ops[offset + 0], op0c, mode2);
+ create_input_operand (&ops[offset + 1], op1c, mode2);
+ if (masked_op)
+ {
+ rtx mask_op = expand_normal (masked_op);
+ auto mask_mode = GET_MODE (mask_op);
+ create_input_operand (&ops[0], mask_op, mask_mode);
+ create_input_operand (&ops[3], CONST0_RTX (mask_mode),
+ mask_mode);
+ }
+ else if (len_op)
+ {
+ rtx len_op2 = expand_normal (len_op);
+ rtx len_else_op2 = expand_normal (len_else_op);
+ create_input_operand (&ops[2], len_op2, GET_MODE (len_op2));
+ create_input_operand (&ops[3], len_else_op2,
+ GET_MODE (len_else_op2));
+ }
+
+ int unsignedp2 = TYPE_UNSIGNED (TREE_TYPE (val));
+ auto inner_code = gimple_assign_rhs_code (def_stmt);
+ rtx test2 = NULL_RTX;
+
+ enum rtx_code comparison2 = get_rtx_code (inner_code, unsignedp2);
+ if (unsignedp2)
+ comparison2 = unsigned_condition (comparison2);
+ if (comparison == NE)
+ optab = masked_op ? cond_vec_cbranch_any_optab
+ : len_op ? cond_len_vec_cbranch_any_optab
+ : vec_cbranch_any_optab;
+ else
+ optab = masked_op ? cond_vec_cbranch_all_optab
+ : len_op ? cond_len_vec_cbranch_all_optab
+ : vec_cbranch_all_optab;
+
+ if ((icode = optab_handler (optab, mode2))
+ != CODE_FOR_nothing
+ && maybe_legitimize_operands (icode, 1, nops, ops))
+ {
+ rtx cond = masked_op ? ops[0].value
+ : len_op ? ops[2].value : NULL_RTX;
+ rtx inactive
+ = masked_op || len_op ? ops[3].value : NULL_RTX;
+ test2 = gen_rtx_fmt_ee (comparison2, VOIDmode,
+ ops[offset + 0].value,
+ ops[offset + 1].value);
+ if (insn_operand_matches (icode, 0, test2))
+ {
+ emit_cmp_and_jump_insn_1 (test2, cond, inactive, mode2,
+ label, optab, prob, false,
+ len_op);
+ tmp = get_insns ();
+ }
+ }
+
+ end_sequence ();
+ if (tmp)
+ {
+ emit_insn (tmp);
+ return;
+ }
+ }
+ }
+ }
+
+ /* cbranch should only be used for VECTOR_BOOLEAN_TYPE_P values. */
+ direct_optab base_optab = cbranch_optab;
+ if (VECTOR_MODE_P (GET_MODE (op0)))
+ {
+ /* If cbranch is provided, use it. If we get here it means we have an
+ instruction in between what created the boolean value and the gcond
+ that is not a masking operation. This can happen for instance during
+ unrolling of early-break where we have an OR-reduction to reduce the
+ masks. In this case knowing we have a mask can let us generate better
+ code. If it's not there there then check the vector specific
+ optabs. */
+ if (optab_handler (cbranch_optab, mode) == CODE_FOR_nothing)
+ {
+ if (comparison == NE)
+ base_optab = vec_cbranch_any_optab;
+ else
+ base_optab = vec_cbranch_all_optab;
+
+ prepare_cmp_insn (op0, op1, NULL, comparison, size, unsignedp,
+ OPTAB_DIRECT, &test, &mode, base_optab);
+
+ enum insn_code icode = optab_handler (base_optab, mode);
+
+ /* If the new cbranch isn't supported, degrade back to old one. */
+ if (icode == CODE_FOR_nothing
+ || !test
+ || !insn_operand_matches (icode, 0, test))
+ base_optab = cbranch_optab;
+ }
+
+ prepare_cmp_insn (op0, op1, NULL, comparison, size, unsignedp,
+ OPTAB_LIB_WIDEN, &test, &mode, base_optab);
}
- emit_cmp_and_jump_insn_1 (test, mode, label, cbranch_optab, prob, false);
+ emit_cmp_and_jump_insn_1 (test, NULL_RTX, NULL_RTX, mode, label, base_optab,
+ prob, false, false);
}
/* Overloaded version of emit_cmp_and_jump_insns in which VAL is unknown. */
@@ -5099,9 +5293,9 @@ emit_conditional_move (rtx target, struct rtx_comparison
comp,
else if (rtx_equal_p (orig_op1, op3))
op3p = XEXP (comparison, 1) = force_reg (cmpmode, orig_op1);
}
- prepare_cmp_insn (XEXP (comparison, 0), XEXP (comparison, 1),
+ prepare_cmp_insn (XEXP (comparison, 0), XEXP (comparison, 1), NULL,
GET_CODE (comparison), NULL_RTX, unsignedp,
- OPTAB_WIDEN, &comparison, &cmpmode);
+ OPTAB_WIDEN, &comparison, &cmpmode, cbranch_optab);
if (comparison)
{
rtx res = emit_conditional_move_1 (target, comparison,
@@ -5316,9 +5510,9 @@ emit_conditional_add (rtx target, enum rtx_code code, rtx
op0, rtx op1,
do_pending_stack_adjust ();
last = get_last_insn ();
- prepare_cmp_insn (XEXP (comparison, 0), XEXP (comparison, 1),
- GET_CODE (comparison), NULL_RTX, unsignedp, OPTAB_WIDEN,
- &comparison, &cmode);
+ prepare_cmp_insn (XEXP (comparison, 0), XEXP (comparison, 1), NULL,
+ GET_CODE (comparison), NULL_RTX, unsignedp, OPTAB_WIDEN,
+ &comparison, &cmode, cbranch_optab);
if (comparison)
{
class expand_operand ops[4];
@@ -6132,8 +6326,8 @@ gen_cond_trap (enum rtx_code code, rtx op1, rtx op2, rtx
tcode)
do_pending_stack_adjust ();
start_sequence ();
- prepare_cmp_insn (op1, op2, code, NULL_RTX, false, OPTAB_DIRECT,
- &trap_rtx, &mode);
+ prepare_cmp_insn (op1, op2, NULL, code, NULL_RTX, false, OPTAB_DIRECT,
+ &trap_rtx, &mode, cbranch_optab);
if (!trap_rtx)
insn = NULL;
else
diff --git a/gcc/optabs.def b/gcc/optabs.def
index
b6f290a95130cd53e94af2249c02a53f01ca3890..371514f3dbe41f1336475f99d1b837c24fa3b818
100644
--- a/gcc/optabs.def
+++ b/gcc/optabs.def
@@ -268,6 +268,8 @@ OPTAB_D (cond_fms_optab, "cond_fms$a")
OPTAB_D (cond_fnma_optab, "cond_fnma$a")
OPTAB_D (cond_fnms_optab, "cond_fnms$a")
OPTAB_D (cond_neg_optab, "cond_neg$a")
+OPTAB_D (cond_vec_cbranch_any_optab, "cond_vec_cbranch_any$a")
+OPTAB_D (cond_vec_cbranch_all_optab, "cond_vec_cbranch_all$a")
OPTAB_D (cond_one_cmpl_optab, "cond_one_cmpl$a")
OPTAB_D (cond_len_add_optab, "cond_len_add$a")
OPTAB_D (cond_len_sub_optab, "cond_len_sub$a")
@@ -295,6 +297,8 @@ OPTAB_D (cond_len_fnma_optab, "cond_len_fnma$a")
OPTAB_D (cond_len_fnms_optab, "cond_len_fnms$a")
OPTAB_D (cond_len_neg_optab, "cond_len_neg$a")
OPTAB_D (cond_len_one_cmpl_optab, "cond_len_one_cmpl$a")
+OPTAB_D (cond_len_vec_cbranch_any_optab, "cond_len_vec_cbranch_any$a")
+OPTAB_D (cond_len_vec_cbranch_all_optab, "cond_len_vec_cbranch_all$a")
OPTAB_D (vcond_mask_len_optab, "vcond_mask_len_$a")
OPTAB_D (cstore_optab, "cstore$a4")
OPTAB_D (ctrap_optab, "ctrap$a4")
@@ -427,6 +431,8 @@ OPTAB_D (smulhrs_optab, "smulhrs$a3")
OPTAB_D (umulhs_optab, "umulhs$a3")
OPTAB_D (umulhrs_optab, "umulhrs$a3")
OPTAB_D (sdiv_pow2_optab, "sdiv_pow2$a3")
+OPTAB_D (vec_cbranch_any_optab, "vec_cbranch_any$a")
+OPTAB_D (vec_cbranch_all_optab, "vec_cbranch_all$a")
OPTAB_D (vec_pack_sfix_trunc_optab, "vec_pack_sfix_trunc_$a")
OPTAB_D (vec_pack_ssat_optab, "vec_pack_ssat_$a")
OPTAB_D (vec_pack_trunc_optab, "vec_pack_trunc_$a")
--
diff --git a/gcc/doc/md.texi b/gcc/doc/md.texi
index ae5d709bd47945272e6f45f83840e21c68bb6534..e668048a387e146b072d414168c5ed6db3707609 100644
--- a/gcc/doc/md.texi
+++ b/gcc/doc/md.texi
@@ -7664,8 +7664,65 @@ position of Operand 1 to test. Operand 3 is the @code{code_label} to jump to.
Conditional branch instruction combined with a compare instruction.
Operand 0 is a comparison operator. Operand 1 and operand 2 are the
first and second operands of the comparison, respectively. Operand 3
+is the @code{code_label} to jump to. This optab is only used for comparisons of
+VECTOR_BOOLEAN_TYPE_P values and it never called for data-registers. Data
+vector operands should use one of the patterns below instead.
+
+@cindex @code{vec_cbranch_any@var{mode}} instruction pattern
+@item @samp{vec_cbranch_any@var{mode}}
+Conditional branch instruction based on a vector compare that branches
+when at least one of the elementwise comparisons of the two input
+vectors is true.
+Operand 0 is a comparison operator. Operand 1 and operand 2 are the
+first and second operands of the comparison, respectively. Operand 3
is the @code{code_label} to jump to.
+@cindex @code{vec_cbranch_all@var{mode}} instruction pattern
+@item @samp{vec_cbranch_all@var{mode}}
+Conditional branch instruction based on a vector compare that branches
+when all of the elementwise comparisons of the two input vectors is true.
+Operand 0 is a comparison operator. Operand 1 and operand 2 are the
+first and second operands of the comparison, respectively. Operand 3
+is the @code{code_label} to jump to.
+
+@cindex @code{cond_vec_cbranch_any@var{mode}} instruction pattern
+@item @samp{cond_vec_cbranch_any@var{mode}}
+Masked conditional branch instruction based on a vector compare that branches
+when at least one of the elementwise comparisons of the two input
+vectors is true.
+Operand 0 is a comparison operator. Operand 1 is the mask operand.
+Operand 2 and operand 3 are the first and second operands of the comparison,
+respectively. Operand 4 is the else value for the masked operation.
+Operand 5 is the @code{code_label} to jump to.
+
+@cindex @code{cond_vec_cbranch_all@var{mode}} instruction pattern
+@item @samp{cond_vec_cbranch_all@var{mode}}
+Masked conditional branch instruction based on a vector compare that branches
+when all of the elementwise comparisons of the two input vectors is true.
+Operand 0 is a comparison operator. Operand 1 is the mask operand.
+Operand 2 and operand 3 are the first and second operands of the comparison,
+respectively. Operand 4 is the else value for the masked operation.
+Operand 5 is the @code{code_label} to jump to.
+
+@cindex @code{cond_len_vec_cbranch_any@var{mode}} instruction pattern
+@item @samp{cond_len_vec_cbranch_any@var{mode}}
+Len based conditional branch instruction based on a vector compare that branches
+when at least one of the elementwise comparisons of the two input
+vectors is true.
+Operand 0 is a comparison operator. Operand 1 and operand 2 are the first and
+second operands of the comparison, respectively. Operand 3 is the len operand.
+Operand 4 is the else value for the masked operation. Operand 5 is the
+@code{code_label} to jump to.
+
+@cindex @code{cond_len_vec_cbranch_all@var{mode}} instruction pattern
+@item @samp{cond_len_vec_cbranch_all@var{mode}}
+Len based conditional branch instruction based on a vector compare that branches
+when all of the elementwise comparisons of the two input vectors is true.
+Operand 0 is a comparison operator. Operand 1 and operand 2 are the first and
+second operands of the comparison, respectively. Operand 3 is the len operand.
+Operand 4 is the else value for the masked operation. Operand 5 is the
+@code{code_label} to jump to.
+
@cindex @code{jump} instruction pattern
@item @samp{jump}
A jump inside a function; an unconditional branch. Operand 0 is the
diff --git a/gcc/optabs.cc b/gcc/optabs.cc
index 0865fc2e19aeb2b3056c8634334d6c1644a3cc96..1072239fef086e4ed959e472f299ed048fd507ad 100644
--- a/gcc/optabs.cc
+++ b/gcc/optabs.cc
@@ -48,6 +48,8 @@ along with GCC; see the file COPYING3. If not see
#include "langhooks.h"
#include "gimple.h"
#include "ssa.h"
+#include "tree-ssa-live.h"
+#include "tree-outof-ssa.h"
static void prepare_float_lib_cmp (rtx, rtx, enum rtx_code, rtx *,
machine_mode *);
@@ -4405,6 +4407,9 @@ can_vec_extract_var_idx_p (machine_mode vec_mode, machine_mode extr_mode)
*PMODE is the mode of the inputs (in case they are const_int).
+ *OPTAB is the optab to check for OPTAB_DIRECT support. Defaults to
+ cbranch_optab.
+
This function performs all the setup necessary so that the caller only has
to emit a single comparison insn. This setup can involve doing a BLKmode
comparison or emitting a library call to perform the comparison if no insn
@@ -4414,9 +4419,9 @@ can_vec_extract_var_idx_p (machine_mode vec_mode, machine_mode extr_mode)
comparisons must have already been folded. */
static void
-prepare_cmp_insn (rtx x, rtx y, enum rtx_code comparison, rtx size,
+prepare_cmp_insn (rtx x, rtx y, rtx *mask, enum rtx_code comparison, rtx size,
int unsignedp, enum optab_methods methods,
- rtx *ptest, machine_mode *pmode)
+ rtx *ptest, machine_mode *pmode, optab optab)
{
machine_mode mode = *pmode;
rtx libfunc, test;
@@ -4534,7 +4539,7 @@ prepare_cmp_insn (rtx x, rtx y, enum rtx_code comparison, rtx size,
FOR_EACH_WIDER_MODE_FROM (cmp_mode, mode)
{
enum insn_code icode;
- icode = optab_handler (cbranch_optab, cmp_mode);
+ icode = optab_handler (optab, cmp_mode);
if (icode != CODE_FOR_nothing
&& insn_operand_matches (icode, 0, test))
{
@@ -4566,8 +4571,8 @@ prepare_cmp_insn (rtx x, rtx y, enum rtx_code comparison, rtx size,
/* Small trick if UNORDERED isn't implemented by the hardware. */
if (comparison == UNORDERED && rtx_equal_p (x, y))
{
- prepare_cmp_insn (x, y, UNLT, NULL_RTX, unsignedp, OPTAB_WIDEN,
- ptest, pmode);
+ prepare_cmp_insn (x, y, mask, UNLT, NULL_RTX, unsignedp, OPTAB_WIDEN,
+ ptest, pmode, optab);
if (*ptest)
return;
}
@@ -4618,8 +4623,8 @@ prepare_cmp_insn (rtx x, rtx y, enum rtx_code comparison, rtx size,
}
*pmode = ret_mode;
- prepare_cmp_insn (x, y, comparison, NULL_RTX, unsignedp, methods,
- ptest, pmode);
+ prepare_cmp_insn (x, y, mask, comparison, NULL_RTX, unsignedp, methods,
+ ptest, pmode, optab);
}
return;
@@ -4657,9 +4662,10 @@ prepare_operand (enum insn_code icode, rtx x, int opnum, machine_mode mode,
we can do the branch. */
static void
-emit_cmp_and_jump_insn_1 (rtx test, machine_mode mode, rtx label,
- direct_optab cmp_optab, profile_probability prob,
- bool test_branch)
+emit_cmp_and_jump_insn_1 (rtx test, rtx cond, rtx inactive, machine_mode mode,
+ rtx label, direct_optab cmp_optab,
+ profile_probability prob, bool test_branch,
+ bool len_op)
{
machine_mode optab_mode;
enum mode_class mclass;
@@ -4672,12 +4678,20 @@ emit_cmp_and_jump_insn_1 (rtx test, machine_mode mode, rtx label,
gcc_assert (icode != CODE_FOR_nothing);
gcc_assert (test_branch || insn_operand_matches (icode, 0, test));
+ gcc_assert (cond == NULL_RTX || (cond != NULL_RTX && !test_branch));
if (test_branch)
insn = emit_jump_insn (GEN_FCN (icode) (XEXP (test, 0),
XEXP (test, 1), label));
- else
+ else if (cond == NULL_RTX)
insn = emit_jump_insn (GEN_FCN (icode) (test, XEXP (test, 0),
XEXP (test, 1), label));
+ else if (len_op)
+ insn = emit_jump_insn (GEN_FCN (icode) (test, XEXP (test, 0),
+ XEXP (test, 1), cond, inactive,
+ label));
+ else
+ insn = emit_jump_insn (GEN_FCN (icode) (test, cond, XEXP (test, 0),
+ XEXP (test, 1), inactive, label));
if (prob.initialized_p ()
&& profile_status_for_fn (cfun) != PROFILE_ABSENT
@@ -4796,22 +4810,202 @@ emit_cmp_and_jump_insns (rtx x, rtx y, enum rtx_code comparison, rtx size,
if (unsignedp)
comparison = unsigned_condition (comparison);
- prepare_cmp_insn (op0, op1, comparison, size, unsignedp, OPTAB_LIB_WIDEN,
- &test, &mode);
+ /* cbranch is no longer allowed for vectors, so when using a vector mode
+ check vec_cbranch variants instead. */
+ if (!VECTOR_MODE_P (GET_MODE (op0)))
+ prepare_cmp_insn (op0, op1, NULL, comparison, size, unsignedp,
+ OPTAB_LIB_WIDEN, &test, &mode, cbranch_optab);
/* Check if we're comparing a truth type with 0, and if so check if
the target supports tbranch. */
machine_mode tmode = mode;
direct_optab optab;
- if (op1 == CONST0_RTX (GET_MODE (op1))
- && validate_test_and_branch (val, &test, &tmode,
- &optab) != CODE_FOR_nothing)
+ if (op1 == CONST0_RTX (GET_MODE (op1)))
{
- emit_cmp_and_jump_insn_1 (test, tmode, label, optab, prob, true);
- return;
+ if (!VECTOR_MODE_P (GET_MODE (op1))
+ && validate_test_and_branch (val, &test, &tmode,
+ &optab) != CODE_FOR_nothing)
+ {
+ emit_cmp_and_jump_insn_1 (test, NULL_RTX, NULL_RTX, tmode, label,
+ optab, prob, true, false);
+ return;
+ }
+
+ /* If we are comparing equality with 0, check if VAL is another equality
+ comparison and if the target supports it directly. */
+ gimple *def_stmt = NULL;
+ if (val && TREE_CODE (val) == SSA_NAME
+ && VECTOR_BOOLEAN_TYPE_P (TREE_TYPE (val))
+ && (comparison == NE || comparison == EQ)
+ && (def_stmt = get_gimple_for_ssa_name (val)))
+ {
+ tree masked_op = NULL_TREE;
+ tree len_op = NULL_TREE;
+ tree len_else_op = NULL_TREE;
+ /* First determine if the operation should be masked or unmasked. */
+ if (is_gimple_assign (def_stmt)
+ && gimple_assign_rhs_code (def_stmt) == BIT_AND_EXPR)
+ {
+ /* See if one side if a comparison, if so use the other side as
+ the mask. */
+ gimple *mask_def = NULL;
+ tree rhs1 = gimple_assign_rhs1 (def_stmt);
+ tree rhs2 = gimple_assign_rhs2 (def_stmt);
+ if ((mask_def = get_gimple_for_ssa_name (rhs1))
+ && is_gimple_assign (mask_def)
+ && TREE_CODE_CLASS (gimple_assign_rhs_code (mask_def)))
+ masked_op = rhs2;
+ else if ((mask_def = get_gimple_for_ssa_name (rhs2))
+ && is_gimple_assign (mask_def)
+ && TREE_CODE_CLASS (gimple_assign_rhs_code (mask_def)))
+ masked_op = rhs1;
+
+ if (masked_op)
+ def_stmt = mask_def;
+ }
+ /* Else check to see if we're a LEN target. */
+ else if (is_gimple_call (def_stmt)
+ && gimple_call_internal_p (def_stmt)
+ && gimple_call_internal_fn (def_stmt) == IFN_VCOND_MASK_LEN)
+ {
+ /* Example to consume:
+
+ a = _59 != vect__4.17_75;
+ vcmp = .VCOND_MASK_LEN (a, { -1, ... }, { 0, ... }, _90, 0);
+ if (vcmp != { 0, ... })
+
+ and transform into
+
+ if (cond_len_vec_cbranch_any (a, _90, 0)). */
+ gcall *call = dyn_cast <gcall *> (def_stmt);
+ tree true_branch = gimple_call_arg (call, 1);
+ tree false_branch = gimple_call_arg (call, 2);
+ if (integer_minus_onep (true_branch)
+ && integer_zerop (false_branch))
+ {
+ len_op = gimple_call_arg (call, 3);
+ len_else_op = gimple_call_arg (call, 4);
+
+ def_stmt = SSA_NAME_DEF_STMT (gimple_call_arg (call, 0));
+ }
+ }
+
+ bool cond_op = masked_op || len_op;
+ enum insn_code icode;
+ if (is_gimple_assign (def_stmt)
+ && TREE_CODE_CLASS (gimple_assign_rhs_code (def_stmt))
+ == tcc_comparison)
+ {
+ class expand_operand ops[4];
+ rtx_insn *tmp = NULL;
+ start_sequence ();
+ rtx op0c = expand_normal (gimple_assign_rhs1 (def_stmt));
+ rtx op1c = expand_normal (gimple_assign_rhs2 (def_stmt));
+ machine_mode mode2 = GET_MODE (op0c);
+
+ int nops = cond_op ? 4 : 2;
+ int offset = masked_op ? 1 : 0;
+ create_input_operand (&ops[offset + 0], op0c, mode2);
+ create_input_operand (&ops[offset + 1], op1c, mode2);
+ if (masked_op)
+ {
+ rtx mask_op = expand_normal (masked_op);
+ auto mask_mode = GET_MODE (mask_op);
+ create_input_operand (&ops[0], mask_op, mask_mode);
+ create_input_operand (&ops[3], CONST0_RTX (mask_mode),
+ mask_mode);
+ }
+ else if (len_op)
+ {
+ rtx len_op2 = expand_normal (len_op);
+ rtx len_else_op2 = expand_normal (len_else_op);
+ create_input_operand (&ops[2], len_op2, GET_MODE (len_op2));
+ create_input_operand (&ops[3], len_else_op2,
+ GET_MODE (len_else_op2));
+ }
+
+ int unsignedp2 = TYPE_UNSIGNED (TREE_TYPE (val));
+ auto inner_code = gimple_assign_rhs_code (def_stmt);
+ rtx test2 = NULL_RTX;
+
+ enum rtx_code comparison2 = get_rtx_code (inner_code, unsignedp2);
+ if (unsignedp2)
+ comparison2 = unsigned_condition (comparison2);
+ if (comparison == NE)
+ optab = masked_op ? cond_vec_cbranch_any_optab
+ : len_op ? cond_len_vec_cbranch_any_optab
+ : vec_cbranch_any_optab;
+ else
+ optab = masked_op ? cond_vec_cbranch_all_optab
+ : len_op ? cond_len_vec_cbranch_all_optab
+ : vec_cbranch_all_optab;
+
+ if ((icode = optab_handler (optab, mode2))
+ != CODE_FOR_nothing
+ && maybe_legitimize_operands (icode, 1, nops, ops))
+ {
+ rtx cond = masked_op ? ops[0].value
+ : len_op ? ops[2].value : NULL_RTX;
+ rtx inactive
+ = masked_op || len_op ? ops[3].value : NULL_RTX;
+ test2 = gen_rtx_fmt_ee (comparison2, VOIDmode,
+ ops[offset + 0].value,
+ ops[offset + 1].value);
+ if (insn_operand_matches (icode, 0, test2))
+ {
+ emit_cmp_and_jump_insn_1 (test2, cond, inactive, mode2,
+ label, optab, prob, false,
+ len_op);
+ tmp = get_insns ();
+ }
+ }
+
+ end_sequence ();
+ if (tmp)
+ {
+ emit_insn (tmp);
+ return;
+ }
+ }
+ }
+ }
+
+ /* cbranch should only be used for VECTOR_BOOLEAN_TYPE_P values. */
+ direct_optab base_optab = cbranch_optab;
+ if (VECTOR_MODE_P (GET_MODE (op0)))
+ {
+ /* If cbranch is provided, use it. If we get here it means we have an
+ instruction in between what created the boolean value and the gcond
+ that is not a masking operation. This can happen for instance during
+ unrolling of early-break where we have an OR-reduction to reduce the
+ masks. In this case knowing we have a mask can let us generate better
+ code. If it's not there there then check the vector specific
+ optabs. */
+ if (optab_handler (cbranch_optab, mode) == CODE_FOR_nothing)
+ {
+ if (comparison == NE)
+ base_optab = vec_cbranch_any_optab;
+ else
+ base_optab = vec_cbranch_all_optab;
+
+ prepare_cmp_insn (op0, op1, NULL, comparison, size, unsignedp,
+ OPTAB_DIRECT, &test, &mode, base_optab);
+
+ enum insn_code icode = optab_handler (base_optab, mode);
+
+ /* If the new cbranch isn't supported, degrade back to old one. */
+ if (icode == CODE_FOR_nothing
+ || !test
+ || !insn_operand_matches (icode, 0, test))
+ base_optab = cbranch_optab;
+ }
+
+ prepare_cmp_insn (op0, op1, NULL, comparison, size, unsignedp,
+ OPTAB_LIB_WIDEN, &test, &mode, base_optab);
}
- emit_cmp_and_jump_insn_1 (test, mode, label, cbranch_optab, prob, false);
+ emit_cmp_and_jump_insn_1 (test, NULL_RTX, NULL_RTX, mode, label, base_optab,
+ prob, false, false);
}
/* Overloaded version of emit_cmp_and_jump_insns in which VAL is unknown. */
@@ -5099,9 +5293,9 @@ emit_conditional_move (rtx target, struct rtx_comparison comp,
else if (rtx_equal_p (orig_op1, op3))
op3p = XEXP (comparison, 1) = force_reg (cmpmode, orig_op1);
}
- prepare_cmp_insn (XEXP (comparison, 0), XEXP (comparison, 1),
+ prepare_cmp_insn (XEXP (comparison, 0), XEXP (comparison, 1), NULL,
GET_CODE (comparison), NULL_RTX, unsignedp,
- OPTAB_WIDEN, &comparison, &cmpmode);
+ OPTAB_WIDEN, &comparison, &cmpmode, cbranch_optab);
if (comparison)
{
rtx res = emit_conditional_move_1 (target, comparison,
@@ -5316,9 +5510,9 @@ emit_conditional_add (rtx target, enum rtx_code code, rtx op0, rtx op1,
do_pending_stack_adjust ();
last = get_last_insn ();
- prepare_cmp_insn (XEXP (comparison, 0), XEXP (comparison, 1),
- GET_CODE (comparison), NULL_RTX, unsignedp, OPTAB_WIDEN,
- &comparison, &cmode);
+ prepare_cmp_insn (XEXP (comparison, 0), XEXP (comparison, 1), NULL,
+ GET_CODE (comparison), NULL_RTX, unsignedp, OPTAB_WIDEN,
+ &comparison, &cmode, cbranch_optab);
if (comparison)
{
class expand_operand ops[4];
@@ -6132,8 +6326,8 @@ gen_cond_trap (enum rtx_code code, rtx op1, rtx op2, rtx tcode)
do_pending_stack_adjust ();
start_sequence ();
- prepare_cmp_insn (op1, op2, code, NULL_RTX, false, OPTAB_DIRECT,
- &trap_rtx, &mode);
+ prepare_cmp_insn (op1, op2, NULL, code, NULL_RTX, false, OPTAB_DIRECT,
+ &trap_rtx, &mode, cbranch_optab);
if (!trap_rtx)
insn = NULL;
else
diff --git a/gcc/optabs.def b/gcc/optabs.def
index b6f290a95130cd53e94af2249c02a53f01ca3890..371514f3dbe41f1336475f99d1b837c24fa3b818 100644
--- a/gcc/optabs.def
+++ b/gcc/optabs.def
@@ -268,6 +268,8 @@ OPTAB_D (cond_fms_optab, "cond_fms$a")
OPTAB_D (cond_fnma_optab, "cond_fnma$a")
OPTAB_D (cond_fnms_optab, "cond_fnms$a")
OPTAB_D (cond_neg_optab, "cond_neg$a")
+OPTAB_D (cond_vec_cbranch_any_optab, "cond_vec_cbranch_any$a")
+OPTAB_D (cond_vec_cbranch_all_optab, "cond_vec_cbranch_all$a")
OPTAB_D (cond_one_cmpl_optab, "cond_one_cmpl$a")
OPTAB_D (cond_len_add_optab, "cond_len_add$a")
OPTAB_D (cond_len_sub_optab, "cond_len_sub$a")
@@ -295,6 +297,8 @@ OPTAB_D (cond_len_fnma_optab, "cond_len_fnma$a")
OPTAB_D (cond_len_fnms_optab, "cond_len_fnms$a")
OPTAB_D (cond_len_neg_optab, "cond_len_neg$a")
OPTAB_D (cond_len_one_cmpl_optab, "cond_len_one_cmpl$a")
+OPTAB_D (cond_len_vec_cbranch_any_optab, "cond_len_vec_cbranch_any$a")
+OPTAB_D (cond_len_vec_cbranch_all_optab, "cond_len_vec_cbranch_all$a")
OPTAB_D (vcond_mask_len_optab, "vcond_mask_len_$a")
OPTAB_D (cstore_optab, "cstore$a4")
OPTAB_D (ctrap_optab, "ctrap$a4")
@@ -427,6 +431,8 @@ OPTAB_D (smulhrs_optab, "smulhrs$a3")
OPTAB_D (umulhs_optab, "umulhs$a3")
OPTAB_D (umulhrs_optab, "umulhrs$a3")
OPTAB_D (sdiv_pow2_optab, "sdiv_pow2$a3")
+OPTAB_D (vec_cbranch_any_optab, "vec_cbranch_any$a")
+OPTAB_D (vec_cbranch_all_optab, "vec_cbranch_all$a")
OPTAB_D (vec_pack_sfix_trunc_optab, "vec_pack_sfix_trunc_$a")
OPTAB_D (vec_pack_ssat_optab, "vec_pack_ssat_$a")
OPTAB_D (vec_pack_trunc_optab, "vec_pack_trunc_$a")
