On Tue Jul 14, 2026 at 3:57 PM IST, Raghesh Aloor wrote:
> On Thu Jul 9, 2026 at 2:49 PM IST, Richard Biener wrote:
> > On Tue, 30 Jun 2026, Raghesh Aloor wrote:
> >
> > > ---
> > > gcc/tree-vect-loop.cc | 396 ++++++++++++++++++++++++++++++++++++++++-
> > > gcc/tree-vect-stmts.cc | 26 +++
> > > gcc/tree-vectorizer.h | 5 +
> > > 3 files changed, 426 insertions(+), 1 deletion(-)
> >
> >
> > This matches too much I think. Consider the following (which
> > I don't see how to vectorize, but that's besides the point)
> >
> > void foo (int *p, int * __restrict q, int n)
> > {
> > int res = 0;
> > for (int i = 0; i < n; ++i)
> > {
> > if (p[i] > 0)
> > res++;
> > q[i] = res;
> > }
> > }
> >
> > There's still that very same "conditional induction" but without
> > the masked store index. We are not going to be able to code-generate
> > a vector containing 'res' (at least I don't see an easy way).
> >
> > Meta-info for this should be on the stmt_info of the PHI, not
> > in loop-vinfo.
> >
>
> Thanks Richard and Robin for your suggestions. I will first address the
> comments on the detection patch which has less confusions.
>
> Hi Richard,
>
> On the above example you gave: the current code already rejects it during
> detection. We do recognise the conditional increment
> (res = mask ? res+1 : res), but we then look for a matching .MASK_STORE
> indexed by that value. The store to q is not picked up — it is an ordinary
> store indexed by i, where res is the value being stored, not the index.
> Since we only classify the PHI when such a masked store is found,
> that example is never matched.
>
> Or are you suggesting a restructuring here? If so, is the following what you
> have in mind:
>
> Recognise the conditional-increment induction on its own, from the PHI alone,
> and store that information on the PHI's stmt_info (not in loop-vinfo).
>
Hi Richard,
This updated patch implements recognition only, as you suggested: during
scalar-cycle analysis we classify the mask ? n+1 : n loop-header PHI and
record the increment mask on its stmt_vec_info. Masked-store pairing is
deferred to a follow-up patch; we do not record anything in
loop_vec_info. Until we can vectorize this def type, vect_is_simple_use
rejects these PHIs so the loop fails cleanly.
This patch, we believe, is self-contained with a graceful exit so it
can land upstream on its own and mark clear progress toward the full
compress-store path.
Your foo example (res++ under a condition, then q[i] = res) is covered
by vect-predicated-index-2.c: the matcher does not fire because
if-conversion leaves a guarded add on the backedge, not mask ? res+1 :
res (predicated-index PHI: backedge is not COND_EXPR.). Even if we later
extend recognition to more increment forms, a follow-up pairing pass
would still need to reject uses where the counter is stored as a value
rather than serving only as a compress index.
One refactoring we may do - If-conversion will likely need the same
shape check (mask ? n+1 : n). Would you prefer the matcher reuse that
if possible, avoiding code duplication across two passes?
Thanks,
Raghesh
>From 584e2e5bdef4c308867c4d51a4733d2b9b965128 Mon Sep 17 00:00:00 2001
From: Raghesh Aloor <[email protected]>
Date: Mon, 4 May 2026 16:06:12 +0530
Subject: [PATCH v3 1/4] Detect predicated-index PHI
After if-conversion, a loop with a compress-store pattern can leave a
loop-header PHI whose latch value has the form mask ? index + 1 : index
(with the index used only to address a masked store). Recognize this
conditional-increment cycle during scalar-cycle analysis and classify
the PHI as vect_predicated_index_def, recording the increment mask on
the PHI's stmt_vec_info for a follow-up pass.
Until predicated-index PHIs can be vectorized, reject them in
vect_is_simple_use so the vectorizer fails gracefully instead of
mis-handling the def-use cycle.
gcc/ChangeLog:
* tree-vectorizer.h (enum vect_def_type): Add
vect_predicated_index_def.
(struct _stmt_vec_info): Add
predicated_index_mask.
* tree-vect-loop.cc (vect_analyze_scalar_cycles_1): Detect
predicated-index PHIs.
(vect_analyze_predicated_index_phi,
vect_predicated_index_plus_one, vect_pred_phi_strip_conversion):
New helpers.
* tree-vect-stmts.cc (vect_is_simple_use): Reject
vect_predicated_index_def operands.
gcc/testsuite/ChangeLog:
* gcc.dg/vect/vect-predicated-index-1.c: New test.
* gcc.dg/vect/vect-predicated-index-2.c: New test.
---
.../gcc.dg/vect/vect-predicated-index-1.c | 25 ++
.../gcc.dg/vect/vect-predicated-index-2.c | 27 +++
gcc/tree-vect-loop.cc | 219 ++++++++++++++++++
gcc/tree-vect-stmts.cc | 12 +
gcc/tree-vectorizer.h | 8 +
5 files changed, 291 insertions(+)
create mode 100644 gcc/testsuite/gcc.dg/vect/vect-predicated-index-1.c
create mode 100644 gcc/testsuite/gcc.dg/vect/vect-predicated-index-2.c
diff --git a/gcc/testsuite/gcc.dg/vect/vect-predicated-index-1.c
b/gcc/testsuite/gcc.dg/vect/vect-predicated-index-1.c
new file mode 100644
index 00000000000..3797e52617c
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-predicated-index-1.c
@@ -0,0 +1,25 @@
+/* { dg-do compile } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-require-effective-target avx512f } */
+/* { dg-additional-options "-march=znver5" { target { x86_64-*-* i?86-*-* } }
} */
+/* { dg-additional-options "-march=armv9-a" { target aarch64-*-* } } */
+
+/* Positive test for predicated-index PHI detection (vpcompress / compress
+ store pattern). After if-conversion the pack index has the shape:
+
+ # j = PHI <j_1, 0>
+ ...
+ .MASK_STORE (base + j * size, mask, value)
+ j_1 = mask ? j + 1 : j; */
+
+void
+foo (int * restrict q, const int * restrict a, int n)
+{
+ int j = 0;
+ for (int i = 0; i < n; i++)
+ if (a[i])
+ q[j++] = a[i];
+}
+
+/* { dg-final { scan-tree-dump "Detected predicated-index PHI." "vect" } } */
+/* { dg-final { scan-tree-dump "unsupported: predicated-index PHI." "vect" } }
*/
diff --git a/gcc/testsuite/gcc.dg/vect/vect-predicated-index-2.c
b/gcc/testsuite/gcc.dg/vect/vect-predicated-index-2.c
new file mode 100644
index 00000000000..b04e133ea8b
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-predicated-index-2.c
@@ -0,0 +1,27 @@
+/* { dg-do compile } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-require-effective-target avx512f } */
+/* { dg-additional-options "-march=znver5" { target { x86_64-*-* i?86-*-* } }
} */
+/* { dg-additional-options "-march=armv9-a" { target aarch64-*-* } } */
+
+/* Negative test: the same conditional increment exists, but the counter is
+ consumed as a stored value (q[i] = res) rather than only as a
+ compress-store index. If-conversion keeps a plain add of a guarded
+ zero/one on the backedge instead of mask ? res + 1 : res, so the
+ predicated-index PHI matcher must not fire. */
+
+void
+foo (int * restrict q, const int * restrict a, int n)
+{
+ int res = 0;
+
+ for (int i = 0; i < n; i++)
+ {
+ if (a[i])
+ res++;
+ q[i] = res;
+ }
+}
+
+/* { dg-final { scan-tree-dump-not "Detected predicated-index PHI." "vect" } }
*/
+/* { dg-final { scan-tree-dump "predicated-index PHI: backedge is not
COND_EXPR." "vect" } } */
diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
index a9335ed68bb..4abc8bbeec8 100644
--- a/gcc/tree-vect-loop.cc
+++ b/gcc/tree-vect-loop.cc
@@ -165,6 +165,138 @@ static stmt_vec_info vect_is_simple_reduction
(loop_vec_info, stmt_vec_info,
gphi **);
+/* Function vect_pred_phi_strip_conversion.
+
+ This function check if CUR is an assigment statement and if it is an SSA
+ whose defining statement is NOP/CONVERT/VIEW_CONVERT/NON_LVALUE to SSA.
+ Stores that rhs SSA into CUR
+
+ t_1 = (unsigned int) n;
+
+ Stores n into CUR. */
+static bool
+vect_pred_phi_strip_conversion (tree *cur)
+{
+ if (TREE_CODE (*cur) != SSA_NAME)
+ return false;
+
+ gimple *def = SSA_NAME_DEF_STMT (*cur);
+ gassign *assign = dyn_cast <gassign *> (def);
+ if (!assign)
+ return false;
+
+ tree_code rhs_code = gimple_assign_rhs_code (assign);
+ if (rhs_code != NOP_EXPR
+ && rhs_code != CONVERT_EXPR
+ && rhs_code != VIEW_CONVERT_EXPR
+ && rhs_code != NON_LVALUE_EXPR)
+ return false;
+
+ tree rhs1 = gimple_assign_rhs1 (assign);
+ if (TREE_CODE (rhs1) != SSA_NAME)
+ return false;
+
+ *cur = rhs1;
+ return true;
+}
+
+/* Function vect_predicated_index_plus_one.
+
+ This function checks if one of the arms of the COND_EXPR,
+ PRED_PHI_BACK_EDGE_ASGN_STMT is PRED_PHI itself and the other is
+ PRED_PHI + 1. Note that the increment can be passed through cast
+ instructions as in the below example. So we need to traverse through them.
+
+ # n = PHI <n_2, ..>
+ ...
+ addr = base + n;
+ .MASK_STORE (addr, mask, ..)
+ t_1 = (unsigned int) n;
+ t_2 = t1_1 + 1;
+ n_1 = int (t_2);
+ n_2 = mask ? n_1 : n; */
+static bool
+vect_predicated_index_plus_one (tree &pred_phi,
+ gassign *pred_phi_back_edge_asgn_stmt)
+{
+ /* Check that one of the arms of COND_EXPR is pred-phi (n) and the other is
+ an increment value (n_1 = n + 1). */
+ tree true_val = gimple_assign_rhs2 (pred_phi_back_edge_asgn_stmt);
+ tree false_val = gimple_assign_rhs3 (pred_phi_back_edge_asgn_stmt);
+ tree pred_phi_inc_val = NULL_TREE;
+ if (true_val == pred_phi)
+ pred_phi_inc_val = false_val;
+ else if (false_val == pred_phi)
+ pred_phi_inc_val = true_val;
+ else
+ return false;
+ if (TREE_CODE (pred_phi_inc_val) != SSA_NAME)
+ return false;
+
+ /* Check if n1 = n + 1?. We may need to strip the cast instructions to reach
+ the PLUS_EXPR, traversing until CUR goes out of current basic block. */
+ basic_block bb = gimple_bb (pred_phi_back_edge_asgn_stmt);
+ tree cur = pred_phi_inc_val;
+
+ while (TREE_CODE (cur) == SSA_NAME)
+ {
+ gimple *def = SSA_NAME_DEF_STMT (cur);
+ if (gimple_bb (def) != bb)
+ return false;
+ /* CUR should be an assignment statement. */
+ gassign *cur_asgn = dyn_cast <gassign *> (def);
+ if (!cur_asgn)
+ return false;
+ /* Stop traversing further if we reach a PLUS_EXPR. */
+ if (gimple_assign_rhs_code (cur_asgn) == PLUS_EXPR)
+ {
+ /* Check for +1 increment. It can be either t_1 + 1 or 1 + t_1. */
+ tree rhs1 = gimple_assign_rhs1 (cur_asgn);
+ tree rhs2 = gimple_assign_rhs2 (cur_asgn);
+ tree plus_base = NULL_TREE;
+ if (TREE_CODE (rhs1) == SSA_NAME && TREE_CODE (rhs2) == INTEGER_CST
+ && integer_onep (rhs2))
+ plus_base = rhs1;
+ else if (TREE_CODE (rhs2) == SSA_NAME
+ && TREE_CODE (rhs1) == INTEGER_CST && integer_onep (rhs1))
+ plus_base = rhs2;
+ /* If plus_base is not set we dont have a +1 pattern. */
+ if (!plus_base)
+ return false;
+
+ /* Check if plus_base is pred_phi itself. We may still need to strip
+ some casts to reach the actual base, traversing until plus_base
+ goes out of the current basic block. */
+ while (TREE_CODE (plus_base) == SSA_NAME)
+ {
+ gimple *base_def = SSA_NAME_DEF_STMT (plus_base);
+ if (gimple_bb (base_def) != bb)
+ break;
+ if (!vect_pred_phi_strip_conversion (&plus_base))
+ break;
+ }
+ /* The stripped base should be pred_phi. */
+ if (plus_base == pred_phi)
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, vect_location,
+ "predicated-index PHI: update is +1.\n");
+ return true;
+ }
+ /* We don't have +1 pattern. */
+ return false;
+ }
+ /* If we reach a non-PLUS_EXPR check if we can continue walk through
+ casts. */
+ if (vect_pred_phi_strip_conversion (&cur))
+ continue;
+ /* We cannot reach a PLUS_EXPR. */
+ return false;
+ }
+
+ return false;
+}
+
/* Function vect_is_simple_iv_evolution.
FORNOW: A simple evolution of an induction variables in the loop is
@@ -333,6 +465,85 @@ vect_phi_first_order_recurrence_p (loop_vec_info
loop_vinfo, class loop *loop,
return true;
}
+/* Function vect_analyze_predicated_index_phi.
+
+ Recognize the conditional-increment induction cycle: a loop-header PHI whose
+ back-edge value is the PHI incremented by one under a mask. On success
+ STMT_INFO is annotated with the mask for a follow-up pass that will pair
+ the cycle with a matching IFN_MASK_STORE. An outline of the shape
recognized
+ here is:
+
+ # n = PHI <0, n_2>
+ ...
+ n_1 = n + 1;
+ ...
+ n_2 = mask ? n_1 : n; */
+static bool
+vect_analyze_predicated_index_phi (class loop *loop, gphi *phi,
+ stmt_vec_info stmt_info)
+{
+ /* Require a loop-header PHI. */
+ if (gimple_bb (phi) != loop->header)
+ return false;
+
+ tree pred_phi = PHI_RESULT (phi);
+ /* PHI should be an integer. */
+ if (TREE_CODE (pred_phi) != SSA_NAME
+ || TREE_CODE (TREE_TYPE (pred_phi)) != INTEGER_TYPE)
+ return false;
+
+ /* Restricting to zero initializatin from preheader for now. TODO: Can be
+ generalized to arbitrary integer starts. */
+ tree pre_init = PHI_ARG_DEF_FROM_EDGE (phi, loop_preheader_edge (loop));
+ if (TREE_CODE (pre_init) != INTEGER_CST || !integer_zerop (pre_init))
+ return false;
+
+ /* The incoming value for pred-phi must be in the if-converted form.
+ n_2 = mask ? n_1 : n; TODO: This may be generalized later for more
+ patterns. */
+ /* Get the incoming value (n_2) of pred-phi through backedge. */
+ tree pred_phi_back_edge_val
+ = PHI_ARG_DEF_FROM_EDGE (phi, loop_latch_edge (loop));
+ if (TREE_CODE (pred_phi_back_edge_val) != SSA_NAME)
+ return false;
+ /* Check if pred_phi_back_edge_val is a COND_EXPR. */
+ gimple *pred_phi_back_edge_stmt = SSA_NAME_DEF_STMT (pred_phi_back_edge_val);
+ if (!pred_phi_back_edge_stmt)
+ return false;
+ gassign *pred_phi_back_edge_asgn_stmt
+ = dyn_cast <gassign *> (pred_phi_back_edge_stmt);
+ if (!pred_phi_back_edge_asgn_stmt
+ || gimple_assign_rhs_code (pred_phi_back_edge_asgn_stmt) != COND_EXPR)
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+ "predicated-index PHI: backedge is not COND_EXPR.\n");
+ return false;
+ }
+
+ /* Check that one of the arms of COND_EXPR is pred-phi (n) and the other is
+ an increment value (n_1 = n + 1). */
+ if (!vect_predicated_index_plus_one (pred_phi, pred_phi_back_edge_asgn_stmt))
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+ "predicated-index PHI: update is not +1.\n");
+ return false;
+ }
+
+ /* The conditional-increment cycle is recognized. Record the mask governing
+ the increment on the PHI's stmt_info. */
+ tree mask = gimple_assign_rhs1 (pred_phi_back_edge_asgn_stmt);
+ STMT_VINFO_PREDICATED_INDEX_MASK (stmt_info) = mask;
+
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, vect_location,
+ "predicated-index PHI: recognized conditional-increment "
+ "cycle %G\n", (gimple *) phi);
+
+ return true;
+}
+
/* Function vect_analyze_scalar_cycles_1.
Examine the cross iteration def-use cycles of scalar variables
@@ -479,6 +690,14 @@ vect_analyze_scalar_cycles_1 (loop_vec_info loop_vinfo,
class loop *loop)
}
else if (vect_phi_first_order_recurrence_p (loop_vinfo, loop, phi))
STMT_VINFO_DEF_TYPE (stmt_vinfo) = vect_first_order_recurrence;
+ else if (loop == LOOP_VINFO_LOOP (loop_vinfo)
+ && vect_analyze_predicated_index_phi (loop, phi, stmt_vinfo))
+ {
+ STMT_VINFO_DEF_TYPE (stmt_vinfo) = vect_predicated_index_def;
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, vect_location,
+ "Detected predicated-index PHI.\n");
+ }
else
if (dump_enabled_p ())
dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index 488b9754bed..8dfb1ca8522 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -13937,12 +13937,24 @@ vect_is_simple_use (tree operand, vec_info *vinfo,
enum vect_def_type *dt,
case vect_condition_def:
dump_printf (MSG_NOTE, "control flow\n");
break;
+ case vect_predicated_index_def:
+ dump_printf (MSG_NOTE, "predicated index\n");
+ break;
case vect_unknown_def_type:
dump_printf (MSG_NOTE, "unknown\n");
break;
}
}
+ /* TODO: Till predicated-index PHI is handled, exit gracefully. */
+ if (*dt == vect_predicated_index_def)
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+ "unsupported: predicated-index PHI.\n");
+ return false;
+ }
+
if (*dt == vect_unknown_def_type)
{
if (dump_enabled_p ())
diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
index d4eb20f1a10..332a19aed28 100644
--- a/gcc/tree-vectorizer.h
+++ b/gcc/tree-vectorizer.h
@@ -76,6 +76,7 @@ enum vect_def_type {
vect_nested_cycle,
vect_first_order_recurrence,
vect_condition_def,
+ vect_predicated_index_def,
vect_unknown_def_type
};
@@ -1666,6 +1667,12 @@ public:
/* True if this is only suitable for SLP vectorization. */
bool slp_vect_only_p;
+
+ /* On a predicated-index PHI (vect_predicated_index_def) the mask under which
+ the index is conditionally incremented, as recognized by
+ vect_analyze_predicated_index_phi. Reserved for a follow-up pass that
+ pairs the cycle with a matching masked store. */
+ tree predicated_index_mask;
};
/* Information about a gather/scatter call. */
@@ -1745,6 +1752,7 @@ struct gather_scatter_info {
#define STMT_VINFO_REDUC_DEF(S) (S)->reduc_def
#define STMT_VINFO_SLP_VECT_ONLY(S) (S)->slp_vect_only_p
#define STMT_VINFO_REDUC_VECTYPE_IN(S) (S)->reduc_vectype_in
+#define STMT_VINFO_PREDICATED_INDEX_MASK(S) (S)->predicated_index_mask
#define DR_GROUP_FIRST_ELEMENT(S) \
(gcc_checking_assert ((S)->dr_aux.dr), (S)->first_element)
