On Thu, 6 Aug 2026, Tamar Christina wrote:
> > -----Original Message-----
> > From: Richard Biener <[email protected]>
> > Sent: 03 August 2026 13:47
> > To: [email protected]
> > Cc: Tamar Christina <[email protected]>;
> > [email protected]
> > Subject: [PATCH] tree-optimization/126028 - vector placement and live lane
> > extracts
> >
> > We currently conservatively assume vector stmt placement when
> > determining whether we can place a lane extraction for a live
> > scalar value. The following attempts to improve this by
> > pre-computing an insertion place for all nodes and using that
> > to improve live lane extract feasibility computation.
> >
> > While we'd like to re-use such computation for the actual insertion
> > there are issues preventing that such as scalar stmts eventually
> > getting elided and thus iterators invalidated. So the following
> > re-computes the scheduling again but at least validates the earlier
> > schedule against it.
> >
> > For PR126099 I have to turn this into schedule validation, but
> > not as part of this initial work.
> >
>
> What I've been trying to figure out here is why the insertion point
> isn't just at the end of the SLP region of the tree with the lane extract.
> Since after scheduling, we should know the dependency order, so for a live
> lane it feels like placing the extract immediately after the vector stmt for
> that node
> would be the natural/latest-needed point. At least for bb-slp-46.c I don't
> immediately
> see why a more global/conservative placement is needed.
This is because for bb-slp-46.c we can insert the vector operation for the
plus at the earliest scalar operation. That's not always possible,
in most cases because of externs (aka from-scalar vector CTOR) which
have to be placed after the last scalar def. That can then trivially
interleave:
x = ...
tem0 = a[0] + x;
... = tem0;
y = ...
tem1 = a[1] + y;
so a vector plus would be inserted after y = ..., which is after
the use of tem0.
> So I'm pretty sure I'm missing something. Other than that the patch looks
> sensible to me.
Thanks.
> > The major pain-point of chosing a gimple * as scheduling anchor
> > are that for placement in empty BBs that is NULL (this hits us
> > with region boundaries), a gsi would be able to handle this
>
> You could store a tuple of bb and gimple* maybe? The bb would
> Be the fallback if insertion point is empty at the time and so you
> can't get the gimple *.
So I've (for now) settled with documenting that a NULL gimple *
means insertion at region start.
I have an updated series I'll post shortly
Richard.
> Thanks,
> Tamar
>
> > case. But then, as you can see in the blob that attempts to
> > verify the early schedule constraints hold at transform time
> > there's issues like the scalar stmts vanishing and statements
> > being inserted. As most viable and ugly option I briefly
> > considered inserting GIMPLE_NOPs as anchors. Instead of
> > vect_schedule_slp_node computing the insertion place this
> > could be also handled (or just overridden?) by vectorizable_*
> > which sometimes has its own idea where to place.
> >
> > Bootstrapped and tested on x86_64-unknown-linux-gnu.
> >
> > Any comments?
> >
> > Thanks,
> > Richard.
> >
> > PR tree-optimization/126028
> > * tree-vectorizer.h (_slp_tree::si): New member.
> > (vect_schedule_slp): Adjust prototype.
> > * tree-vect-loop.cc (vect_transform_loop): Adjust.
> > * tree-vect-slp.cc (_slp_tree::_slp_tree): Initialize si.
> > (vect_bb_slp_mark_live_stmts): Use vector stmt placement
> > constraint computed by pre-scheduling of SLP nodes.
> > (vect_slp_analyze_bb_1): Pre-schedule SLP nodes before
> > computing live lane extraction points.
> > (vect_slp_region): Adjust.
> > (vect_schedule_slp_node): Add parameter indicating to whether
> > we should perform pre-scehduling or not. Compute the
> > insertion place suitable for dominance checks when
> > pre-scheduling.
> > (vect_schedule_scc): Adjust.
> > (vect_schedule_slp): Likewise.
> >
> > * gcc.dg/vect/bb-slp-46.c: Un-XFAIL.
> > * gcc.dg/vect/bb-slp-pr126053.c: New testcase.
> > * gcc.dg/vect/costmodel/x86_64/costmodel-pr126028.c: Likewise.
> > ---
> > gcc/testsuite/gcc.dg/vect/bb-slp-46.c | 12 +-
> > gcc/testsuite/gcc.dg/vect/bb-slp-pr126053.c | 63 +++++++
> > .../costmodel/x86_64/costmodel-pr126028.c | 63 +++++++
> > gcc/tree-vect-loop.cc | 3 +-
> > gcc/tree-vect-slp.cc | 162 +++++++++++++++---
> > gcc/tree-vectorizer.h | 4 +-
> > 6 files changed, 275 insertions(+), 32 deletions(-)
> > create mode 100644 gcc/testsuite/gcc.dg/vect/bb-slp-pr126053.c
> > create mode 100644
> > gcc/testsuite/gcc.dg/vect/costmodel/x86_64/costmodel-pr126028.c
> >
> > diff --git a/gcc/testsuite/gcc.dg/vect/bb-slp-46.c
> > b/gcc/testsuite/gcc.dg/vect/bb-slp-46.c
> > index 4eceea44efc..ff4a64a011b 100644
> > --- a/gcc/testsuite/gcc.dg/vect/bb-slp-46.c
> > +++ b/gcc/testsuite/gcc.dg/vect/bb-slp-46.c
> > @@ -6,7 +6,7 @@ int a[4], b[4];
> > int foo ()
> > {
> > int tem0 = a[0] + b[0];
> > - int temx = tem0 * 17; /* this fails without a real need */
> > + int temx = tem0 * 17;
> > int tem1 = a[1] + b[1];
> > int tem2 = a[2] + b[2];
> > int tem3 = a[3] + b[3];
> > @@ -18,11 +18,9 @@ int foo ()
> > return temx / temy;
> > }
> >
> > -/* We should extract the live lane from the vectorized add rather than
> > - keeping the original scalar add.
> > - ??? Because of a too conservative check we fail for temx here. */
> > +/* We should extract the live lanes from the vectorized mul rather than
> > + keeping the original scalar add. */
> > /* { dg-final { scan-tree-dump "optimized: basic block" "slp2" } } */
> > -/* { dg-final { scan-tree-dump "extracting lane for live stmt" "slp2" } }
> > */
> > -/* { dg-final { scan-tree-dump-times "extracting lane for live stmt" 2
> > "slp2" {
> > xfail *-*-* } } } */
> > +/* { dg-final { scan-tree-dump-times "extracting lane for live stmt" 2
> > "slp2" } }
> > */
> > /* { dg-final { scan-tree-dump-not "tem3_\[0-9\]\+ = " "optimized" } } */
> > -/* { dg-final { scan-tree-dump-not "tem0_\[0-9\]\+ = " "optimized" { xfail
> > *-*-
> > * } } } */
> > +/* { dg-final { scan-tree-dump-not "tem0_\[0-9\]\+ = " "optimized" } } */
> > diff --git a/gcc/testsuite/gcc.dg/vect/bb-slp-pr126053.c
> > b/gcc/testsuite/gcc.dg/vect/bb-slp-pr126053.c
> > new file mode 100644
> > index 00000000000..018810d37b8
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.dg/vect/bb-slp-pr126053.c
> > @@ -0,0 +1,63 @@
> > +/* { dg-do compile } */
> > +
> > +typedef double BigReal;
> > +
> > +struct Position { BigReal x, y, z; };
> > +
> > +struct CompAtom {
> > + struct Position position;
> > + float charge;
> > + short vdwType;
> > + unsigned char partition;
> > + unsigned char nonbondedGroupSize;
> > +};
> > +
> > +void cull_goodgroups(const struct CompAtom * __restrict p_1,
> > + const int * __restrict glist,
> > + int gu,
> > + BigReal p_i_x, BigReal p_i_y, BigReal p_i_z,
> > + BigReal groupplcutoff2,
> > + int * __restrict goodglist)
> > +{
> > + int hu = 0;
> > + int g = 0;
> > + int jprev0 = glist[0];
> > + int jprev1 = glist[1];
> > + int j0, j1;
> > +
> > + BigReal pj_x_0, pj_x_1, pj_y_0, pj_y_1, pj_z_0, pj_z_1;
> > + BigReal t_0, t_1, r2_0, r2_1;
> > +
> > + pj_x_0 = p_1[jprev0].position.x; pj_x_1 = p_1[jprev1].position.x;
> > + pj_y_0 = p_1[jprev0].position.y; pj_y_1 = p_1[jprev1].position.y;
> > + pj_z_0 = p_1[jprev0].position.z; pj_z_1 = p_1[jprev1].position.z;
> > + g += 2;
> > +
> > + for ( ; g < gu - 2; g += 2 ) {
> > + j0 = jprev0;
> > + j1 = jprev1;
> > +
> > + t_0 = p_i_x - pj_x_0; t_1 = p_i_x - pj_x_1;
> > + r2_0 = t_0 * t_0; r2_1 = t_1 * t_1;
> > + t_0 = p_i_y - pj_y_0; t_1 = p_i_y - pj_y_1;
> > + r2_0 += t_0 * t_0; r2_1 += t_1 * t_1;
> > + t_0 = p_i_z - pj_z_0; t_1 = p_i_z - pj_z_1;
> > + r2_0 += t_0 * t_0; r2_1 += t_1 * t_1;
> > +
> > + jprev0 = glist[g];
> > + jprev1 = glist[g + 1];
> > + pj_x_0 = p_1[jprev0].position.x; pj_x_1 = p_1[jprev1].position.x;
> > + pj_y_0 = p_1[jprev0].position.y; pj_y_1 = p_1[jprev1].position.y;
> > + pj_z_0 = p_1[jprev0].position.z; pj_z_1 = p_1[jprev1].position.z;
> > +
> > + bool test0 = (r2_0 < groupplcutoff2);
> > + bool test1 = (r2_1 < groupplcutoff2);
> > +
> > + goodglist[hu] = j0;
> > + goodglist[hu + test0] = j1;
> > + hu += test0 + test1;
> > + }
> > +}
> > +
> > +/* { dg-final { scan-tree-dump "58:12: optimized: basic block part
> > vectorized"
> > "slp1" { target x86_64-*-* } } } */
> > +/* { dg-final { scan-tree-dump-not "Cannot determine insertion place"
> > "slp1"
> > } } */
> > diff --git a/gcc/testsuite/gcc.dg/vect/costmodel/x86_64/costmodel-
> > pr126028.c b/gcc/testsuite/gcc.dg/vect/costmodel/x86_64/costmodel-
> > pr126028.c
> > new file mode 100644
> > index 00000000000..15db1d390d9
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.dg/vect/costmodel/x86_64/costmodel-pr126028.c
> > @@ -0,0 +1,63 @@
> > +/* { dg-do compile } */
> > +/* { dg-additional-options "-fdump-tree-slp1 -fopt-info-vec" } */
> > +
> > +typedef double BigReal;
> > +
> > +struct Position { BigReal x, y, z; };
> > +
> > +struct CompAtom {
> > + struct Position position;
> > + float charge;
> > + short vdwType;
> > + unsigned char partition;
> > + unsigned char nonbondedGroupSize;
> > +};
> > +
> > +void cull_goodgroups(const struct CompAtom * __restrict p_1,
> > + const int * __restrict glist,
> > + int gu,
> > + BigReal p_i_x, BigReal p_i_y, BigReal p_i_z,
> > + BigReal groupplcutoff2,
> > + int * __restrict goodglist)
> > +{
> > + int hu = 0;
> > + int g = 0;
> > + int jprev0 = glist[0];
> > + int jprev1 = glist[1];
> > + int j0, j1;
> > +
> > + BigReal pj_x_0, pj_x_1, pj_y_0, pj_y_1, pj_z_0, pj_z_1;
> > + BigReal t_0, t_1, r2_0, r2_1;
> > +
> > + pj_x_0 = p_1[jprev0].position.x; pj_x_1 = p_1[jprev1].position.x;
> > + pj_y_0 = p_1[jprev0].position.y; pj_y_1 = p_1[jprev1].position.y;
> > + pj_z_0 = p_1[jprev0].position.z; pj_z_1 = p_1[jprev1].position.z;
> > + g += 2;
> > +
> > + for ( ; g < gu - 2; g += 2 ) {
> > + j0 = jprev0;
> > + j1 = jprev1;
> > +
> > + t_0 = p_i_x - pj_x_0; t_1 = p_i_x - pj_x_1;
> > + r2_0 = t_0 * t_0; r2_1 = t_1 * t_1;
> > + t_0 = p_i_y - pj_y_0; t_1 = p_i_y - pj_y_1;
> > + r2_0 += t_0 * t_0; r2_1 += t_1 * t_1;
> > + t_0 = p_i_z - pj_z_0; t_1 = p_i_z - pj_z_1;
> > + r2_0 += t_0 * t_0; r2_1 += t_1 * t_1;
> > +
> > + jprev0 = glist[g];
> > + jprev1 = glist[g + 1];
> > + pj_x_0 = p_1[jprev0].position.x; pj_x_1 = p_1[jprev1].position.x;
> > + pj_y_0 = p_1[jprev0].position.y; pj_y_1 = p_1[jprev1].position.y;
> > + pj_z_0 = p_1[jprev0].position.z; pj_z_1 = p_1[jprev1].position.z;
> > +
> > + bool test0 = (r2_0 < groupplcutoff2);
> > + bool test1 = (r2_1 < groupplcutoff2);
> > +
> > + goodglist[hu] = j0;
> > + goodglist[hu + test0] = j1;
> > + hu += test0 + test1; /* { dg-optimized "basic block part vectorized
> > using 8
> > byte vectors" } */
> > + }
> > +}
> > +
> > +/* { dg-final { scan-tree-dump-times " = vect_t\[^ \]* \\\* vect_t" 3
> > "slp1" } }
> > */
> > diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
> > index 31de29c036b..a336df81bc7 100644
> > --- a/gcc/tree-vect-loop.cc
> > +++ b/gcc/tree-vect-loop.cc
> > @@ -11406,7 +11406,8 @@ vect_transform_loop (loop_vec_info loop_vinfo,
> > gimple *loop_vectorized_call)
> > if (!loop_vinfo->slp_instances.is_empty ())
> > {
> > DUMP_VECT_SCOPE ("scheduling SLP instances");
> > - vect_schedule_slp (loop_vinfo, LOOP_VINFO_SLP_INSTANCES
> > (loop_vinfo));
> > + vect_schedule_slp (loop_vinfo, LOOP_VINFO_SLP_INSTANCES
> > (loop_vinfo),
> > + false);
> > }
> >
> > /* Generate the loop invariant statements. */
> > diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc
> > index 8ed7a0da377..dda9e046ade 100644
> > --- a/gcc/tree-vect-slp.cc
> > +++ b/gcc/tree-vect-slp.cc
> > @@ -135,6 +135,7 @@ _slp_tree::_slp_tree ()
> > this->lanes = 0;
> > SLP_TREE_TYPE (this) = undef_vec_info_type;
> > this->data = NULL;
> > + this->si = NULL;
> > }
> >
> > /* Tear down a SLP node. */
> > @@ -9277,7 +9278,7 @@ vect_bb_slp_mark_live_stmts (bb_vec_info
> > bb_vinfo, slp_tree node,
> >
> > unsigned i;
> > stmt_vec_info stmt_info;
> > - stmt_vec_info last_stmt = vect_find_last_scalar_stmt_in_slp (node);
> > + gimple *last_stmt = NULL;
> > FOR_EACH_VEC_ELT (SLP_TREE_SCALAR_STMTS (node), i, stmt_info)
> > {
> > if (!stmt_info || svisited.contains (stmt_info))
> > @@ -9329,7 +9330,11 @@ vect_bb_slp_mark_live_stmts (bb_vec_info
> > bb_vinfo, slp_tree node,
> > || !PURE_SLP_STMT (use_stmt_info)))
> > {
> > live_p = true;
> > - if (!vect_stmt_dominates_stmt_p (last_stmt->stmt,
> > use_stmt))
> > + if (!last_stmt)
> > + last_stmt
> > + = (node->si ? node->si
> > + : vect_find_last_scalar_stmt_in_slp (node)->stmt);
> > + if (!vect_stmt_dominates_stmt_p (last_stmt, use_stmt))
> > {
> > if (dump_enabled_p ())
> > dump_printf_loc (MSG_MISSED_OPTIMIZATION,
> > vect_location,
> > @@ -10448,6 +10453,9 @@ vect_slp_analyze_bb_1 (bb_vec_info bb_vinfo,
> > int n_stmts, bool &fatal,
> > /* Mark all the statements that we vectorize. */
> > vect_bb_slp_mark_stmts_vectorized (bb_vinfo);
> >
> > + /* Compute vector stmt placement. */
> > + vect_schedule_slp (bb_vinfo, BB_VINFO_SLP_INSTANCES (bb_vinfo), true);
> > +
> > /* Compute vectorizable live stmts. */
> > vect_bb_slp_mark_live_stmts (bb_vinfo);
> >
> > @@ -10602,12 +10610,11 @@ vect_slp_region (vec<basic_block> bbs,
> > vec<data_reference_p> datarefs,
> > dump_user_location_t saved_vect_location = vect_location;
> > vect_location = instance->location ();
> >
> > - vect_schedule_slp (bb_vinfo, instance->subgraph_entries);
> > + vect_schedule_slp (bb_vinfo, instance->subgraph_entries, false);
> >
> > vect_location = saved_vect_location;
> > }
> >
> > -
> > /* Generate the invariant statements. */
> > if (!gimple_seq_empty_p (bb_vinfo->inv_pattern_def_seq))
> > {
> > @@ -12049,13 +12056,13 @@ vectorizable_slp_permutation (vec_info
> > *vinfo, gimple_stmt_iterator *gsi,
> > return true;
> > }
> >
> > -/* Vectorize SLP NODE. */
> > +/* Vectorize SLP NODE. Only compute the vector insertion places when
> > + PLACE_ONLY is true. */
> >
> > static void
> > vect_schedule_slp_node (vec_info *vinfo,
> > - slp_tree node, slp_instance instance)
> > + slp_tree node, slp_instance instance, bool place_only)
> > {
> > - gimple_stmt_iterator si;
> > int i;
> > slp_tree child;
> >
> > @@ -12063,6 +12070,9 @@ vect_schedule_slp_node (vec_info *vinfo,
> > if (SLP_TREE_DEF_TYPE (node) == vect_constant_def
> > || SLP_TREE_DEF_TYPE (node) == vect_external_def)
> > {
> > + if (place_only)
> > + return;
> > +
> > /* ??? vectorizable_shift can end up using a scalar operand which is
> > currently denoted as !SLP_TREE_VECTYPE. No need to vectorize the
> > node in this case. */
> > @@ -12082,9 +12092,15 @@ vect_schedule_slp_node (vec_info *vinfo,
> > stmt_vec_info stmt_info = SLP_TREE_REPRESENTATIVE (node);
> >
> > gcc_assert (SLP_TREE_VEC_DEFS (node).is_empty ());
> > - if (SLP_TREE_VECTYPE (node))
> > + if (!place_only && SLP_TREE_VECTYPE (node))
> > SLP_TREE_VEC_DEFS (node).create (vect_get_num_copies (vinfo, node));
> >
> > + gimple *last_stmt;
> > + gimple_stmt_iterator si;
> > + /* ??? When !place_only we'd like to re-use place_only computed info,
> > + but this is a bit awkward due to using gsi_insert_before and the
> > + requirement to insert after vector defs. So we compute last_stmt
> > + during pre-scheduling and si during scheduling. */
> > if (!SLP_TREE_PERMUTE_P (node) && STMT_VINFO_DATA_REF (stmt_info))
> > {
> > /* Vectorized loads go before the first scalar load to make it
> > @@ -12095,7 +12111,8 @@ vect_schedule_slp_node (vec_info *vinfo,
> > last_stmt_info = vect_find_first_scalar_stmt_in_slp (node);
> > else /* DR_IS_WRITE */
> > last_stmt_info = vect_find_last_scalar_stmt_in_slp (node);
> > - si = gsi_for_stmt (last_stmt_info->stmt);
> > + last_stmt = last_stmt_info->stmt;
> > + si = gsi_for_stmt (last_stmt);
> > }
> > else if (!SLP_TREE_PERMUTE_P (node)
> > && (SLP_TREE_TYPE (node) == cycle_phi_info_type
> > @@ -12103,15 +12120,59 @@ vect_schedule_slp_node (vec_info *vinfo,
> > || SLP_TREE_TYPE (node) == phi_info_type))
> > {
> > /* For PHI node vectorization we do not use the insertion iterator.
> > */
> > + last_stmt = SLP_TREE_SCALAR_STMTS (node)[0]->stmt;
> > si = gsi_none ();
> > }
> > else
> > {
> > /* Emit other stmts after the children vectorized defs which is
> > earliest possible. */
> > - gimple *last_stmt = NULL;
> > + last_stmt = NULL;
> > FOR_EACH_VEC_ELT (SLP_TREE_CHILDREN (node), i, child)
> > - if (SLP_TREE_DEF_TYPE (child) == vect_internal_def)
> > + if (place_only)
> > + {
> > + gimple *vstmt = child->si;
> > + if (!vstmt)
> > + {
> > + if (SLP_TREE_DEF_TYPE (child) == vect_external_def)
> > + {
> > + vec<tree> &defs
> > + = (!SLP_TREE_SCALAR_OPS (child).is_empty ()
> > + ? SLP_TREE_SCALAR_OPS (child)
> > + : SLP_TREE_VEC_DEFS (child));
> > + for (tree def : defs)
> > + /* If the stmt is not inside the region do not
> > + use it as possible insertion point. */
> > + if (auto stmt_info = vinfo->lookup_def (def))
> > + {
> > + gimple *stmt = stmt_info->stmt;
> > + if (!last_stmt)
> > + last_stmt = stmt;
> > + else if (vect_stmt_dominates_stmt_p (last_stmt,
> > stmt))
> > + last_stmt = stmt;
> > + else if (vect_stmt_dominates_stmt_p (stmt,
> > last_stmt))
> > + ;
> > + else
> > + gcc_unreachable ();
> > + }
> > + }
> > + else
> > + /* vect_constant_def and defs at region boundary do not
> > + constrain placement. */
> > + gcc_assert (SLP_TREE_DEF_TYPE (child) == vect_constant_def
> > + /* ??? Region boundary is not representated. */
> > + || true);
> > + }
> > + else if (!last_stmt)
> > + last_stmt = vstmt;
> > + else if (vect_stmt_dominates_stmt_p (last_stmt, vstmt))
> > + last_stmt = vstmt;
> > + else if (vect_stmt_dominates_stmt_p (vstmt, last_stmt))
> > + ;
> > + else
> > + gcc_unreachable ();
> > + }
> > + else if (SLP_TREE_DEF_TYPE (child) == vect_internal_def)
> > {
> > /* For fold-left reductions we are retaining the scalar
> > reduction PHI but we still have SLP_TREE_NUM_VEC_STMTS
> > @@ -12225,11 +12286,16 @@ vect_schedule_slp_node (vec_info *vinfo,
> > gimple_bb (stmt_info->stmt),
> > gimple_bb (last_stmt)));
> > si = gsi_after_labels (gimple_bb (stmt_info->stmt));
> > + last_stmt = gsi_stmt (si);
> > }
> > /* When there is no in-region child def to guide placement, insert
> > at region boundary. */
> > else if (!last_stmt)
> > - si = gsi_after_labels (vinfo->bbs[0]);
> > + {
> > + si = gsi_after_labels (vinfo->bbs[0]);
> > + /* ??? last_stmt can be NULL if the block is empty. */
> > + last_stmt = gsi_stmt (si);
> > + }
> > else if (is_a <gphi *> (last_stmt))
> > si = gsi_after_labels (gimple_bb (last_stmt));
> > else
> > @@ -12241,6 +12307,8 @@ vect_schedule_slp_node (vec_info *vinfo,
> >
> > if (auto loop_vinfo = dyn_cast <loop_vec_info> (vinfo))
> > {
> > + /* We'll have to fix this up for loop vect. */
> > + gcc_assert (!place_only);
> > /* Avoid scheduling stmts to random places in the CFG, any
> > stmt dominance check we performed is possibly wrong as
> > UIDs
> > are not initialized for all of the function for loop
> > @@ -12274,6 +12342,44 @@ vect_schedule_slp_node (vec_info *vinfo,
> > }
> > }
> >
> > + if (place_only)
> > + {
> > + if (dump_enabled_p () && last_stmt)
> > + dump_printf_loc (MSG_NOTE, vect_location,
> > + "placing node %p at %G:", (void *)node, last_stmt);
> > + gcc_assert ((last_stmt && gimple_bb (last_stmt)) || gsi_bb (si));
> > + node->si = last_stmt;
> > + return;
> > + }
> > +
> > + /* ??? Asserting vect_stmt_dominates_stmt_p (gsi_stmt (si), node->si)
> > + does not work because in some cases we advance si from last_stmt (as
> > + we want to insert after vector stmts) and because vector stmts of
> > + children have been inserted possibly at the same location constraint,
> > + moving si even further. */
> > + if (flag_checking && node->si && gimple_bb (node->si) && !gsi_end_p (si))
> > + {
> > + auto gsi2 = si;
> > + while (1)
> > + {
> > + if (vect_stmt_dominates_stmt_p (gsi_stmt (gsi2), node->si))
> > + break;
> > + /* As we have possibly advanced si it might now point to the
> > + scalar stmt immediately following node->si. That's OK. */
> > + if (gsi_stmt (gsi2) != gsi_stmt (si)
> > + && gimple_uid (gsi_stmt (gsi2)) != 0)
> > + gcc_unreachable ();
> > + gsi_prev (&gsi2);
> > + if (gsi_end_p (gsi2))
> > + {
> > + if (is_a <gphi *> (node->si)
> > + && gimple_bb (node->si) == gsi_bb (si))
> > + break;
> > + gcc_unreachable ();
> > + }
> > + }
> > + }
> > +
> > if (dump_enabled_p ())
> > {
> > if (stmt_info)
> > @@ -12499,7 +12605,7 @@ struct slp_scc_info
> > static void
> > vect_schedule_scc (vec_info *vinfo, slp_tree node, slp_instance instance,
> > hash_map<slp_tree, slp_scc_info> &scc_info,
> > - int &maxdfs, vec<slp_tree> &stack)
> > + int &maxdfs, vec<slp_tree> &stack, bool place_only)
> > {
> > bool existed_p;
> > slp_scc_info *info = &scc_info.get_or_insert (node, &existed_p);
> > @@ -12512,7 +12618,7 @@ vect_schedule_scc (vec_info *vinfo, slp_tree
> > node, slp_instance instance,
> > if (SLP_TREE_DEF_TYPE (node) != vect_internal_def)
> > {
> > info->on_stack = false;
> > - vect_schedule_slp_node (vinfo, node, instance);
> > + vect_schedule_slp_node (vinfo, node, instance, place_only);
> > return;
> > }
> >
> > @@ -12529,7 +12635,8 @@ vect_schedule_scc (vec_info *vinfo, slp_tree
> > node, slp_instance instance,
> > slp_scc_info *child_info = scc_info.get (child);
> > if (!child_info)
> > {
> > - vect_schedule_scc (vinfo, child, instance, scc_info, maxdfs, stack);
> > + vect_schedule_scc (vinfo, child, instance, scc_info, maxdfs, stack,
> > + place_only);
> > /* Recursion might have re-allocated the node. */
> > info = scc_info.get (node);
> > child_info = scc_info.get (child);
> > @@ -12548,7 +12655,7 @@ vect_schedule_scc (vec_info *vinfo, slp_tree
> > node, slp_instance instance,
> > {
> > stack.pop ();
> > info->on_stack = false;
> > - vect_schedule_slp_node (vinfo, node, instance);
> > + vect_schedule_slp_node (vinfo, node, instance, place_only);
> > if (!SLP_TREE_PERMUTE_P (node)
> > && is_a <gphi *> (SLP_TREE_REPRESENTATIVE (node)->stmt))
> > phis_to_fixup.quick_push (node);
> > @@ -12600,7 +12707,7 @@ vect_schedule_scc (vec_info *vinfo, slp_tree
> > node, slp_instance instance,
> > }
> > if (ready)
> > {
> > - vect_schedule_slp_node (vinfo, entry, instance);
> > + vect_schedule_slp_node (vinfo, entry, instance, place_only);
> > scc_info.get (entry)->on_stack = false;
> > stack[idx] = NULL;
> > todo--;
> > @@ -12615,6 +12722,9 @@ vect_schedule_scc (vec_info *vinfo, slp_tree
> > node, slp_instance instance,
> > stack.truncate (last_idx);
> > }
> >
> > + if (place_only)
> > + return;
> > +
> > /* Now fixup the backedge def of the vectorized PHIs in this SCC. */
> > slp_tree phi_node;
> > FOR_EACH_VEC_ELT (phis_to_fixup, i, phi_node)
> > @@ -12665,10 +12775,12 @@ vect_schedule_scc (vec_info *vinfo, slp_tree
> > node, slp_instance instance,
> > }
> > }
> >
> > -/* Generate vector code for SLP_INSTANCES in the loop/basic block. */
> > +/* Generate vector code for SLP_INSTANCES in the loop/basic block. Perform
> > + vector stmt placement only when PLACE_ONLY is true. */
> >
> > void
> > -vect_schedule_slp (vec_info *vinfo, const vec<slp_instance> &slp_instances)
> > +vect_schedule_slp (vec_info *vinfo, const vec<slp_instance> &slp_instances,
> > + bool place_only)
> > {
> > slp_instance instance;
> > unsigned int i;
> > @@ -12678,7 +12790,7 @@ vect_schedule_slp (vec_info *vinfo, const
> > vec<slp_instance> &slp_instances)
> > FOR_EACH_VEC_ELT (slp_instances, i, instance)
> > {
> > slp_tree node = SLP_INSTANCE_TREE (instance);
> > - if (dump_enabled_p ())
> > + if (!place_only && dump_enabled_p ())
> > {
> > dump_printf_loc (MSG_NOTE, vect_location,
> > "Vectorizing SLP tree:\n");
> > @@ -12693,16 +12805,20 @@ vect_schedule_slp (vec_info *vinfo, const
> > vec<slp_instance> &slp_instances)
> > have a PHI be the node breaking the cycle. */
> > auto_vec<slp_tree> stack;
> > if (!scc_info.get (node))
> > - vect_schedule_scc (vinfo, node, instance, scc_info, maxdfs, stack);
> > + vect_schedule_scc (vinfo, node, instance, scc_info, maxdfs, stack,
> > + place_only);
> >
> > - if (!SLP_INSTANCE_ROOT_STMTS (instance).is_empty ())
> > + if (!place_only && !SLP_INSTANCE_ROOT_STMTS (instance).is_empty ())
> > vectorize_slp_instance_root_stmt (vinfo, node, instance);
> >
> > - if (dump_enabled_p ())
> > + if (!place_only && dump_enabled_p ())
> > dump_printf_loc (MSG_NOTE, vect_location,
> > "vectorizing stmts using SLP.\n");
> > }
> >
> > + if (place_only)
> > + return;
> > +
> > FOR_EACH_VEC_ELT (slp_instances, i, instance)
> > {
> > slp_tree root = SLP_INSTANCE_TREE (instance);
> > diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
> > index 94ef81383f4..e79b9087b21 100644
> > --- a/gcc/tree-vectorizer.h
> > +++ b/gcc/tree-vectorizer.h
> > @@ -358,6 +358,8 @@ struct _slp_tree {
> > tree vectype;
> > /* Vectorized defs. */
> > vec<tree> vec_defs;
> > + /* Insertion place. */
> > + gimple *si;
> >
> > /* Reference count in the SLP graph. */
> > unsigned int refcnt;
> > @@ -2775,7 +2777,7 @@ extern bool vect_transform_slp_perm_load
> > (vec_info *, slp_tree, const vec<tree>
> > extern bool vectorizable_slp_permutation (vec_info *, gimple_stmt_iterator
> > *,
> > slp_tree, stmt_vector_for_cost *);
> > extern bool vect_slp_analyze_operations (vec_info *);
> > -extern void vect_schedule_slp (vec_info *, const vec<slp_instance> &);
> > +extern void vect_schedule_slp (vec_info *, const vec<slp_instance> &,
> > bool);
> > extern opt_result vect_analyze_slp (vec_info *, unsigned, bool);
> > extern bool vect_make_slp_decision (loop_vec_info);
> > extern bool vect_detect_hybrid_slp (loop_vec_info);
> > --
> > 2.51.0
>
--
Richard Biener <[email protected]>
SUSE Software Solutions Germany GmbH,
Frankenstrasse 146, 90461 Nuernberg, Germany;
GF: Jochen Jaser, Andrew McDonald, Abhinav Puri; (HRB 36809, AG Nuernberg)
