We currently make vect_check_gather_scatter happy by replacing SSA
name references in DR_REF for gather/scatter DRs but the replacement
process only works once since for the second epilogue we have SSA
names from the first epilogue in DR_REF but as we copied from the
original loop the SSA mapping doesn't work.
The following simply punts for non-first epilogues, those gather/scatter
recognized by patterns to IFNs are already analyzed and should work
fine.
Bootstrapped and tested on x86_64-unknown-linux-gnu.
* tree-vect-data-refs.cc (vect_check_gather_scatter): Refuse
to analyze DR_REF if from an epilogue that's not first.
* tree-vect-loop.cc (update_epilogue_loop_vinfo): Add comment
how the substitution in DR_REF is broken.
---
gcc/tree-vect-data-refs.cc | 7 +++++++
gcc/tree-vect-loop.cc | 3 +++
2 files changed, 10 insertions(+)
diff --git a/gcc/tree-vect-data-refs.cc b/gcc/tree-vect-data-refs.cc
index 54ad5c8f3dc..6276139b71f 100644
--- a/gcc/tree-vect-data-refs.cc
+++ b/gcc/tree-vect-data-refs.cc
@@ -4319,6 +4319,13 @@ vect_check_gather_scatter (stmt_vec_info stmt_info,
loop_vec_info loop_vinfo,
masked_p = (ifn == IFN_MASK_LOAD || ifn == IFN_MASK_STORE);
}
+ /* ??? For epilogues we adjust DR_REF to make the following stmt-based
+ analysis work, but this adjustment doesn't work for epilogues of
+ epilogues during transform, so disable gather/scatter in that case. */
+ if (LOOP_VINFO_EPILOGUE_P (loop_vinfo)
+ && LOOP_VINFO_EPILOGUE_P (LOOP_VINFO_ORIG_LOOP_INFO (loop_vinfo)))
+ return false;
+
/* True if we should aim to use internal functions rather than
built-in functions. */
bool use_ifn_p = (DR_IS_READ (dr)
diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
index 5de90b6573a..41875683595 100644
--- a/gcc/tree-vect-loop.cc
+++ b/gcc/tree-vect-loop.cc
@@ -12294,6 +12294,9 @@ update_epilogue_loop_vinfo (class loop *epilogue, tree
advance)
if (STMT_VINFO_MEMORY_ACCESS_TYPE (vstmt_vinfo) == VMAT_GATHER_SCATTER
|| STMT_VINFO_GATHER_SCATTER_P (vstmt_vinfo))
{
+ /* ??? As we copy epilogues from the main loop incremental
+ replacement from an already replaced DR_REF from vectorizing
+ the first epilogue will fail. */
DR_REF (dr)
= simplify_replace_tree (DR_REF (dr), NULL_TREE, NULL_TREE,
&find_in_mapping, &mapping);
--
2.43.0
