https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104658
--- Comment #2 from CVS Commits <cvs-commit at gcc dot gnu.org> --- The master branch has been updated by Richard Biener <rgue...@gcc.gnu.org>: https://gcc.gnu.org/g:eca04dc8555f5fae462fbd16386da9aaf38a0711 commit r13-111-geca04dc8555f5fae462fbd16386da9aaf38a0711 Author: Richard Biener <rguent...@suse.de> Date: Tue Feb 22 16:02:27 2022 +0100 tree-optimization/104658 - avoid mixing mask & non-mask vector defs When pattern recognition fails to sanitize all defs of a mask producing operation and the respective def is external or constant we end up trying to produce a VECTOR_BOOLEAN_TYPE_P constructor which in turn ends up exposing stmts like <signed-boolean:1> _135 = _49 ? -1 : 0; which isn't handled well in followup SLP and generates awful code. We do rely heavily on pattern recognition to sanitize mask vs. data uses of bools but that fails here which means we also should fail vectorization. That avoids ICEing because of such stmts and it also avoids generating weird code which makes the vectorization not profitable. The following patch simply disallows external VECTOR_BOOLEAN_TYPE_P defs and arranges the promote to external code to instead promote mask uses to extern (that's just a short-cut here). I've also looked at aarch64 and with SVE and a fixed vector length for the gcc.target/i386/pr101636.c testcase. I see similar vectorization (using <signed-boolean:4>) there but it's hard to decide whether the old, the new or no vectorization is better for this. The code generated with traditional integer masks isn't as awkward but we still get the != 0 promotion done for each scalar element which doesn't look like intended - this operation should be visible upfront. That also means some cases will now become a missed optimization that needs to be fixed by bool pattern recognition. But that can possibly be delayed to GCC 13. 2022-02-22 Richard Biener <rguent...@suse.de> PR tree-optimization/104658 * tree-vect-slp.cc (vect_slp_convert_to_external): Do not create VECTOR_BOOLEAN_TYPE_P extern defs. Reset the vector type on nodes we promote. (vectorizable_bb_reduc_epilogue): Deal with externalized root. * tree-vect-stmts.cc (vect_maybe_update_slp_op_vectype): Do not allow VECTOR_BOOLEAN_TYPE_P extern defs. * gcc.target/i386/pr104658.c: New testcase.
