On Tue, 11 Aug 2026, Tamar Christina wrote:
> > -----Original Message-----
> > From: Richard Biener <[email protected]>
> > Sent: 11 August 2026 10:06
> > To: Tamar Christina <[email protected]>
> > Cc: Alfie Richards <[email protected]>; [email protected]; nd
> > <[email protected]>; [email protected]
> > Subject: RE: [patch][vect]: convert do-loop edges properly for
> > LOOP_VINFO_EARLY_BREAKS_VECT_PEELED [PR126301]
> >
> > On Tue, 11 Aug 2026, Tamar Christina wrote:
> >
> > > > -----Original Message-----
> > > > From: Tamar Christina <[email protected]>
> > > > Sent: 10 August 2026 15:14
> > > > To: Richard Biener <[email protected]>; Alfie Richards
> > > > <[email protected]>
> > > > Cc: [email protected]; nd <[email protected]>; [email protected]
> > > > Subject: RE: [patch][vect]: convert do-loop edges properly for
> > > > LOOP_VINFO_EARLY_BREAKS_VECT_PEELED [PR126301]
> > > >
> > > > > -----Original Message-----
> > > > > From: Richard Biener <[email protected]>
> > > > > Sent: 10 August 2026 13:21
> > > > > To: Alfie Richards <[email protected]>
> > > > > Cc: Tamar Christina <[email protected]>; gcc-
> > [email protected];
> > > > > nd <[email protected]>; [email protected]
> > > > > Subject: Re: [patch][vect]: convert do-loop edges properly for
> > > > > LOOP_VINFO_EARLY_BREAKS_VECT_PEELED [PR126301]
> > > > >
> > > > > On Wed, 29 Jul 2026, Alfie Richards wrote:
> > > > >
> > > > > > On 29/07/2026 14:46, Tamar Christina wrote:
> > > > > > > The example
> > > > > > >
> > > > > > > char b[100];
> > > > > > > int c(int a) {
> > > > > > > unsigned d = 0;
> > > > > > > for (; d < a; ++d)
> > > > > > > {
> > > > > > > if (b[0] + b[d + 1])
> > > > > > > return 0;
> > > > > > > }
> > > > > > > return 1;
> > > > > > > }
> > > > > > >
> > > > > > > when compiled with a partial masked target, e.g. -march=armv9-a
> > > > > > > -O3
> > > > > > > generates
> > > > > > >
> > > > > > > ptrue p6.b, all
> > > > > > > whilelo p15.s, wzr, w3
> > > > > > > b .L5
> > > > > > > .L11:
> > > > > > > ld1b z31.s, p7/z, [x5, x1]
> > > > > > > add z31.h, z31.h, z30.h
> > > > > > > uxth z31.s, p6/m, z31.s
> > > > > > > cmpne p7.s, p7/z, z31.s, #0
> > > > > > > b.any .L4
> > > > > > > .L5:
> > > > > > > mov x2, x1
> > > > > > > mov p7.b, p15.b
> > > > > > > incw x1
> > > > > > > whilelo p15.s, w1, w3
> > > > > > > b.any .L11
> > > > > > > mov w1, w2
> > > > > > > add x3, x4, :lo12:.LANCHOR0
> > > > > > > b .L8
> > > > > > >
> > > > > > > Warped BB rotation due to the incorrect profiles aside (different
> > problem)
> > > > > > > this loop will exit one iteration early since the IV check (d <
> > > > > > > a) is
> > > > > > > checked
> > > > > > > as (++d < a).
> > > > > > >
> > > > > > > This means that when branched to the scalar code we do 2 vector
> > > > > iterations
> > > > > > > in
> > > > > > > the worst case rather than 1.
> > > > > > >
> > > > > > > This is due to the fact that the loop is essentially a do-while
> > > > > > > loop
> > coming
> > > > > > > into
> > > > > > > the vectorizer. We detect this loop as a
> > > > > > > LOOP_VINFO_EARLY_BREAKS_VECT_PEELED
> > > > > > > because it's the same form as a normal while-do loop but where we
> > > > picked
> > > > > a
> > > > > > > different exit than the loop latch exit.
> > > > > > >
> > > > > > > Because of the versioning and other checks in the preheader we
> > > > > > > know
> > > > that
> > > > > if
> > > > > > > you
> > > > > > > reach the loop body you'll always do at least 1 iteration of the
> > > > > > > body
> > (but
> > > > > > > possibly never the latch). i.e. the d != 0 check is always true
> > > > > > > should
> > you
> > > > > > > get
> > > > > > > to the end of the pre-header.
> > > > > > >
> > > > > > > The codegen reflects this but because of the early IV exit check
> > > > > > > we
> > never
> > > > > > > get to
> > > > > > > the body.
> > > > > > >
> > > > > > > Alfie's patch to fix this changed the `d++ < a` check to the
> > > > > > > correct (d <
> > a)
> > > > > > > check, but due to how the control flow becomes we end up doing
> > > > > > >
> > > > > > > mask_1 = PHI <mask_0, next_mask>
> > > > > > > if (mask_1)
> > > > > > >
> > > > > > > and causes us to retest the mask, even though the mask generation
> > > > which
> > > > > > > happens
> > > > > > > in the pre-header has guaranteed that for the first iteration it's
> > > > > > > non-empty.
> > > > > > >
> > > > > > > So we lose information because the check in the header is used by
> > both
> > > > the
> > > > > > > value
> > > > > > > from the pre-header and the latch on loop back.
> > > > > > >
> > > > > > > Instead of doing that this patch just changed the control flow to
> > reflect
> > > > > > > that
> > > > > > > we can always execute the body at least once. In effect it moves
> > > > > > > the
> > IV
> > > > > > > latch
> > > > > > > check to the end.
> > > > > > >
> > > > > > > This generates:
> > > > > > >
> > > > > > > ptrue p6.b, all
> > > > > > > whilelo p7.s, wzr, w4
> > > > > > > b .L4
> > > > > > > .p2align 2,,3
> > > > > > > .L11:
> > > > > > > whilelo p7.s, w1, w4
> > > > > > > b.none .L15
> > > > > > > .L4:
> > > > > > > mov x2, x1
> > > > > > > incw x1
> > > > > > > ld1b z31.s, p7/z, [x5, x1]
> > > > > > > add z31.h, z31.h, z30.h
> > > > > > > uxth z31.s, p6/m, z31.s
> > > > > > > cmpne p7.s, p7/z, z31.s, #0
> > > > > > > b.none .L11
> > > > > > > .L7:
> > > > > > > mov w0, 0
> > > > > > > ret
> > > > > > >
> > > > > > > and when the profiles are fixed (PR117790) we'd get
> > > > > > >
> > > > > > > ptrue p6.b, all
> > > > > > > whilelo p7.s, wzr, w5
> > > > > > > .p2align 5,,15
> > > > > > > .L4:
> > > > > > > mov x2, x1
> > > > > > > incw x1
> > > > > > > ld1b z31.s, p7/z, [x3, x1]
> > > > > > > add z31.h, z31.h, z30.h
> > > > > > > uxth z31.s, p6/m, z31.s
> > > > > > > cmpne p7.s, p7/z, z31.s, #0
> > > > > > > b.any .L7
> > > > > > > whilelo p7.s, w1, w5
> > > > > > > b.any .L4
> > > > > > >
> > > > > > > Bootstrapped Regtested on aarch64-none-linux-gnu,
> > > > > > > arm-none-linux-gnueabihf, x86_64-pc-linux-gnu
> > > > > > > -m32, -m64 and no issues.
> > > > > > >
> > > > > > > Any comments?
> > > > > >
> > > > > > FWIW LGTM, fixes the issues I was having and markedly improved code
> > gen
> > > > > for
> > > > > > the cases I was looking at.
> > > > > >
> > > > > > There are some minor extra changes needed for FFR but I will submit
> > those
> > > > > with
> > > > > > FFR series and related changes once I finish squashing bugs.
> > > > >
> > > > > Sorry for chiming in late - so how can this be valid without actually
> > > > > peeling up to the IV exit, that is, actually performing loop-ch with
> > > > > the "correct" exit/latch in mind? And why would that be not
> > > > > valid for vect_partial_vectors_avx512? IMO the terminology in the
> > > > > added comments is confusing. We have
> > > > > LOOP_VINFO_EARLY_BREAKS_VECT_PEELED
> > > > > also for two early exits and the order
> > > > >
> > > > > if (early)
> > > > >
> > > > > if (IV-exit)
> > > > >
> > > > > if (early)
> > > > >
> > > > > no?
> > > >
> > > > Yes, but LOOP_VINFO_EARLY_BREAKS_VECT_PEELED always replay
> > > > the loop from the start of the current vector iteration, and it won't
> > > > have any side effects.
> > > >
> > > > So when you exit you will always restart the scalar loop.
> > > >
> > > > When this isn't the case you need more updates, or peeling up to the IV
> > > > exit as you mentioned.
> > > >
> > > > The other exception is UNCOUNTED loops, but uncounted loops today
> > > > can't use masking because masking atm needs a counted exit.
> > > >
> > > > > And why would that be not valid for vect_partial_vectors_avx512?
> > > >
> > > > I looked at the codegen for both AVX and RISCV for this before, in both
> > > > cases they seem to be able to generate good code already since the
> > > > loop control code is a bit different. As I tried to explain the
> > > > majority
> > > > of the issues here is because using loop masks as control mechanisms
> > > > are expensive.
> > > >
> > > > If concerned I'd suggest just tightening and perhaps renaming
> > > > vect_use_loop_latch_condition_p to check for this specific condition.
> > > >
> > > > But that means this loop https://godbolt.org/z/4z6hK7szY will stay
> > > > suboptimal and throwing away expensive mask comparisons (in L14
> > > > the result of the mask generation is not used at all this iteration for
> > > > the
> > > > block.)
> > > >
> > > > Since I don't think the peeling of a partial loop is going to be easy.
> > >
> > > Ok looks like as long as the region is identified,
> > gimple_duplicate_seme_region
> > > makes it rather simple... at least at first glance..
> > >
> > > I can do this if you prefer.
> >
> > Well, let's see. If we can elide all _PEELED variants we could ensure
> > this in the pre-vectorizer loop header copying pass by rotating the
> > loop? Just copying parts does not perform the rotation.
>
> Of course... I was merely saying that copying parts of the loop isn't
> as hard as I thought. Doing it pre-vectorizer is harder I think as that
> needs to be done immediately after the exit is chosen... but do-able...
>
> >
> > But I think I am not fully getting the problem you are solving :/
>
> PEELED loops check the loop's iteration exit early.
> https://godbolt.org/z/P5oafrcjj
>
> The check for d < a is performed at the start of the loop, rather than at the
> end.
> This means, you do all the loop setup code, and then check if you will exit
> from
> that exit in the *next* iteration. So you will exit one iteration early even
> though
> you can do the current iteration.
>
> If you look at GCC 16, the loop is
>
> whilelo p15.s, wzr, w3 <-- mask for iteration 0
> b .L5
> .L12:
> ld1b z31.s, p7/z, [x5, x1] <-- use mask of iteration n
> add z31.h, z31.h, z30.h
> uxth z31.s, p6/m, z31.s
> cmpne p7.s, p7/z, z31.s, #0
> b.any .L4
> .L5:
> mov w2, w1
> mov p7.b, p15.b <-- copy mask for iteration n
> incw x1
> whilelo p15.s, w1, w3 <-- mask for next iteration n+1
> b.any .L12
>
> and so the loop is very inefficient, because it's constantly generation
> and saving masks because it's pre-computing the next iteration mask early
> and we don't do iteration n when we know we'll exit at the end of n+1.
>
> The rewrite in trunk transforms this into
>
> whilelo p7.s, wzr, w0 <-- mask for iteration 0
> ptrue p6.b, all
> b .L4
> .L11:
> whilelo p7.s, w1, w0 <-- check and generate mask n+1
> b.none .L15
> .L4:
> mov x2, x1
> incw x1
> ld1b z31.s, p7/z, [x4, x1] <--- use mask of iteration n
> add z31.h, z31.h, z30.h
> uxth z31.s, p6/m, z31.s
> cmpne p7.s, p7/z, z31.s, #0
> b.none .L11
> .L7:
> mov w0, 0
> ret
>
> My reasoning for not needing the header copying before is that
> 1. we know iteration `n` can be performed, that's guaranteed by the loop
> pre-header.
> Where skip_vector means we won't get there when p7 in the pre-header is
> empty.
> and even if we did, the mask would just be empty and so all operations
> no-ops.
> 2. Any exit leads to the restart of the scalar loop at iteration n. *even*
> the early one checking n+1.
>
> These two is why I don't think
>
> If (early)
>
> If (IV-exit)
>
> If (early)
>
> Are a problem.
Ah, OK. Thanks for explaining - I think the code is good as-is.
Richard.
> Tamar
> >
> > Richard.
> >
> > >
> > > Tamar
> > > >
> > > > Thanks,
> > > > Tamar
> > > >
> > > > >
> > > > > Richard.
> > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Tamar
> > > > > > >
> > > > > > > gcc/ChangeLog:
> > > > > > >
> > > > > > > * tree-vect-loop-manip.cc (vect_use_loop_latch_condition_p): New.
> > > > > > > (vect_set_loop_condition_partial_vectors,
> > > > > > > vect_set_loop_condition):
> > > > Use
> > > > > > > it to rewrite latch and condition.
> > > > > > > (vect_set_loop_controls_directly): Adjust IV for do-loop
> > > > > > > conversion.
> > > > > > >
> > > > > > > gcc/testsuite/ChangeLog:
> > > > > > >
> > > > > > > * gcc.target/aarch64/sve/peeled.c: New test.
> > > > > > > * gcc.target/aarch64/sve/peeled1.c: New test.
> > > > > > > * gcc.target/aarch64/sve/peeled1_run.c: New test.
> > > > > > > * gcc.target/aarch64/sve/peeled2.c: New test.
> > > > > > > * gcc.target/aarch64/sve/peeled2_run.c: New test.
> > > > > > > * gcc.target/aarch64/sve/peeled_run.c: New test.
> > > > > > >
> > > > > > > ---
> > > > > > > diff --git a/gcc/testsuite/gcc.target/aarch64/sve/peeled.c
> > > > > > > b/gcc/testsuite/gcc.target/aarch64/sve/peeled.c
> > > > > > > new file mode 100644
> > > > > > > index
> > > > > > >
> > > > >
> > > >
> > 0000000000000000000000000000000000000000..f40ffc8f0bcde40aeb75
> > > > > 4887ed6d5047e9806bc7
> > > > > > > --- /dev/null
> > > > > > > +++ b/gcc/testsuite/gcc.target/aarch64/sve/peeled.c
> > > > > > > @@ -0,0 +1,20 @@
> > > > > > > +/* { dg-do compile } */
> > > > > > > +/* { dg-options "-O3 -mautovec-preference=sve-only
> > > > > > > -msve-vector-bits=scalable" } */
> > > > > > > +
> > > > > > > +char b[100];
> > > > > > > +
> > > > > > > +int __attribute__ ((noipa))
> > > > > > > +c (int a)
> > > > > > > +{
> > > > > > > + unsigned d = 0;
> > > > > > > + for (; d < a; ++d)
> > > > > > > + {
> > > > > > > + if (b[0] + b[d + 1])
> > > > > > > + return 0;
> > > > > > > + }
> > > > > > > + return 1;
> > > > > > > +}
> > > > > > > +
> > > > > > > +/* { dg-final { scan-assembler-times {\twhilelo\t} 2 } } */
> > > > > > > +/* { dg-final { scan-assembler-times {\tptest\t} 0 } } */
> > > > > > > +/* { dg-final { scan-assembler {\tld1b\tz[0-9]+\.s, p[0-9]+/z,}
> > > > > > > } } */
> > > > > > > diff --git a/gcc/testsuite/gcc.target/aarch64/sve/peeled1.c
> > > > > > > b/gcc/testsuite/gcc.target/aarch64/sve/peeled1.c
> > > > > > > new file mode 100644
> > > > > > > index
> > > > > > >
> > > > >
> > > >
> > 0000000000000000000000000000000000000000..92e5bca44c48cefa20e
> > > > > 829510a8914e4852145ed
> > > > > > > --- /dev/null
> > > > > > > +++ b/gcc/testsuite/gcc.target/aarch64/sve/peeled1.c
> > > > > > > @@ -0,0 +1,24 @@
> > > > > > > +/* { dg-do compile } */
> > > > > > > +/* { dg-options "-O3 -mautovec-preference=sve-only
> > > > > > > -msve-vector-bits=scalable" } */
> > > > > > > +
> > > > > > > +char b[100];
> > > > > > > +char e[100];
> > > > > > > +
> > > > > > > +int __attribute__ ((noipa))
> > > > > > > +c (int a)
> > > > > > > +{
> > > > > > > + unsigned d = 0;
> > > > > > > + for (; d < a; ++d)
> > > > > > > + {
> > > > > > > + if (b[0] + b[d + 1])
> > > > > > > + return 0;
> > > > > > > +
> > > > > > > + if (e[0] + e[d + 1])
> > > > > > > + return 0;
> > > > > > > + }
> > > > > > > + return 1;
> > > > > > > +}
> > > > > > > +
> > > > > > > +/* { dg-final { scan-assembler-times {\twhilelo\t} 2 } } */
> > > > > > > +/* { dg-final { scan-assembler-times {\tptest\t} 2 } } */
> > > > > > > +/* { dg-final { scan-assembler-times {\tld1b\tz[0-9]+\.h,
> > > > > > > p[0-9]+/z,}
> > 2 } }
> > > > > > > */
> > > > > > > diff --git a/gcc/testsuite/gcc.target/aarch64/sve/peeled1_run.c
> > > > > > > b/gcc/testsuite/gcc.target/aarch64/sve/peeled1_run.c
> > > > > > > new file mode 100644
> > > > > > > index
> > > > > > >
> > > > >
> > > >
> > 0000000000000000000000000000000000000000..8ec95a187ab0bce43d
> > > > > 7bcf9e8d83ea8ddae7ccf5
> > > > > > > --- /dev/null
> > > > > > > +++ b/gcc/testsuite/gcc.target/aarch64/sve/peeled1_run.c
> > > > > > > @@ -0,0 +1,34 @@
> > > > > > > +/* { dg-do run { target aarch64_sve_hw } } */
> > > > > > > +/* { dg-options "-O3 -mautovec-preference=sve-only
> > > > > > > -msve-vector-bits=scalable" } */
> > > > > > > +
> > > > > > > +#include "peeled1.c"
> > > > > > > +
> > > > > > > +static void
> > > > > > > +clear_arrays (void)
> > > > > > > +{
> > > > > > > + for (int i = 0; i < 100; ++i)
> > > > > > > + {
> > > > > > > + b[i] = 0;
> > > > > > > + e[i] = 0;
> > > > > > > + }
> > > > > > > +}
> > > > > > > +
> > > > > > > +int
> > > > > > > +main (void)
> > > > > > > +{
> > > > > > > + clear_arrays ();
> > > > > > > + if (c (99) != 1)
> > > > > > > + __builtin_abort ();
> > > > > > > +
> > > > > > > + clear_arrays ();
> > > > > > > + b[37] = 1;
> > > > > > > + if (c (99) != 0)
> > > > > > > + __builtin_abort ();
> > > > > > > +
> > > > > > > + clear_arrays ();
> > > > > > > + e[45] = 1;
> > > > > > > + if (c (99) != 0)
> > > > > > > + __builtin_abort ();
> > > > > > > +
> > > > > > > + return 0;
> > > > > > > +}
> > > > > > > diff --git a/gcc/testsuite/gcc.target/aarch64/sve/peeled2.c
> > > > > > > b/gcc/testsuite/gcc.target/aarch64/sve/peeled2.c
> > > > > > > new file mode 100644
> > > > > > > index
> > > > > > >
> > > > >
> > > >
> > 0000000000000000000000000000000000000000..c29dab4b5536d318b4
> > > > > 5a0b70c407ef4d9fa32ed5
> > > > > > > --- /dev/null
> > > > > > > +++ b/gcc/testsuite/gcc.target/aarch64/sve/peeled2.c
> > > > > > > @@ -0,0 +1,22 @@
> > > > > > > +/* { dg-do compile } */
> > > > > > > +/* { dg-options "-O3 -mautovec-preference=sve-only
> > > > > > > -msve-vector-bits=scalable" } */
> > > > > > > +
> > > > > > > +char b[100];
> > > > > > > +
> > > > > > > +int __attribute__ ((noipa))
> > > > > > > +c (int a)
> > > > > > > +{
> > > > > > > + unsigned d = 0;
> > > > > > > + do
> > > > > > > + {
> > > > > > > + if (b[0] + b[d + 1])
> > > > > > > + return 0;
> > > > > > > + d++;
> > > > > > > + }
> > > > > > > + while (__builtin_expect (d < a, 1));
> > > > > > > + return 1;
> > > > > > > +}
> > > > > > > +
> > > > > > > +/* { dg-final { scan-assembler-times {\twhilelo\t} 2 } } */
> > > > > > > +/* { dg-final { scan-assembler-times {\tptest\t} 0 } } */
> > > > > > > +/* { dg-final { scan-assembler {\tld1b\tz[0-9]+\.s, p[0-9]+/z,}
> > > > > > > } } */
> > > > > > > diff --git a/gcc/testsuite/gcc.target/aarch64/sve/peeled2_run.c
> > > > > > > b/gcc/testsuite/gcc.target/aarch64/sve/peeled2_run.c
> > > > > > > new file mode 100644
> > > > > > > index
> > > > > > >
> > > > >
> > > >
> > 0000000000000000000000000000000000000000..2075b0803eb0cab439
> > > > > 32bc9e60f80f0822ffa83b
> > > > > > > --- /dev/null
> > > > > > > +++ b/gcc/testsuite/gcc.target/aarch64/sve/peeled2_run.c
> > > > > > > @@ -0,0 +1,26 @@
> > > > > > > +/* { dg-do run { target aarch64_sve_hw } } */
> > > > > > > +/* { dg-options "-O3 -mautovec-preference=sve-only
> > > > > > > -msve-vector-bits=scalable" } */
> > > > > > > +
> > > > > > > +#include "peeled2.c"
> > > > > > > +
> > > > > > > +static void
> > > > > > > +clear_b (void)
> > > > > > > +{
> > > > > > > + for (int i = 0; i < 100; ++i)
> > > > > > > + b[i] = 0;
> > > > > > > +}
> > > > > > > +
> > > > > > > +int
> > > > > > > +main (void)
> > > > > > > +{
> > > > > > > + clear_b ();
> > > > > > > + if (c (99) != 1)
> > > > > > > + __builtin_abort ();
> > > > > > > +
> > > > > > > + clear_b ();
> > > > > > > + b[37] = 1;
> > > > > > > + if (c (99) != 0)
> > > > > > > + __builtin_abort ();
> > > > > > > +
> > > > > > > + return 0;
> > > > > > > +}
> > > > > > > diff --git a/gcc/testsuite/gcc.target/aarch64/sve/peeled3.c
> > > > > > > b/gcc/testsuite/gcc.target/aarch64/sve/peeled3.c
> > > > > > > new file mode 100644
> > > > > > > index
> > > > > > >
> > > > >
> > > >
> > 0000000000000000000000000000000000000000..8d82828b770d16868c
> > > > > 99930c115571c9128c3fa5
> > > > > > > --- /dev/null
> > > > > > > +++ b/gcc/testsuite/gcc.target/aarch64/sve/peeled3.c
> > > > > > > @@ -0,0 +1,21 @@
> > > > > > > +/* { dg-do compile } */
> > > > > > > +/* { dg-options "-O3 -mautovec-preference=sve-only
> > > > > > > -msve-vector-bits=scalable -fdump-tree-vect-details" } */
> > > > > > > +
> > > > > > > +int __attribute__ ((noipa))
> > > > > > > +c (int *restrict x, int *restrict y, int n)
> > > > > > > +{
> > > > > > > + unsigned d = 5;
> > > > > > > + for (; __builtin_expect (d < n, 1); ++d)
> > > > > > > + {
> > > > > > > + if (x[d] != y[d])
> > > > > > > + return 0;
> > > > > > > + }
> > > > > > > + return 1;
> > > > > > > +}
> > > > > > > +
> > > > > > > +/* { dg-final { scan-tree-dump "Both peeling and versioning will
> > > > > > > be
> > > > > > > applied" "vect" } } */
> > > > > > > +/* { dg-final { scan-tree-dump "misalignment for fully-masked
> > > > > > > loop"
> > > > > "vect"
> > > > > > > } } */
> > > > > > > +/* { dg-final { scan-assembler {\tsub\tw[0-9]+, w[0-9]+, #6} } }
> > > > > > > */
> > > > > > > +/* { dg-final { scan-assembler-times {\twhilelo\t} 3 } } */
> > > > > > > +/* { dg-final { scan-assembler-times {\tptest\t} 0 } } */
> > > > > > > +/* { dg-final { scan-assembler-times {\tld1w\t} 2 } } */
> > > > > > > diff --git a/gcc/testsuite/gcc.target/aarch64/sve/peeled3_run.c
> > > > > > > b/gcc/testsuite/gcc.target/aarch64/sve/peeled3_run.c
> > > > > > > new file mode 100644
> > > > > > > index
> > > > > > >
> > > > >
> > > >
> > 0000000000000000000000000000000000000000..83015b3bfe1861e951
> > > > > 71f59f4a8ddd679b9583d2
> > > > > > > --- /dev/null
> > > > > > > +++ b/gcc/testsuite/gcc.target/aarch64/sve/peeled3_run.c
> > > > > > > @@ -0,0 +1,35 @@
> > > > > > > +/* { dg-do run { target aarch64_sve_hw } } */
> > > > > > > +/* { dg-options "-O3 -mautovec-preference=sve-only
> > > > > > > -msve-vector-bits=scalable" } */
> > > > > > > +
> > > > > > > +#include "peeled3.c"
> > > > > > > +
> > > > > > > +#define N 128
> > > > > > > +
> > > > > > > +int a[N] __attribute__ ((aligned (64)));
> > > > > > > +int b[N] __attribute__ ((aligned (64)));
> > > > > > > +
> > > > > > > +static void
> > > > > > > +clear_arrays (void)
> > > > > > > +{
> > > > > > > + for (int i = 0; i < N; ++i)
> > > > > > > + {
> > > > > > > + a[i] = 0;
> > > > > > > + b[i] = 0;
> > > > > > > + }
> > > > > > > +}
> > > > > > > +
> > > > > > > +int
> > > > > > > +main (void)
> > > > > > > +{
> > > > > > > + clear_arrays ();
> > > > > > > + b[16] = 1;
> > > > > > > + if (c (a, b, 16) != 1)
> > > > > > > + __builtin_abort ();
> > > > > > > +
> > > > > > > + clear_arrays ();
> > > > > > > + b[15] = 1;
> > > > > > > + if (c (a, b, 16) != 0)
> > > > > > > + __builtin_abort ();
> > > > > > > +
> > > > > > > + return 0;
> > > > > > > +}
> > > > > > > diff --git a/gcc/testsuite/gcc.target/aarch64/sve/peeled_run.c
> > > > > > > b/gcc/testsuite/gcc.target/aarch64/sve/peeled_run.c
> > > > > > > new file mode 100644
> > > > > > > index
> > > > > > >
> > > > >
> > > >
> > 0000000000000000000000000000000000000000..7e289f3c0c29fda37f3
> > > > > 138f3655e79389d76a2d0
> > > > > > > --- /dev/null
> > > > > > > +++ b/gcc/testsuite/gcc.target/aarch64/sve/peeled_run.c
> > > > > > > @@ -0,0 +1,26 @@
> > > > > > > +/* { dg-do run { target aarch64_sve_hw } } */
> > > > > > > +/* { dg-options "-O3 -mautovec-preference=sve-only
> > > > > > > -msve-vector-bits=scalable" } */
> > > > > > > +
> > > > > > > +#include "peeled.c"
> > > > > > > +
> > > > > > > +static void
> > > > > > > +clear_b (void)
> > > > > > > +{
> > > > > > > + for (int i = 0; i < 100; ++i)
> > > > > > > + b[i] = 0;
> > > > > > > +}
> > > > > > > +
> > > > > > > +int
> > > > > > > +main (void)
> > > > > > > +{
> > > > > > > + clear_b ();
> > > > > > > + if (c (99) != 1)
> > > > > > > + __builtin_abort ();
> > > > > > > +
> > > > > > > + clear_b ();
> > > > > > > + b[37] = 1;
> > > > > > > + if (c (99) != 0)
> > > > > > > + __builtin_abort ();
> > > > > > > +
> > > > > > > + return 0;
> > > > > > > +}
> > > > > > > diff --git a/gcc/tree-vect-loop-manip.cc b/gcc/tree-vect-loop-
> > manip.cc
> > > > > > > index
> > > > > > >
> > > > >
> > > >
> > ec9dec1b7135916ea69fd9eabfdfe1645d6ea11b..96c10340efd94e7db503af
> > > > > 208ca21cf3fd807dcd
> > > > > > > 100644
> > > > > > > --- a/gcc/tree-vect-loop-manip.cc
> > > > > > > +++ b/gcc/tree-vect-loop-manip.cc
> > > > > > > @@ -464,6 +464,19 @@ vect_iv_increment_position (edge
> > loop_exit,
> > > > > > > @@ gimple_stmt_iterator *bsi,
> > > > > > > *insert_after = false;
> > > > > > > }
> > > > > > >
> > > > > > > +/* If this is a loop where the latch condition should be
> > > > > > > rewritten to
> > > > > > > reflect
> > > > > > > + a control flow change from a while-do to a do-while loop. */
> > > > > > > +
> > > > > > > +static bool
> > > > > > > +vect_use_loop_latch_condition_p (loop_vec_info loop_vinfo)
> > > > > > > +{
> > > > > > > + return (loop_vinfo
> > > > > > > + && LOOP_VINFO_EARLY_BREAKS_VECT_PEELED (loop_vinfo)
> > > > > > > + && LOOP_VINFO_USING_PARTIAL_VECTORS_P (loop_vinfo)
> > > > > > > + && (LOOP_VINFO_PARTIAL_VECTORS_STYLE (loop_vinfo)
> > > > > > > + != vect_partial_vectors_avx512));
> > > > > > > +}
> > > > > > > +
> > > > > > > /* Helper for vect_set_loop_condition_partial_vectors. Generate
> > > > > > > definitions
> > > > > > > for all the rgroup controls in RGC and return a control that
> > > > > > > is
> > nonzero
> > > > > > > when the loop needs to iterate. Add any new preheader
> > statements to
> > > > > > > @@ -744,6 +757,13 @@ vect_set_loop_controls_directly (class loop
> > > > > *loop,
> > > > > > > @@ loop_vec_info loop_vinfo,
> > > > > > > bias_tree);
> > > > > > > }
> > > > > > > + /* A do-while loop always executes the body once, as
> > > > > > > such the
> > limit
> > > > > > > + the end counter should be lowered by 1 iteration. */
> > > > > > > + if (vect_use_loop_latch_condition_p (loop_vinfo))
> > > > > > > + this_test_limit = gimple_build (preheader_seq, MINUS_EXPR,
> > > > > > > + compare_type,
> > this_test_limit,
> > > > > > > + build_one_cst
> > (compare_type));
> > > > > > > +
> > > > > > > /* Create the initial control. First include all items
> > > > > > > that
> > > > > > > are within the loop limit. */
> > > > > > > tree init_ctrl = NULL_TREE;
> > > > > > > @@ -968,7 +988,62 @@ vect_set_loop_condition_partial_vectors
> > (class
> > > > > loop
> > > > > > > @@ *loop, edge exit_edge,
> > > > > > > cond_stmt
> > > > > > > = gimple_build_cond (code, test_ctrl, zero_ctrl, NULL_TREE,
> > > > NULL_TREE);
> > > > > > > }
> > > > > > > - gsi_insert_before (&loop_cond_gsi, cond_stmt, GSI_SAME_STMT);
> > > > > > > + edge latch_exit_edge = NULL;
> > > > > > > + /* Convert the loop into a do-while form similar to what
> > > > > > > ch_vect
> > would
> > > > > > > have
> > > > > > > + done. We know that after the checks and peeling that we
> > > > > > > have at
> > > > least
> > > > > > > one
> > > > > > > + iteration to perform of the loop because the loop is
> > > > > > > PEELED. A
> > PEELED
> > > > > > > loop
> > > > > > > + has the increment exit before the early ones, i.e. it's a
> > > > > > > do-while
> > > > > > > loop but
> > > > > > > + if we materialize the IV edge in that place we are
> > > > > > > essentially
> > > > > > > checking one
> > > > > > > + iteration ahead so we exit early. Instead when using masks
> > > > > > > and
> > the
> > > > > > > loop
> > > > > > > + is PEELED we remove the existing loop latch and make it a
> > > > > > > fall
> > through
> > > > > > > + edge and place the latch back to the end of the loop. So
> > > > > > > effectively
> > > > > > > + transform:
> > > > > > > +
> > > > > > > + header
> > > > > > > + |
> > > > > > > + latch
> > > > > > > + |
> > > > > > > + body
> > > > > > > + |
> > > > > > > + branch to header
> > > > > > > +
> > > > > > > + into
> > > > > > > +
> > > > > > > + header
> > > > > > > + |
> > > > > > > + body
> > > > > > > + |
> > > > > > > + newlatch
> > > > > > > + |
> > > > > > > + branch to header
> > > > > > > +
> > > > > > > + because the conditions in the pre-header makes it safe to
> > > > > > > do so for
> > > > > > > some
> > > > > > > + cases. */
> > > > > > > + if (vect_use_loop_latch_condition_p (loop_vinfo))
> > > > > > > + {
> > > > > > > + basic_block latch = loop->latch;
> > > > > > > + edge latch_e = single_succ_edge (latch);
> > > > > > > + int exit_flags = exit_edge->flags & (EDGE_TRUE_VALUE |
> > > > > > > EDGE_FALSE_VALUE);
> > > > > > > +
> > > > > > > + latch_e->flags &= ~(EDGE_FALLTHRU | EDGE_TRUE_VALUE |
> > > > > > > EDGE_FALSE_VALUE);
> > > > > > > + latch_e->flags |= (EDGE_TRUE_VALUE | EDGE_FALSE_VALUE) ^
> > > > > exit_flags;
> > > > > > > + latch_exit_edge = make_edge (latch, exit_edge->dest,
> > > > > > > exit_flags);
> > > > > > > + latch_exit_edge->probability = exit_edge->probability;
> > > > > > > + latch_exit_edge->count () = exit_edge->count ();
> > > > > > > + copy_phi_arg_into_existing_phi (exit_edge,
> > > > > > > latch_exit_edge);
> > > > > > > + gimple_stmt_iterator latch_gsi = gsi_last_bb (latch);
> > > > > > > + gsi_insert_after (&latch_gsi, cond_stmt, GSI_NEW_STMT);
> > > > > > > + LOOP_VINFO_MAIN_EXIT (loop_vinfo) = latch_exit_edge;
> > > > > > > +
> > > > > > > + gcond *old_cond = as_a <gcond *> (gsi_stmt
> > > > > > > (loop_cond_gsi));
> > > > > > > + if (exit_edge->flags & EDGE_TRUE_VALUE)
> > > > > > > + gimple_cond_make_false (old_cond);
> > > > > > > + else
> > > > > > > + gimple_cond_make_true (old_cond);
> > > > > > > + update_stmt (old_cond);
> > > > > > > + }
> > > > > > > + else
> > > > > > > + gsi_insert_before (&loop_cond_gsi, cond_stmt, GSI_SAME_STMT);
> > > > > > >
> > > > > > > /* The loop iterates (NITERS - 1) / VF + 1 times.
> > > > > > > Subtract one from this to get the latch count. */
> > > > > > > @@ -993,7 +1068,7 @@ vect_set_loop_condition_partial_vectors
> > (class
> > > > > loop
> > > > > > > @@ *loop, edge exit_edge,
> > > > > > > }
> > > > > > > else
> > > > > > > assign = gimple_build_assign (final_iv, orig_niters);
> > > > > > > - gsi_insert_on_edge_immediate (exit_edge, assign);
> > > > > > > + gsi_insert_on_edge_immediate (LOOP_VINFO_MAIN_EXIT
> > > > > (loop_vinfo),
> > > > > > > assign);
> > > > > > > }
> > > > > > >
> > > > > > > return cond_stmt;
> > > > > > > @@ -1470,11 +1545,14 @@ vect_set_loop_condition (class loop
> > *loop,
> > > > > edge
> > > > > > > @@ loop_e, loop_vec_info loop_vinfo
> > > > > > >
> > > > > > > /* Remove old loop exit test. */
> > > > > > > stmt_vec_info orig_cond_info;
> > > > > > > - if (loop_vinfo
> > > > > > > - && (orig_cond_info = loop_vinfo->lookup_stmt (orig_cond)))
> > > > > > > - loop_vinfo->remove_stmt (orig_cond_info);
> > > > > > > - else
> > > > > > > - gsi_remove (&loop_cond_gsi, true);
> > > > > > > + if (!vect_use_loop_latch_condition_p (loop_vinfo))
> > > > > > > + {
> > > > > > > + if (loop_vinfo
> > > > > > > + && (orig_cond_info = loop_vinfo->lookup_stmt (orig_cond)))
> > > > > > > + loop_vinfo->remove_stmt (orig_cond_info);
> > > > > > > + else
> > > > > > > + gsi_remove (&loop_cond_gsi, true);
> > > > > > > + }
> > > > > > >
> > > > > > > if (dump_enabled_p ())
> > > > > > > dump_printf_loc (MSG_NOTE, vect_location, "New loop exit
> > > > condition:
> > > > > > > %G",
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Richard Biener <[email protected]>
> > > > > SUSE Software Solutions Germany GmbH,
> > > > > Frankenstrasse 146, 90461 Nuernberg, Germany;
> > > > > GF: Jochen Jaser, Andrew McDonald, Abhinav Puri; (HRB 36809, AG
> > > > > Nuernberg)
> > >
> >
> > --
> > Richard Biener <[email protected]>
> > SUSE Software Solutions Germany GmbH,
> > Frankenstrasse 146, 90461 Nuernberg, Germany;
> > GF: Jochen Jaser, Andrew McDonald, Abhinav Puri; (HRB 36809, AG
> > Nuernberg)
>
--
Richard Biener <[email protected]>
SUSE Software Solutions Germany GmbH,
Frankenstrasse 146, 90461 Nuernberg, Germany;
GF: Jochen Jaser, Andrew McDonald, Abhinav Puri; (HRB 36809, AG Nuernberg)
