Ping. This is the last substantial patch needed to if-convert the hot Snappy decompress loop on aarch64 (and maybe other targets?) https://gcc.gnu.org/pipermail/gcc-patches/2026-July/724006.html
(Well that and the profile count fix at https://gcc.gnu.org/pipermail/gcc-patches/2026-July/724378.html) Thanks, Kyrill > On 31 Jul 2026, at 14:49, Kyrylo Tkachov <[email protected]> wrote: > > Ping. > Thanks, > Kyrill > >> On 24 Jul 2026, at 14:28, Kyrylo Tkachov <[email protected]> wrote: >> >> Ping. >> Thanks, >> Kyrill >> >>> On 15 Jul 2026, at 11:06, Kyrylo Tkachov <[email protected]> wrote: >>> >>> From: Kyrylo Tkachov <[email protected]> >>> >>> This patch supersedes this one: >>> https://gcc.gnu.org/pipermail/gcc-patches/2026-July/722391.html >>> Consider that patch discarded. >>> >>> noce_convert_multiple_sets handles multi-set IF-THEN-JOIN regions but not >>> IF-THEN-ELSE-JOIN diamonds. After GIMPLE factors the selected load in >>> Snappy's tag decoder, such a diamond still defines the next tag and input >>> advance and leaves a data-dependent branch. >>> >>> Validate both arms using the existing multiple-set checks. Choose a >>> primary arm that defines every live-out pseudo defined by the secondary arm. >>> Require the primary arm to define at least two distinct live-out pseudos. >>> This leaves single-result diamonds to the existing specialized transforms. >>> Collect the live-out destinations while validating each arm and compare the >>> resulting bitmaps when choosing the primary arm. >>> >>> Evaluate secondary-arm SET_SRCs into fresh pseudos in dependency order >>> and preserve their source locations. For each primary destination, use the >>> final secondary value as the other conditional-move input. Use the incoming >>> value when the secondary arm does not define that destination. >>> >>> Either CFG arm may be primary. If the branch-target arm is primary, >>> swap the arms in a local noce_if_info and reverse the select orientation. >>> Reject speculative sequences that modify condition inputs. If they clobber >>> the condition code, stop sharing the comparison. Rematerialize it for each >>> conditional move instead. >>> >>> In outline, convert >>> >>> secondary: >>> (set advance secondary_advance) >>> primary: >>> (set next primary_next) >>> (set advance primary_advance) >>> >>> to >>> >>> (set secondary_tmp secondary_advance) >>> (set next >>> (if_then_else primary_p primary_next incoming_next)) >>> (set advance >>> (if_then_else primary_p primary_advance secondary_tmp)) >>> >>> where primary_p selects the primary arm. >>> >>> Weight arm costs by edge probability for speed and sum them for size. The >>> target noce_conversion_profitable_p hook retains the final profitability >>> decision. >>> >>> Add execution and code-generation tests for symmetric and asymmetric >>> diamonds, arm dependencies, repeated definitions, unmatched live-outs, >>> condition-code clobbers, and both arm orientations. >>> >>> This removes the badly-predicted conditional branch in the Snappy decode >>> loop and gives ~20% on the BM_UFlatMedley workload on my aarch64 >>> machine. >>> >>> On SPEC2026 it triggers a few thousand times with small improvements but >>> nothing I would call out above noise levels. >>> >>> Bootstrapped and tested on aarch64-linux-gnu and x86_64-pc-linux-gnu. >>> >>> Ideally this would land after the fix for the wrong-code PR126184 >>> (https://gcc.gnu.org/pipermail/gcc-patches/2026-July/723467.html) >>> >>> Ok for trunk? >>> Thanks, >>> Kyrill >>> >>> gcc/ChangeLog: >>> >>> PR tree-optimization/125557 >>> * ifcvt.cc: Include "explow.h". >>> (noce_convert_multiple_sets): Handle diamond CFG cleanup. >>> (noce_convert_multiple_sets_1): Evaluate secondary-arm values in fresh >>> pseudos, preserve their source locations, and use their final values >>> as conditional-move inputs. Use copy_to_mode_reg. Reject secondary >>> sequences that modify condition inputs and handle condition-code >>> clobbers. >>> (bb_ok_for_noce_convert_multiple_sets): Add REQUIRE_MULTIPLE and >>> LIVE_OUT_DESTS. Record distinct live-out destinations. >>> (noce_process_if_block): Recognize and cost multi-set diamonds. Choose >>> the compatible primary arm from the validated live-out destinations. >>> >>> gcc/testsuite/ChangeLog: >>> >>> PR tree-optimization/125557 >>> * gcc.c-torture/execute/ifcvt-diamond-1.c: New test. >>> * gcc.target/aarch64/ifcvt_multiple_sets_diamond.c: New test. >>> * gcc.target/aarch64/ifcvt_multiple_sets_diamond_2.c: New test. >>> * gcc.target/aarch64/ifcvt_multiple_sets_diamond_3.c: New test. >>> * gcc.target/aarch64/ifcvt_multiple_sets_diamond_4.c: New test. >>> * gcc.target/aarch64/ifcvt_multiple_sets_diamond_5.c: New test. >>> * gcc.target/i386/ifcvt-multiple-sets-diamond-1.c: New test. >>> >>> Signed-off-by: Kyrylo Tkachov <[email protected]> >>> --- >>> gcc/ifcvt.cc | 216 +++++++++++++++--- >>> .../gcc.c-torture/execute/ifcvt-diamond-1.c | 143 ++++++++++++ >>> .../aarch64/ifcvt_multiple_sets_diamond.c | 66 ++++++ >>> .../aarch64/ifcvt_multiple_sets_diamond_2.c | 52 +++++ >>> .../aarch64/ifcvt_multiple_sets_diamond_3.c | 57 +++++ >>> .../aarch64/ifcvt_multiple_sets_diamond_4.c | 64 ++++++ >>> .../aarch64/ifcvt_multiple_sets_diamond_5.c | 31 +++ >>> .../i386/ifcvt-multiple-sets-diamond-1.c | 44 ++++ >>> 8 files changed, 644 insertions(+), 29 deletions(-) >>> create mode 100644 gcc/testsuite/gcc.c-torture/execute/ifcvt-diamond-1.c >>> create mode 100644 >>> gcc/testsuite/gcc.target/aarch64/ifcvt_multiple_sets_diamond.c >>> create mode 100644 >>> gcc/testsuite/gcc.target/aarch64/ifcvt_multiple_sets_diamond_2.c >>> create mode 100644 >>> gcc/testsuite/gcc.target/aarch64/ifcvt_multiple_sets_diamond_3.c >>> create mode 100644 >>> gcc/testsuite/gcc.target/aarch64/ifcvt_multiple_sets_diamond_4.c >>> create mode 100644 >>> gcc/testsuite/gcc.target/aarch64/ifcvt_multiple_sets_diamond_5.c >>> create mode 100644 >>> gcc/testsuite/gcc.target/i386/ifcvt-multiple-sets-diamond-1.c >>> >>> diff --git a/gcc/ifcvt.cc b/gcc/ifcvt.cc >>> index 03d9a4c6ab3..925e4e331b9 100644 >>> --- a/gcc/ifcvt.cc >>> +++ b/gcc/ifcvt.cc >>> @@ -37,6 +37,7 @@ >>> #include "cfgrtl.h" >>> #include "cfganal.h" >>> #include "cfgcleanup.h" >>> +#include "explow.h" >>> #include "expr.h" >>> #include "output.h" >>> #include "cfgloop.h" >>> @@ -3771,13 +3772,17 @@ try_emit_cmove_seq (struct noce_if_info *if_info, >>> rtx temp, >>> conditional set to use the temporary we introduced earlier. >>> >>> IF_INFO contains the useful information about the block structure and >>> - jump instructions. */ >>> + jump instructions. For an IF-THEN-ELSE-JOIN, first evaluate the >>> secondary >>> + arm's sets into temporaries and retain their final values for the >>> + conditional moves. Return true if the replacement is valid and >>> profitable >>> + and its CFG changes have been committed, otherwise return false. */ >>> >>> static bool >>> noce_convert_multiple_sets (struct noce_if_info *if_info) >>> { >>> basic_block test_bb = if_info->test_bb; >>> basic_block then_bb = if_info->then_bb; >>> + basic_block else_bb = if_info->else_bb; >>> basic_block join_bb = if_info->join_bb; >>> rtx_insn *jump = if_info->jump; >>> rtx_insn *cond_earliest; >>> @@ -3887,8 +3892,16 @@ noce_convert_multiple_sets (struct noce_if_info >>> *if_info) >>> emit_insn_before_setloc (seq, if_info->jump, >>> INSN_LOCATION (insn_info.last ()->unmodified_insn)); >>> >>> - /* Clean up THEN_BB and the edges in and out of it. */ >>> - remove_edge (find_edge (test_bb, join_bb)); >>> + /* Clean up the THEN (and, for a diamond, ELSE) block and the edges into >>> and >>> + out of the if-region. An IF-THEN-ELSE-JOIN has no test->join edge. >>> + Deleting ELSE_BB removes the test->else and else->join edges instead. >>> */ >>> + if (else_bb) >>> + { >>> + delete_basic_block (else_bb); >>> + num_true_changes++; >>> + } >>> + else >>> + remove_edge (find_edge (test_bb, join_bb)); >>> remove_edge (find_edge (then_bb, join_bb)); >>> redirect_edge_and_branch_force (single_succ_edge (test_bb), join_bb); >>> delete_basic_block (then_bb); >>> @@ -3906,8 +3919,15 @@ noce_convert_multiple_sets (struct noce_if_info >>> *if_info) >>> return true; >>> } >>> >>> -/* This goes through all relevant insns of IF_INFO->then_bb and tries to >>> create >>> - conditional moves. Information for the insns is kept in INSN_INFO. */ >>> +/* Try to emit the multiple-set conversion described by IF_INFO. INSN_INFO >>> + holds the primary-arm metadata. For a diamond, evaluate the secondary >>> arm >>> + first and retain its final values for the conditional moves. >>> + >>> + LAST_NEEDS_COMPARISON is -1 on the first attempt. Record in it the >>> last set >>> + that needs a temporary to preserve the comparison, then use that >>> boundary >>> + on the second attempt. Set USE_COND_EARLIEST if the emitted sequence >>> uses >>> + IF_INFO->cond_earliest. Return true if the complete sequence was >>> + emitted. */ >>> >>> static bool >>> noce_convert_multiple_sets_1 (struct noce_if_info *if_info, >>> @@ -3933,6 +3953,73 @@ noce_convert_multiple_sets_1 (struct noce_if_info >>> *if_info, >>> int count = 0; >>> bool second_try = *last_needs_comparison != -1; >>> *use_cond_earliest = false; >>> + auto_delete_vec<noce_multiple_sets_info> else_insn_info; >>> + >>> + /* For an IF-THEN-ELSE-JOIN, emit the else block's computations first >>> into >>> + fresh temporaries. This leaves the incoming register values >>> available to >>> + the then block. The conditional moves below select the then values or >>> + these else values. */ >>> + if (if_info->else_bb) >>> + { >>> + init_noce_multiple_sets_info (if_info->else_bb, else_insn_info); >>> + int else_count = 0; >>> + rtx_insn *before_else = get_last_insn (); >>> + location_t saved_location = curr_insn_location (); >>> + rtx_insn *else_insn; >>> + FOR_BB_INSNS (if_info->else_bb, else_insn) >>> + { >>> + if (!active_insn_p (else_insn)) >>> + continue; >>> + >>> + noce_multiple_sets_info *info = else_insn_info[else_count]; >>> + rtx set = single_set (else_insn); >>> + gcc_checking_assert (set && REG_P (SET_DEST (set)) >>> + && !HARD_REGISTER_P (SET_DEST (set))); >>> + >>> + rtx target = SET_DEST (set); >>> + rtx value = copy_rtx (SET_SRC (set)); >>> + int i, ii; >>> + FOR_EACH_VEC_ELT (info->rewired_src, i, ii) >>> + value = simplify_replace_rtx (value, >>> + else_insn_info[ii]->target, >>> + else_insn_info[ii]->temporary); >>> + >>> + set_curr_insn_location (INSN_LOCATION (else_insn)); >>> + rtx temporary = copy_to_mode_reg (GET_MODE (target), value); >>> + >>> + info->target = target; >>> + info->temporary = temporary; >>> + info->unmodified_insn = else_insn; >>> + else_count++; >>> + } >>> + >>> + set_curr_insn_location (saved_location); >>> + >>> + gcc_checking_assert (else_count == (int) else_insn_info.length ()); >>> + >>> + /* These insns run ahead of the conditional moves. If they change a >>> + register the comparison reads we cannot reuse it, so bail. If they >>> + only clobber the condition code, drop the shared compare so that every >>> + move re-materializes its own. */ >>> + >>> + rtx_insn *first_else >>> + = before_else ? NEXT_INSN (before_else) : get_insns (); >>> + for (rtx_insn *ei = first_else; ei; ei = NEXT_INSN (ei)) >>> + { >>> + if (modified_in_p (cond, ei)) >>> + { >>> + end_sequence (); >>> + return false; >>> + } >>> + if (cc_cmp >>> + && (modified_in_p (cc_cmp, ei) >>> + || (rev_cc_cmp && modified_in_p (rev_cc_cmp, ei)))) >>> + { >>> + cc_cmp = NULL_RTX; >>> + rev_cc_cmp = NULL_RTX; >>> + } >>> + } >>> + } >>> >>> FOR_BB_INSNS (then_bb, insn) >>> { >>> @@ -3957,6 +4044,17 @@ noce_convert_multiple_sets_1 (struct noce_if_info >>> *if_info, >>> >>> rtx old_val = target; >>> >>> + /* Use the final value assigned to TARGET on the else arm. Scanning >>> in >>> + reverse is important when the arm assigns the same register more than >>> + once. */ >>> + if (if_info->else_bb) >>> + for (int j = else_insn_info.length () - 1; j >= 0; --j) >>> + if (rtx_equal_p (target, else_insn_info[j]->target)) >>> + { >>> + old_val = else_insn_info[j]->temporary; >>> + break; >>> + } >>> + >>> /* As we are transforming >>> if (x > y) >>> { >>> @@ -4244,19 +4342,29 @@ init_noce_multiple_sets_info (basic_block bb, >>> } >>> >>> /* Return true iff basic block TEST_BB is suitable for conversion to a >>> - series of conditional moves. Also check that we have more than one >>> - set (other routines can handle a single set better than we would), >>> - and fewer than PARAM_MAX_RTL_IF_CONVERSION_INSNS sets. While going >>> - through the insns store the sum of their potential costs in COST. */ >>> + series of conditional moves. Unless REQUIRE_MULTIPLE is false, also >>> check >>> + that we have more than one set (other routines can handle a single set >>> + better than we would). A diamond arm may have a single set when it is >>> + selected as the secondary arm. Require fewer than >>> + PARAM_MAX_RTL_IF_CONVERSION_INSNS sets. While going through the insns >>> store >>> + the sum of their potential costs in COST. On success, if >>> LIVE_OUT_DESTS is >>> + nonnull, record the distinct pseudo destinations that are live out of >>> + TEST_BB. */ >>> >>> static bool >>> -bb_ok_for_noce_convert_multiple_sets (basic_block test_bb, unsigned *cost) >>> +bb_ok_for_noce_convert_multiple_sets (basic_block test_bb, unsigned *cost, >>> + bool require_multiple = true, >>> + bitmap live_out_dests = NULL) >>> { >>> rtx_insn *insn; >>> unsigned count = 0; >>> unsigned param = param_max_rtl_if_conversion_insns; >>> bool speed_p = optimize_bb_for_speed_p (test_bb); >>> unsigned potential_cost = 0; >>> + if (live_out_dests) >>> + bitmap_clear (live_out_dests); >>> + bitmap bb_live_out >>> + = live_out_dests ? df_get_live_out (test_bb) : NULL; >>> >>> FOR_BB_INSNS (test_bb, insn) >>> { >>> @@ -4289,6 +4397,9 @@ bb_ok_for_noce_convert_multiple_sets (basic_block >>> test_bb, unsigned *cost) >>> if (!can_conditionally_move_p (GET_MODE (dest))) >>> return false; >>> >>> + if (live_out_dests && bitmap_bit_p (bb_live_out, REGNO (dest))) >>> + bitmap_set_bit (live_out_dests, REGNO (dest)); >>> + >>> potential_cost += insn_cost (insn, speed_p); >>> >>> count++; >>> @@ -4297,11 +4408,13 @@ bb_ok_for_noce_convert_multiple_sets (basic_block >>> test_bb, unsigned *cost) >>> *cost += potential_cost; >>> >>> /* If we would only put out one conditional move, the other strategies >>> - this pass tries are better optimized and will be more appropriate. >>> + this pass tries are better optimized and will be more appropriate, so >>> + require more than one set unless REQUIRE_MULTIPLE is false. A diamond >>> + arm may have one set when it is selected as the secondary arm. >>> Some targets want to strictly limit the number of conditional moves >>> that are emitted, they set this through PARAM, we need to respect >>> that. */ >>> - return count > 1 && count <= param; >>> + return count >= (require_multiple ? 2u : 1u) && count <= param; >>> } >>> >>> /* Compute average of two given costs weighted by relative probabilities >>> @@ -4342,10 +4455,11 @@ noce_process_if_block (struct noce_if_info *if_info) >>> (2) x = b; if (...) x = a; >>> (3) if (...) x = a; // as if with an initial x = x. >>> (4) if (...) { x = a; y = b; z = c; } // Like 3, for multiple SETS. >>> + (5) A multi-set IF-THEN-ELSE-JOIN. >>> The later patterns require jumps to be more expensive. >>> - For the if (...) x = a; else x = b; case we allow multiple insns >>> - inside the then and else blocks as long as their only effect is >>> - to calculate a value for x. >>> + For the diamond case, use an arm that sets at least two distinct >>> live-out >>> + pseudos as the primary arm. Every live-out destination set by the >>> + secondary arm must also be set by the primary arm. >>> ??? For future expansion, further expand the "multiple X" rules. */ >>> >>> /* First look for multiple SETS. >>> @@ -4354,25 +4468,69 @@ noce_process_if_block (struct noce_if_info *if_info) >>> If a target re-uses the existing CC comparison we keep track of that >>> and add the costs before default noce_conversion_profitable_p. */ >>> >>> - unsigned potential_cost = if_info->original_cost; >>> unsigned old_cost = if_info->original_cost; >>> - if (!else_bb >>> - && HAVE_conditional_move >>> - && bb_ok_for_noce_convert_multiple_sets (then_bb, &potential_cost)) >>> - { >>> - /* Temporarily set the original costs to what we estimated so >>> - we can determine if the transformation is worth it. */ >>> - if_info->original_cost = potential_cost; >>> - if (noce_convert_multiple_sets (if_info)) >>> + unsigned ms_then_cost = 0, ms_else_cost = 0; >>> + auto_bitmap ms_then_live_out_dests, ms_else_live_out_dests; >>> + noce_if_info ms_if_info = *if_info; >>> + bool multiple_sets_p = false; >>> + >>> + if (HAVE_conditional_move) >>> + { >>> + if (!else_bb) >>> + multiple_sets_p >>> + = bb_ok_for_noce_convert_multiple_sets (then_bb, &ms_then_cost); >>> + else if (!if_info->then_else_reversed >>> + && bb_ok_for_noce_convert_multiple_sets (then_bb, >>> + &ms_then_cost, false, >>> + ms_then_live_out_dests) >>> + && bb_ok_for_noce_convert_multiple_sets (else_bb, >>> + &ms_else_cost, false, >>> + ms_else_live_out_dests)) >>> { >>> - if (dump_file && if_info->transform_name) >>> + if (bitmap_count_bits (ms_then_live_out_dests) >= 2 >>> + && !bitmap_intersect_compl_p (ms_else_live_out_dests, >>> + ms_then_live_out_dests)) >>> + multiple_sets_p = true; >>> + else if (bitmap_count_bits (ms_else_live_out_dests) >= 2 >>> + && !bitmap_intersect_compl_p (ms_then_live_out_dests, >>> + ms_else_live_out_dests)) >>> + { >>> + /* The branch-target arm is the compatible multi-set superset. >>> + Make it the primary arm and reverse the select orientation. */ >>> + std::swap (ms_if_info.then_bb, ms_if_info.else_bb); >>> + ms_if_info.then_else_reversed >>> + = !ms_if_info.then_else_reversed; >>> + std::swap (ms_then_cost, ms_else_cost); >>> + multiple_sets_p = true; >>> + } >>> + } >>> + } >>> + >>> + if (multiple_sets_p) >>> + { >>> + /* The original code runs the comparison and one arm. Estimate that >>> cost >>> + (for a diamond weight the two arms by their probabilities) and let >>> + noce_convert_multiple_sets convert only if the conditional moves come >>> + out cheaper. */ >>> + unsigned potential_cost = old_cost + ms_then_cost; >>> + if (ms_if_info.else_bb) >>> + { >>> + if (optimize_bb_for_speed_p (test_bb)) >>> + potential_cost >>> + = old_cost + average_cost (ms_then_cost, ms_else_cost, >>> + find_edge (test_bb, >>> + ms_if_info.then_bb)); >>> + else >>> + potential_cost = old_cost + ms_then_cost + ms_else_cost; >>> + } >>> + ms_if_info.original_cost = potential_cost; >>> + if (noce_convert_multiple_sets (&ms_if_info)) >>> + { >>> + if (dump_file && ms_if_info.transform_name) >>> fprintf (dump_file, "if-conversion succeeded through %s\n", >>> - if_info->transform_name); >>> + ms_if_info.transform_name); >>> return true; >>> } >>> - >>> - /* Restore the original costs. */ >>> - if_info->original_cost = old_cost; >>> } >>> >>> bool speed_p = optimize_bb_for_speed_p (test_bb); >>> diff --git a/gcc/testsuite/gcc.c-torture/execute/ifcvt-diamond-1.c >>> b/gcc/testsuite/gcc.c-torture/execute/ifcvt-diamond-1.c >>> new file mode 100644 >>> index 00000000000..b6a883e3de4 >>> --- /dev/null >>> +++ b/gcc/testsuite/gcc.c-torture/execute/ifcvt-diamond-1.c >>> @@ -0,0 +1,143 @@ >>> +/* Runtime correctness of if-converted IF-THEN-ELSE-JOIN diamonds with >>> + multiple output registers (noce_convert_multiple_sets). */ >>> + >>> +long g1, g2, g3; >>> + >>> +__attribute__ ((noipa)) void >>> +diamond2 (long c, long x, long y) >>> +{ >>> + long a, b; >>> + if (c & 3) >>> + { >>> + a = x + 1; >>> + b = y - 2; >>> + } >>> + else >>> + { >>> + a = x * 4; >>> + b = y + 9; >>> + } >>> + g1 = a; >>> + g2 = b; >>> +} >>> + >>> +/* Then arm reads an earlier then output (arm-internal dependency). */ >>> +__attribute__ ((noipa)) void >>> +diamond3 (long c, long p, long q) >>> +{ >>> + long a, b, d; >>> + if (c > 0) >>> + { >>> + a = p ^ q; >>> + b = a + 7; >>> + d = q * 2; >>> + } >>> + else >>> + { >>> + a = p & q; >>> + b = q | 1; >>> + d = p - 3; >>> + } >>> + g1 = a; >>> + g2 = b; >>> + g3 = d; >>> +} >>> + >>> +/* A single output in the else arm. The other register keeps its incoming >>> + value on the else path. */ >>> +__attribute__ ((noipa)) void >>> +diamond_then2_else1 (long c, long x, long y) >>> +{ >>> + long a = x, b = y; >>> + if (c < 0) >>> + { >>> + a = x + 100; >>> + b = y + 200; >>> + } >>> + else >>> + a = x - 50; >>> + g1 = a; >>> + g2 = b; >>> +} >>> + >>> +/* Keep the single-set arm as the likely fallthrough block. The multi-set >>> + arm must become the primary arm of the conversion. */ >>> +__attribute__ ((noipa)) void >>> +diamond_reversed_then2_else1 (long c, long x) >>> +{ >>> + long type = c & 3; >>> + long next = type; >>> + long advance; >>> + if (__builtin_expect (type != 0, 1)) >>> + advance = type + 1; >>> + else >>> + { >>> + next = x + 1; >>> + advance = x + 2; >>> + } >>> + g1 = next; >>> + g2 = advance; >>> +} >>> + >>> +/* Each arm produces a live-out value that the other arm does not. */ >>> +__attribute__ ((noipa)) void >>> +diamond_unmatched_liveouts (long c, long p, long q) >>> +{ >>> + long a = 100, t = 300, e = 200; >>> + if (c & 4) >>> + { >>> + a = p + q; >>> + t = p * 2; >>> + } >>> + else >>> + { >>> + a = p - q; >>> + e = q * 2; >>> + } >>> + g1 = a; >>> + g2 = e; >>> + g3 = t; >>> +} >>> + >>> +int >>> +main (void) >>> +{ >>> + for (long c = -4; c <= 12; c++) >>> + for (long p = -6; p <= 6; p++) >>> + for (long q = -6; q <= 6; q++) >>> + { >>> + diamond2 (c, p, q); >>> + if (g1 != ((c & 3) ? p + 1 : p * 4) >>> + || g2 != ((c & 3) ? q - 2 : q + 9)) >>> + __builtin_abort (); >>> + >>> + diamond3 (c, p, q); >>> + { >>> + long ea = (c > 0) ? (p ^ q) : (p & q); >>> + long eb = (c > 0) ? ea + 7 : (q | 1); >>> + long ed = (c > 0) ? (q * 2) : (p - 3); >>> + if (g1 != ea || g2 != eb || g3 != ed) >>> + __builtin_abort (); >>> + } >>> + >>> + diamond_then2_else1 (c, p, q); >>> + if (g1 != ((c < 0) ? p + 100 : p - 50) >>> + || g2 != ((c < 0) ? q + 200 : q)) >>> + __builtin_abort (); >>> + >>> + diamond_reversed_then2_else1 (c, p); >>> + { >>> + long type = c & 3; >>> + if (g1 != (type ? type : p + 1) >>> + || g2 != (type ? type + 1 : p + 2)) >>> + __builtin_abort (); >>> + } >>> + >>> + diamond_unmatched_liveouts (c, p, q); >>> + if (g1 != ((c & 4) ? p + q : p - q) >>> + || g2 != ((c & 4) ? 200 : q * 2) >>> + || g3 != ((c & 4) ? p * 2 : 300)) >>> + __builtin_abort (); >>> + } >>> + return 0; >>> +} >>> diff --git a/gcc/testsuite/gcc.target/aarch64/ifcvt_multiple_sets_diamond.c >>> b/gcc/testsuite/gcc.target/aarch64/ifcvt_multiple_sets_diamond.c >>> new file mode 100644 >>> index 00000000000..58105001fd2 >>> --- /dev/null >>> +++ b/gcc/testsuite/gcc.target/aarch64/ifcvt_multiple_sets_diamond.c >>> @@ -0,0 +1,66 @@ >>> +/* Test if-conversion of IF-THEN-ELSE-JOIN diamonds with multiple output >>> + registers through noce_convert_multiple_sets. */ >>> +/* { dg-do compile } */ >>> +/* { dg-options "-O2 -fdump-rtl-ce1" } */ >>> + >>> +void sink2 (long, long); >>> + >>> +/* Two outputs, both arms write the same registers. */ >>> +void >>> +diamond_arith (long c, long x, long y) >>> +{ >>> + long a, b; >>> + if (c > 7) >>> + { >>> + a = x + 1; >>> + b = y - 2; >>> + } >>> + else >>> + { >>> + a = x * 4; >>> + b = y + 9; >>> + } >>> + sink2 (a, b); >>> +} >>> + >>> +/* Two outputs computed from constants on each arm. */ >>> +void >>> +diamond_const (long c, long x, long y) >>> +{ >>> + long a, b; >>> + if (c == 3) >>> + { >>> + a = 5; >>> + b = 7; >>> + } >>> + else >>> + { >>> + a = 9; >>> + b = 11; >>> + } >>> + sink2 (a, b); >>> +} >>> + >>> +/* Two outputs in the then arm, a single output in the else arm. The >>> second >>> + register keeps its incoming value on the else path. */ >>> +void >>> +diamond_then2_else1 (long c, long x, long y) >>> +{ >>> + long a = x, b = y; >>> + if (c < 0) >>> + { >>> + a = x + 100; >>> + b = y + 200; >>> + } >>> + else >>> + a = x - 50; >>> + sink2 (a, b); >>> +} >>> + >>> +/* { dg-final { scan-rtl-dump-times "if-conversion succeeded through >>> noce_convert_multiple_sets" 3 "ce1" } } */ >>> + >>> +/* The converted diamonds are branchless: no conditional branch remains. >>> */ >>> +/* { dg-final { scan-assembler-not >>> {\tb(eq|ne|cs|cc|mi|pl|vs|vc|hi|ls|ge|lt|gt|le)\t} } } */ >>> +/* { dg-final { scan-assembler-not "\tcbn?z\t" } } */ >>> +/* { dg-final { scan-assembler-not "\ttbn?z\t" } } */ >>> +/* { dg-final { scan-assembler "\tcsel\t" } } */ >>> diff --git >>> a/gcc/testsuite/gcc.target/aarch64/ifcvt_multiple_sets_diamond_2.c >>> b/gcc/testsuite/gcc.target/aarch64/ifcvt_multiple_sets_diamond_2.c >>> new file mode 100644 >>> index 00000000000..f5891730cad >>> --- /dev/null >>> +++ b/gcc/testsuite/gcc.target/aarch64/ifcvt_multiple_sets_diamond_2.c >>> @@ -0,0 +1,52 @@ >>> +/* Each arm of the diamond loads from a selected address and advances a >>> + pointer by a selected amount. Once the two arm loads are commoned the >>> + diamond writes two registers on both arms (the load result and the >>> + advance), which noce_convert_multiple_sets turns into conditional >>> moves. */ >>> +/* { dg-do compile } */ >>> +/* { dg-options "-O2 -fdump-rtl-ce1" } */ >>> + >>> +#include <stddef.h> >>> +#include <stdint.h> >>> + >>> +extern const int16_t lentab[256]; >>> + >>> +static inline uint32_t >>> +extract (uint32_t val, size_t type) >>> +{ >>> + const uint64_t masks = 0x0000FFFF00FF0000ull; >>> + return val & (uint32_t) ((masks >> (type * 16)) & 0xFFFF); >>> +} >>> + >>> +ptrdiff_t >>> +f (const uint8_t *ip, size_t tag, const uint8_t *end, ptrdiff_t op) >>> +{ >>> + do >>> + { >>> + const uint8_t *old_ip = ip; >>> + ptrdiff_t lmo = lentab[tag]; >>> + size_t type = tag & 3; >>> + if (type == 0) >>> + { >>> + size_t n = (tag >> 2) + 1; >>> + tag = ip[n]; >>> + ip += n + 1; >>> + } >>> + else >>> + { >>> + tag = ip[type]; >>> + ip += type + 1; >>> + } >>> + uint32_t next = (uint32_t) old_ip[0] | ((uint32_t) old_ip[1] << 8); >>> + ptrdiff_t extracted = extract (next, type); >>> + op += lmo - extracted; >>> + } >>> + while (ip < end); >>> + return op; >>> +} >>> + >>> +/* { dg-final { scan-rtl-dump "if-conversion succeeded through >>> noce_convert_multiple_sets" "ce1" } } */ >>> +/* { dg-final { scan-assembler-times "\tcsinc\t" 2 } } */ >>> +/* { dg-final { scan-assembler-times "\tldrb\t" 1 } } */ >>> +/* { dg-final { scan-assembler-not {\tb(eq|ne)\t} } } */ >>> +/* { dg-final { scan-assembler-not {\tcbn?z\t} } } */ >>> +/* { dg-final { scan-assembler-not {\ttbn?z\t} } } */ >>> diff --git >>> a/gcc/testsuite/gcc.target/aarch64/ifcvt_multiple_sets_diamond_3.c >>> b/gcc/testsuite/gcc.target/aarch64/ifcvt_multiple_sets_diamond_3.c >>> new file mode 100644 >>> index 00000000000..c459703e9bd >>> --- /dev/null >>> +++ b/gcc/testsuite/gcc.target/aarch64/ifcvt_multiple_sets_diamond_3.c >>> @@ -0,0 +1,57 @@ >>> +/* Test dependencies between sets in both arms of an IF-THEN-ELSE-JOIN >>> + diamond. */ >>> +/* { dg-do run } */ >>> +/* { dg-options "-O2 --param=max-rtl-if-conversion-unpredictable-cost=100 >>> -fdump-rtl-ce1" } */ >>> +/* Keep both assignments to x in the same RTL pseudo. */ >>> +/* { dg-additional-options "-fno-tree-ter -fno-tree-coalesce-vars" } */ >>> + >>> +volatile long gx, gy; >>> + >>> +__attribute__ ((noipa)) void >>> +diamond_dependencies (long c, long a, long b) >>> +{ >>> + long x, y; >>> + if (c & 1) >>> + { >>> + x = a + 1; >>> + y = x ^ b; >>> + x = y + 3; >>> + } >>> + else >>> + { >>> + x = b - 1; >>> + y = x ^ a; >>> + x = y - 3; >>> + } >>> + gx = x; >>> + gy = y; >>> +} >>> + >>> +__attribute__ ((optimize ("O0"))) int >>> +main (void) >>> +{ >>> + for (long c = -3; c <= 3; ++c) >>> + for (long a = -5; a <= 5; ++a) >>> + for (long b = -5; b <= 5; ++b) >>> + { >>> + long x, y; >>> + diamond_dependencies (c, a, b); >>> + if (c & 1) >>> + { >>> + long first_x = a + 1; >>> + y = first_x ^ b; >>> + x = y + 3; >>> + } >>> + else >>> + { >>> + long first_x = b - 1; >>> + y = first_x ^ a; >>> + x = y - 3; >>> + } >>> + if (gx != x || gy != y) >>> + __builtin_abort (); >>> + } >>> + return 0; >>> +} >>> + >>> +/* { dg-final { scan-rtl-dump-times "if-conversion succeeded through >>> noce_convert_multiple_sets" 1 "ce1" } } */ >>> diff --git >>> a/gcc/testsuite/gcc.target/aarch64/ifcvt_multiple_sets_diamond_4.c >>> b/gcc/testsuite/gcc.target/aarch64/ifcvt_multiple_sets_diamond_4.c >>> new file mode 100644 >>> index 00000000000..22617c4d098 >>> --- /dev/null >>> +++ b/gcc/testsuite/gcc.target/aarch64/ifcvt_multiple_sets_diamond_4.c >>> @@ -0,0 +1,64 @@ >>> +/* { dg-do run } */ >>> +/* { dg-options "-O2 --param=max-rtl-if-conversion-unpredictable-cost=100 >>> -fdump-rtl-ce1" } */ >>> + >>> +volatile long ga, gt, ge; >>> + >>> +__attribute__ ((noipa)) void >>> +convertible (long c, long p, long q) >>> +{ >>> + long a, t; >>> + if (c > 7) >>> + { >>> + a = p + 1; >>> + t = q + 2; >>> + } >>> + else >>> + { >>> + a = p - 3; >>> + t = q - 4; >>> + } >>> + ga = a; >>> + gt = t; >>> +} >>> + >>> +__attribute__ ((noipa)) void >>> +reject_arm_only_values (long c, long p, long q, long t, long e) >>> +{ >>> + long a; >>> + if (c > 7) >>> + { >>> + a = p + 1; >>> + t = q + 2; >>> + } >>> + else >>> + { >>> + a = p - 3; >>> + e = q - 4; >>> + } >>> + ga = a; >>> + gt = t; >>> + ge = e; >>> +} >>> + >>> +__attribute__ ((optimize ("O0"))) int >>> +main (void) >>> +{ >>> + convertible (8, 10, 20); >>> + if (ga != 11 || gt != 22) >>> + __builtin_abort (); >>> + convertible (7, 10, 20); >>> + if (ga != 7 || gt != 16) >>> + __builtin_abort (); >>> + >>> + reject_arm_only_values (8, 10, 20, 31, 47); >>> + if (ga != 11 || gt != 22 || ge != 47) >>> + __builtin_abort (); >>> + reject_arm_only_values (7, 10, 20, 31, 47); >>> + if (ga != 7 || gt != 31 || ge != 16) >>> + __builtin_abort (); >>> + return 0; >>> +} >>> + >>> +/* The first diamond converts. The second must be rejected because each >>> arm >>> + has a live-out value not assigned by the other arm. */ >>> +/* { dg-final { scan-rtl-dump-times "if-conversion succeeded through >>> noce_convert_multiple_sets" 1 "ce1" } } */ >>> diff --git >>> a/gcc/testsuite/gcc.target/aarch64/ifcvt_multiple_sets_diamond_5.c >>> b/gcc/testsuite/gcc.target/aarch64/ifcvt_multiple_sets_diamond_5.c >>> new file mode 100644 >>> index 00000000000..7490df7ddc0 >>> --- /dev/null >>> +++ b/gcc/testsuite/gcc.target/aarch64/ifcvt_multiple_sets_diamond_5.c >>> @@ -0,0 +1,31 @@ >>> +/* Test a diamond whose likely fallthrough arm has one set and whose other >>> + arm has multiple sets, including a live-out not changed by the >>> fallthrough >>> + arm. */ >>> +/* { dg-do compile } */ >>> +/* { dg-options "-O2 -fdump-rtl-ce1" } */ >>> +/* { dg-additional-options >>> "--param=max-rtl-if-conversion-predictable-cost=100" } */ >>> + >>> +unsigned long >>> +f (const unsigned char *p, unsigned long tag) >>> +{ >>> + unsigned long type = tag & 3; >>> + unsigned long next = type; >>> + unsigned long advance; >>> + if (__builtin_expect (type != 0, 1)) >>> + advance = type + 1; >>> + else >>> + { >>> + unsigned long base = tag >> 2; >>> + next = base + 1; >>> + advance = base + 2; >>> + } >>> + return p[next] + (advance << 8); >>> +} >>> + >>> +/* { dg-final { scan-rtl-dump-times "if-conversion succeeded through >>> noce_convert_multiple_sets" 1 "ce1" } } */ >>> +/* { dg-final { scan-assembler-times "\tcsinc\t" 1 } } */ >>> +/* { dg-final { scan-assembler-times "\tcsel\t" 1 } } */ >>> +/* { dg-final { scan-assembler-times "\tldrb\t" 1 } } */ >>> +/* { dg-final { scan-assembler-not {\tb(eq|ne)\t} } } */ >>> +/* { dg-final { scan-assembler-not {\tcbn?z\t} } } */ >>> +/* { dg-final { scan-assembler-not {\ttbn?z\t} } } */ >>> diff --git a/gcc/testsuite/gcc.target/i386/ifcvt-multiple-sets-diamond-1.c >>> b/gcc/testsuite/gcc.target/i386/ifcvt-multiple-sets-diamond-1.c >>> new file mode 100644 >>> index 00000000000..fb4491074f3 >>> --- /dev/null >>> +++ b/gcc/testsuite/gcc.target/i386/ifcvt-multiple-sets-diamond-1.c >>> @@ -0,0 +1,44 @@ >>> +/* { dg-do run } */ >>> +/* { dg-require-effective-target lp64 } */ >>> +/* { dg-options "-O2 -mtune=generic -fdump-rtl-ce1" } */ >>> +/* { dg-additional-options >>> "--param=max-rtl-if-conversion-predictable-cost=100" } */ >>> + >>> +/* The single-set arm is the likely fallthrough block. Speculative >>> arithmetic >>> + clobbers FLAGS, so each conditional move must re-materialize the >>> + comparison. */ >>> + >>> +volatile long ga, gb; >>> + >>> +__attribute__ ((noipa)) void >>> +f (long c, long x, long y, long b) >>> +{ >>> + long a; >>> + if (__builtin_expect (c <= 7, 1)) >>> + a = y + 3; >>> + else >>> + { >>> + a = x + 1; >>> + b = y + 2; >>> + } >>> + ga = a; >>> + gb = b; >>> +} >>> + >>> +/* Keep the runtime driver out of noce so that the dump count is specific >>> to >>> + F. */ >>> +__attribute__ ((optimize ("O0"))) int >>> +main (void) >>> +{ >>> + for (long c = 5; c != 11; ++c) >>> + for (long x = -8; x != 9; ++x) >>> + for (long y = -8; y != 9; ++y) >>> + { >>> + f (c, x, y, 4); >>> + if (ga != (c > 7 ? x + 1 : y + 3) >>> + || gb != (c > 7 ? y + 2 : 4)) >>> + __builtin_abort (); >>> + } >>> + return 0; >>> +} >>> + >>> +/* { dg-final { scan-rtl-dump-times "if-conversion succeeded through >>> noce_convert_multiple_sets" 1 "ce1" } } */ >>> -- >>> 2.50.1 (Apple Git-155) >>> >> >
