Hi,
Gentle ping: https://gcc.gnu.org/pipermail/gcc-patches/2022-July/598601.html
BR,
Kewen
on 2022/7/20 17:30, Kewen.Lin via Gcc-patches wrote:
> Hi,
>
> Commit r12-6679-g7ca1582ca60dc8 made vectorizer accept one
> unroll factor to be applied to vectorization factor when
> vectorizing the main loop, it would be suggested by target
> when doing costing.
>
> This patch introduces function determine_suggested_unroll_factor
> for rs6000 port, to make it be able to suggest the unroll factor
> for a given loop being vectorized. Referring to aarch64 port
> and basing on the analysis on SPEC2017 performance evaluation
> results, it mainly considers these aspects:
> 1) unroll option and pragma which can disable unrolling for the
> given loop;
> 2) simple hardware resource model with issued non memory access
> vector insn per cycle;
> 3) aggressive heuristics when iteration count is unknown:
> - reduction case to break cross iteration dependency;
> - emulated gather load;
> 4) estimated iteration count when iteration count is unknown;
>
> With this patch, SPEC2017 performance evaluation results on
> Power8/9/10 are listed below (speedup pct.):
>
> * Power10
> - O2: all are neutral (excluding some noises);
> - Ofast: 510.parest_r +6.67%, the others are neutral
> (use ... for the followings);
> - Ofast + unroll: 510.parest_r +5.91%, ...
> - Ofast + LTO + PGO: 510.parest_r +3.00%, ...
> - Ofast + cheap vect cost: 510.parest_r +6.23%, ...
> - Ofast + very-cheap vect cost: all are neutral;
>
> * Power9
> - Ofast: 510.parest_r +8.73%, 538.imagick_r +11.18%
> (likely noise), 500.perlbench_r +1.84%, ...
>
> * Power8
> - Ofast: 510.parest_r +5.43%, ...;
>
> This patch also introduces one documented parameter
> rs6000-vect-unroll-limit= similar to what aarch64 proposes,
> by evaluating on P8/P9/P10, the default value 4 is slightly
> better than the other choices like 2 and 8.
>
> It also parameterizes two other values as undocumented
> parameters for future tweaking. One parameter is
> rs6000-vect-unroll-issue, it's to simply model hardware
> resource for non memory access vector instructions to avoid
> excessive unrolling, initially I tried to use the value in
> the hook rs6000_issue_rate, but the evaluation showed it's
> bad, so I evaluated different values 2/4/6/8 on P8/P9/P10 at
> Ofast, the results showed the default value 4 is good enough
> on these different architectures. For a record, choice 8
> could make 510.parest_r's gain become smaller or gone on
> P8/P9/P10; choice 6 could make 503.bwaves_r degrade by more
> than 1% on P8/P10; and choice 2 could make 538.imagick_r
> degrade by 3.8%. The other parameter is
> rs6000-vect-unroll-reduc-threshold. It's mainly inspired by
> 510.parest_r and tweaked as it, evaluating with different
> values 0/1/2/3 for the threshold, it showed value 1 is the
> best choice. For a record, choice 0 could make 525.x264_r
> degrade by 2% and 527.cam4_r degrade by 2.95% on P10,
> 548.exchange2_r degrade by 1.41% and 527.cam4_r degrade by
> 2.54% on P8; choice 2 and bigger values could make
> 510.parest_r's gain become smaller.
>
> Bootstrapped and regtested on powerpc64-linux-gnu P7 and P8,
> and powerpc64le-linux-gnu P9. Bootstrapped on
> powerpc64le-linux-gnu P10, but one failure was exposed during
> regression testing there, it's identified as one miss
> optimization and can be reproduced without this support,
> PR106365 was opened for further tracking.
>
> Is it for trunk?
>
> BR,
> Kewen
> ------
> gcc/ChangeLog:
>
> * config/rs6000/rs6000.cc (class rs6000_cost_data): Add new members
> m_nstores, m_reduc_factor, m_gather_load and member function
> determine_suggested_unroll_factor.
> (rs6000_cost_data::update_target_cost_per_stmt): Update for m_nstores,
> m_reduc_factor and m_gather_load.
> (rs6000_cost_data::determine_suggested_unroll_factor): New function.
> (rs6000_cost_data::finish_cost): Use determine_suggested_unroll_factor.
> * config/rs6000/rs6000.opt (rs6000-vect-unroll-limit): New parameter.
> (rs6000-vect-unroll-issue): Likewise.
> (rs6000-vect-unroll-reduc-threshold): Likewise.
> * doc/invoke.texi (rs6000-vect-unroll-limit): Document new parameter.
>
> ---
> gcc/config/rs6000/rs6000.cc | 125 ++++++++++++++++++++++++++++++++++-
> gcc/config/rs6000/rs6000.opt | 18 +++++
> gcc/doc/invoke.texi | 7 ++
> 3 files changed, 147 insertions(+), 3 deletions(-)
>
> diff --git a/gcc/config/rs6000/rs6000.cc b/gcc/config/rs6000/rs6000.cc
> index 3ff16b8ae04..d0f107d70a8 100644
> --- a/gcc/config/rs6000/rs6000.cc
> +++ b/gcc/config/rs6000/rs6000.cc
> @@ -5208,16 +5208,23 @@ protected:
> vect_cost_model_location, unsigned int);
> void density_test (loop_vec_info);
> void adjust_vect_cost_per_loop (loop_vec_info);
> + unsigned int determine_suggested_unroll_factor (loop_vec_info);
>
> /* Total number of vectorized stmts (loop only). */
> unsigned m_nstmts = 0;
> /* Total number of loads (loop only). */
> unsigned m_nloads = 0;
> + /* Total number of stores (loop only). */
> + unsigned m_nstores = 0;
> + /* Reduction factor for suggesting unroll factor (loop only). */
> + unsigned m_reduc_factor = 0;
> /* Possible extra penalized cost on vector construction (loop only). */
> unsigned m_extra_ctor_cost = 0;
> /* For each vectorized loop, this var holds TRUE iff a non-memory vector
> instruction is needed by the vectorization. */
> bool m_vect_nonmem = false;
> + /* If this loop gets vectorized with emulated gather load. */
> + bool m_gather_load = false;
> };
>
> /* Test for likely overcommitment of vector hardware resources. If a
> @@ -5368,9 +5375,34 @@ rs6000_cost_data::update_target_cost_per_stmt
> (vect_cost_for_stmt kind,
> {
> m_nstmts += orig_count;
>
> - if (kind == scalar_load || kind == vector_load
> - || kind == unaligned_load || kind == vector_gather_load)
> - m_nloads += orig_count;
> + if (kind == scalar_load
> + || kind == vector_load
> + || kind == unaligned_load
> + || kind == vector_gather_load)
> + {
> + m_nloads += orig_count;
> + if (stmt_info && STMT_VINFO_GATHER_SCATTER_P (stmt_info))
> + m_gather_load = true;
> + }
> + else if (kind == scalar_store
> + || kind == vector_store
> + || kind == unaligned_store
> + || kind == vector_scatter_store)
> + m_nstores += orig_count;
> + else if ((kind == scalar_stmt
> + || kind == vector_stmt
> + || kind == vec_to_scalar)
> + && stmt_info
> + && vect_is_reduction (stmt_info))
> + {
> + /* Loop body contains normal int or fp operations and epilogue
> + contains vector reduction. For simplicity, we assume int
> + operation takes one cycle and fp operation takes one more. */
> + tree lhs = gimple_get_lhs (stmt_info->stmt);
> + bool is_float = FLOAT_TYPE_P (TREE_TYPE (lhs));
> + unsigned int basic_cost = is_float ? 2 : 1;
> + m_reduc_factor = MAX (basic_cost * orig_count, m_reduc_factor);
> + }
>
> /* Power processors do not currently have instructions for strided
> and elementwise loads, and instead we must generate multiple
> @@ -5462,6 +5494,90 @@ rs6000_cost_data::adjust_vect_cost_per_loop
> (loop_vec_info loop_vinfo)
> }
> }
>
> +/* Determine suggested unroll factor by considering some below factors:
> +
> + - unroll option/pragma which can disable unrolling for this loop;
> + - simple hardware resource model for non memory vector insns;
> + - aggressive heuristics when iteration count is unknown:
> + - reduction case to break cross iteration dependency;
> + - emulated gather load;
> + - estimated iteration count when iteration count is unknown;
> +*/
> +
> +
> +unsigned int
> +rs6000_cost_data::determine_suggested_unroll_factor (loop_vec_info
> loop_vinfo)
> +{
> + class loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
> +
> + /* Don't unroll if it's specified explicitly not to be unrolled. */
> + if (loop->unroll == 1
> + || (OPTION_SET_P (flag_unroll_loops) && !flag_unroll_loops)
> + || (OPTION_SET_P (flag_unroll_all_loops) && !flag_unroll_all_loops))
> + return 1;
> +
> + unsigned int nstmts_nonldst = m_nstmts - m_nloads - m_nstores;
> + /* Don't unroll if no vector instructions excepting for memory access. */
> + if (nstmts_nonldst == 0)
> + return 1;
> +
> + /* Consider breaking cross iteration dependency for reduction. */
> + unsigned int reduc_factor = m_reduc_factor > 1 ? m_reduc_factor : 1;
> +
> + /* Use this simple hardware resource model that how many non ld/st
> + vector instructions can be issued per cycle. */
> + unsigned int issue_width = rs6000_vect_unroll_issue;
> + unsigned int uf = CEIL (reduc_factor * issue_width, nstmts_nonldst);
> + uf = MIN ((unsigned int) rs6000_vect_unroll_limit, uf);
> + /* Make sure it is power of 2. */
> + uf = 1 << ceil_log2 (uf);
> +
> + /* If the iteration count is known, the costing would be exact enough,
> + don't worry it could be worse. */
> + if (LOOP_VINFO_NITERS_KNOWN_P (loop_vinfo))
> + return uf;
> +
> + /* Inspired by SPEC2017 parest_r, we want to aggressively unroll the
> + loop if either condition is satisfied:
> + - reduction factor exceeds the threshold;
> + - emulated gather load adopted. */
> + if (reduc_factor > (unsigned int) rs6000_vect_unroll_reduc_threshold
> + || m_gather_load)
> + return uf;
> +
> + /* Check if we can conclude it's good to unroll from the estimated
> + iteration count. */
> + HOST_WIDE_INT est_niter = get_estimated_loop_iterations_int (loop);
> + unsigned int vf = vect_vf_for_cost (loop_vinfo);
> + unsigned int unrolled_vf = vf * uf;
> + if (est_niter == -1 || est_niter < unrolled_vf)
> + /* When the estimated iteration of this loop is unknown, it's possible
> + that we are able to vectorize this loop with the original VF but fail
> + to vectorize it with the unrolled VF any more if the actual iteration
> + count is in between. */
> + return 1;
> + else
> + {
> + unsigned int epil_niter_unr = est_niter % unrolled_vf;
> + unsigned int epil_niter = est_niter % vf;
> + /* Even if we have partial vector support, it can be still inefficent
> + to calculate the length when the iteration count is unknown, so
> + only expect it's good to unroll when the epilogue iteration count
> + is not bigger than VF (only one time length calculation). */
> + if (LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo)
> + && epil_niter_unr <= vf)
> + return uf;
> + /* Without partial vector support, conservatively unroll this when
> + the epilogue iteration count is less than the original one
> + (epilogue execution time wouldn't be longer than before). */
> + else if (!LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo)
> + && epil_niter_unr <= epil_niter)
> + return uf;
> + }
> +
> + return 1;
> +}
> +
> void
> rs6000_cost_data::finish_cost (const vector_costs *scalar_costs)
> {
> @@ -5478,6 +5594,9 @@ rs6000_cost_data::finish_cost (const vector_costs
> *scalar_costs)
> && LOOP_VINFO_VECT_FACTOR (loop_vinfo) == 2
> && LOOP_REQUIRES_VERSIONING (loop_vinfo))
> m_costs[vect_body] += 10000;
> +
> + m_suggested_unroll_factor
> + = determine_suggested_unroll_factor (loop_vinfo);
> }
>
> vector_costs::finish_cost (scalar_costs);
> diff --git a/gcc/config/rs6000/rs6000.opt b/gcc/config/rs6000/rs6000.opt
> index 4931d781c4e..80c2c61a9de 100644
> --- a/gcc/config/rs6000/rs6000.opt
> +++ b/gcc/config/rs6000/rs6000.opt
> @@ -624,6 +624,14 @@ mieee128-constant
> Target Var(TARGET_IEEE128_CONSTANT) Init(1) Save
> Generate (do not generate) code that uses the LXVKQ instruction.
>
> +; Documented parameters
> +
> +-param=rs6000-vect-unroll-limit=
> +Target Joined UInteger Var(rs6000_vect_unroll_limit) Init(4) IntegerRange(1,
> 64) Param
> +Used to limit unroll factor which indicates how much the autovectorizer may
> +unroll a loop. The default value is 4.
> +
> +; Undocumented parameters
> -param=rs6000-density-pct-threshold=
> Target Undocumented Joined UInteger Var(rs6000_density_pct_threshold)
> Init(85) IntegerRange(0, 100) Param
> When costing for loop vectorization, we probably need to penalize the loop
> body
> @@ -661,3 +669,13 @@ Like parameter rs6000-density-load-pct-threshold, we
> also check if the total
> number of load statements exceeds the threshold specified by this parameter,
> and penalize only if it's satisfied. The default value is 20.
>
> +-param=rs6000-vect-unroll-issue=
> +Target Undocumented Joined UInteger Var(rs6000_vect_unroll_issue) Init(4)
> IntegerRange(1, 128) Param
> +Indicate how many non memory access vector instructions can be issued per
> +cycle, it's used in unroll factor determination for autovectorizer. The
> +default value is 4.
> +
> +-param=rs6000-vect-unroll-reduc-threshold=
> +Target Undocumented Joined UInteger Var(rs6000_vect_unroll_reduc_threshold)
> Init(1) Param
> +When reduction factor computed for a loop exceeds the threshold specified by
> +this parameter, prefer to unroll this loop. The default value is 1.
> diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
> index 84d6f0f9860..097ab1d5563 100644
> --- a/gcc/doc/invoke.texi
> +++ b/gcc/doc/invoke.texi
> @@ -29658,6 +29658,13 @@ Generate (do not generate) code that will run in
> privileged state.
> @opindex no-block-ops-unaligned-vsx
> Generate (do not generate) unaligned vsx loads and stores for
> inline expansion of @code{memcpy} and @code{memmove}.
> +
> +@item --param rs6000-vect-unroll-limit=
> +The vectorizer will check with target information to determine whether it
> +would be beneficial to unroll the main vectorized loop and by how much. This
> +parameter sets the upper bound of how much the vectorizer will unroll the
> main
> +loop. The default value is four.
> +
> @end table
>
> @node RX Options
> --
> 2.27.0
