Re: [PATCH V5 1/2] topology: Allow multiple entities to provide sched_freq_tick() callback

2021-03-09 Thread Ionela Voinescu
f arm64 specific code and is added > to arch_topology.c. > > Note that this also defines SCALE_FREQ_SOURCE_CPUFREQ but doesn't use it > and it is added to show that cpufreq is also acts as source of > information for FIE and will be used by default if no other counters are > su

Re: [PATCH V5 2/2] cpufreq: CPPC: Add support for frequency invariance

2021-03-09 Thread Ionela Voinescu
his CPPC counter-based frequency invariance support.. (disabling this config will not disable cpufreq or arch counter-based FIE) > this is all done under CONFIG_ACPI_CPPC_CPUFREQ_FIE, which is enabled by > default. > > This also exports sched_setattr_no

Re: [PATCH V3 2/2] cpufreq: cppc: Add support for frequency invariance

2021-02-22 Thread Ionela Voinescu
Hey, Some test results: On Thursday 18 Feb 2021 at 16:35:38 (+), Ionela Voinescu wrote: [..] > > +static void __init cppc_freq_invariance_init(void) > > +{ [..] > > + > > + ret = cppc_get_perf_ctrs(i, &fb_ctrs); > > + if (!ret) >

Re: [PATCH V3 1/2] topology: Allow multiple entities to provide sched_freq_tick() callback

2021-02-19 Thread Ionela Voinescu
On Friday 19 Feb 2021 at 10:28:23 (+0530), Viresh Kumar wrote: > On 18-02-21, 16:36, Ionela Voinescu wrote: > > Yes, we don't care if there is no cpufreq driver, as the use of AMUs won't > > get initialised either. But we do care if there is a cpufreq driver that >

Re: [PATCH V3 2/2] cpufreq: cppc: Add support for frequency invariance

2021-02-18 Thread Ionela Voinescu
Hi, On Thursday 28 Jan 2021 at 16:18:56 (+0530), Viresh Kumar wrote: > The Frequency Invariance Engine (FIE) is providing a frequency scaling > correction factor that helps achieve more accurate load-tracking. > > Normally, this scaling factor can be obtained directly with the help of > the cpufr

Re: [PATCH V3 1/2] topology: Allow multiple entities to provide sched_freq_tick() callback

2021-02-18 Thread Ionela Voinescu
Hey, On Thursday 18 Feb 2021 at 15:03:04 (+0530), Viresh Kumar wrote: > On 17-02-21, 00:24, Ionela Voinescu wrote: > > > diff --git a/arch/arm64/kernel/topology.c b/arch/arm64/kernel/topology.c > > > index 1e47dfd465f8..47fca7376c93 100644 > > > --- a/arch/arm64/kern

Re: [PATCH V3 1/2] topology: Allow multiple entities to provide sched_freq_tick() callback

2021-02-17 Thread Ionela Voinescu
On Wednesday 17 Feb 2021 at 17:10:27 (+0530), Viresh Kumar wrote: > On 17-02-21, 11:30, Ionela Voinescu wrote: > > The problem is not topology_scale_freq_invariant() but whether a scale > > factor is set for some CPUs. > > > > Scenario (test system above): > > -

Re: [PATCH V3 1/2] topology: Allow multiple entities to provide sched_freq_tick() callback

2021-02-17 Thread Ionela Voinescu
iresh Kumar wrote: > On 17-02-21, 00:24, Ionela Voinescu wrote: > > I think it could be merged in patch 1/2 as it's part of enabling the use > > of multiple sources of information for FIE. Up to you! > > Sure. > > > > static void amu_fie_setup(const struct cp

Re: [PATCH V3 1/2] topology: Allow multiple entities to provide sched_freq_tick() callback

2021-02-16 Thread Ionela Voinescu
Hey, On Friday 05 Feb 2021 at 14:44:24 (+0530), Viresh Kumar wrote: > On 03-02-21, 11:45, Ionela Voinescu wrote: > > Therefore, I think system level invariance management (checks and > > call to rebuild_sched_domains_energy()) also needs to move from arm64 > > code

Re: [PATCH V3 1/2] topology: Allow multiple entities to provide sched_freq_tick() callback

2021-02-03 Thread Ionela Voinescu
Hi Viresh, Many thanks for the renaming of functions/variables/enums. I've cropped all the code that looks good to me, and I kept some portions of interest. On Thursday 28 Jan 2021 at 16:18:55 (+0530), Viresh Kumar wrote: > This patch attempts to make it generic enough so other parts of the > ke

Re: [PATCH -next v2] sched/topology: Mark some symbols with static keyword

2021-02-02 Thread Ionela Voinescu
*sd, struct > sched_domain *parent) > #if defined(CONFIG_ENERGY_MODEL) && defined(CONFIG_CPU_FREQ_GOV_SCHEDUTIL) > DEFINE_STATIC_KEY_FALSE(sched_energy_present); > unsigned int sysctl_sched_energy_aware = 1; > -DEFINE_MUTEX(sched_energy_mutex); > -bool sched_energy_up

Re: [RFC V2 2/2] cpufreq: cppc: Add support for frequency invariance

2021-01-19 Thread Ionela Voinescu
Hi, Do you know of a current platform that would benefit from this, that we could run some tests on? I've Cc-ed Jeremy as well, as he might be interested in this. Also, please find some initial comments below: On Tuesday 15 Dec 2020 at 16:46:36 (+0530), Viresh Kumar wrote: > The Frequency Invar

Re: [RFC V2 1/2] topology: Allow multiple entities to provide sched_freq_tick() callback

2021-01-19 Thread Ionela Voinescu
Hi, On Friday 15 Jan 2021 at 13:18:47 (+0530), Viresh Kumar wrote: > On 13-01-21, 16:18, Ionela Voinescu wrote: > > On Tuesday 15 Dec 2020 at 16:46:35 (+0530), Viresh Kumar wrote: > > > +void topology_scale_freq_tick(void) > > > +{ > > > + struct scale_

Re: [RFC V2 1/2] topology: Allow multiple entities to provide sched_freq_tick() callback

2021-01-13 Thread Ionela Voinescu
Hi, I focused for now on the design of the solution and mostly on the generic code in arch_topology.c, as that will then impact the arm64 code. On Tuesday 15 Dec 2020 at 16:46:35 (+0530), Viresh Kumar wrote: [..] > > diff --git a/drivers/base/arch_topology.c b/drivers/base/arch_topology.c > in

Re: [PATCH V4 0/3] arm64: topology: improvements

2021-01-08 Thread Ionela Voinescu
15 +-- > 1 file changed, 56 insertions(+), 59 deletions(-) > Tested-by: Ionela Voinescu ..for the full set. Thanks, Ionela. > -- > 2.25.0.rc1.19.g042ed3e048af >

Re: [PATCH V4 3/3] arm64: topology: Make AMUs work with modular cpufreq drivers

2021-01-08 Thread Ionela Voinescu
mask_var(&amu_fie_cpus, GFP_KERNEL)) > + return -ENOMEM; > > -free_valid_mask: > - free_cpumask_var(valid_cpus); > + ret = cpufreq_register_notifier(&init_amu_fie_notifier, > + CPUFREQ_POLICY_NOTIFIER); > + if (ret) > + free_cpumask_var(amu_fie_cpus); > > return ret; > } > -late_initcall_sync(init_amu_fie); > +core_initcall(init_amu_fie); > > bool arch_freq_counters_available(const struct cpumask *cpus) > { > -- > 2.25.0.rc1.19.g042ed3e048af > Reviewed-by: Ionela Voinescu Thanks, Ionela.

[PATCH] cpufreq: Revert "cpufreq: qcom-hw: Use devm_platform_ioremap_resource() to simplify code"

2021-01-08 Thread Ionela Voinescu
[mem 0x17d43000-0x17d443ff] Signed-off-by: Ionela Voinescu --- Hi guys, I noticed the issue described in the commit message on DB845c platforms on 5.11-rc2. I tried to fix this, rather than revert it, but I did not find any better way to fix this, other than possibly calling devm_release_mem_r

Re: [PATCH V3 3/3] arm64: topology: Make AMUs work with modular cpufreq drivers

2021-01-08 Thread Ionela Voinescu
On Friday 08 Jan 2021 at 09:44:16 (+), Ionela Voinescu wrote: > On Thursday 17 Dec 2020 at 16:20:49 (+0530), Viresh Kumar wrote: > > On 16-12-20, 19:37, Ionela Voinescu wrote: > > > I did not yet test this, but reading this comment made me wonder.. > > > > >

Re: [PATCH V3 3/3] arm64: topology: Make AMUs work with modular cpufreq drivers

2021-01-08 Thread Ionela Voinescu
On Thursday 17 Dec 2020 at 16:20:49 (+0530), Viresh Kumar wrote: > On 16-12-20, 19:37, Ionela Voinescu wrote: > > I did not yet test this, but reading this comment made me wonder.. > > > > arch_scale_freq_invariant() (or topology_scale_freq_invariant()) is also > >

[PATCH 3/3] acpi: cppc_acpi: initialise vaddr pointers to NULL

2021-01-07 Thread Ionela Voinescu
-by: Ionela Voinescu Cc: "Rafael J. Wysocki" Cc: Len Brown --- drivers/acpi/cppc_acpi.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/acpi/cppc_acpi.c b/drivers/acpi/cppc_acpi.c index fd71020f5d5f..69057fcd2c04 100644 --- a/drivers/acpi/cppc_acpi.c +++

[PATCH 0/3] acpi: cppc_acpi: fix sparse warnings

2021-01-07 Thread Ionela Voinescu
Hi guys, These patches just fix some trivial sparse warnings. Hope they help, Ionela. Ionela Voinescu (3): acpi: cppc_acpi: remove __iomem annotation for cpc_reg's address acpi: cppc_acpi: add __iomem annotation to generic_comm_base pointer acpi: cppc_acpi: initialise vaddr pointe

[PATCH 2/3] acpi: cppc_acpi: add __iomem annotation to generic_comm_base pointer

2021-01-07 Thread Ionela Voinescu
Signed-off-by: Ionela Voinescu Cc: "Rafael J. Wysocki" Cc: Len Brown --- drivers/acpi/cppc_acpi.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/acpi/cppc_acpi.c b/drivers/acpi/cppc_acpi.c index 75aaf94ae0a9..fd71020f5d5f 100644 --- a/drivers/acpi/cppc_

[PATCH 1/3] acpi: cppc_acpi: remove __iomem annotation for cpc_reg's address

2021-01-07 Thread Ionela Voinescu
12:13: warning: dereference of noderef expression Suggested-by: Al Viro Signed-off-by: Ionela Voinescu Cc: Robert Moore Cc: Erik Kaneda Cc: "Rafael J. Wysocki" --- include/acpi/cppc_acpi.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/acpi/cppc_acpi.h

Re: arch/arm64/kernel/topology.c:367:22: sparse: sparse: dereference of noderef expression

2021-01-06 Thread Ionela Voinescu
On Wednesday 06 Jan 2021 at 17:47:58 (+), Al Viro wrote: > On Wed, Jan 06, 2021 at 03:07:24PM +0000, Ionela Voinescu wrote: > > > > > > 367 switch ((u64)reg->address) { > > > > > > That's not a dereference but I guess sparse complains

Re: arch/arm64/kernel/topology.c:367:22: sparse: sparse: dereference of noderef expression

2021-01-06 Thread Ionela Voinescu
Hi, On Wednesday 06 Jan 2021 at 16:13:53 (+), Al Viro wrote: > On Wed, Jan 06, 2021 at 03:52:14PM +0000, Ionela Voinescu wrote: > > > > > > vim +367 arch/arm64/kernel/topology.c > > > > > > > > > > > >362 > > > &g

Re: arch/arm64/kernel/topology.c:367:22: sparse: sparse: dereference of noderef expression

2021-01-06 Thread Ionela Voinescu
On Wednesday 06 Jan 2021 at 15:21:19 (+), Catalin Marinas wrote: > On Wed, Jan 06, 2021 at 03:07:24PM +0000, Ionela Voinescu wrote: > > On Friday 18 Dec 2020 at 10:44:10 (+), Catalin Marinas wrote: > > > On Fri, Dec 18, 2020 at 05:00:16AM +0800, kernel test robot w

Re: arch/arm64/kernel/topology.c:367:22: sparse: sparse: dereference of noderef expression

2021-01-06 Thread Ionela Voinescu
Hi, On Friday 18 Dec 2020 at 10:44:10 (+), Catalin Marinas wrote: > On Fri, Dec 18, 2020 at 05:00:16AM +0800, kernel test robot wrote: > > tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git > > master > > head: 74f602dc96dd854c7b2034947798c1e2a6b84066 > > commit: 68c

Re: [RFC V2 0/2] cpufreq: cppc: Add support for frequency invariance

2020-12-16 Thread Ionela Voinescu
Hi, On Tuesday 15 Dec 2020 at 16:46:34 (+0530), Viresh Kumar wrote: > Hello, > > CPPC cpufreq driver is used for ARM servers and this patch series tries > to provide frequency invariance support for them. > Nitpick: cppc_cpufreq already supports cpufreq-based frequency invariance, but not count

Re: [PATCH V3 3/3] arm64: topology: Make AMUs work with modular cpufreq drivers

2020-12-16 Thread Ionela Voinescu
Hi, On Wednesday 16 Dec 2020 at 10:08:05 (+0530), Viresh Kumar wrote: [..] > > > +static void amu_fie_setup(const struct cpumask *cpus) > > > { > > > - cpumask_var_t valid_cpus; > > > bool invariant; > > > - int ret = 0; > > > int cpu; > > > > > > - if (!zalloc_cpumask_var(&valid_cpus, GFP_

Re: [PATCH v2 0/4] cppc_cpufreq: fix, clarify and improve support

2020-12-15 Thread Ionela Voinescu
On Tuesday 15 Dec 2020 at 19:21:01 (+0100), Rafael J. Wysocki wrote: > On Mon, Dec 14, 2020 at 5:14 PM Mian Yousaf Kaukab wrote: > > > > On Mon, Dec 14, 2020 at 12:38:19PM +, Ionela Voinescu wrote: > > > Hi guys, > > > > > > I'm sending v

Re: [PATCH V3 3/3] arm64: topology: Make AMUs work with modular cpufreq drivers

2020-12-15 Thread Ionela Voinescu
Hi, On Tuesday 15 Dec 2020 at 11:04:16 (+0530), Viresh Kumar wrote: > The AMU counters won't get used today if the cpufreq driver is built as > a module as the amu core requires everything to be ready by late init. > > Fix that properly by registering for cpufreq policy notifier. Note that > the

Re: [PATCH V3 3/3] arm64: topology: Make AMUs work with modular cpufreq drivers

2020-12-15 Thread Ionela Voinescu
Hi, On Tuesday 15 Dec 2020 at 11:04:16 (+0530), Viresh Kumar wrote: > The AMU counters won't get used today if the cpufreq driver is built as > a module as the amu core requires everything to be ready by late init. > > Fix that properly by registering for cpufreq policy notifier. Note that > the

Re: [PATCH V3 2/3] arm64: topology: Reorder init_amu_fie() a bit

2020-12-15 Thread Ionela Voinescu
counter initialisation and use, retrigger the build of >* scheduling domains to ensure the information is propagated properly. > */ > - if (invariance_status != topology_scale_freq_invariant()) > + if (!invariant) > rebuild_sched_domains_energy(); > > free_valid_mask: > -- > 2.25.0.rc1.19.g042ed3e048af > Looks good! Reviewed-by: Ionela Voinescu

Re: [PATCH] arm64: topology: Cleanup init_amu_fie() a bit

2020-12-14 Thread Ionela Voinescu
Hi, Sorry, I missed this. On Friday 11 Dec 2020 at 16:35:55 (+0530), Viresh Kumar wrote: > On 10-12-20, 10:38, Ionela Voinescu wrote: > > Basically, that's functions purpose is only to make sure that invariance > > at the level of the policy is consistent: either all CPUs in

Re: [PATCH V2 2/2] arm64: topology: Reorder init_amu_fie() a bit

2020-12-14 Thread Ionela Voinescu
Hey, On Thursday 10 Dec 2020 at 21:59:23 (+0530), Viresh Kumar wrote: > This patch does a couple of optimizations in init_amu_fie(), like early > exits from paths where we don't need to continue any further, moving the > calls to topology_scale_freq_invariant() just when we need > them, instead of

Re: [PATCH V2 1/2] arm64: topology: Avoid the have_policy check

2020-12-14 Thread Ionela Voinescu
; + /* Overwrite amu_fie_cpus if all CPUs support AMU */ > + if (cpumask_equal(valid_cpus, cpu_present_mask)) > + cpumask_copy(amu_fie_cpus, cpu_present_mask); > > if (!cpumask_empty(amu_fie_cpus)) { > pr_info("CPUs[%*pbl]: counters will be used for FIE.", > -- > 2.25.0.rc1.19.g042ed3e048af > Reviewed-by: Ionela Voinescu Thanks, Ionela.

[PATCH v2 4/4] cppc_cpufreq: replace per-cpu data array with a list

2020-12-14 Thread Ionela Voinescu
ed if _CPC entries are not present. Signed-off-by: Ionela Voinescu Cc: Rafael J. Wysocki Cc: Viresh Kumar --- drivers/acpi/cppc_acpi.c | 141 -- drivers/cpufreq/cppc_cpufreq.c | 174 + include/acpi/cppc_acpi.h | 6 +- 3 files c

[PATCH v2 0/4] cppc_cpufreq: fix, clarify and improve support

2020-12-14 Thread Ionela Voinescu
4409-1-ionela.voine...@arm.com/#t [2] https://lore.kernel.org/linux-pm/20201210142139.20490-1-yousaf.kau...@suse.com/ [3] https://lore.kernel.org/linux-pm/20201214120740.10948-1-ionela.voine...@arm.com/ Ionela Voinescu (4): cppc_cpufreq: use policy->cpu as driver of frequency setting

[PATCH v2 3/4] cppc_cpufreq: expose information on frequency domains

2020-12-14 Thread Ionela Voinescu
Use the existing sysfs attribute "freqdomain_cpus" to expose information to userspace about CPUs in the same frequency domain. Signed-off-by: Ionela Voinescu Acked-by: Viresh Kumar Cc: Rafael J. Wysocki Cc: Viresh Kumar --- Documentation/ABI/testing/sysfs-devices-system-cpu | 3 ++

[PATCH v2 2/4] cppc_cpufreq: clarify support for coordination types

2020-12-14 Thread Ionela Voinescu
upport for coordination types while describing in comments the intended behavior. Signed-off-by: Ionela Voinescu Acked-by: Viresh Kumar Cc: Rafael J. Wysocki Cc: Viresh Kumar --- drivers/cpufreq/cppc_cpufreq.c | 19 --- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git

[PATCH v2 1/4] cppc_cpufreq: use policy->cpu as driver of frequency setting

2020-12-14 Thread Ionela Voinescu
at is hotplugged out. Signed-off-by: Ionela Voinescu Acked-by: Viresh Kumar Cc: Rafael J. Wysocki Cc: Viresh Kumar --- drivers/cpufreq/cppc_cpufreq.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/drivers/cpufreq/cppc_cpufreq.c b/drivers/cpufreq/cppc_cpufreq.

[PATCH v2] ACPI: processor: fix NONE coordination for domain mapping failure

2020-12-14 Thread Ionela Voinescu
intention. Signed-off-by: Ionela Voinescu Acked-by: Viresh Kumar [ rjw: Subject edit ] Signed-off-by: Rafael J. Wysocki --- Hi guys, I'm sending v2 of some of the patches at [1] in light of the discussions at [2]. This patch is a trivial rebase on linux next 20201211, submitted separ

Re: [PATCH 1/2] acpi: cppc: add cpufreq device

2020-12-10 Thread Ionela Voinescu
Hi Rafael, On Thursday 10 Dec 2020 at 17:55:56 (+0100), Rafael J. Wysocki wrote: > On Thursday, December 10, 2020 4:04:40 PM CET Mian Yousaf Kaukab wrote: > > On Thu, Dec 10, 2020 at 03:32:09PM +0100, Rafael J. Wysocki wrote: > > > On Thu, Dec 10, 2020 at 3:23 PM Mian Yousaf Kaukab > > > wrote: >

Re: [PATCH 1/2] acpi: cppc: add cpufreq device

2020-12-10 Thread Ionela Voinescu
Hi guys, On Thursday 10 Dec 2020 at 15:32:09 (+0100), Rafael J. Wysocki wrote: > On Thu, Dec 10, 2020 at 3:23 PM Mian Yousaf Kaukab > wrote: > > > > From: Mian Yousaf Kaukab > > > > Since commit 28f06f770454 ("cppc_cpufreq: replace per-cpu structures with > > lists"), cppc-cpufreq driver doesn't

Re: [PATCH] arm64: topology: Cleanup init_amu_fie() a bit

2020-12-10 Thread Ionela Voinescu
On Thursday 10 Dec 2020 at 18:04:39 (+0530), Viresh Kumar wrote: > On 10-12-20, 11:29, Ionela Voinescu wrote: > > On Thursday 10 Dec 2020 at 16:25:06 (+0530), Viresh Kumar wrote: > > > - But right after that we call static_branch_disable() if we aren't >

Re: [PATCH] arm64: topology: Cleanup init_amu_fie() a bit

2020-12-10 Thread Ionela Voinescu
Hey, On Thursday 10 Dec 2020 at 12:48:20 (+0530), Viresh Kumar wrote: > Every time I have stumbled upon this routine, I get confused with the > way 'have_policy' is used and I have to dig in to understand why is it > so. > > Here is an attempt to make it easier to understand, and hopefully it is

Re: [PATCH] arm64: topology: Cleanup init_amu_fie() a bit

2020-12-10 Thread Ionela Voinescu
iece of code that confused things. > On 10-12-20, 10:38, Ionela Voinescu wrote: > > I'm first of all replying to discuss the need of this policy analysis > > that enable_policy_freq_counters() does which results in the setting of > > have_policy. > > > > B

Re: [PATCH] arm64: topology: Avoid the static_branch_{enable|disable} dance

2020-12-10 Thread Ionela Voinescu
On Thursday 10 Dec 2020 at 15:12:25 (+0530), Viresh Kumar wrote: > Avoid the static_branch_enable() and static_branch_disable() dance by > redoing the code in a different way. We will be fully invariant here > only if amu_fie_cpus is set with all present CPUs, use that instead of > yet another call

Re: [PATCH] arm64: topology: Cleanup init_amu_fie() a bit

2020-12-10 Thread Ionela Voinescu
Hi, On Thursday 10 Dec 2020 at 12:48:20 (+0530), Viresh Kumar wrote: > Every time I have stumbled upon this routine, I get confused with the > way 'have_policy' is used and I have to dig in to understand why is it > so. > I'm first of all replying to discuss the need of this policy analysis that

Re: [PATCH] arm64: topology: Drop the useless update to per-cpu cycles

2020-12-10 Thread Ionela Voinescu
; - > -store_and_exit: > - this_cpu_write(arch_core_cycles_prev, core_cnt); > - this_cpu_write(arch_const_cycles_prev, const_cnt); > } Right, no need for this anymore! Reviewed-by: Ionela Voinescu Thank you for the fix, Ionela. > > #ifdef CONFIG_ACPI_CPPC_LIB > -- > 2.25.0.rc1.19.g042ed3e048af >

Re: [PATCH v2 3/5] thermal: devfreq_cooling: add new registration functions with Energy Model

2020-12-02 Thread Ionela Voinescu
On Wednesday 02 Dec 2020 at 11:14:02 (+), Lukasz Luba wrote: > Hi Ionela, > > On 12/2/20 10:24 AM, Ionela Voinescu wrote: > > Hi Lukasz, > > > > On Wednesday 18 Nov 2020 at 12:03:56 (+), Lukasz Luba wrote: > > [snip] > > > >

Re: [PATCH v2 4/5] thermal: devfreq_cooling: remove old power model and use EM

2020-12-02 Thread Ionela Voinescu
remove_qos_req: > dev_pm_qos_remove_request(&dfc->req_max_freq); > - > -free_tables: > - kfree(dfc->power_table); > +free_table: > kfree(dfc->freq_table); > free_dfc: > kfree(dfc); > @@ -696,9 +588,7 @@ void devfreq_cooling_unregister(struct > thermal_cooling_device *cdev) > if (dfc->em_registered) > em_dev_unregister_perf_domain(dev); > > - kfree(dfc->power_table); > kfree(dfc->freq_table); > - > kfree(dfc); > } > EXPORT_SYMBOL_GPL(devfreq_cooling_unregister); > diff --git a/include/linux/devfreq_cooling.h b/include/linux/devfreq_cooling.h > index 19868fb922f1..4890b12b54b4 100644 > --- a/include/linux/devfreq_cooling.h > +++ b/include/linux/devfreq_cooling.h > @@ -17,17 +17,6 @@ > > /** > * struct devfreq_cooling_power - Devfreq cooling power ops > - * @get_static_power:Take voltage, in mV, and return the static power > - * in mW. If NULL, the static power is assumed > - * to be 0. > - * @get_dynamic_power: Take voltage, in mV, and frequency, in HZ, and > - * return the dynamic power draw in mW. If NULL, > - * a simple power model is used. > - * @dyn_power_coeff: Coefficient for the simple dynamic power model in > - * mW/(MHz mV mV). > - * If get_dynamic_power() is NULL, then the > - * dynamic power is calculated as > - * @dyn_power_coeff * frequency * voltage^2 > * @get_real_power: When this is set, the framework uses it to ask the > * device driver for the actual power. > * Some devices have more sophisticated methods > @@ -47,14 +36,8 @@ > * max total (static + dynamic) power value for each OPP. > */ > struct devfreq_cooling_power { > - unsigned long (*get_static_power)(struct devfreq *devfreq, > - unsigned long voltage); > - unsigned long (*get_dynamic_power)(struct devfreq *devfreq, > -unsigned long freq, > -unsigned long voltage); > int (*get_real_power)(struct devfreq *df, u32 *power, > unsigned long freq, unsigned long voltage); > - unsigned long dyn_power_coeff; > }; > > #ifdef CONFIG_DEVFREQ_THERMAL > -- > 2.17.1 > Reviewed-by: Ionela Voinescu Thanks, Ionela.

Re: [PATCH v2 3/5] thermal: devfreq_cooling: add new registration functions with Energy Model

2020-12-02 Thread Ionela Voinescu
); > > #else /* !CONFIG_DEVFREQ_THERMAL */ > > @@ -87,6 +95,20 @@ devfreq_cooling_register(struct devfreq *df) > return ERR_PTR(-EINVAL); > } > > +static inline struct thermal_cooling_device * > +devfreq_cooling_em_register_power(struct devfreq *df, > +

Re: [PATCH v2 2/5] thermal: devfreq_cooling: get a copy of device status

2020-12-02 Thread Ionela Voinescu
est_power = power * dfc->res_util; > @@ -328,8 +362,8 @@ static int devfreq_cooling_power2state(struct > thermal_cooling_device *cdev, > dyn_power = dyn_power > 0 ? dyn_power : 0; > > /* Scale dynamic power for utilization */ > - busy_time = status->busy_time ?: 1; > - est_power = (dyn_power * status->total_time) / busy_time; > + busy_time = status.busy_time ?: 1; > + est_power = (dyn_power * status.total_time) / busy_time; > } > > /* > -- > 2.17.1 > Reviewed-by: Ionela Voinescu Thanks, Ionela.

Re: [PATCH v2 1/5] thermal: devfreq_cooling: change tracing function and arguments

2020-12-02 Thread Ionela Voinescu
u load=%u dynamic_power=%u static_power=%u > power=%u", > + TP_printk("type=%s freq=%lu load=%u power=%u", > __get_str(type), __entry->freq, > - __entry->load, __entry->dynamic_power, __entry->static_power, > + __entry->total_time == 0 ? 0 : > + (100 * __entry->busy_time) / __entry->total_time, > __entry->power) > ); > > -- > 2.17.1 > Reviewed-by: Ionela Voinescu Regards, Ionela.

Re: [PATCH 3/5] thermal: devfreq_cooling: add new registration functions with Energy Model

2020-12-01 Thread Ionela Voinescu
On Tuesday 01 Dec 2020 at 14:37:58 (+), Lukasz Luba wrote: > > > On 12/1/20 2:05 PM, Ionela Voinescu wrote: > > Hi, > > > > On Thursday 22 Oct 2020 at 12:17:31 (+0100), Lukasz Luba wrote: > > [..] > > > > > > > +/** > > >

Re: [PATCH 2/5] thermal: devfreq_cooling: get a copy of device status

2020-12-01 Thread Ionela Voinescu
On Tuesday 01 Dec 2020 at 12:19:18 (+), Lukasz Luba wrote: > > > On 12/1/20 10:36 AM, Ionela Voinescu wrote: > > Hi, > > > > Sorry for the delay and for the noise on this older version. I first > > want to understand the code better. > > > &g

Re: [PATCH 3/5] thermal: devfreq_cooling: add new registration functions with Energy Model

2020-12-01 Thread Ionela Voinescu
Hi, On Thursday 22 Oct 2020 at 12:17:31 (+0100), Lukasz Luba wrote: [..] > > > +/** > > > + * devfreq_cooling_em_register_power() - Register devfreq cooling device > > > with > > > + * power information and attempt to register Energy Model > > > (EM) > > > > It took me a while to

Re: [PATCH 2/5] thermal: devfreq_cooling: get a copy of device status

2020-12-01 Thread Ionela Voinescu
Hi, Sorry for the delay and for the noise on this older version. I first want to understand the code better. On Thursday 22 Oct 2020 at 11:55:28 (+0100), Lukasz Luba wrote: [..] > > > > > > +{ > > > + /* Make some space if needed */ > > > + if (status->busy_time > 0x) { > > > + stat

Re: [PATCH v4 3/3] thermal: power allocator: change the 'k_*' always in estimate_pid_constants()

2020-11-26 Thread Ionela Voinescu
tz->tzp->sustainable_power = sustainable_power; > @@ -640,7 +630,7 @@ static int power_allocator_bind(struct > thermal_zone_device *tz) > if (!ret) > estimate_pid_constants(tz, tz->tzp->sustainable_power, > params->trip_switch_on, > -control_temp, false); > +control_temp); > } > > reset_pid_controller(params); > -- I was actually wondering why we still need force, while reading 2/3. It looks good! Reviewed-by: Ionela Voinescu > 2.17.1 >

Re: [PATCH v4 1/3] thermal: power allocator: change the 'k_i' coefficient estimation

2020-11-26 Thread Ionela Voinescu
temperature_threshold; > > - if (!tz->tzp->k_i || force) > - tz->tzp->k_i = int_to_frac(10) / 1000; > + if (!tz->tzp->k_i || force) { > + k_i = tz->tzp->k_pu / 10; > + tz->tzp->k_i = k_i > 0 ? k_i : 1; > + } > + > /* >* The default for k_d and integral_cutoff is 0, so we can >* leave them as they are. > -- I see this patch did not change so: Reviewed-by: Ionela Voinescu > 2.17.1 >

Re: [PATCH v4 2/3] thermal: power allocator: refactor sustainable power estimation

2020-11-26 Thread Ionela Voinescu
sustainable_power = tz->tzp->sustainable_power; > - } else { > - sustainable_power = estimate_sustainable_power(tz); > - estimate_pid_constants(tz, sustainable_power, > -params->trip_switch_on, control_temp, > -true); > - } > + sustainable_power = get_sustainable_power(tz, params, control_temp); > > err = control_temp - tz->temperature; > err = int_to_frac(err); > -- The logic seems sane so: Reviewed-by: Ionela Voinescu Thank you, Ionela. > 2.17.1 >

[tip: sched/core] sched/topology,schedutil: Wrap sched domains rebuild

2020-11-20 Thread tip-bot2 for Ionela Voinescu
The following commit has been merged into the sched/core branch of tip: Commit-ID: 31f6a8c0a471be7d7d05c93eac50fcb729e79b9d Gitweb: https://git.kernel.org/tip/31f6a8c0a471be7d7d05c93eac50fcb729e79b9d Author:Ionela Voinescu AuthorDate:Tue, 27 Oct 2020 18:07:11 Committer

[tip: sched/core] sched/topology: Condition EAS enablement on FIE support

2020-11-20 Thread tip-bot2 for Ionela Voinescu
The following commit has been merged into the sched/core branch of tip: Commit-ID: fa50e2b452c60cff9f4000de5b372a61d6695c26 Gitweb: https://git.kernel.org/tip/fa50e2b452c60cff9f4000de5b372a61d6695c26 Author:Ionela Voinescu AuthorDate:Tue, 27 Oct 2020 18:07:13 Committer

[tip: sched/core] arm64: Rebuild sched domains on invariance status changes

2020-11-20 Thread tip-bot2 for Ionela Voinescu
The following commit has been merged into the sched/core branch of tip: Commit-ID: ecec9e86d1a366f97c827ab4a8134ec06ccf031a Gitweb: https://git.kernel.org/tip/ecec9e86d1a366f97c827ab4a8134ec06ccf031a Author:Ionela Voinescu AuthorDate:Tue, 27 Oct 2020 18:07:12 Committer

Re: [PATCH RESEND v2 0/3] condition EAS enablement on FI support

2020-11-18 Thread Ionela Voinescu
Hi guys, On Tuesday 27 Oct 2020 at 18:07:10 (+), Ionela Voinescu wrote: > Given the maturity gained by cpufreq-based Frequency Invariance (FI) > support following the patches at [1], this series conditions Energy > Aware Scheduling (EAS) enablement on a frequency invaria

Re: [PATCH 0/8] cppc_cpufreq: fix, clarify and improve support

2020-11-17 Thread Ionela Voinescu
On Tuesday 17 Nov 2020 at 17:30:33 (+0100), Rafael J. Wysocki wrote: [..] > > > > Ionela Voinescu (8): > > > > cppc_cpufreq: fix misspelling, code style and readability issues > > > > cppc_cpufreq: clean up cpu, cpu_num and cpunum variable use >

[PATCH] cppc_cpufreq: optimise memory allocation for HW and NONE coordination

2020-11-17 Thread Ionela Voinescu
_kernel_size_le_hi32+0x0x88ffd274 - (1) ANY coordination: totalslack req alloc/free caller 2560 256 2/0 _kernel_size_le_hi32+0x0x88fed410 000 0/2 _kernel_size_le_hi32+0x0x88fed274 Signed-off-by: Ionela Voinescu

Re: [PATCH 0/8] cppc_cpufreq: fix, clarify and improve support

2020-11-17 Thread Ionela Voinescu
Hi Rafael, On Tuesday 17 Nov 2020 at 15:59:24 (+0100), Rafael J. Wysocki wrote: > On Thu, Nov 5, 2020 at 1:56 PM Ionela Voinescu > wrote: > > > > Hi guys, > > > > I found myself staring a bit too much at this driver in the past weeks > > and that's li

Re: [PATCH] arm64: abort counter_read_on_cpu() when irqs_disabled()

2020-11-13 Thread Ionela Voinescu
On Friday 13 Nov 2020 at 16:02:34 (+), Mark Rutland wrote: > On Fri, Nov 13, 2020 at 03:53:28PM +0000, Ionela Voinescu wrote: > > Given that smp_call_function_single() can deadlock when interrupts are > > disabled, abort the SMP call if irqs_disabled(). This scenario is &g

Re: [PATCH v4 3/3] arm64: implement CPPC FFH support using AMUs

2020-11-13 Thread Ionela Voinescu
Hi Sudeep, On Friday 13 Nov 2020 at 14:16:58 (+), Sudeep Holla wrote: [..] > > diff --git a/arch/arm64/kernel/topology.c b/arch/arm64/kernel/topology.c > > index b8cb16e3a2cc..7c9b6a0ecd6a 100644 > > --- a/arch/arm64/kernel/topology.c > > +++ b/arch/arm64/kernel/topology.c > > @@ -147,6 +147,9

[PATCH] arm64: abort counter_read_on_cpu() when irqs_disabled()

2020-11-13 Thread Ionela Voinescu
Given that smp_call_function_single() can deadlock when interrupts are disabled, abort the SMP call if irqs_disabled(). This scenario is currently not possible given the function's uses, but safeguard this for potential future uses. Signed-off-by: Ionela Voinescu Cc: Catalin Marinas Cc:

Re: [PATCH v4 3/3] arm64: implement CPPC FFH support using AMUs

2020-11-13 Thread Ionela Voinescu
Hi, On Thursday 12 Nov 2020 at 18:00:46 (+), Catalin Marinas wrote: > On Fri, Nov 06, 2020 at 12:53:34PM +0000, Ionela Voinescu wrote: > > +static inline > > +int counters_read_on_cpu(int cpu, smp_call_func_t func, u64 *val) > > +{ > > + if (!cpu_has_amu_feat(c

[PATCH v4 3/3] arm64: implement CPPC FFH support using AMUs

2020-11-06 Thread Ionela Voinescu
ister values or write operation for all registers are unsupported. Therefore, keep CPPC's FFH unsupported if no CPUs have valid AMU frequency counters. For this purpose, the get_cpu_with_amu_feat() is introduced. Signed-off-by: Ionela Voinescu Cc: Catalin Marinas Cc: Will Deacon ---

[PATCH v4 2/3] arm64: split counter validation function

2020-11-06 Thread Ionela Voinescu
used by topology_scale_freq_tick() Signed-off-by: Ionela Voinescu Cc: Catalin Marinas Cc: Will Deacon --- arch/arm64/kernel/topology.c | 44 +--- 1 file changed, 26 insertions(+), 18 deletions(-) diff --git a/arch/arm64/kernel/topology.c b/arch/arm64/kernel

[PATCH v4 0/3] arm64: cppc: add FFH support using AMUs

2020-11-06 Thread Ionela Voinescu
lkml/20200826130309.28027-1-ionela.voine...@arm.com/ [3] https://lore.kernel.org/linux-arm-kernel/20201027163624.20747-1-ionela.voine...@arm.com/ [4] https://lore.kernel.org/linux-arm-kernel/20201105122702.13916-1-ionela.voine...@arm.com/ Thank you, Ionela. Ionela Voinescu (3): arm64: wrap and genera

[PATCH v4 1/3] arm64: wrap and generalise counter read functions

2020-11-06 Thread Ionela Voinescu
and update the per-cpu reference variables. Signed-off-by: Ionela Voinescu Cc: Catalin Marinas Cc: Will Deacon --- arch/arm64/include/asm/cpufeature.h | 5 + arch/arm64/include/asm/topology.h | 4 +++- arch/arm64/kernel/cpufeature.c | 5 + arch/arm64/kernel/topology.c

Re: [PATCH 4/8] cppc_cpufreq: replace per-cpu structures with lists

2020-11-05 Thread Ionela Voinescu
Hi Jeremy, On Thursday 05 Nov 2020 at 09:50:30 (-0600), Jeremy Linton wrote: > Hi, > > On 11/5/20 6:55 AM, Ionela Voinescu wrote: > > The cppc_cpudata per-cpu storage was inefficient (1) additional to causing > > functional issues (2) when CPUs are hotplugged out, due to p

Re: [PATCH v3 0/3] CPUFreq: Add support for cpu performance dependencies

2020-11-05 Thread Ionela Voinescu
Hi guys, On Thursday 05 Nov 2020 at 15:25:53 (+0100), Vincent Guittot wrote: [..] > > > - Because of hardware co-ordination of otherwise co-ordinated CPUs, > > > few things break. Thermal and EAS are some of the examples and so > > > you are trying to fix them here by proving them the missing

Re: [PATCH v3 3/3] arm64: implement CPPC FFH support using AMUs

2020-11-05 Thread Ionela Voinescu
Hi Mark, On Thursday 05 Nov 2020 at 13:28:23 (+), Mark Rutland wrote: [..] > > +#ifdef CONFIG_ACPI_CPPC_LIB > > +#include > > As mentioned on patch 1, I think it'd be better to open-code the smp > call functions here, e.g. > > static void cpu_read_corecnt(void *val) > { > *(u64 *)val

Re: [PATCH RESEND v2 1/3] sched/topology,schedutil: wrap sched domains rebuild

2020-11-05 Thread Ionela Voinescu
Hi Rafael, On Friday 30 Oct 2020 at 16:26:32 (+0100), Rafael J. Wysocki wrote: > On Tue, Oct 27, 2020 at 7:08 PM Ionela Voinescu > wrote: > > > > Add the rebuild_sched_domains_energy() function to wrap the functionality > > that rebuilds the scheduling domains if

Re: [PATCH 8/8] acpi: fix NONE coordination for domain mapping failure

2020-11-05 Thread Ionela Voinescu
Hi Rafael, On Thursday 05 Nov 2020 at 14:05:55 (+0100), Rafael J. Wysocki wrote: > On Thu, Nov 5, 2020 at 1:57 PM Ionela Voinescu > wrote: > > > > For errors parsing the _PSD domains, a separate domain is returned for > > each CPU in the failed _PSD domain with no coord

[PATCH 8/8] acpi: fix NONE coordination for domain mapping failure

2020-11-05 Thread Ionela Voinescu
intention. Signed-off-by: Ionela Voinescu Cc: Rafael J. Wysocki Cc: Len Brown Cc: Viresh Kumar --- drivers/acpi/cppc_acpi.c | 2 +- drivers/acpi/processor_perflib.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/acpi/cppc_acpi.c b/drivers/acpi

[PATCH 1/8] cppc_cpufreq: fix misspelling, code style and readability issues

2020-11-05 Thread Ionela Voinescu
improve readability Signed-off-by: Ionela Voinescu Cc: Rafael J. Wysocki Cc: Viresh Kumar --- drivers/cpufreq/cppc_cpufreq.c | 32 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/drivers/cpufreq/cppc_cpufreq.c b/drivers/cpufreq/cppc_cpufreq.c

[PATCH 4/8] cppc_cpufreq: replace per-cpu structures with lists

2020-11-05 Thread Ionela Voinescu
ed or retrieved. For this purpose acpi_get_psd_map() was changed to create a map of the domain of a single CPU, rather than the full system domain map. - When CPU's are hotplugged out and in, old information can be retrieved. Signed-off-by: Ionela Voinescu Cc: Rafael J. Wysocki Cc: Len Brow

[PATCH 2/8] cppc_cpufreq: clean up cpu, cpu_num and cpunum variable use

2020-11-05 Thread Ionela Voinescu
ables unsigned int Where pertinent, also move the initialisation of cpu_data variable to its declaration and make consistent use of the local "cpu" variable. Signed-off-by: Ionela Voinescu Cc: Rafael J. Wysocki Cc: Viresh Kumar --- drivers/cpufreq/cppc_cpufreq.c | 143 -

[PATCH 7/8] cppc_cpufreq: expose information on frequency domains

2020-11-05 Thread Ionela Voinescu
Use the existing sysfs attribute "freqdomain_cpus" to expose information to userspace about CPUs in the same frequency domain. Signed-off-by: Ionela Voinescu Cc: Rafael J. Wysocki Cc: Viresh Kumar --- Documentation/ABI/testing/sysfs-devices-system-cpu | 3 ++- drivers/cpufreq/cppc

[PATCH 0/8] cppc_cpufreq: fix, clarify and improve support

2020-11-05 Thread Ionela Voinescu
.org/linux-pm/20200922162540.gb...@arm.com/ Ionela Voinescu (8): cppc_cpufreq: fix misspelling, code style and readability issues cppc_cpufreq: clean up cpu, cpu_num and cpunum variable use cppc_cpufreq: simplify use of performance capabilities cppc_cpufreq: replace per-cpu structures

[PATCH 6/8] cppc_cpufreq: clarify support for coordination types

2020-11-05 Thread Ionela Voinescu
upport for coordination types while describing in comments the intended behavior. Signed-off-by: Ionela Voinescu Cc: Rafael J. Wysocki Cc: Viresh Kumar --- drivers/cpufreq/cppc_cpufreq.c | 19 +++ 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/drivers/cpufreq/cppc_c

[PATCH 5/8] cppc_cpufreq: use policy->cpu as driver of frequency setting

2020-11-05 Thread Ionela Voinescu
at is hotplugged out. Signed-off-by: Ionela Voinescu Cc: Rafael J. Wysocki Cc: Viresh Kumar --- drivers/cpufreq/cppc_cpufreq.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/drivers/cpufreq/cppc_cpufreq.c b/drivers/cpufreq/cppc_cpufreq.c index ac95b4424a2e..fd2daeb59b49

[PATCH 3/8] cppc_cpufreq: simplify use of performance capabilities

2020-11-05 Thread Ionela Voinescu
The CPPC performance capabilities are used significantly throughout the driver. Simplify the use of them by introducing a local pointer "caps" to point to cpu_data->perf_caps, in functions that access performance capabilities often. Signed-off-by: Ionela Voinescu Cc: Rafael J

[PATCH v3 0/3] arm64: cppc: add FFH support using AMUs

2020-11-05 Thread Ionela Voinescu
://documentation-service.arm.com/static/5f106ad60daa596235e80081 [2] https://lore.kernel.org/lkml/20200826130309.28027-1-ionela.voine...@arm.com/ [3] https://lore.kernel.org/linux-arm-kernel/20201027163624.20747-1-ionela.voine...@arm.com/ Thank you, Ionela. Ionela Voinescu (3): arm64: wra

[PATCH v3 1/3] arm64: wrap and generalise counter read functions

2020-11-05 Thread Ionela Voinescu
and update the per-cpu reference variables. Signed-off-by: Ionela Voinescu Cc: Catalin Marinas Cc: Will Deacon --- arch/arm64/include/asm/cpufeature.h | 5 arch/arm64/include/asm/topology.h | 4 +++- arch/arm64/kernel/cpufeature.c | 5 +--- arch/arm64/kernel/topology.c

[PATCH v3 2/3] arm64: split counter validation function

2020-11-05 Thread Ionela Voinescu
used by topology_scale_freq_tick() Signed-off-by: Ionela Voinescu Cc: Catalin Marinas Cc: Will Deacon --- arch/arm64/kernel/topology.c | 44 +--- 1 file changed, 26 insertions(+), 18 deletions(-) diff --git a/arch/arm64/kernel/topology.c b/arch/arm64/kernel

[PATCH v3 3/3] arm64: implement CPPC FFH support using AMUs

2020-11-05 Thread Ionela Voinescu
other register values or write operation for all registers are unsupported. Therefore, keep CPPC's FFH unsupported if no CPUs have valid AMU frequency counters. For this purpose, the get_cpu_with_amu_feat() is introduced. Signed-off-by: Ionela Voinescu Cc: Catalin Marinas Cc: Will Deacon ---

[PATCH RESEND v2 2/3] arm64: rebuild sched domains on invariance status changes

2020-10-27 Thread Ionela Voinescu
e of FI support for the use of EAS, ensure the following: if there is a change in FI support status after counter init, use the existing rebuild_sched_domains_energy() function to trigger a rebuild of the scheduling and performance domains that in turn will determine the enablement of EAS. Signed

[PATCH RESEND v2 3/3] sched/topology: condition EAS enablement on FIE support

2020-10-27 Thread Ionela Voinescu
s the case when initializing EAS. Warn and bail out otherwise. Signed-off-by: Ionela Voinescu Suggested-by: Quentin Perret Cc: Ingo Molnar Cc: Peter Zijlstra --- kernel/sched/topology.c | 9 + 1 file changed, 9 insertions(+) diff --git a/kernel/sched/topology.c b/kernel/sched/t

[PATCH RESEND v2 0/3] condition EAS enablement on FI support

2020-10-27 Thread Ionela Voinescu
rning under sched_debug() as per Quentin and Dietmar's recommendations. [1] Most recent version at: https://lore.kernel.org/lkml/20200901205549.30096-1-ionela.voine...@arm.com/ [2] https://lore.kernel.org/lkml/20200924123937.20938-1-ionela.voine...@arm.com/ Many thanks, Ionela. Ionela Vo

[PATCH RESEND v2 1/3] sched/topology,schedutil: wrap sched domains rebuild

2020-10-27 Thread Ionela Voinescu
sched_energy_aware sysctl. Therefore, create a single function that is used in both these cases and that can be later reused. Signed-off-by: Ionela Voinescu Acked-by: Quentin Perret Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Rafael J. Wysocki Cc: Viresh Kumar --- include/linux/sched/topology.h

[PATCH RESEND v2 2/3] arm64: split counter validation function

2020-10-27 Thread Ionela Voinescu
used by topology_scale_freq_tick() Signed-off-by: Ionela Voinescu Cc: Catalin Marinas Cc: Will Deacon --- arch/arm64/kernel/topology.c | 44 +--- 1 file changed, 26 insertions(+), 18 deletions(-) diff --git a/arch/arm64/kernel/topology.c b/arch/arm64/kernel

  1   2   3   >