Re: [PATCH v3 32/52] powercap, intel-rapl: Fix CPU hotplug callback registration
On Tue, 11 Mar 2014 02:09:26 +0530
"Srivatsa S. Bhat" wrote:
> Subsystems that want to register CPU hotplug callbacks, as well as
> perform initialization for the CPUs that are already online, often do
> it as shown below:
>
> get_online_cpus();
>
> for_each_online_cpu(cpu)
> init_cpu(cpu);
>
> register_cpu_notifier(&foobar_cpu_notifier);
>
> put_online_cpus();
>
> This is wrong, since it is prone to ABBA deadlocks involving the
> cpu_add_remove_lock and the cpu_hotplug.lock (when running
> concurrently with CPU hotplug operations).
>
> Instead, the correct and race-free way of performing the callback
> registration is:
>
> cpu_notifier_register_begin();
>
> for_each_online_cpu(cpu)
> init_cpu(cpu);
>
> /* Note the use of the double underscored version of the API
> */ __register_cpu_notifier(&foobar_cpu_notifier);
>
> cpu_notifier_register_done();
>
>
> Fix the intel-rapl code in the powercap driver by using this latter
> form of callback registration. But retain the calls to
> get/put_online_cpus(), since they also protect the function
> rapl_cleanup_data(). By nesting get/put_online_cpus() *inside*
> cpu_notifier_register_begin/done(), we avoid the ABBA deadlock
> possibility mentioned above.
>
> Cc: "Rafael J. Wysocki"
> Cc: Jacob Pan
> Cc: Srinivas Pandruvada
> Cc: Ingo Molnar
> Signed-off-by: Srivatsa S. Bhat
> ---
>
Tested-by: Jacob Pan
> drivers/powercap/intel_rapl.c | 10 --
> 1 file changed, 8 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/powercap/intel_rapl.c
> b/drivers/powercap/intel_rapl.c index 3c67683..d6c74c1 100644
> --- a/drivers/powercap/intel_rapl.c
> +++ b/drivers/powercap/intel_rapl.c
> @@ -1369,6 +1369,9 @@ static int __init rapl_init(void)
>
> return -ENODEV;
> }
> +
> + cpu_notifier_register_begin();
> +
> /* prevent CPU hotplug during detection */
> get_online_cpus();
> ret = rapl_detect_topology();
> @@ -1380,20 +1383,23 @@ static int __init rapl_init(void)
> ret = -ENODEV;
> goto done;
> }
> - register_hotcpu_notifier(&rapl_cpu_notifier);
> + __register_hotcpu_notifier(&rapl_cpu_notifier);
> done:
> put_online_cpus();
> + cpu_notifier_register_done();
>
> return ret;
> }
>
> static void __exit rapl_exit(void)
> {
> + cpu_notifier_register_begin();
> get_online_cpus();
> - unregister_hotcpu_notifier(&rapl_cpu_notifier);
> + __unregister_hotcpu_notifier(&rapl_cpu_notifier);
> rapl_unregister_powercap();
> rapl_cleanup_data();
> put_online_cpus();
> + cpu_notifier_register_done();
> }
>
> module_init(rapl_init);
>
[Jacob Pan]
___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [PATCH 04/36] cpuidle,intel_idle: Fix CPUIDLE_FLAG_IRQ_ENABLE
, Arnd Bergmann , ulli.kr...@googlemail.com, vgu...@kernel.org,
linux-...@vger.kernel.org, j...@joshtriplett.org, rost...@goodmis.org,
r...@vger.kernel.org, b...@alien8.de, bc...@quicinc.com,
tsbog...@alpha.franken.de, linux-par...@vger.kernel.org, sudeep.ho...@arm.com,
shawn...@kernel.org, da...@davemloft.net, dal...@libc.org, t...@atomide.com,
amakha...@vmware.com, bjorn.anders...@linaro.org, h...@zytor.com,
sparcli...@vger.kernel.org, linux-hexa...@vger.kernel.org,
linux-ri...@lists.infradead.org, anton.iva...@cambridgegreys.com,
jo...@southpole.se, yury.no...@gmail.com, rich...@nod.at, x...@kernel.org,
li...@armlinux.org.uk, mi...@redhat.com, a...@eecs.berkeley.edu,
paul...@kernel.org, h...@linux.ibm.com, stefan.kristians...@saunalahti.fi,
openr...@lists.librecores.org, paul.walms...@sifive.com,
linux-te...@vger.kernel.org, namhy...@kernel.org,
andriy.shevche...@linux.intel.com, jpoim...@kernel.org, jgr...@suse.com,
mon...@monstr.eu, linux-m...@vger.kernel.org, palmer@dab
belt.com, a...@brainfault.org, i...@jurassic.park.msu.ru,
johan...@sipsolutions.net, linuxppc-dev@lists.ozlabs.org
Errors-To: linuxppc-dev-bounces+archive=mail-archive@lists.ozlabs.org
Sender: "Linuxppc-dev"
Hi Peter,
On Wed, 08 Jun 2022 16:27:27 +0200, Peter Zijlstra
wrote:
> Commit c227233ad64c ("intel_idle: enable interrupts before C1 on
> Xeons") wrecked intel_idle in two ways:
>
> - must not have tracing in idle functions
> - must return with IRQs disabled
>
> Additionally, it added a branch for no good reason.
>
> Fixes: c227233ad64c ("intel_idle: enable interrupts before C1 on Xeons")
> Signed-off-by: Peter Zijlstra (Intel)
> ---
> drivers/idle/intel_idle.c | 48
> +++--- 1 file changed, 37
> insertions(+), 11 deletions(-)
>
> --- a/drivers/idle/intel_idle.c
> +++ b/drivers/idle/intel_idle.c
> @@ -129,21 +137,37 @@ static unsigned int mwait_substates __in
> *
> * Must be called under local_irq_disable().
> */
nit: this comment is no long true, right?
> +
> -static __cpuidle int intel_idle(struct cpuidle_device *dev,
> - struct cpuidle_driver *drv, int index)
> +static __always_inline int __intel_idle(struct cpuidle_device *dev,
> + struct cpuidle_driver *drv, int
> index) {
> struct cpuidle_state *state = &drv->states[index];
> unsigned long eax = flg2MWAIT(state->flags);
> unsigned long ecx = 1; /* break on interrupt flag */
>
> - if (state->flags & CPUIDLE_FLAG_IRQ_ENABLE)
> - local_irq_enable();
> -
> mwait_idle_with_hints(eax, ecx);
>
> return index;
> }
>
> +static __cpuidle int intel_idle(struct cpuidle_device *dev,
> + struct cpuidle_driver *drv, int index)
> +{
> + return __intel_idle(dev, drv, index);
> +}
> +
> +static __cpuidle int intel_idle_irq(struct cpuidle_device *dev,
> + struct cpuidle_driver *drv, int
> index) +{
> + int ret;
> +
> + raw_local_irq_enable();
> + ret = __intel_idle(dev, drv, index);
> + raw_local_irq_disable();
> +
> + return ret;
> +}
> +
> /**
> * intel_idle_s2idle - Ask the processor to enter the given idle state.
> * @dev: cpuidle device of the target CPU.
> @@ -1801,6 +1824,9 @@ static void __init intel_idle_init_cstat
> /* Structure copy. */
> drv->states[drv->state_count] =
> cpuidle_state_table[cstate];
> + if (cpuidle_state_table[cstate].flags &
> CPUIDLE_FLAG_IRQ_ENABLE)
> + drv->states[drv->state_count].enter =
> intel_idle_irq; +
> if ((disabled_states_mask & BIT(drv->state_count)) ||
> ((icpu->use_acpi || force_use_acpi) &&
>intel_idle_off_by_default(mwait_hint) &&
>
>
Thanks,
Jacob
Re: [PATCH 04/36] cpuidle,intel_idle: Fix CPUIDLE_FLAG_IRQ_ENABLE
, Arnd Bergmann , ulli.kr...@googlemail.com, vgu...@kernel.org,
linux-...@vger.kernel.org, j...@joshtriplett.org, rost...@goodmis.org,
r...@vger.kernel.org, b...@alien8.de, bc...@quicinc.com,
tsbog...@alpha.franken.de, linux-par...@vger.kernel.org, sudeep.ho...@arm.com,
shawn...@kernel.org, da...@davemloft.net, dal...@libc.org, t...@atomide.com,
amakha...@vmware.com, bjorn.anders...@linaro.org, h...@zytor.com,
sparcli...@vger.kernel.org, linux-hexa...@vger.kernel.org,
linux-ri...@lists.infradead.org, anton.iva...@cambridgegreys.com,
jo...@southpole.se, yury.no...@gmail.com, rich...@nod.at, x...@kernel.org,
li...@armlinux.org.uk, mi...@redhat.com, a...@eecs.berkeley.edu,
paul...@kernel.org, h...@linux.ibm.com, stefan.kristians...@saunalahti.fi,
openr...@lists.librecores.org, paul.walms...@sifive.com,
linux-te...@vger.kernel.org, namhy...@kernel.org,
andriy.shevche...@linux.intel.com, jpoim...@kernel.org, jgr...@suse.com,
mon...@monstr.eu, linux-m...@vger.kernel.org, palmer@dab
belt.com, a...@brainfault.org, i...@jurassic.park.msu.ru,
johan...@sipsolutions.net, linuxppc-dev@lists.ozlabs.org
Errors-To: linuxppc-dev-bounces+archive=mail-archive@lists.ozlabs.org
Sender: "Linuxppc-dev"
Hi Peter,
On Mon, 13 Jun 2022 10:44:22 +0200, Peter Zijlstra
wrote:
> On Thu, Jun 09, 2022 at 04:49:21PM -0700, Jacob Pan wrote:
> > Hi Peter,
> >
> > On Wed, 08 Jun 2022 16:27:27 +0200, Peter Zijlstra
> > wrote:
> >
> > > Commit c227233ad64c ("intel_idle: enable interrupts before C1 on
> > > Xeons") wrecked intel_idle in two ways:
> > >
> > > - must not have tracing in idle functions
> > > - must return with IRQs disabled
> > >
> > > Additionally, it added a branch for no good reason.
> > >
> > > Fixes: c227233ad64c ("intel_idle: enable interrupts before C1 on
> > > Xeons") Signed-off-by: Peter Zijlstra (Intel)
> > > ---
> > > drivers/idle/intel_idle.c | 48
> > > +++--- 1 file changed, 37
> > > insertions(+), 11 deletions(-)
> > >
> > > --- a/drivers/idle/intel_idle.c
> > > +++ b/drivers/idle/intel_idle.c
> > > @@ -129,21 +137,37 @@ static unsigned int mwait_substates __in
> > > *
> > > * Must be called under local_irq_disable().
> > > */
> > nit: this comment is no long true, right?
>
> It still is, all the idle routines are called with interrupts disabled,
> but must also exit with interrupts disabled.
>
> If the idle method requires interrupts to be enabled, it must be sure to
> disable them again before returning. Given all the RCU/tracing concerns
> it must use raw_local_irq_*() for this though.
Makes sense, it is just little confusing when the immediate caller does
raw_local_irq_enable() which does not cancel out local_irq_disable().
Thanks,
Jacob
