The arming code in rcu_read_unlock_special() has two paths for
deferring QS reporting: a softirq raise and an irq_work/set_need_resched
combination.
The irq_work path always calls set_need_resched_current() before
queuing irq_work. The softirq path does not, relying solely on the
softirq firing to do the rightthing.
This results in a problem as follows:
Consider 2 rcu_read_lock/unlock segments:
rcu_read_lock(); // segment 1 starts
// needs_exp becomes true
preempt_disable();
rcu_read_unlock(); // segment 1 ends; IRQs on, preempt
// off, needs_exp=true =>
// raise_softirq(RCU_SOFTIRQ);
// arms defer_qs_pending.
// Before this fix: no
// set_need_resched_current().
local_irq_disable();
preempt_enable(); // softirq pending but IRQs disabled
// hold it off.
rcu_read_lock(); // segment 2 starts
local_irq_enable(); // softirq fires: rcu_core runs, but
// we are inside a reader (depth>0)
// so no QS report; on softirq-exit
// preempt-check finds no
// need_resched -- still no nudge.
preempt_disable();
rcu_read_unlock(); // arming attempt suppressed
// incorrectly: defer_qs_pending
// already PENDING. Without this
// fix, no fresh
// set_need_resched_current() on
// this path either.
preempt_enable();
There, add set_need_resched_current() to the softirq deferral path to
avoid long latencies in situations where GP needs to end sooner.
Signed-off-by: Joel Fernandes <[email protected]>
---
kernel/rcu/tree_plugin.h | 1 +
1 file changed, 1 insertion(+)
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index bb5f955a06df..86ac1e8228cf 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -747,6 +747,7 @@ static void rcu_read_unlock_special(struct task_struct *t)
// Using softirq, safe to awaken, and either the
// wakeup is free or there is either an expedited
// GP in flight or a potential need to deboost.
+ set_need_resched_current();
if (rdp->defer_qs_pending != DEFER_QS_PENDING) {
rdp->defer_qs_pending = DEFER_QS_PENDING;
raise_softirq_irqoff(RCU_SOFTIRQ);
--
2.34.1
