Le Wed, Jun 24, 2026 at 06:23:50AM -0700, Puranjay Mohan a écrit : > When an expedited grace period completes, rcu_exp_wait_wake() wakes > waiters on rnp->exp_wq[] but does not notify the NOCB rcuog kthreads. An > rcuog kthread that is waiting for a grace period sleeps on the leaf > rcu_node's ->nocb_gp_wq[] with a wait condition based on the grace-period > state, so without a wakeup, callbacks on offloaded CPUs that could > benefit from the expedited GP wait until the rcuog kthread wakes for some > other reason (e.g. the next normal GP or a timer). > > Make the rcuog grace-period wait honour expedited GPs and wake it when > one completes: > > - nocb_gp_wait() now records the grace period to wait for as a struct > rcu_gp_seq (both normal and expedited), tracks the earliest pending > normal and expedited sequence across the group, and releases the wait > via poll_state_synchronize_rcu_full() so it wakes for whichever > completes first. ->nocb_gp_seq is widened to struct rcu_gp_seq > accordingly. > > - rcu_exp_wait_wake() calls the new rcu_nocb_exp_cleanup() on leaf > nodes, which wakes both ->nocb_gp_wq[0] and ->nocb_gp_wq[1] (the > expedited sequence does not share parity with the normal ->gp_seq the > waiter indexed with). Both this path and rcu_nocb_gp_cleanup() use > the shared rcu_nocb_cleanup_wake() helper, which checks swait_active() > first; the smp_mb() in rcu_gp_cleanup()/rcu_exp_wait_wake() orders the > grace-period state update before that check. > > A stub rcu_nocb_exp_cleanup() is provided for CONFIG_RCU_NOCB_CPU=n. > > Signed-off-by: Puranjay Mohan <[email protected]>
Reviewed-by: Frederic Weisbecker <[email protected]> -- Frederic Weisbecker SUSE Labs
