Argh, a random syzbot fix has polluted my recipient cache; adding Sean for real.
On Wed, 2026-08-12 at 13:32 +0100, David Woodhouse wrote: > From: David Woodhouse <[email protected]> > > For a dedicated srcu_struct whose read-side critical sections are > short, atomic, and usually absent — such as one converted from a > spinning lock — the common case at synchronize time is that there are > no readers at all. Even synchronize_srcu_expedited() still costs the > caller an unconditional sleep and two trips through the SRCU workqueue > to discover that: tens of microseconds of machinery to wait for > nothing. And synchronize_srcu()'s auto-expedite heuristic explicitly > declines to expedite within exp_holdoff (25µs) of the previous grace > period, which is exactly the regime a burst of back-to-back > invalidations puts such a domain in. > > Provide try_synchronize_srcu(), which proves the no-readers case > inline and returns true without sleeping, without the workqueue, and > without advancing the grace-period sequence (so cookies from > get_state_synchronize_srcu() and queued callbacks are unaffected). If > the proof fails, it returns false and the caller falls back: > > if (!try_synchronize_srcu(ssp)) > synchronize_srcu_expedited(ssp); > > For Tree SRCU the proof sums both epochs' unlock counters, executes a > full barrier, then sums both epochs' lock counters. Equality proves a > moment within this function at which no reader existed: a reader > entering between the sums inflates only the lock sum (spurious > fallback, safe), and a reader whose increment is unobserved has not > yet returned from srcu_read_lock() — the barrier pairing with > __srcu_read_lock() guarantees such a reader sees every store the > caller made beforehand, so it is not a reader the caller is obliged > to wait for. Summing both epochs means no index flip is required, and > without a flip the counter-wrap concerns of > srcu_readers_active_idx_check() do not arise. Readers of the _fast() > flavors elide the read-side barrier this depends on; any sign of them > (in the unlock-side rdm mask, which is gathered unconditionally) > disqualifies the fast path. > > For Tiny SRCU (!SMP) both nesting counts being zero already proves no > reader exists — a mid-section reader could only be preempted or in an > interrupt, either of which leaves its count visibly elevated — and > program order on the sole CPU provides all the required ordering. > > The immediate motivation is a proposed conversion of KVM's > gfn_to_pfn_cache to use SRCU¹, where the mmu_notifier invalidation > path must drain readers of a cache (in the manner of a TLB shootdown) > before the primary MMU zaps the backing page. Those readers are short > non-sleeping fast paths, some in contexts which cannot sleep (hardirq > event channel delivery, the scheduler's sched-out hook); measurement > under a worst-case invalidation flood shows 98.8% of drains complete inline > in 4-16µs where the expedited grace period took 32-128µs, with the > wait dominated by workqueue round-trip latency, not by readers. > > ¹ https://lore.kernel.org/all/[email protected]/ > > More potential use cases already exist in the tree with the same > no-readers-common-case profile: kvm->irq_srcu takes half a dozen > expedited grace periods in the irqfd and routing-update paths (bursts > of which, at VM boot, fall inside the exp_holdoff window), and mshv's > pt_irq_srcu is the same shape. > > Signed-off-by: David Woodhouse <[email protected]> > Assisted-by: Claude:claude-mythos-5 > --- > Compile-tested for Tiny SRCU (tinyconfig); the Tree version has had > the KVM gfn_to_pfn_cache conversion soaking on it under an adversarial > invalidation flood. > > include/linux/srcu.h | 1 + > kernel/rcu/srcutiny.c | 23 ++++++++++++++ > kernel/rcu/srcutree.c | 70 +++++++++++++++++++++++++++++++++++++++++++ > 3 files changed, 94 insertions(+) > > diff --git a/include/linux/srcu.h b/include/linux/srcu.h > index a54ce9e808b9..0d7543c7becc 100644 > --- a/include/linux/srcu.h > +++ b/include/linux/srcu.h > @@ -91,6 +91,7 @@ void call_srcu(struct srcu_struct *ssp, struct rcu_head > *head, > void (*func)(struct rcu_head *head)); > void cleanup_srcu_struct(struct srcu_struct *ssp); > void synchronize_srcu(struct srcu_struct *ssp); > +bool try_synchronize_srcu(struct srcu_struct *ssp); > > #define SRCU_GET_STATE_COMPLETED 0x1 > > diff --git a/kernel/rcu/srcutiny.c b/kernel/rcu/srcutiny.c > index a2e2d516e51b..f07159cae241 100644 > --- a/kernel/rcu/srcutiny.c > +++ b/kernel/rcu/srcutiny.c > @@ -260,6 +260,29 @@ void synchronize_srcu(struct srcu_struct *ssp) > } > EXPORT_SYMBOL_GPL(synchronize_srcu); > > +/** > + * try_synchronize_srcu - inline grace period for a reader-free srcu_struct > + * @ssp: srcu_struct with which to synchronize. > + * > + * If @ssp provably has no readers in either epoch, provide the > + * synchronize_srcu() guarantee to the caller immediately, without > + * sleeping. Returns true on success; on failure the caller must fall > + * back to synchronize_srcu(). > + * > + * On !SMP a reader can only be mid-critical-section if it was > + * preempted (or is running in an interrupt which preempted us), in > + * which case its nesting count is visibly non-zero. Both counts being > + * zero therefore proves that no reader exists, and any reader which > + * begins after this function returns will, by program order on this > + * sole CPU, observe every store the caller made before calling it. > + */ > +bool try_synchronize_srcu(struct srcu_struct *ssp) > +{ > + return !READ_ONCE(ssp->srcu_lock_nesting[0]) && > + !READ_ONCE(ssp->srcu_lock_nesting[1]); > +} > +EXPORT_SYMBOL_GPL(try_synchronize_srcu); > + > /* > * get_state_synchronize_srcu - Provide an end-of-grace-period cookie > */ > diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c > index 2601566c254a..76a78336feaa 100644 > --- a/kernel/rcu/srcutree.c > +++ b/kernel/rcu/srcutree.c > @@ -1645,6 +1645,76 @@ void synchronize_srcu(struct srcu_struct *ssp) > } > EXPORT_SYMBOL_GPL(synchronize_srcu); > > +/** > + * try_synchronize_srcu - inline grace period for a reader-free srcu_struct > + * @ssp: srcu_struct with which to synchronize. > + * > + * If @ssp provably has no readers in either epoch, provide the > + * synchronize_srcu() guarantee to the caller immediately: without > + * sleeping, without a trip through the SRCU workqueue, and without > + * advancing the grace-period sequence. Returns true on success; on > + * failure the caller must fall back to synchronize_srcu() or > + * synchronize_srcu_expedited(). > + * > + * This serves dedicated srcu_struct structures whose read-side critical > + * sections are short, atomic, and usually absent — where even an > + * expedited grace period costs two trips through the workqueue and an > + * unconditional sleep of the caller, three orders of magnitude more > + * than the check below. > + * > + * Only readers of the srcu_read_lock() and srcu_read_lock_nmisafe() > + * flavors are compatible with this proof; if the _fast() flavors have > + * ever been used on @ssp, this function always returns false. > + */ > +bool try_synchronize_srcu(struct srcu_struct *ssp) > +{ > + unsigned long unlocks0, unlocks1; > + unsigned long rdm0, rdm1; > + > + check_init_srcu_struct(ssp); > + > + /* > + * Order the caller's prior stores before the counter reads below. > + * Pairs (store-buffering pattern) with the smp_mb() in > + * __srcu_read_lock(): any reader whose lock increment is not > + * observed by the sums below is guaranteed to observe, within its > + * critical section, every store the caller made before calling > + * this function. > + */ > + smp_mb(); > + > + unlocks0 = srcu_readers_unlock_idx(ssp, 0, &rdm0); > + unlocks1 = srcu_readers_unlock_idx(ssp, 1, &rdm1); > + > + /* > + * Reader flavors which elide the read-side smp_mb() that the > + * pairing above depends on cannot be proven absent this way; > + * they need a real grace period. > + */ > + if ((rdm0 | rdm1) & SRCU_READ_FLAVOR_SLOWGP) > + return false; > + > + /* > + * As in srcu_readers_active_idx_check(): ensure that a lock is > + * always counted if the corresponding unlock is counted, so that > + * a reader racing with these sums can only inflate the lock sum > + * and force the (safe) fallback. Summing both epochs means no > + * index flip is needed: a stable equality proves there was a > + * moment in this function at which no readers existed at all. > + */ > + smp_mb(); > + > + if (!srcu_readers_lock_idx(ssp, 0, false, unlocks0)) > + return false; > + if (!srcu_readers_lock_idx(ssp, 1, false, unlocks1)) > + return false; > + > + /* Order the caller's subsequent accesses after the proof. */ > + smp_mb(); > + return true; > +} > +EXPORT_SYMBOL_GPL(try_synchronize_srcu); > + > /** > * get_state_synchronize_srcu - Provide an end-of-grace-period cookie > * @ssp: srcu_struct to provide cookie for.
smime.p7s
Description: S/MIME cryptographic signature
