27/08/2023 10:34, Morten Brørup: > +CC Honnappa and Konstantin, Ring lib maintainers > +CC Mattias, PRNG lib maintainer > > > From: Bruce Richardson [mailto:bruce.richard...@intel.com] > > Sent: Friday, 25 August 2023 11.24 > > > > On Fri, Aug 25, 2023 at 11:06:01AM +0200, Morten Brørup wrote: > > > +CC mempool maintainers > > > > > > > From: Bruce Richardson [mailto:bruce.richard...@intel.com] > > > > Sent: Friday, 25 August 2023 10.23 > > > > > > > > On Fri, Aug 25, 2023 at 08:45:12AM +0200, Morten Brørup wrote: > > > > > Bruce, > > > > > > > > > > With this patch [1], it is noted that the ring producer and > > consumer data > > > > should not be on adjacent cache lines, for performance reasons. > > > > > > > > > > [1]: > > > > > > https://git.dpdk.org/dpdk/commit/lib/librte_ring/rte_ring.h?id=d9f0d3a1f > > fd4b66 > > > > e75485cc8b63b9aedfbdfe8b0 > > > > > > > > > > (It's obvious that they cannot share the same cache line, because > > they are > > > > accessed by two different threads.) > > > > > > > > > > Intuitively, I would think that having them on different cache > > lines would > > > > suffice. Why does having an empty cache line between them make a > > difference? > > > > > > > > > > And does it need to be an empty cache line? Or does it suffice > > having the > > > > second structure start at two cache lines after the start of the > > first > > > > structure (e.g. if the size of the first structure is two cache > > lines)? > > > > > > > > > > I'm asking because the same principle might apply to other code > > too. > > > > > > > > > Hi Morten, > > > > > > > > this was something we discovered when working on the distributor > > library. > > > > If we have cachelines per core where there is heavy access, having > > some > > > > cachelines as a gap between the content cachelines can help > > performance. We > > > > believe this helps due to avoiding issues with the HW prefetchers > > (e.g. > > > > adjacent cacheline prefetcher) bringing in the second cacheline > > > > speculatively when an operation is done on the first line. > > > > > > I guessed that it had something to do with speculative prefetching, > > but wasn't sure. Good to get confirmation, and that it has a measureable > > effect somewhere. Very interesting! > > > > > > NB: More comments in the ring lib about stuff like this would be nice. > > > > > > So, for the mempool lib, what do you think about applying the same > > technique to the rte_mempool_debug_stats structure (which is an array > > indexed per lcore)... Two adjacent lcores heavily accessing their local > > mempool caches seems likely to me. But how heavy does the access need to > > be for this technique to be relevant? > > > > > > > No idea how heavy the accesses need to be for this to have a noticable > > effect. For things like debug stats, I wonder how worthwhile making such > > a > > change would be, but then again, any change would have very low impact > > too > > in that case. > > I just tried adding padding to some of the hot structures in our own > application, and observed a significant performance improvement for those. > > So I think this technique should have higher visibility in DPDK by adding a > new cache macro to rte_common.h:
+1 to make more visibility in doc and adding a macro, good idea!
