I have been playing around with the idea to make some changes to avoid using the mbuf's 2nd cache line in many common cases, which would reduce the cache pressure significantly, and thus improve performance. I would like to discuss if it is doable. (And let's just assume that ABI breakage is an acceptable tradeoff.)
Move 'tx_offload' to the 1st cache line --------------------------------------- Under all circumstances: We would need to move the 'tx_offload' field to the 1st cache line. This field is set by the application's packet forwarding pipeline stage, and read by the PMD TX function. In most cases, these two stages directly follow each other. This also means that we must make room for it by moving a 64 bit field from the 1st to the 2nd cache line. It could be the 'next' or the 'pool' field, as discussed below. The 'next' field - make it conditional -------------------------------------- Optimization for (1) non-segmented packets: We could avoid touching the 'next' field by making the 'next' field depend on something in the first cache line. E.g.: - Use the 'ol_flags' field. Add a RTE_MBUF_F_MORE_SEGS flag, to be set/cleared when setting/clearing the 'next' field. - Use the 'nb_segs' field. Set the 'nb_segs' field to a value >1 when setting the 'next' field, and set it to 1 when clearing the 'next' field. The 'pool' field - use it less frequently ----------------------------------------- Optimizations for (2) single-mempool TX queues and (3) single-mempool applications: The 'pool' field seems to be only used by when a PMD frees a burst of mbufs that it has finished transmitting. Please correct me if I am wrong here. We could introduce a sibling to RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE, with the only requirement that the mbufs come from the same mempool. When set, only the first mbuf in a burst gets its 'pool' field read, thus avoiding reading it in the remaining mbufs in the burst. For single-mempool applications, we could introduce a global 'mbuf_pool' variable, to be used instead of the mbuf's 'pool' field, if set. Med venlig hilsen / Kind regards, -Morten Brørup
