> From: dev [mailto:[email protected]] On Behalf Of Leyi Rong
> Sent: Thursday, 14 January 2021 07.40
>
> Optimize Tx path by using AVX512 instructions and vectorize the
> tx free bufs process.
>
> Signed-off-by: Leyi Rong <[email protected]>
> Signed-off-by: Bruce Richardson <[email protected]>
> ---
[...]
> +static __rte_always_inline int
> +i40e_tx_free_bufs_avx512(struct i40e_tx_queue *txq)
> +{
> + struct i40e_vec_tx_entry *txep;
> + uint32_t n;
> + uint32_t i;
> + int nb_free = 0;
> + struct rte_mbuf *m, *free[RTE_I40E_TX_MAX_FREE_BUF_SZ];
> +
> + /* check DD bits on threshold descriptor */
> + if ((txq->tx_ring[txq->tx_next_dd].cmd_type_offset_bsz &
> + rte_cpu_to_le_64(I40E_TXD_QW1_DTYPE_MASK)) !=
> + rte_cpu_to_le_64(I40E_TX_DESC_DTYPE_DESC_DONE))
> + return 0;
> +
> + n = txq->tx_rs_thresh;
> +
> + /* first buffer to free from S/W ring is at index
> + * tx_next_dd - (tx_rs_thresh-1)
> + */
> + txep = (void *)txq->sw_ring;
> + txep += txq->tx_next_dd - (n - 1);
> +
> + if (txq->offloads & DEV_TX_OFFLOAD_MBUF_FAST_FREE && (n & 31) ==
> 0) {
> + struct rte_mempool *mp = txep[0].mbuf->pool;
> + void **cache_objs;
> + struct rte_mempool_cache *cache =
> rte_mempool_default_cache(mp,
> + rte_lcore_id());
> +
> + if (!cache || cache->len == 0)
> + goto normal;
> +
> + cache_objs = &cache->objs[cache->len];
> +
> + if (n > RTE_MEMPOOL_CACHE_MAX_SIZE) {
> + rte_mempool_ops_enqueue_bulk(mp, (void *)txep, n);
> + goto done;
> + }
> +
> + /* The cache follows the following algorithm
> + * 1. Add the objects to the cache
> + * 2. Anything greater than the cache min value (if it
> + * crosses the cache flush threshold) is flushed to the
> ring.
> + */
> + /* Add elements back into the cache */
> + uint32_t copied = 0;
> + /* n is multiple of 32 */
> + while (copied < n) {
> + const __m512i a = _mm512_load_si512(&txep[copied]);
> + const __m512i b = _mm512_load_si512(&txep[copied +
> 8]);
> + const __m512i c = _mm512_load_si512(&txep[copied +
> 16]);
> + const __m512i d = _mm512_load_si512(&txep[copied +
> 24]);
> +
> + _mm512_storeu_si512(&cache_objs[copied], a);
> + _mm512_storeu_si512(&cache_objs[copied + 8], b);
> + _mm512_storeu_si512(&cache_objs[copied + 16], c);
> + _mm512_storeu_si512(&cache_objs[copied + 24], d);
> + copied += 32;
> + }
Do you have any indications about how this copy loop performs, compared to e.g.
the 64-byte block copy in rte_memcpy() [1]?
[1]:
https://github.com/DPDK/dpdk/blob/v26.07-rc4/lib/eal/x86/include/rte_memcpy.h#L657
I'm wondering if it would be worth adding something similar to the mempool
library, for speeding up mbuf alloc/free.
> + cache->len += n;
> +
> + if (cache->len >= cache->flushthresh) {
> + rte_mempool_ops_enqueue_bulk
> + (mp, &cache->objs[cache->size],
> + cache->len - cache->size);
> + cache->len = cache->size;
> + }
> + goto done;
> + }
> +
> +normal:
