Hi everyone,
> > > >>> I tried this. On x86 (Xeon(R) Gold 6132 CPU @ 2.60GHz), the results > > > >>> are as > > > >> follows. The numbers in brackets are with the code on master. > > > >>> gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0 > > > >>> > > > >>> RTE>>ring_perf_elem_autotest > > > >>> ### Testing single element and burst enq/deq ### SP/SC single > > > >>> enq/dequeue: 5 MP/MC single enq/dequeue: 40 (35) SP/SC burst > > > >>> enq/dequeue (size: 8): 2 MP/MC burst enq/dequeue (size: 8): 6 SP/SC > > > >>> burst enq/dequeue (size: 32): 1 (2) MP/MC burst enq/dequeue (size: > > > >>> 32): 2 > > > >>> > > > >>> ### Testing empty dequeue ### > > > >>> SC empty dequeue: 2.11 > > > >>> MC empty dequeue: 1.41 (2.11) > > > >>> > > > >>> ### Testing using a single lcore ### SP/SC bulk enq/dequeue (size: > > > >>> 8): 2.15 (2.86) MP/MC bulk enq/dequeue > > > >>> (size: 8): 6.35 (6.91) SP/SC bulk enq/dequeue (size: 32): 1.35 > > > >>> (2.06) MP/MC bulk enq/dequeue (size: 32): 2.38 (2.95) > > > >>> > > > >>> ### Testing using two physical cores ### SP/SC bulk enq/dequeue (size: > > > >>> 8): 73.81 (15.33) MP/MC bulk enq/dequeue (size: 8): 75.10 (71.27) > > > >>> SP/SC bulk enq/dequeue (size: 32): 21.14 (9.58) MP/MC bulk > > > >>> enq/dequeue > > > >>> (size: 32): 25.74 (20.91) > > > >>> > > > >>> ### Testing using two NUMA nodes ### SP/SC bulk enq/dequeue (size: > > > >>> 8): 164.32 (50.66) MP/MC bulk enq/dequeue (size: 8): 176.02 (173.43) > > > >>> SP/SC bulk enq/dequeue (size: > > > >>> 32): 50.78 (23) MP/MC bulk enq/dequeue (size: 32): 63.17 (46.74) > > > >>> > > > >>> On one of the Arm platform > > > >>> MP/MC bulk enq/dequeue (size: 32): 0.37 (0.33) (~12% hit, the rest > > > >>> are > > > >>> ok) > > > > > > Tried this on a Power9 platform (3.6GHz), with two numa nodes and 16 > > > cores/node (SMT=4). Applied all 3 patches in v5, test results are as > > > follows: > > > > > > RTE>>ring_perf_elem_autotest > > > ### Testing single element and burst enq/deq ### SP/SC single enq/dequeue: > > > 42 MP/MC single enq/dequeue: 59 SP/SC burst enq/dequeue (size: 8): 5 > > > MP/MC burst enq/dequeue (size: 8): 7 SP/SC burst enq/dequeue (size: 32): 2 > > > MP/MC burst enq/dequeue (size: 32): 2 > > > > > > ### Testing empty dequeue ### > > > SC empty dequeue: 7.81 > > > MC empty dequeue: 7.81 > > > > > > ### Testing using a single lcore ### > > > SP/SC bulk enq/dequeue (size: 8): 5.76 > > > MP/MC bulk enq/dequeue (size: 8): 7.66 > > > SP/SC bulk enq/dequeue (size: 32): 2.10 > > > MP/MC bulk enq/dequeue (size: 32): 2.57 > > > > > > ### Testing using two hyperthreads ### > > > SP/SC bulk enq/dequeue (size: 8): 13.13 > > > MP/MC bulk enq/dequeue (size: 8): 13.98 > > > SP/SC bulk enq/dequeue (size: 32): 3.41 > > > MP/MC bulk enq/dequeue (size: 32): 4.45 > > > > > > ### Testing using two physical cores ### SP/SC bulk enq/dequeue (size: 8): > > > 11.00 MP/MC bulk enq/dequeue (size: 8): 10.95 SP/SC bulk enq/dequeue > > > (size: 32): 3.08 MP/MC bulk enq/dequeue (size: 32): 3.40 > > > > > > ### Testing using two NUMA nodes ### > > > SP/SC bulk enq/dequeue (size: 8): 63.41 > > > MP/MC bulk enq/dequeue (size: 8): 62.70 > > > SP/SC bulk enq/dequeue (size: 32): 15.39 MP/MC bulk enq/dequeue (size: > > > 32): 22.96 > > > > > Thanks for running this. There is another test 'ring_perf_autotest' which > > provides the numbers with the original implementation. The goal > is to make sure the numbers with the original implementation are the same as > these. Can you please run that as well? > > Honnappa, > > Your earlier perf report shows the cycles are in less than 1. That's > is due to it is using 50 or 100MHz clock in EL0. > Please check with PMU counter. See "ARM64 profiling" in > > http://doc.dpdk.org/guides/prog_guide/profile_app.html > > > Here is the octeontx2 values. There is a regression in two core cases > as you reported earlier in x86. > > > RTE>>ring_perf_autotest > ### Testing single element and burst enq/deq ### > SP/SC single enq/dequeue: 288 > MP/MC single enq/dequeue: 452 > SP/SC burst enq/dequeue (size: 8): 39 > MP/MC burst enq/dequeue (size: 8): 61 > SP/SC burst enq/dequeue (size: 32): 13 > MP/MC burst enq/dequeue (size: 32): 21 > > ### Testing empty dequeue ### > SC empty dequeue: 6.33 > MC empty dequeue: 6.67 > > ### Testing using a single lcore ### > SP/SC bulk enq/dequeue (size: 8): 38.35 > MP/MC bulk enq/dequeue (size: 8): 67.36 > SP/SC bulk enq/dequeue (size: 32): 13.10 > MP/MC bulk enq/dequeue (size: 32): 21.64 > > ### Testing using two physical cores ### > SP/SC bulk enq/dequeue (size: 8): 75.94 > MP/MC bulk enq/dequeue (size: 8): 107.66 > SP/SC bulk enq/dequeue (size: 32): 24.51 > MP/MC bulk enq/dequeue (size: 32): 33.23 > Test OK > RTE>> > > ---- after applying v5 of the patch ------ > > RTE>>ring_perf_autotest > ### Testing single element and burst enq/deq ### > SP/SC single enq/dequeue: 289 > MP/MC single enq/dequeue: 452 > SP/SC burst enq/dequeue (size: 8): 40 > MP/MC burst enq/dequeue (size: 8): 64 > SP/SC burst enq/dequeue (size: 32): 13 > MP/MC burst enq/dequeue (size: 32): 22 > > ### Testing empty dequeue ### > SC empty dequeue: 6.33 > MC empty dequeue: 6.67 > > ### Testing using a single lcore ### > SP/SC bulk enq/dequeue (size: 8): 39.73 > MP/MC bulk enq/dequeue (size: 8): 69.13 > SP/SC bulk enq/dequeue (size: 32): 13.44 > MP/MC bulk enq/dequeue (size: 32): 22.00 > > ### Testing using two physical cores ### > SP/SC bulk enq/dequeue (size: 8): 76.02 > MP/MC bulk enq/dequeue (size: 8): 112.50 > SP/SC bulk enq/dequeue (size: 32): 24.71 > MP/MC bulk enq/dequeue (size: 32): 33.34 > Test OK > RTE>> > > RTE>>ring_perf_elem_autotest > ### Testing single element and burst enq/deq ### > SP/SC single enq/dequeue: 290 > MP/MC single enq/dequeue: 503 > SP/SC burst enq/dequeue (size: 8): 39 > MP/MC burst enq/dequeue (size: 8): 63 > SP/SC burst enq/dequeue (size: 32): 11 > MP/MC burst enq/dequeue (size: 32): 19 > > ### Testing empty dequeue ### > SC empty dequeue: 6.33 > MC empty dequeue: 6.67 > > ### Testing using a single lcore ### > SP/SC bulk enq/dequeue (size: 8): 38.92 > MP/MC bulk enq/dequeue (size: 8): 62.54 > SP/SC bulk enq/dequeue (size: 32): 11.46 > MP/MC bulk enq/dequeue (size: 32): 19.89 > > ### Testing using two physical cores ### > SP/SC bulk enq/dequeue (size: 8): 87.55 > MP/MC bulk enq/dequeue (size: 8): 99.10 > SP/SC bulk enq/dequeue (size: 32): 26.63 > MP/MC bulk enq/dequeue (size: 32): 29.91 > Test OK > RTE>> > As I can see, there is copy&paste bug in patch #3 (that's why it probably produced some weird numbers for me first). After fix applied (see patch below), things look pretty good on my box. As I can see there are only 3 results noticably lower: SP/SC (size=8) over 2 physical cores same numa socket MP/MC (size=8) over 2 physical cores on different numa sockets. All others seems about same or better. Anyway I went ahead and reworked code a bit (as I suggested before) to get rid of these huge ENQUEUE/DEQUEUE macros. Results are very close to fixed patch #3 version (patch is also attached). Though I suggest people hold on to re-run perf tests till we'll make ring functional test to run for _elem_ functions too. I started to work on that, but not sure I'll finish today (most likely Monday). Perf results from my box, plus patches below. Konstantin perf results ========== Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz A - ring_perf_autotest B - ring_perf_elem_autotest + patch #3 + fix C - B + update ### Testing using a single lcore ### A B C SP/SC bulk enq/dequeue (size: 8): 4.06 3.06 3.22 MP/MC bulk enq/dequeue (size: 8): 10.05 9.04 9.38 SP/SC bulk enq/dequeue (size: 32): 2.93 1.91 1.84 MP/MC bulk enq/dequeue (size: 32): 4.12 3.39 3.35 ### Testing using two hyperthreads ### SP/SC bulk enq/dequeue (size: 8): 9.24 8.92 8.89 MP/MC bulk enq/dequeue (size: 8): 15.47 15.39 16.02 SP/SC bulk enq/dequeue (size: 32): 5.78 3.87 3.86 MP/MC bulk enq/dequeue (size: 32): 6.41 4.57 4.45 ### Testing using two physical cores ### SP/SC bulk enq/dequeue (size: 8): 24.14 29.89 27.05 MP/MC bulk enq/dequeue (size: 8): 68.61 70.55 69.85 SP/SC bulk enq/dequeue (size: 32): 12.11 12.99 13.04 MP/MC bulk enq/dequeue (size: 32): 22.14 17.86 18.25 ### Testing using two NUMA nodes ### SP/SC bulk enq/dequeue (size: 8): 48.78 31.98 33.57 MP/MC bulk enq/dequeue (size: 8): 167.53 197.29 192.13 SP/SC bulk enq/dequeue (size: 32): 31.28 21.68 21.61 MP/MC bulk enq/dequeue (size: 32): 53.45 49.94 48.81 fix patch ======= From a2be5a9b136333a56d466ef042c655e522ca7012 Mon Sep 17 00:00:00 2001 From: Konstantin Ananyev <konstantin.anan...@intel.com> Date: Fri, 18 Oct 2019 15:50:43 +0100 Subject: [PATCH] fix1 Signed-off-by: Konstantin Ananyev <konstantin.anan...@intel.com> --- lib/librte_ring/rte_ring_elem.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/librte_ring/rte_ring_elem.h b/lib/librte_ring/rte_ring_elem.h index 92e92f150..5e1819069 100644 --- a/lib/librte_ring/rte_ring_elem.h +++ b/lib/librte_ring/rte_ring_elem.h @@ -118,7 +118,7 @@ struct rte_ring *rte_ring_create_elem(const char *name, unsigned count, uint32_t sz = n * (esize / sizeof(uint32_t)); \ if (likely(idx + n < size)) { \ for (i = 0; i < (sz & ((~(unsigned)0x7))); i += 8, idx += 8) { \ - memcpy (ring + i, obj + i, 8 * sizeof (uint32_t)); \ + memcpy (ring + idx, obj + i, 8 * sizeof (uint32_t)); \ } \ switch (n & 0x7) { \ case 7: \ @@ -153,7 +153,7 @@ struct rte_ring *rte_ring_create_elem(const char *name, unsigned count, uint32_t sz = n * (esize / sizeof(uint32_t)); \ if (likely(idx + n < size)) { \ for (i = 0; i < (sz & ((~(unsigned)0x7))); i += 8, idx += 8) { \ - memcpy (obj + i, ring + i, 8 * sizeof (uint32_t)); \ + memcpy (obj + i, ring + idx, 8 * sizeof (uint32_t)); \ } \ switch (n & 0x7) { \ case 7: \ -- 2.17.1 update patch (remove macros) ========================= From 18b388e877b97e243f807f27a323e876b30869dd Mon Sep 17 00:00:00 2001 From: Konstantin Ananyev <konstantin.anan...@intel.com> Date: Fri, 18 Oct 2019 17:35:43 +0100 Subject: [PATCH] update1 Signed-off-by: Konstantin Ananyev <konstantin.anan...@intel.com> --- lib/librte_ring/rte_ring_elem.h | 141 ++++++++++++++++---------------- 1 file changed, 70 insertions(+), 71 deletions(-) diff --git a/lib/librte_ring/rte_ring_elem.h b/lib/librte_ring/rte_ring_elem.h index 5e1819069..eb706b12f 100644 --- a/lib/librte_ring/rte_ring_elem.h +++ b/lib/librte_ring/rte_ring_elem.h @@ -109,75 +109,74 @@ __rte_experimental struct rte_ring *rte_ring_create_elem(const char *name, unsigned count, unsigned esize, int socket_id, unsigned flags); -#define ENQUEUE_PTRS_GEN(r, ring_start, prod_head, obj_table, esize, n) do { \ - unsigned int i; \ - const uint32_t size = (r)->size; \ - uint32_t idx = prod_head & (r)->mask; \ - uint32_t *ring = (uint32_t *)ring_start; \ - uint32_t *obj = (uint32_t *)obj_table; \ - uint32_t sz = n * (esize / sizeof(uint32_t)); \ - if (likely(idx + n < size)) { \ - for (i = 0; i < (sz & ((~(unsigned)0x7))); i += 8, idx += 8) { \ - memcpy (ring + idx, obj + i, 8 * sizeof (uint32_t)); \ - } \ - switch (n & 0x7) { \ - case 7: \ - ring[idx++] = obj[i++]; /* fallthrough */ \ - case 6: \ - ring[idx++] = obj[i++]; /* fallthrough */ \ - case 5: \ - ring[idx++] = obj[i++]; /* fallthrough */ \ - case 4: \ - ring[idx++] = obj[i++]; /* fallthrough */ \ - case 3: \ - ring[idx++] = obj[i++]; /* fallthrough */ \ - case 2: \ - ring[idx++] = obj[i++]; /* fallthrough */ \ - case 1: \ - ring[idx++] = obj[i++]; /* fallthrough */ \ - } \ - } else { \ - for (i = 0; idx < size; i++, idx++)\ - ring[idx] = obj[i]; \ - for (idx = 0; i < n; i++, idx++) \ - ring[idx] = obj[i]; \ - } \ -} while (0) - -#define DEQUEUE_PTRS_GEN(r, ring_start, cons_head, obj_table, esize, n) do { \ - unsigned int i; \ - uint32_t idx = cons_head & (r)->mask; \ - const uint32_t size = (r)->size; \ - uint32_t *ring = (uint32_t *)ring_start; \ - uint32_t *obj = (uint32_t *)obj_table; \ - uint32_t sz = n * (esize / sizeof(uint32_t)); \ - if (likely(idx + n < size)) { \ - for (i = 0; i < (sz & ((~(unsigned)0x7))); i += 8, idx += 8) { \ - memcpy (obj + i, ring + idx, 8 * sizeof (uint32_t)); \ - } \ - switch (n & 0x7) { \ - case 7: \ - obj[i++] = ring[idx++]; /* fallthrough */ \ - case 6: \ - obj[i++] = ring[idx++]; /* fallthrough */ \ - case 5: \ - obj[i++] = ring[idx++]; /* fallthrough */ \ - case 4: \ - obj[i++] = ring[idx++]; /* fallthrough */ \ - case 3: \ - obj[i++] = ring[idx++]; /* fallthrough */ \ - case 2: \ - obj[i++] = ring[idx++]; /* fallthrough */ \ - case 1: \ - obj[i++] = ring[idx++]; /* fallthrough */ \ - } \ - } else { \ - for (i = 0; idx < size; i++, idx++) \ - obj[i] = ring[idx]; \ - for (idx = 0; i < n; i++, idx++) \ - obj[i] = ring[idx]; \ - } \ -} while (0) +static __rte_always_inline void +copy_elems(uint32_t du32[], const uint32_t su32[], uint32_t num, uint32_t esize) +{ + uint32_t i, sz; + + sz = (num * esize) / sizeof(uint32_t); + + for (i = 0; i < (sz & ~7); i += 8) + memcpy(du32 + i, su32 + i, 8 * sizeof(uint32_t)); + + switch (sz & 7) { + case 7: du32[sz - 7] = su32[sz - 7]; /* fallthrough */ + case 6: du32[sz - 6] = su32[sz - 6]; /* fallthrough */ + case 5: du32[sz - 5] = su32[sz - 5]; /* fallthrough */ + case 4: du32[sz - 4] = su32[sz - 4]; /* fallthrough */ + case 3: du32[sz - 3] = su32[sz - 3]; /* fallthrough */ + case 2: du32[sz - 2] = su32[sz - 2]; /* fallthrough */ + case 1: du32[sz - 1] = su32[sz - 1]; /* fallthrough */ + } +} + +static __rte_always_inline void +enqueue_elems(struct rte_ring *r, void *ring_start, uint32_t prod_head, + void *obj_table, uint32_t num, uint32_t esize) +{ + uint32_t idx, n; + uint32_t *du32; + const uint32_t *su32; + + const uint32_t size = r->size; + + idx = prod_head & (r)->mask; + + du32 = (uint32_t *)ring_start + idx; + su32 = obj_table; + + if (idx + num < size) + copy_elems(du32, su32, num, esize); + else { + n = size - idx; + copy_elems(du32, su32, n, esize); + copy_elems(ring_start, su32 + n, num - n, esize); + } +} + +static __rte_always_inline void +dequeue_elems(struct rte_ring *r, void *ring_start, uint32_t cons_head, + void *obj_table, uint32_t num, uint32_t esize) +{ + uint32_t idx, n; + uint32_t *du32; + const uint32_t *su32; + + const uint32_t size = r->size; + + idx = cons_head & (r)->mask; + + su32 = (uint32_t *)ring_start + idx; + du32 = obj_table; + + if (idx + num < size) + copy_elems(du32, su32, num, esize); + else { + n = size - idx; + copy_elems(du32, su32, n, esize); + copy_elems(du32 + n, ring_start, num - n, esize); + } +} /* Between load and load. there might be cpu reorder in weak model * (powerpc/arm). @@ -232,7 +231,7 @@ __rte_ring_do_enqueue_elem(struct rte_ring *r, void * const obj_table, if (n == 0) goto end; - ENQUEUE_PTRS_GEN(r, &r[1], prod_head, obj_table, esize, n); + enqueue_elems(r, &r[1], prod_head, obj_table, n, esize); update_tail(&r->prod, prod_head, prod_next, is_sp, 1); end: @@ -279,7 +278,7 @@ __rte_ring_do_dequeue_elem(struct rte_ring *r, void *obj_table, if (n == 0) goto end; - DEQUEUE_PTRS_GEN(r, &r[1], cons_head, obj_table, esize, n); + dequeue_elems(r, &r[1], cons_head, obj_table, n, esize); update_tail(&r->cons, cons_head, cons_next, is_sc, 0); -- 2.17.1
