RE: [EXT] Re: [PATCH v5 0/4] add support for self monitoring
> From: Tyler Retzlaff [mailto:roret...@linux.microsoft.com] > Sent: Friday, 13 January 2023 20.22 > > On Fri, Jan 13, 2023 at 07:44:57AM +, Tomasz Duszynski wrote: > > > > >From: Tyler Retzlaff > > >Sent: Wednesday, January 11, 2023 10:06 PM > > > > > >On Wed, Jan 11, 2023 at 09:39:35AM +, Tomasz Duszynski wrote: > > >> Hi Tyler, > > >> > > >> >From: Tyler Retzlaff > > >> >Sent: Wednesday, January 11, 2023 1:32 AM > > >> > > > >> >hi, > > >> > > > >> >don't interpret this as an objection to the functionality but > this > > >> >looks like a clear example of something that doesn't belong in > the > > >> >EAL. has there been a discussion as to whether or not this should > be in a separate library? > > >> > > >> No, I don't recall anybody having any concerns about the code > > >> placement. Rationale behind making this part of eal was based on > the > > >> fact that tracing itself is a part of eal and since this was meant > to be extension to tracing, > > >code placement decision came out naturally. > > >> > > >> During development phase idea evolved a bit and what initially was > > >> supposed to be solely yet another tracepoint become generic API to > > >> read pmu and tracepoint based on that. Which means both can be > used independently. > > >> > > >> That said, since this code has both platform agnostic and platform > specific parts this can either > > >be split into: > > >> 1. library + eal platform code > > >> 2. all under eal > > >> > > >> Either approach seems legit. Thoughts? > > >> > > >> > > > >> >a basic test is whether or not an implementation exists or can be > > >> >reasonably provided for all platforms and that isn't strictly > evident > > >> >here. red flag is to see yet more code being added conditionally > compiled for a single platform. > > >> > > >> Even libs are not entirely pristine and have platform specific > ifdefs > > >> lurking so not sure where this red flag is coming from. > > > > > >i think red flag was probably the wrong term to use sorry for that. > > >rather i should say it is an indicator that the api probably doesn't > belong in the eal. > > > > > >fundamentally the purpose of the abstraction library is to relieve > the application from having to > > >do conditional compilation and/or execution for the subject apis > coming from eal. including and > > >exporting apis that work for only one platform is in direct > contradiction. > > > > > >please explore adding this as a separate library, it is understood > that there are tradeoffs > > >involved. > > > > > >thanks! > > > > Any ideas how to name the library? > > naming is always so hard and i'm definitely not authoritative. > > it seems like lib/pmu would be the least churn against your existing > patch series, here are some other suggestions that might work. +1 to lib/pmu Less work, as Tyler already mentioned. Furthermore: Both Intel and ARM use the term Performance Monitoring Unit (abbreviated PMU). Microsoft does too [1]. [1]: https://learn.microsoft.com/en-us/windows-hardware/test/wpt/recording-pmu-events RISC-V uses the term Hardware Performance Monitor (abbreviated HPM). I haven't checked other CPU vendors.
RE: [PATCH 0/7] replace zero length arrays
> From: Stephen Hemminger [mailto:step...@networkplumber.org] > Sent: Friday, 13 January 2023 22.52 > > Zero length arrays are a GNU extension that has been > superseded by flex arrays. > > https://gcc.gnu.org/onlinedocs/gcc/Zero-Length.html > > These are places found by cocci/zero_length_array.cocci Series-acked-by: Morten Brørup
[PATCH v12 6/6] test/memarea: support dump API test
This patch supports rte_memarea_dump() API test.
Signed-off-by: Chengwen Feng
Reviewed-by: Dongdong Liu
---
app/test/test_memarea.c | 34 ++
1 file changed, 34 insertions(+)
diff --git a/app/test/test_memarea.c b/app/test/test_memarea.c
index efabc594b7..f8c32ca9dd 100644
--- a/app/test/test_memarea.c
+++ b/app/test/test_memarea.c
@@ -246,6 +246,39 @@ test_memarea_alloc_free(void)
return 0;
}
+static int
+test_memarea_dump(void)
+{
+ struct rte_memarea_param init;
+ struct rte_memarea *ma;
+ int ret;
+
+ test_memarea_init_param(&init);
+ init.source = RTE_MEMAREA_SOURCE_LIBC;
+ init.total_sz = MEMAREA_TEST_DEFAULT_SIZE;
+ ma = rte_memarea_create(&init);
+ RTE_TEST_ASSERT(ma != NULL, "Expected Non-NULL");
+
+ /* test for invalid parameters */
+ ret = rte_memarea_dump(NULL, stderr, false);
+ RTE_TEST_ASSERT(ret == -EINVAL, "Expected EINVAL");
+ ret = rte_memarea_dump(ma, NULL, false);
+ RTE_TEST_ASSERT(ret == -EINVAL, "Expected EINVAL");
+
+ /* test for dump */
+ (void)rte_memarea_alloc(ma, 1, 0);
+ (void)rte_memarea_alloc(ma, 1, 0);
+ (void)rte_memarea_alloc(ma, 1, 0);
+ (void)rte_memarea_alloc(ma, MEMAREA_TEST_DEFAULT_SIZE, 0);
+ (void)rte_memarea_alloc(ma, MEMAREA_TEST_DEFAULT_SIZE, 0);
+ ret = rte_memarea_dump(ma, stderr, true);
+ RTE_TEST_ASSERT(ret == 0, "Expected ZERO");
+
+ rte_memarea_destroy(ma);
+
+ return 0;
+}
+
static int
test_memarea(void)
{
@@ -256,6 +289,7 @@ test_memarea(void)
MEMAREA_TEST_API_RUN(test_memarea_alloc_fail);
MEMAREA_TEST_API_RUN(test_memarea_free_fail);
MEMAREA_TEST_API_RUN(test_memarea_alloc_free);
+ MEMAREA_TEST_API_RUN(test_memarea_dump);
return test_memarea_retcode();
}
--
2.33.0
[PATCH v12 0/6] introduce memarea library
The memarea library is an allocator of variable-size object which based on a memory region. The main features are as follows: - The memory region can be initialized from the following memory sources: 1. HEAP: e.g. invoke rte_malloc_socket. 2. LIBC: e.g. invoke posix_memalign. 3. Another memarea: it can be from another memarea. - It supports MT-safe as long as it's specified at creation time. Note: a) The memarea is oriented towards the application layer, which could provides 'region-based memory management' [1] function. b) The eal library also provide memory zone/heap management, but these are tied to huge pages management. [1] https://en.wikipedia.org/wiki/Region-based_memory_management Chengwen Feng (6): memarea: introduce memarea library test/memarea: support memarea test memarea: support alloc and free API test/memarea: support alloc and free API test memarea: support dump API test/memarea: support dump API test --- v12: * remove rte_memarea_refcnt_update() API which address Dongdong's comments. * refine the variable naming. * fix some bugs. v11: * rebase 23.03 * remove "app/test: add memarea to malloc-perf-autotest" because the two algorithm are not comparable which also address previous comments. v10: * support windows platform. * add rte_memarea.libc perftest to malloc-perf-autotest. v9: * address Dmitry's comments. * drop features of SOURCE_USER and backup memarea mechanism. * rename rte_memarea_update_refcnt to rte_memarea_refcnt_update to keep with rte_mbuf_refcnt_update name style. * fix memarea perftest compile failed at windows platform. * fix spell warning. v8: * address Mattias's comments (rename ALG_DEFAULT with ALG_NEXTFIT). * small feature patches are combined. * enhanced backup memory mechanism. * add memarea to malloc-perf-autotest. * other tiny naming optimize. v7: * repost patches as there are spread over different series in patchwork. v6: * address Mattias's comments. v5: * fix 09/10 patch spell warning. v4: * repost patches as there are spread over different series in patchwork. v3: * add memory source of RTE memory. * add algorithm field to facilitate the introduction of new algorithms. * fix memarea log don't output problem. v2: * fix compile issues reported by dpdk-test-report. * address Dimitry and Jerin's comments. * add no MT-safe test. MAINTAINERS| 6 + app/test/meson.build | 2 + app/test/test_memarea.c| 297 + doc/api/doxy-api-index.md | 3 +- doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/index.rst| 1 + doc/guides/prog_guide/memarea_lib.rst | 46 +++ doc/guides/rel_notes/release_23_03.rst | 5 + lib/eal/common/eal_common_log.c| 1 + lib/eal/include/rte_log.h | 1 + lib/memarea/memarea_private.h | 56 lib/memarea/meson.build| 10 + lib/memarea/rte_memarea.c | 420 + lib/memarea/rte_memarea.h | 189 +++ lib/memarea/version.map| 15 + lib/meson.build| 1 + 16 files changed, 1053 insertions(+), 1 deletion(-) create mode 100644 app/test/test_memarea.c create mode 100644 doc/guides/prog_guide/memarea_lib.rst create mode 100644 lib/memarea/memarea_private.h create mode 100644 lib/memarea/meson.build create mode 100644 lib/memarea/rte_memarea.c create mode 100644 lib/memarea/rte_memarea.h create mode 100644 lib/memarea/version.map -- 2.33.0
[PATCH v12 3/6] memarea: support alloc and free API
This patch supports rte_memarea_alloc() and rte_memarea_free() API.
Signed-off-by: Chengwen Feng
Reviewed-by: Dongdong Liu
---
doc/guides/prog_guide/memarea_lib.rst | 6 ++
lib/memarea/memarea_private.h | 3 +
lib/memarea/rte_memarea.c | 146 ++
lib/memarea/rte_memarea.h | 46
lib/memarea/version.map | 2 +
5 files changed, 203 insertions(+)
diff --git a/doc/guides/prog_guide/memarea_lib.rst
b/doc/guides/prog_guide/memarea_lib.rst
index 156ff35cfd..01187f7ccb 100644
--- a/doc/guides/prog_guide/memarea_lib.rst
+++ b/doc/guides/prog_guide/memarea_lib.rst
@@ -31,6 +31,12 @@ returns the pointer to the created memarea or ``NULL`` if
the creation failed.
The ``rte_memarea_destroy()`` function is used to destroy a memarea.
+The ``rte_memarea_alloc()`` function is used to alloc one memory object from
+the memarea.
+
+The ``rte_memarea_free()`` function is used to free one memory object which
+allocated by ``rte_memarea_alloc()``.
+
Reference
-
diff --git a/lib/memarea/memarea_private.h b/lib/memarea/memarea_private.h
index 3d152ec780..509cbc7bc7 100644
--- a/lib/memarea/memarea_private.h
+++ b/lib/memarea/memarea_private.h
@@ -48,6 +48,9 @@ struct rte_memarea {
void*area_addr;
struct memarea_obj_list obj_list;
struct memarea_obj_list free_list;
+
+ uint64_t alloc_fails;
+ uint64_t magic_check_fails;
} __rte_cache_aligned;
#endif /* MEMAREA_PRIVATE_H */
diff --git a/lib/memarea/rte_memarea.c b/lib/memarea/rte_memarea.c
index f3b3bdb09a..76f8083d96 100644
--- a/lib/memarea/rte_memarea.c
+++ b/lib/memarea/rte_memarea.c
@@ -2,8 +2,10 @@
* Copyright(c) 2023 HiSilicon Limited
*/
+#include
#include
#include
+#include
#include
#include
@@ -84,6 +86,8 @@ memarea_alloc_area(const struct rte_memarea_param *init)
init->numa_socket);
else if (init->source == RTE_MEMAREA_SOURCE_LIBC)
ptr = memarea_alloc_from_libc(init->total_sz);
+ else if (init->source == RTE_MEMAREA_SOURCE_MEMAREA)
+ ptr = rte_memarea_alloc(init->src_memarea, init->total_sz, 0);
return ptr;
}
@@ -95,6 +99,8 @@ memarea_free_area(const struct rte_memarea_param *init, void
*ptr)
rte_free(ptr);
else if (init->source == RTE_MEMAREA_SOURCE_LIBC)
free(ptr);
+ else if (init->source == RTE_MEMAREA_SOURCE_MEMAREA)
+ rte_memarea_free(init->src_memarea, ptr);
}
struct rte_memarea *
@@ -156,3 +162,143 @@ rte_memarea_destroy(struct rte_memarea *ma)
rte_free(ma);
}
+static inline void
+memarea_lock(struct rte_memarea *ma)
+{
+ if (ma->init.mt_safe)
+ rte_spinlock_lock(&ma->lock);
+}
+
+static inline void
+memarea_unlock(struct rte_memarea *ma)
+{
+ if (ma->init.mt_safe)
+ rte_spinlock_unlock(&ma->lock);
+}
+
+static inline uint32_t
+memarea_calc_align_space(struct memarea_obj *obj, uint32_t align)
+{
+ if (align == 0)
+ return 0;
+ return align - (((uintptr_t)obj + sizeof(struct memarea_obj) +
sizeof(uint32_t)) &
+ (align - 1));
+}
+
+static inline bool
+memarea_whether_add_node(size_t obj_size, size_t need_size)
+{
+ return (obj_size - need_size) > sizeof(struct memarea_obj) +
RTE_CACHE_LINE_SIZE;
+}
+
+static inline void
+memarea_add_node(struct rte_memarea *ma, struct memarea_obj *obj, size_t
used_size)
+{
+ size_t align_size = RTE_ALIGN_CEIL(used_size, sizeof(void *));
+ struct memarea_obj *new_obj;
+
+ new_obj = (struct memarea_obj *)RTE_PTR_ADD(obj, sizeof(struct
memarea_obj) +
+align_size);
+ new_obj->size = obj->size - align_size - sizeof(struct memarea_obj);
+ new_obj->alloc_size = 0;
+ new_obj->magic = MEMAREA_OBJECT_FREE_MAGIC;
+ TAILQ_INSERT_AFTER(&ma->obj_list, obj, new_obj, obj_node);
+ TAILQ_INSERT_AFTER(&ma->free_list, obj, new_obj, free_node);
+ obj->size = align_size;
+}
+
+void *
+rte_memarea_alloc(struct rte_memarea *ma, size_t size, uint32_t align)
+{
+ size_t size_req = size + align + sizeof(uint32_t); /* use to check size
overflow */
+ struct memarea_obj *obj;
+ uint32_t align_space;
+ void *ptr = NULL;
+
+ if (unlikely(ma == NULL || size == 0 || size_req < size ||
+ (align && !rte_is_power_of_2(align
+ return ptr;
+
+ memarea_lock(ma);
+ TAILQ_FOREACH(obj, &ma->free_list, free_node) {
+ if (unlikely(obj->magic != MEMAREA_OBJECT_FREE_MAGIC)) {
+ ma->magic_check_fails++;
+ RTE_LOG(ERR, MEMAREA, "memarea: %s magic: 0x%x check
fail when alloc object!\n",
+ ma->init.name, obj->magic);
+
[PATCH v12 4/6] test/memarea: support alloc and free API test
This patch supports rte_memarea_alloc() and rte_memarea_free() API
test.
Signed-off-by: Chengwen Feng
Reviewed-by: Dongdong Liu
---
app/test/test_memarea.c | 135 +++-
1 file changed, 134 insertions(+), 1 deletion(-)
diff --git a/app/test/test_memarea.c b/app/test/test_memarea.c
index c9de779a1d..efabc594b7 100644
--- a/app/test/test_memarea.c
+++ b/app/test/test_memarea.c
@@ -47,6 +47,12 @@ test_memarea_init_param(struct rte_memarea_param *init)
init->mt_safe = 1;
}
+static void
+test_memarea_fill_region(void *ptr, size_t size)
+{
+ memset(ptr, 0xff, size);
+}
+
static int
test_memarea_create_bad_param(void)
{
@@ -95,8 +101,8 @@ test_memarea_create_bad_param(void)
static int
test_memarea_create_destroy(void)
{
+ struct rte_memarea *ma, *src_ma;
struct rte_memarea_param init;
- struct rte_memarea *ma;
/* test for create with HEAP */
test_memarea_init_param(&init);
@@ -113,6 +119,130 @@ test_memarea_create_destroy(void)
RTE_TEST_ASSERT(ma != NULL, "Expected Non-NULL");
rte_memarea_destroy(ma);
+ /* test for create with another memarea */
+ test_memarea_init_param(&init);
+ init.source = RTE_MEMAREA_SOURCE_LIBC;
+ src_ma = rte_memarea_create(&init);
+ RTE_TEST_ASSERT(src_ma != NULL, "Expected Non-NULL");
+ test_memarea_init_param(&init);
+ init.source = RTE_MEMAREA_SOURCE_MEMAREA;
+ init.total_sz = init.total_sz >> 1;
+ init.src_memarea = src_ma;
+ ma = rte_memarea_create(&init);
+ RTE_TEST_ASSERT(ma != NULL, "Expected Non-NULL");
+ rte_memarea_destroy(ma);
+ rte_memarea_destroy(src_ma);
+
+ return 0;
+}
+
+static int
+test_memarea_alloc_fail(void)
+{
+ struct rte_memarea_param init;
+ struct rte_memarea *ma;
+ void *ptr[2];
+
+ test_memarea_init_param(&init);
+ init.source = RTE_MEMAREA_SOURCE_LIBC;
+ init.total_sz = MEMAREA_TEST_DEFAULT_SIZE;
+ ma = rte_memarea_create(&init);
+ RTE_TEST_ASSERT(ma != NULL, "Expected Non-NULL");
+
+ /* test alloc fail with big size */
+ ptr[0] = rte_memarea_alloc(ma, MEMAREA_TEST_DEFAULT_SIZE, 0);
+ RTE_TEST_ASSERT(ptr[0] == NULL, "Expected NULL");
+
+ /* test alloc fail because no memory */
+ ptr[0] = rte_memarea_alloc(ma, MEMAREA_TEST_DEFAULT_SIZE -
RTE_CACHE_LINE_SIZE, 0);
+ RTE_TEST_ASSERT(ptr[0] != NULL, "Expected Non-NULL");
+ ptr[1] = rte_memarea_alloc(ma, 1, 0);
+ RTE_TEST_ASSERT(ptr[1] == NULL, "Expected NULL");
+ rte_memarea_free(ma, ptr[0]);
+
+ /* test alloc fail when second fail */
+ ptr[0] = rte_memarea_alloc(ma, MEMAREA_TEST_DEFAULT_SIZE >> 1, 0);
+ RTE_TEST_ASSERT(ptr[0] != NULL, "Expected Non-NULL");
+ test_memarea_fill_region(ptr[0], MEMAREA_TEST_DEFAULT_SIZE >> 1);
+ ptr[1] = rte_memarea_alloc(ma, MEMAREA_TEST_DEFAULT_SIZE >> 1, 0);
+ RTE_TEST_ASSERT(ptr[1] == NULL, "Expected NULL");
+ rte_memarea_free(ma, ptr[0]);
+ ptr[1] = rte_memarea_alloc(ma, MEMAREA_TEST_DEFAULT_SIZE >> 1, 0);
+ RTE_TEST_ASSERT(ptr[1] != NULL, "Expected Non-NULL");
+ test_memarea_fill_region(ptr[1], MEMAREA_TEST_DEFAULT_SIZE >> 1);
+ rte_memarea_free(ma, ptr[1]);
+
+ rte_memarea_destroy(ma);
+
+ return 0;
+}
+
+static int
+test_memarea_free_fail(void)
+{
+ struct rte_memarea_param init;
+ struct rte_memarea *ma;
+ void *ptr;
+
+ test_memarea_init_param(&init);
+ init.source = RTE_MEMAREA_SOURCE_LIBC;
+ init.total_sz = MEMAREA_TEST_DEFAULT_SIZE;
+ ma = rte_memarea_create(&init);
+ RTE_TEST_ASSERT(ma != NULL, "Expected Non-NULL");
+
+ /* test repeat free */
+ ptr = rte_memarea_alloc(ma, MEMAREA_TEST_DEFAULT_SIZE >> 1, 0);
+ RTE_TEST_ASSERT(ptr != NULL, "Expected Non-NULL");
+ test_memarea_fill_region(ptr, MEMAREA_TEST_DEFAULT_SIZE >> 1);
+ rte_memarea_free(ma, ptr);
+ rte_memarea_free(ma, ptr);
+
+ rte_memarea_destroy(ma);
+
+ return 0;
+}
+
+static int
+test_memarea_alloc_free(void)
+{
+#define ALLOC_MAX_NUM 8
+ struct rte_memarea_param init;
+ void *ptr[ALLOC_MAX_NUM];
+ struct rte_memarea *ma;
+ int i;
+
+ test_memarea_init_param(&init);
+ init.source = RTE_MEMAREA_SOURCE_LIBC;
+ init.total_sz = MEMAREA_TEST_DEFAULT_SIZE;
+ ma = rte_memarea_create(&init);
+ RTE_TEST_ASSERT(ma != NULL, "Expected Non-NULL");
+ memset(ptr, 0, sizeof(ptr));
+
+ /* test random alloc and free */
+ for (i = 0; i < ALLOC_MAX_NUM; i++)
+ ptr[i] = rte_memarea_alloc(ma, 1, 0);
+
+ /* test merge left */
+ rte_memarea_free(ma, ptr[0]);
+ rte_memarea_free(ma, ptr[1]);
+
+ /* test merge right */
+ rte_memarea_free(ma, ptr[7]);
+ rte_memarea_free(ma, ptr[6]);
+
+ /* test merge left and right */
+ rte_memarea_fr
[PATCH v12 1/6] memarea: introduce memarea library
The memarea library is an allocator of variable-size object which based on a memory region. This patch provides rte_memarea_create() and rte_memarea_destroy() API. Signed-off-by: Chengwen Feng Reviewed-by: Dongdong Liu --- MAINTAINERS| 5 + doc/api/doxy-api-index.md | 3 +- doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/index.rst| 1 + doc/guides/prog_guide/memarea_lib.rst | 37 ++ doc/guides/rel_notes/release_23_03.rst | 5 + lib/eal/common/eal_common_log.c| 1 + lib/eal/include/rte_log.h | 1 + lib/memarea/memarea_private.h | 53 + lib/memarea/meson.build| 10 ++ lib/memarea/rte_memarea.c | 158 + lib/memarea/rte_memarea.h | 122 +++ lib/memarea/version.map| 12 ++ lib/meson.build| 1 + 14 files changed, 409 insertions(+), 1 deletion(-) create mode 100644 doc/guides/prog_guide/memarea_lib.rst create mode 100644 lib/memarea/memarea_private.h create mode 100644 lib/memarea/meson.build create mode 100644 lib/memarea/rte_memarea.c create mode 100644 lib/memarea/rte_memarea.h create mode 100644 lib/memarea/version.map diff --git a/MAINTAINERS b/MAINTAINERS index 9a0f416d2e..5991156227 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1575,6 +1575,11 @@ F: app/test/test_lpm* F: app/test/test_func_reentrancy.c F: app/test/test_xmmt_ops.h +Memarea - EXPERIMENTAL +M: Chengwen Feng +F: lib/memarea +F: doc/guides/prog_guide/memarea_lib.rst + Membership - EXPERIMENTAL M: Yipeng Wang M: Sameh Gobriel diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md index de488c7abf..24456604f8 100644 --- a/doc/api/doxy-api-index.md +++ b/doc/api/doxy-api-index.md @@ -62,7 +62,8 @@ The public API headers are grouped by topics: [memzone](@ref rte_memzone.h), [mempool](@ref rte_mempool.h), [malloc](@ref rte_malloc.h), - [memcpy](@ref rte_memcpy.h) + [memcpy](@ref rte_memcpy.h), + [memarea](@ref rte_memarea.h) - **timers**: [cycles](@ref rte_cycles.h), diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in index f0886c3bd1..8334ebcbd6 100644 --- a/doc/api/doxy-api.conf.in +++ b/doc/api/doxy-api.conf.in @@ -53,6 +53,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \ @TOPDIR@/lib/latencystats \ @TOPDIR@/lib/lpm \ @TOPDIR@/lib/mbuf \ + @TOPDIR@/lib/memarea \ @TOPDIR@/lib/member \ @TOPDIR@/lib/mempool \ @TOPDIR@/lib/meter \ diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst index 8564883018..e9015d65e3 100644 --- a/doc/guides/prog_guide/index.rst +++ b/doc/guides/prog_guide/index.rst @@ -37,6 +37,7 @@ Programmer's Guide hash_lib toeplitz_hash_lib efd_lib +memarea_lib member_lib lpm_lib lpm6_lib diff --git a/doc/guides/prog_guide/memarea_lib.rst b/doc/guides/prog_guide/memarea_lib.rst new file mode 100644 index 00..156ff35cfd --- /dev/null +++ b/doc/guides/prog_guide/memarea_lib.rst @@ -0,0 +1,37 @@ +.. SPDX-License-Identifier: BSD-3-Clause +Copyright(c) 2023 HiSilicon Limited + +Memarea Library +=== + +Introduction + + +The memarea library provides an allocator of variable-size objects, it is +oriented towards the application layer, providing 'region-based memory +management' function [1]. + +The main features are as follows: + +* The memory region can be initialized from the following memory sources: + + - HEAP: e.g. invoke ``rte_malloc_socket``. + + - LIBC: e.g. invoke posix_memalign. + + - Another memarea: it can be allocated from another memarea. + +* It supports MT-safe as long as it's specified at creation time. + +Library API Overview + + +The ``rte_memarea_create()`` function is used to create a memarea, the function +returns the pointer to the created memarea or ``NULL`` if the creation failed. + +The ``rte_memarea_destroy()`` function is used to destroy a memarea. + +Reference +- + +[1] https://en.wikipedia.org/wiki/Region-based_memory_management diff --git a/doc/guides/rel_notes/release_23_03.rst b/doc/guides/rel_notes/release_23_03.rst index b8c5b68d6c..052603dffd 100644 --- a/doc/guides/rel_notes/release_23_03.rst +++ b/doc/guides/rel_notes/release_23_03.rst @@ -55,6 +55,11 @@ New Features Also, make sure to start the actual text at the margin. === +* **Added memarea library.** + + The memarea library is an allocator of variable-size objects, it is oriented + towards the application layer, providing 'region-based memory management' + function. Removed Items - diff --git a/lib/eal/common/eal_common_log.
[PATCH v12 2/6] test/memarea: support memarea test
This patch supports memarea test of rte_memarea_create() and
rte_memarea_destroy() API.
Signed-off-by: Chengwen Feng
Reviewed-by: Dongdong Liu
---
MAINTAINERS | 1 +
app/test/meson.build| 2 +
app/test/test_memarea.c | 130
3 files changed, 133 insertions(+)
create mode 100644 app/test/test_memarea.c
diff --git a/MAINTAINERS b/MAINTAINERS
index 5991156227..7badd4ad1a 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1579,6 +1579,7 @@ Memarea - EXPERIMENTAL
M: Chengwen Feng
F: lib/memarea
F: doc/guides/prog_guide/memarea_lib.rst
+F: app/test/test_memarea*
Membership - EXPERIMENTAL
M: Yipeng Wang
diff --git a/app/test/meson.build b/app/test/meson.build
index f34d19e3c3..fde0155cf9 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -84,6 +84,7 @@ test_sources = files(
'test_malloc.c',
'test_malloc_perf.c',
'test_mbuf.c',
+'test_memarea.c',
'test_member.c',
'test_member_perf.c',
'test_memcpy.c',
@@ -200,6 +201,7 @@ fast_tests = [
['malloc_autotest', false, true],
['mbuf_autotest', false, true],
['mcslock_autotest', false, true],
+['memarea_autotest', true, true],
['memcpy_autotest', true, true],
['memory_autotest', false, true],
['mempool_autotest', false, true],
diff --git a/app/test/test_memarea.c b/app/test/test_memarea.c
new file mode 100644
index 00..c9de779a1d
--- /dev/null
+++ b/app/test/test_memarea.c
@@ -0,0 +1,130 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 HiSilicon Limited
+ */
+
+#include
+#include
+
+#include "test.h"
+
+#include
+#include
+
+#define MEMAREA_TEST_DEFAULT_SIZE 0x1000
+
+#define MEMAREA_TEST_API_RUN(test_func) \
+ do { \
+ int ret = test_func(); \
+ if (ret < 0) { \
+ printf("%s Failed\n", #test_func); \
+ fails++; \
+ } else { \
+ printf("%s Passed\n", #test_func); \
+ } \
+ } while (0)
+
+static int fails;
+
+static void
+test_memarea_prepare(void)
+{
+ fails = 0;
+}
+
+static int
+test_memarea_retcode(void)
+{
+ return fails > 0 ? -1 : 0;
+}
+
+static void
+test_memarea_init_param(struct rte_memarea_param *init)
+{
+ memset(init, 0, sizeof(struct rte_memarea_param));
+ sprintf(init->name, "%s", "test-memarea");
+ init->source = RTE_MEMAREA_SOURCE_LIBC;
+ init->total_sz = MEMAREA_TEST_DEFAULT_SIZE;
+ init->mt_safe = 1;
+}
+
+static int
+test_memarea_create_bad_param(void)
+{
+ struct rte_memarea_param init;
+ struct rte_memarea *ma;
+
+ /* test for NULL */
+ ma = rte_memarea_create(NULL);
+ RTE_TEST_ASSERT(ma == NULL, "Expected NULL");
+
+ /* test for invalid name */
+ memset(&init, 0, sizeof(init));
+ ma = rte_memarea_create(&init);
+ RTE_TEST_ASSERT(ma == NULL, "Expected NULL");
+ memset(&init.name, 1, sizeof(init.name));
+ ma = rte_memarea_create(&init);
+ RTE_TEST_ASSERT(ma == NULL, "Expected NULL");
+
+ /* test for invalid source */
+ test_memarea_init_param(&init);
+ init.source = RTE_MEMAREA_SOURCE_MEMAREA + 1;
+ ma = rte_memarea_create(&init);
+ RTE_TEST_ASSERT(ma == NULL, "Expected NULL");
+
+ /* test for total_sz */
+ test_memarea_init_param(&init);
+ init.total_sz = 0;
+ ma = rte_memarea_create(&init);
+ RTE_TEST_ASSERT(ma == NULL, "Expected NULL");
+
+ /* test for memarea NULL */
+ test_memarea_init_param(&init);
+ init.source = RTE_MEMAREA_SOURCE_MEMAREA;
+ ma = rte_memarea_create(&init);
+ RTE_TEST_ASSERT(ma == NULL, "Expected NULL");
+
+ /* test for algorithm invalid */
+ test_memarea_init_param(&init);
+ init.alg = RTE_MEMAREA_ALGORITHM_NEXTFIT + 1;
+ ma = rte_memarea_create(&init);
+ RTE_TEST_ASSERT(ma == NULL, "Expected NULL");
+
+ return 0;
+}
+
+static int
+test_memarea_create_destroy(void)
+{
+ struct rte_memarea_param init;
+ struct rte_memarea *ma;
+
+ /* test for create with HEAP */
+ test_memarea_init_param(&init);
+ init.source = RTE_MEMAREA_SOURCE_HEAP;
+ init.numa_socket = SOCKET_ID_ANY;
+ ma = rte_memarea_create(&init);
+ RTE_TEST_ASSERT(ma != NULL, "Expected Non-NULL");
+ rte_memarea_destroy(ma);
+
+ /* test for create with LIBC */
+ test_memarea_init_param(&init);
+ init.source = RTE_MEMAREA_SOURCE_LIBC;
+ ma = rte_memarea_create(&init);
+ RTE_TEST_ASSERT(ma != NULL, "Expected Non-NULL");
+ rte_memarea_destroy(ma);
+
+ return 0;
+}
+
+static int
+test_memarea(void)
+{
+ test_memarea_prepare();
+
+ MEMAREA_TEST_API_RUN(test_memarea_create_bad_param);
+ MEMAREA_TEST_API_RUN(test_memarea_create_destroy);
+
+ return test_me
[PATCH v12 5/6] memarea: support dump API
This patch supports rte_memarea_dump() API which could be used for
debug.
Signed-off-by: Chengwen Feng
Reviewed-by: Dongdong Liu
---
doc/guides/prog_guide/memarea_lib.rst | 3 +
lib/memarea/rte_memarea.c | 116 ++
lib/memarea/rte_memarea.h | 21 +
lib/memarea/version.map | 1 +
4 files changed, 141 insertions(+)
diff --git a/doc/guides/prog_guide/memarea_lib.rst
b/doc/guides/prog_guide/memarea_lib.rst
index 01187f7ccb..2b5326e717 100644
--- a/doc/guides/prog_guide/memarea_lib.rst
+++ b/doc/guides/prog_guide/memarea_lib.rst
@@ -37,6 +37,9 @@ the memarea.
The ``rte_memarea_free()`` function is used to free one memory object which
allocated by ``rte_memarea_alloc()``.
++The ``rte_memarea_dump()`` function is used to dump the internal information
++of a memarea.
+
Reference
-
diff --git a/lib/memarea/rte_memarea.c b/lib/memarea/rte_memarea.c
index 76f8083d96..de473abe1d 100644
--- a/lib/memarea/rte_memarea.c
+++ b/lib/memarea/rte_memarea.c
@@ -302,3 +302,119 @@ rte_memarea_free(struct rte_memarea *ma, void *ptr)
}
memarea_unlock(ma);
}
+
+static const char *
+memarea_source_name(enum rte_memarea_source source)
+{
+ if (source == RTE_MEMAREA_SOURCE_HEAP)
+ return "heap";
+ else if (source == RTE_MEMAREA_SOURCE_LIBC)
+ return "libc";
+ else if (source == RTE_MEMAREA_SOURCE_MEMAREA)
+ return "memarea";
+ else
+ return "unknown";
+}
+
+static const char *
+memarea_alg_name(enum rte_memarea_algorithm alg)
+{
+ if (alg == RTE_MEMAREA_ALGORITHM_NEXTFIT)
+ return "nextfit";
+ else
+ return "unknown";
+}
+
+static uint32_t
+memarea_obj_list_num(struct rte_memarea *ma)
+{
+ struct memarea_obj *obj;
+ uint32_t num = 0;
+
+ TAILQ_FOREACH(obj, &ma->obj_list, obj_node) {
+ if (obj->magic != MEMAREA_OBJECT_ALLOCATED_MAGIC &&
+ obj->magic != MEMAREA_OBJECT_FREE_MAGIC)
+ break;
+ num++;
+ }
+
+ return num;
+}
+
+static uint32_t
+memarea_free_list_num(struct rte_memarea *ma)
+{
+ struct memarea_obj *obj;
+ uint32_t num = 0;
+
+ TAILQ_FOREACH(obj, &ma->free_list, free_node) {
+ if (obj->magic != MEMAREA_OBJECT_ALLOCATED_MAGIC &&
+ obj->magic != MEMAREA_OBJECT_FREE_MAGIC)
+ break;
+ num++;
+ }
+
+ return num;
+}
+
+static size_t
+memarea_free_list_size(struct rte_memarea *ma)
+{
+ struct memarea_obj *obj;
+ size_t sz = 0;
+
+ TAILQ_FOREACH(obj, &ma->free_list, free_node) {
+ if (obj->magic != MEMAREA_OBJECT_ALLOCATED_MAGIC &&
+ obj->magic != MEMAREA_OBJECT_FREE_MAGIC)
+ break;
+ sz += obj->size;
+ }
+
+ return sz;
+}
+
+static void
+memarea_dump_all_objects(struct rte_memarea *ma, FILE *f)
+{
+ struct memarea_obj *obj;
+
+ fprintf(f, " objects:\n");
+ TAILQ_FOREACH(obj, &ma->obj_list, obj_node) {
+ if (obj->magic != MEMAREA_OBJECT_ALLOCATED_MAGIC &&
+ obj->magic != MEMAREA_OBJECT_FREE_MAGIC) {
+ fprintf(f, "magic: 0x%x check fail! stop dump!\n",
obj->magic);
+ break;
+ }
+ fprintf(f, "size: 0x%zx alloc-size: 0x%zx %s\n",
+ obj->size, obj->alloc_size,
+ obj->magic == MEMAREA_OBJECT_ALLOCATED_MAGIC ?
"allocated" : "free");
+ }
+}
+
+int
+rte_memarea_dump(struct rte_memarea *ma, FILE *f, bool dump_all)
+{
+ if (ma == NULL || f == NULL)
+ return -EINVAL;
+
+ memarea_lock(ma);
+ fprintf(f, "memarea name: %s\n", ma->init.name);
+ fprintf(f, " source: %s\n", memarea_source_name(ma->init.source));
+ if (ma->init.source == RTE_MEMAREA_SOURCE_HEAP)
+ fprintf(f, " heap-numa-socket: %d\n", ma->init.numa_socket);
+ else if (ma->init.source == RTE_MEMAREA_SOURCE_MEMAREA)
+ fprintf(f, " source-memarea: %s\n",
ma->init.src_memarea->init.name);
+ fprintf(f, " algorithm: %s\n", memarea_alg_name(ma->init.alg));
+ fprintf(f, " total-size: 0x%zx\n", ma->init.total_sz);
+ fprintf(f, " mt-safe: %s\n", ma->init.mt_safe ? "yes" : "no");
+ fprintf(f, " total-objects: %u\n", memarea_obj_list_num(ma));
+ fprintf(f, " total-free-objects: %u\n", memarea_free_list_num(ma));
+ fprintf(f, " total-free-objects-size: 0x%zx\n",
memarea_free_list_size(ma));
+ fprintf(f, " alloc-fails: %" PRIu64 "\n", ma->alloc_fails);
+ fprintf(f, " magic_check_fails: %" PRIu64 "\n", ma->magic_check_fails);
+ if (dump_all)
+ memarea_dump_all_objects(ma, f);
+ memarea_unlock(ma);
+
+ return 0;
+}
diff --git a/lib/memarea/r
Re: [PATCH 4/7] mlx5: replace zero length array with flex array
On Fri, 13 Jan 2023 13:52:02 -0800 Stephen Hemminger wrote: > Zero length arrays are GNU extension. Replace with > standard flex array. > > Signed-off-by: Stephen Hemminger > --- Since Mlx driver enables pedantic checking, this causes build failure on some versions of gcc. In file included from ../drivers/net/mlx5/mlx5.c:40: ../drivers/net/mlx5/mlx5_tx.h:187:23: error: invalid use of structure with flexible array member [-Werror=pedantic] 187 | struct mlx5_txq_data txq; /* Data path structure. */ | ^~~ Understand that the driver wants to enable more checking since Nvidia does good job of keeping the code up to date. But having driver specific compiler flags like this creates more unnecessary complexity and doesn't improve the resulting code.
Re: [RFC] Remove Kernel Network Interface (KNI)
On Fri, Jan 13, 2023 at 03:25:30PM -0800, Stephen Hemminger wrote:
> On Fri, 13 Jan 2023 19:34:24 +0100
> Thomas Monjalon wrote:
>
> > 13/01/2023 18:13, Stephen Hemminger:
> > > On Fri, 13 Jan 2023 09:12:16 +0100
> > > Thomas Monjalon wrote:
> > >
> > > > 13/01/2023 06:03, Stephen Hemminger:
> > > > > The Linux special network driver for kernel networking has been
> > > > > a long term problem for DPDK. The performance benefits of KNI
> > > > > are available via virtio-user and XDP, and the simpler kernel
> > > > > interface via TAP is also available.
> > > > >
> > > > > This driver has required lots of effort to keep up with the
> > > > > kernel API changes. And the overall architecture of the driver
> > > > > is fundamentally insecure and has unfixable locking and data
> > > > > race problems. No developer has been willing to do extensive
> > > > > tests or be the maintainer.
> > > > >
> > > > > In short, the time has come to do some early spring cleaning
> > > > > and remove KNI from DPDK 23.03.
> > > >
> > > > In doc/guides/rel_notes/deprecation.rst it is announced
> > > > to be removed in 23.11. Let's keep this RFC for later :)
> > > >
> > > >
> > >
> > > For 23.03 could we add a deprecation log message when library is
> > > used and when kernel module is loaded.
> >
> > We already have a message in the lib:
> >
> > int
> > rte_kni_init(unsigned int max_kni_ifaces __rte_unused)
> > {
> > RTE_LOG(WARNING, KNI, "WARNING: KNI is deprecated and will be removed
> > in DPDK 23.11\n");
> >
> > It is a good idea to add a message in the kernel module loading.
> >
> >
> >
> >
>
> No matter how much we tell users, guarantee someone will still miss it and
> complain :-)
one of the techniques we use is to remove the headers in advance of
removing the ABI. the warning becomes a lot more clear when you can't
compile anymore but your existing binaries continue to run.
[PATCH] doc: add capability to access physical addresses
CAP_DAC_OVERRIDE capability is required to access /proc/self/pagemap,
but it was missing from the Linux guide, causing issues for users.
Fixes: 979bb5d493fb ("doc: add more instructions for running as non-root")
Cc: sta...@dpdk.org
Signed-off-by: Dmitry Kozlyuk
Reported-by: Boris Ouretskey
Reported-by: Isaac Boukris
---
Mail threads:
*
https://inbox.dpdk.org/users/cag4aaq21hn_ogi0-gfujxvhtwu19bk634bt6ue1bpyzjrxf...@mail.gmail.com
*
https://inbox.dpdk.org/users/CAC-fF8ShK_n1dhPK2KwV7nRx7535simb=fn77yrzr1+fv2j...@mail.gmail.com
doc/guides/linux_gsg/enable_func.rst | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/doc/guides/linux_gsg/enable_func.rst
b/doc/guides/linux_gsg/enable_func.rst
index 829084d80e..b0e123cb7e 100644
--- a/doc/guides/linux_gsg/enable_func.rst
+++ b/doc/guides/linux_gsg/enable_func.rst
@@ -55,12 +55,12 @@ Refer to the `documentation
Re: [PATCH] doc: add capability to access physical addresses
On Sun, Jan 15, 2023 at 4:27 AM Stephen Hemminger
wrote:
>
> On Sun, 15 Jan 2023 01:58:02 +0300
> Dmitry Kozlyuk wrote:
>
> > CAP_DAC_OVERRIDE capability is required to access /proc/self/pagemap,
> > but it was missing from the Linux guide, causing issues for users.
> >
> > Fixes: 979bb5d493fb ("doc: add more instructions for running as non-root")
> > Cc: sta...@dpdk.org
> >
> > Signed-off-by: Dmitry Kozlyuk
> > Reported-by: Boris Ouretskey
> > Reported-by: Isaac Boukris
>
> DAC_OVERRIDE is like having the master key. It opens all doors
> and if so, running as non-root really doesn't matter that much.
The cap_sys_admin also seems heavy but I guessed it is still better
than full root.
> Ideally, a finer grain permission could be used.
> Recommending this to users seems wrong.
>
> According proc.5 man page.
>
>
>/proc/[pid]/pagemap (since Linux 2.6.25)
> This file shows the mapping of each of the process's
> virtual pages into physical page frames or swap area.
> ...
> Permission to access this file is governed by a ptrace
> access mode PTRACE_MODE_READ_FSCREDS check; see ptrace(2).
>
> Which distro is this? What security module are you using.
> For example, on Debian (kernel 5.17) running as non-root it is possible to
> read pagemap.
I tested on fedora (but also on Rocky8 older kernel): uname -a
Linux localhost.localdomain 6.0.17-200.fc36.x86_64 #1 SMP
PREEMPT_DYNAMIC Wed Jan 4 16:00:03 UTC 2023 x86_64 x86_64 x86_64
GNU/Linux
It can be shown by running the 'pagemap.c' demo code from
https://bugs.centos.org/view.php?id=17176 which hinted me to adding
DAC_OVERRIDE.
The strange thing is that running it without any capabilities allows
you to read the file but give the leading zeros, upon adding
cap_ipc_lock,cap_sys_admin you get a read error and only adding
cap_dac_override lets it run successfully.
RE: [PATCH v12 1/6] memarea: introduce memarea library
> From: Chengwen Feng [mailto:fengcheng...@huawei.com]
> Sent: Saturday, 14 January 2023 12.50
>
> The memarea library is an allocator of variable-size object which based
> on a memory region.
>
> This patch provides rte_memarea_create() and rte_memarea_destroy() API.
>
> Signed-off-by: Chengwen Feng
> Reviewed-by: Dongdong Liu
> ---
[...]
> +struct memarea_obj {
> + TAILQ_ENTRY(memarea_obj) obj_node;
> + TAILQ_ENTRY(memarea_obj) free_node;
> + size_t size;
> + size_t alloc_size;
> + uint32_t magic;
> +};
The magic cookie is for debug purposes only, and should be enclosed by #ifdef
RTE_LIBRTE_MEMAREA_DEBUG, like in the mempool library [1].
[1]:
https://elixir.bootlin.com/dpdk/latest/source/lib/mempool/rte_mempool.h#L154
With that added:
Series-acked-by: Morten Brørup
