On 18-Oct-19 11:54 AM, Rajesh Ravi wrote:
+ Srinath
Thanks Anatoly for reviewing this. Please find my reply inline below:
[Anatoly] First of all, what is "iso-cmem"? It doesn't seem to have any
defined
meaning nor any relation to any existing functionality, and it's not
explained anywhere what is "isolated cmem".
[Rajesh] I 'll correct commit message to include clearer explanation.
_
_
_Context & purpose_
We 're using this to improve SPDK performance. When NVMe completion
queues are allocated from a certain memory range,
out of order competions completions are enabled with our PCIe and it
improves performance.
_Usage_
spdk_env_init()-->(calls rte_eal_init(), and then calls
)spdk_env_dpdk_post_init() [file: spdk/lib/env_dpdk/init.c]
--> spdk_mem_map_init()
[lib/env_dpdk/memory.c]-->map_cmem_virtual_area() [this is our custom
function]
In map_cmem_virtual_area(): we 're mmaping anonymous & private region
and then creating iova table which makes iova addresses which fall in
custom memory region.
Then we call rte_malloc_heap_memory_add() and then allocate NVMe
completion queues from this heap.
[Anatoly] More importantly, why is this necessary? Type1 map only bypasses
external segments when adding memory at startup - it doesn't stop you
from calling rte_vfio_dma_map() to map the memory with VFIO when you
create the segment.
[Rajesh] Please correct me if I'm wrong or missing some thing here.
A) We wanted to create a heap from which we can allocate dynamically
B) I see that type1_map()
[file:dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c] getting called once
rte_malloc_heap_memory_add()() was invoked.
SPDK uses type1 dma map [spdk/lib/env_dpdk/memory.c]
vfio_type1_dma_map() is registered for .dma_map_func member
[dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c]
So type1_map is called. Not sure whether we can change this.
I'm not sure i'm following. You mean you *don't* want this mapping to be
performed?
--
Thanks,
Anatoly
