On 22-Dec-19 3:33 PM, Byonggon Chun wrote:
x-posting to dev mailing list.
Hi all.
I'm Kubernetes contributors and I'm working to make container isolation of
hugepages that allows us to set hugepages limit per container cgroup.
(At this point, limits are set on pod level cgroup even though we asked
hugepages as the container level resource)
I tore down testPMD and some parts of DPDK lib and I got a question after i
found there is no usage of "shmget" in DPDK.
My question is that Should "shmget" not be used to consume hugepages in
DPDK?
And here is following questions:
1) If we don't have to use "shmget", Why? Does it affect performance?
2) If I use "shmget" to get hugepages, should I call "mlock" syscall for it?
For more details, as I know there are three ways to consume hugepages in
kubernetes.
1) shmget with SHM_HUGETLB
2) mmap with hugetlbs filebacking
3) mmap with MAP_ANONYMOUS | MAP_HUGETLB
And I found that testPMD calls mlock syscall when it maps an anonymous
hugepages or external allocated
hugepages.https://github.com/DPDK/dpdk/blob/924e55fb340623f03fdf2ff7fbcfd78819d1db25/app/test-pmd/testpmd.c#L896https://github.com/DPDK/dpdk/blob/924e55fb340623f03fdf2ff7fbcfd78819d1db25/app/test-pmd/testpmd.c#L916
The reason we're not using shmget is not because it's "legacy" or
something else. It's because it doesn't give the guarantees that we want
to have in DPDK. Namely, that the same shared object in memory is mapped
at the same addresses. Last time i checked, shmget doesn't allow to map
things at a specific address - each process will have its own pointer to
shared memory, and accesses to shared memory by pointer are not valid
across process boundaries. This is contrary to DPDK's goals, because we
want to avoid address translation when working with multiple processes
(in that sense, DPDK's multiprocess is basically like having multiple
threads).
Thanks.
On Fri, Dec 20, 2019 at 9:42 PM Byonggon Chun <byonggonc...@gmail.com>
wrote:
shmget is a legacy Unix API and there is no point in using it.
Yeah, I agree with it,
I also prefer to use mmap with hugetlbfs in a DPDK container.
The reason why I started this mail thread is some DPDK users still use
shmget to consume hugepages, and I just wanted to find a good rationale to
convince them to use mmap.
But, at this point, I have only one rationale : shmget is a legacy UINIX
API.
On Fri, Dec 20, 2019 at 6:06 AM Stephen Hemminger <
step...@networkplumber.org> wrote:
On Fri, 20 Dec 2019 01:23:50 +0900
Byonggon Chun <byonggonc...@gmail.com> wrote:
Hi all.
I'm Kubernetes contributors and I'm working to make container isolation
of
hugepages that allows us to set hugepages limit per container cgroup.
(At this point, limits are set on pod level cgroup even though we asked
hugepages as the container level resource)
I tore down testPMD and some parts of DPDK lib and I got a question
after i
found there is no usage of "shmget" in DPDK.
My question is that Should "shmget" not be used to consume hugepages in
DPDK?
And here is following questions:
1) If we don't have to use "shmget", Why? Does it affect performance?
2) If I use "shmget" to get hugepages, should I call "mlock" syscall
for it?
For more details, as I know there are three ways to consume hugepages in
kubernetes.
1) shmget with SHM_HUGETLB
2) mmap with hugetlbs filebacking
3) mmap with MAP_ANONYMOUS | MAP_HUGETLB
And I found that testPMD calls mlock syscall when it maps an anonymous
hugepages or external allocated hugepages.
https://github.com/DPDK/dpdk/blob/924e55fb340623f03fdf2ff7fbcfd78819d1db25/app/test-pmd/testpmd.c#L896
https://github.com/DPDK/dpdk/blob/924e55fb340623f03fdf2ff7fbcfd78819d1db25/app/test-pmd/testpmd.c#L916
Thanks.
shmget is a legacy Unix API and there is no point in using it.
For new applications libhugetlbfs is preferable.
--
Thanks,
Anatoly
