Hello,
I'm measuring the overhead of using Linux container for HPC
applications. To do so I was comparing the execution time of NAS
parallel benchmarks on two infrastructures:
1) real: 16 real machines
2) container: 16 containers distributed over 16 real machines
Each machine used is equipped with two Intel Xeon E5-2630v3 processors
(with 8 cores each), 128 GB of RAM and a 10 Gigabit Ethernet adapter.
In my results, I found a particular performance degradation for CG.B
benchmark:
walltime numprocess type ci1 ci2 overhead
1 6615085 16 native 6473340 6756830 1.1271473
2 6349030 32 native 6315947 6382112 2.2187747
3 5811724 64 native 5771509 5851938 0.8983445
4 4002865 128 native 3966314 4039416 *180.7472715*
5 4077885 256 native 4044667 4111103 *402.8036531
* walltime numprocess type ci1 ci2 overhead
6 6540523 16 container 6458503 6622543 0.0000000
7 6208159 32 container 6184888 6231431 0.0000000
8 5759514 64 container 5719453 5799575 0.0000000
9 11237935 128 container 10762906 11712963 0.0000000
10 20503755 256 container 19830425 21177085 0.0000000
(16 MPI processes per machine/container)
When I use containers everything is fine before 128 MPI processes. I got
180% and 400% performance degration with 128 and 256 MPI processes
respectively. I repeated again the meaures and I had statistically the
same results. So, I decided to generate a trace of the execution using
TAU. I discovered that the source of the overhead is the MPI_wait()
method that sometimes takes around 0.2 seconds and this happens around
20 times which adds around 4 seconds to the execution time. The method
is called 25992 times and in avarage takes between 50 and 300 usecs
(values obtained with profiling).
This strange behavior was reported in this paper[1] (page 10) that says:
"We can see two outstanding zones of MPI_Send and MPI_Wait. Such
operations typically take few microseconds to less than a millisecond.
Here they take 0.2 seconds"
They attributed that strange behavior to package loss and network
malfunctioning. In my experiments I measured the number of packets
dropped and nothing unusual happened.
I used two versions of OpenMPI 1.6.5 and 1.8.5 and in both versions I
got the same strange behavior. Any clues of what could be the source of
that strange behavior? could you please suggest any method to
debug this problem?
Thank you in advance
[1] https://hal.inria.fr/hal-00919507/file/smpi_pmbs13.pdf
