Cristian,

If the message takes some extra time to land into the receiver, then
MPI_Wait will take more time.
or even worse, if the sender is late, the receiver will spend even more
time in MPI_Wait.

First, how do you run 128 tasks on 16 nodes ?
if you do a simple mpirun, then you will use sm or vader btl.
containers can only use the tcp btl, even within the same physical node.
so I encourage you to mpirun --mca tcp,self -np 128 ...
and see if you observe any degradation.

I know very few about containers, but if I remember correctly, you can do
stuff such as cgroup (cpu caping, network bandwidth caping, memory limit)
do you use such things ?
a possible explanation is a container reaches it's limit and is given a
very low priority.

regardless the containers, you end up having 16 tasks sharing the same
interconnect.
I can imagine that an unfair share can lead to this kind of behaviour.

on the network, did you measure zero or few errors ?
few errors take some extra time to be fixed, and if your application is
communication intensive, these delays get propagated and you can end up
with huge performance hit.

Cheers,

Gilles

On Tuesday, July 28, 2015, Cristian RUIZ <cristian.r...@inria.fr> wrote:

>  Hello,
>
> I'm measuring the overhead of using Linux container for HPC applications.
> To do so I was comparing the execution time of NAS parallel benchmarks on
> two infrastructures:
>
> 1) real: 16 real machines
> 2) container: 16 containers distributed over 16 real machines
>
> Each machine used is equipped with two Intel Xeon E5-2630v3 processors
> (with 8 cores each), 128 GB of RAM and a 10 Gigabit Ethernet adapter.
>
> In my results, I found a particular performance degradation for CG.B
> benchmark:
>
>     walltime numprocess      type      ci1      ci2    overhead
> 1   6615085         16    native  6473340  6756830   1.1271473
> 2   6349030         32    native  6315947  6382112   2.2187747
> 3   5811724         64    native  5771509  5851938   0.8983445
> 4   4002865        128    native  3966314  4039416 *180.7472715*
> 5   4077885        256    native  4044667  4111103
>
> *402.8036531 *    walltime numprocess      type      ci1      ci2
> overhead
> 6   6540523         16 container  6458503  6622543   0.0000000
> 7   6208159         32 container  6184888  6231431   0.0000000
> 8   5759514         64 container  5719453  5799575   0.0000000
> 9  11237935        128 container 10762906 11712963   0.0000000
> 10 20503755        256 container 19830425 21177085   0.0000000
>
> (16 MPI processes per machine/container)
>
> When I use containers everything is fine before 128 MPI processes.  I got
> 180% and 400% performance degration with 128  and 256 MPI processes
> respectively. I repeated again the meaures and I had statistically the same
> results. So, I decided to generate a trace of the execution using TAU. I
> discovered that the source of the overhead is the MPI_wait() method that
> sometimes takes around 0.2 seconds and this happens around 20 times which
> adds around 4 seconds to the execution time. The method is called 25992
> times and in avarage takes between 50 and 300 usecs (values obtained with
> profiling).
> This strange behavior was reported in this paper[1] (page 10)  that says:
>
> "We can see two outstanding zones of MPI_Send and MPI_Wait. Such
> operations typically take few microseconds to less than a millisecond. Here
> they take 0.2 seconds"
>
> They attributed that strange behavior to package loss and network
> malfunctioning. In my experiments I measured the number of packets dropped
> and nothing unusual happened.
> I used two versions of OpenMPI 1.6.5 and 1.8.5 and in both versions I got
> the same strange behavior. Any clues of what could be the source of that
> strange behavior? could you please suggest any method to
> debug this problem?
>
>
> Thank you in advance
>
> [1] https://hal.inria.fr/hal-00919507/file/smpi_pmbs13.pdf
>
>
>
>

Reply via email to