Re: [OMPI users] Poor performance on Amazon EC2 with TCP

2016-03-13 Thread Jackson, Gary L.
y, March 8, 2016 at 11:07 PM To: Open MPI Users Subject: Re: [OMPI users] Poor performance on Amazon EC2 with TCP Jackson, one more thing, how did you build openmpi ? if you built from git (and without VPATH), then --enable-debug is automatically set, and this is hurting performance. if not a

Re: [OMPI users] Poor performance on Amazon EC2 with TCP

2016-03-11 Thread George Bosilca
> Gary Jackson > > From: users mailto:users-boun...@open-mpi.org>> > on behalf of Gilles Gouaillardet <mailto:gil...@rist.or.jp>> > Reply-To: Open MPI Users mailto:us...@open-mpi.org>> > Date: Tuesday, March 8, 2016 at 11:07 PM > To: Open MPI Users mailto:us..

Re: [OMPI users] Poor performance on Amazon EC2 with TCP

2016-03-11 Thread Gilles Gouaillardet
7;m using are correct with tracepath. >2. I am using a placement group. > > -- > Gary Jackson > > From: users > on behalf > of Gilles Gouaillardet > > Reply-To: Open MPI Users > > Date: Tuesday, March 8, 2016 at 11:07 PM > To: Open MPI Users > > Su

Re: [OMPI users] Poor performance on Amazon EC2 with TCP

2016-03-10 Thread Jackson, Gary L.
>> on behalf of Gilles Gouaillardet mailto:gil...@rist.or.jp>> Reply-To: Open MPI Users mailto:us...@open-mpi.org>> List-Post: users@lists.open-mpi.org Date: Tuesday, March 8, 2016 at 11:07 PM To: Open MPI Users mailto:us...@open-mpi.org>> Subject: Re: [OMPI users] Poor p

Re: [OMPI users] Poor performance on Amazon EC2 with TCP

2016-03-08 Thread Gilles Gouaillardet
Jackson, one more thing, how did you build openmpi ? if you built from git (and without VPATH), then --enable-debug is automatically set, and this is hurting performance. if not already done, i recommend you download the latest openmpi tarball (1.10.2) and ./configure --with-platform=contrib/

Re: [OMPI users] Poor performance on Amazon EC2 with TCP

2016-03-08 Thread Rayson Ho
If you are using instance types that support SR-IOV (aka. "enhanced networking" in AWS), then turn it on. We saw huge differences when SR-IOV is enabled http://blogs.scalablelogic.com/2013/12/enhanced-networking-in-aws-cloud.html http://blogs.scalablelogic.com/2014/01/enhanced-networking-in-aws-cl

Re: [OMPI users] Poor performance on Amazon EC2 with TCP

2016-03-08 Thread Gilles Gouaillardet
> RX bytes:6647 (6.4 KiB) TX bytes:6647 (6.4 KiB) > > > -- > Gary Jackson > > > > > From: users on behalf of Gilles Gouaillardet > > Reply-To: Open MPI Users > Date: Tuesday, March 8, 2016 at 9:39 AM > To: Open MPI Users > Subject: Re: [OMPI user

Re: [OMPI users] Poor performance on Amazon EC2 with TCP

2016-03-08 Thread Jackson, Gary L.
:6647 (6.4 KiB) TX bytes:6647 (6.4 KiB) -- Gary Jackson From: users on behalf of Gilles Gouaillardet Reply-To: Open MPI Users List-Post: users@lists.open-mpi.org Date: Tuesday, March 8, 2016 at 9:39 AM To: Open MPI Users Subject: Re: [OMPI users] Poor performance on Amazon EC2 with TCP

Re: [OMPI users] Poor performance on Amazon EC2 with TCP

2016-03-08 Thread Gilles Gouaillardet
Jason, how many Ethernet interfaces are there ? if several, can you try again with one only mpirun --mca btl_tcp_if_include eth0 ... Cheers, Gilles On Tuesday, March 8, 2016, Jackson, Gary L. wrote: > > I've built OpenMPI 1.10.1 on Amazon EC2. Using NetPIPE, I'm seeing about > half the perfor

[OMPI users] Poor performance on Amazon EC2 with TCP

2016-03-08 Thread Jackson, Gary L.
I've built OpenMPI 1.10.1 on Amazon EC2. Using NetPIPE, I'm seeing about half the performance for MPI over TCP as I do with raw TCP. Before I start digging in to this more deeply, does anyone know what might cause that? For what it's worth, I see the same issues with MPICH, but I do not see it