I forgot add some other details... 1) I am setting the affinity of each process to a specific core, explicitly in my application (...with OS system call) 2) I enabled the 'use_eager_rdma' with the corresponding buffer limit at 32 KBytes( large enough to cover all my message sizes) 3) I set the btl mtu be 2048 bytes. My first objective is to figure out if those large jitter on MPI_Test call is from the MPI library / non-optimal setting, or from OS/ related services taking CPU away. Thanks, Sashi Sashi
List-Post: users@lists.open-mpi.org Date: Sat, 18 Dec 2010 07:00:36 -0500 From: Tim Prince <n...@aol.com> Subject: Re: [OMPI users] Call to MPI_Test has large time-jitter To: us...@open-mpi.org Message-ID: <4d0ca264.4040...@aol.com> Content-Type: text/plain; charset="iso-8859-1"; Format="flowed" On 12/17/2010 6:43 PM, Sashi Balasingam wrote: > Hi, > I recently started on an MPI-based, 'real-time', pipelined-processing > application, and the application fails due to large time-jitter in > sending and receiving messages. Here are related info - > 1) Platform: > a) Intel Box: Two Hex-core, Intel Xeon, 2.668 GHz (...total of 12 cores), > b) OS: SUSE Linux Enterprise Server 11 (x86_64) - Kernel \r (\l) > c) MPI Rev: (OpenRTE) 1.4, (...Installed OFED package) > d) HCA: InfiniBand: Mellanox Technologies MT26428 [ConnectX IB QDR, > PCIe 2.0 5GT/s] (rev a0) > 2) Application detail > a) Launching 7 processes, for pipelined processing, where each process > waits for a message (sizes vary between 1 KBytes to 26 KBytes), > then process the data, and outputs a message (sizes vary between 1 > KBytes to 26 KBytes), to next process. > b) MPI transport functions used : "MPI_Isend", MPI_Irecv, MPI_Test. > i) For Receiving messages, I first make an MPI_Irecv call, followed > by a busy-loop on MPI_Test, waiting for message > ii) For Sending message, there is a busy-loop on MPI_Test to ensure > prior buffer was sent, then use MPI_Isend. > c) When the job starts, all these 7 process are put in High priority > mode ( SCHED_FIFO policy, with priority setting of 99). > The Job entails an input data packet stream (and a series of MPI > messages), continually at 40 micro-sec rate, for a few minutes. > > 3) The Problem: > Most calls to MPI_Test (...which is non-blocking) takes a few > micro-sec, but around 10% of the job, it has a large jitter, that vary > from 1 to 100 odd millisec. This causes > some of the application input queues to fill-up and cause a failure. > Any suggestions to look at on the MPI settings or OS config/issues > will be much appreciated. > I didn't see anything there about your -mca affinity settings. Even if the defaults don't choose optimum mapping, it's way better than allowing them to float as you would with multiple independent jobs running. -- Tim Prince -------------- next part -------------- HTML attachment scrubbed and removed ------------------------------ _______________________________________________ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users End of users Digest, Vol 1772, Issue 1 **************************************
