PR is up https://github.com/open-mpi/ompi/pull/5193
-Nathan
> On May 24, 2018, at 7:09 AM, Nathan Hjelm <hje...@me.com> wrote:
>
> Ok, thanks for testing that. I will open a PR for master changing the default
> backing location to /dev/shm on linux. Will be PR’d to v3.0.x and v3.1.x.
>
> -Nathan
>
>> On May 24, 2018, at 6:46 AM, Joseph Schuchart <schuch...@hlrs.de> wrote:
>>
>> Thank you all for your input!
>>
>> Nathan: thanks for that hint, this seems to be the culprit: With your patch,
>> I do not observe a difference in the performance between the two memory
>> allocations. I remembered that Open MPI allows to change the shmem allocator
>> on the command line. Using vanilla Open MPI 3.1.0 and increasing the
>> priority of the POSIX shmem implementation using `--mca shmem_posix_priority
>> 100` leads to good performance, too. The reason could be that on the Bull
>> machine /tmp is mounted on a disk partition (SSD, iirc). Maybe there is
>> actual I/O involved that hurts performance if the shm backing file is
>> located on a disk (even though the file is unlinked before the memory is
>> accessed)?
>>
>> Regarding the other hints: I tried using MPI_Win_allocate_shared with the
>> noncontig hint. Using POSIX shmem, I do not observe a difference in
>> performance to the other two options. If using the disk-backed shmem file,
>> performance fluctuations are similar to MPI_Win_allocate.
>>
>> On this machine /proc/sys/kernel/numa_balancing is not available, so I
>> assume that this is not the cause in this case. It's good to know for the
>> future that this might become an issue on other systems.
>>
>> Cheers
>> Joseph
>>
>> On 05/23/2018 02:26 PM, Nathan Hjelm wrote:
>>> Odd. I wonder if it is something affected by your session directory. It
>>> might be worth moving the segment to /dev/shm. I don’t expect it will have
>>> an impact but you could try the following patch:
>>> diff --git a/ompi/mca/osc/sm/osc_sm_component.c
>>> b/ompi/mca/osc/sm/osc_sm_component.c
>>> index f7211cd93c..bfc26b39f2 100644
>>> --- a/ompi/mca/osc/sm/osc_sm_component.c
>>> +++ b/ompi/mca/osc/sm/osc_sm_component.c
>>> @@ -262,8 +262,8 @@ component_select(struct ompi_win_t *win, void **base,
>>> size_t size, int disp_unit
>>> posts_size += OPAL_ALIGN_PAD_AMOUNT(posts_size, 64);
>>> if (0 == ompi_comm_rank (module->comm)) {
>>> char *data_file;
>>> - if (asprintf(&data_file,
>>> "%s"OPAL_PATH_SEP"shared_window_%d.%s",
>>> - ompi_process_info.proc_session_dir,
>>> + if (asprintf(&data_file, "/dev/shm/%d.shared_window_%d.%s",
>>> + ompi_process_info.my_name.jobid,
>>> ompi_comm_get_cid(module->comm),
>>> ompi_process_info.nodename) < 0) {
>>> return OMPI_ERR_OUT_OF_RESOURCE;
>>>> On May 23, 2018, at 6:11 AM, Joseph Schuchart <schuch...@hlrs.de> wrote:
>>>>
>>>> I tested with Open MPI 3.1.0 and Open MPI 3.0.0, both compiled with GCC
>>>> 7.1.0 on the Bull Cluster. I only ran on a single node but haven't tested
>>>> what happens if more than one node is involved.
>>>>
>>>> Joseph
>>>>
>>>> On 05/23/2018 02:04 PM, Nathan Hjelm wrote:
>>>>> What Open MPI version are you using? Does this happen when you run on a
>>>>> single node or multiple nodes?
>>>>> -Nathan
>>>>>> On May 23, 2018, at 4:45 AM, Joseph Schuchart <schuch...@hlrs.de> wrote:
>>>>>>
>>>>>> All,
>>>>>>
>>>>>> We are observing some strange/interesting performance issues in
>>>>>> accessing memory that has been allocated through MPI_Win_allocate. I am
>>>>>> attaching our test case, which allocates memory for 100M integer values
>>>>>> on each process both through malloc and MPI_Win_allocate and writes to
>>>>>> the local ranges sequentially.
>>>>>>
>>>>>> On different systems (incl. SuperMUC and a Bull Cluster), we see that
>>>>>> accessing the memory allocated through MPI is significantly slower than
>>>>>> accessing the malloc'ed memory if multiple processes run on a single
>>>>>> node, increasing the effect with increasing number of processes per
>>>>>> node. As an example, running 24 processes per node with the example
>>>>>> attached we see the operations on the malloc'ed memory to take ~0.4s
>>>>>> while the MPI allocated memory takes up to 10s.
>>>>>>
>>>>>> After some experiments, I think there are two factors involved:
>>>>>>
>>>>>> 1) Initialization: it appears that the first iteration is significantly
>>>>>> slower than any subsequent accesses (1.1s vs 0.4s with 12 processes on a
>>>>>> single socket). Excluding the first iteration from the timing or
>>>>>> memsetting the range leads to comparable performance. I assume that this
>>>>>> is due to page faults that stem from first accessing the mmap'ed memory
>>>>>> that backs the shared memory used in the window. The effect of
>>>>>> presetting the malloc'ed memory seems smaller (0.4s vs 0.6s).
>>>>>>
>>>>>> 2) NUMA effects: Given proper initialization, running on two sockets
>>>>>> still leads to fluctuating performance degradation under the MPI window
>>>>>> memory, which ranges up to 20x (in extreme cases). The performance of
>>>>>> accessing the malloc'ed memory is rather stable. The difference seems to
>>>>>> get smaller (but does not disappear) with increasing number of
>>>>>> repetitions. I am not sure what causes these effects as each process
>>>>>> should first-touch their local memory.
>>>>>>
>>>>>> Are these known issues? Does anyone have any thoughts on my analysis?
>>>>>>
>>>>>> It is problematic for us that replacing local memory allocation with MPI
>>>>>> memory allocation leads to performance degradation as we rely on this
>>>>>> mechanism in our distributed data structures. While we can ensure proper
>>>>>> initialization of the memory to mitigate 1) for performance
>>>>>> measurements, I don't see a way to control the NUMA effects. If there is
>>>>>> one I'd be happy about any hints :)
>>>>>>
>>>>>> I should note that we also tested MPICH-based implementations, which
>>>>>> showed similar effects (as they also mmap their window memory). Not
>>>>>> surprisingly, using MPI_Alloc_mem and attaching that memory to a dynamic
>>>>>> window does not cause these effects while using shared memory windows
>>>>>> does. I ran my experiments using Open MPI 3.1.0 with the following
>>>>>> command lines:
>>>>>>
>>>>>> - 12 cores / 1 socket:
>>>>>> mpirun -n 12 --bind-to socket --map-by ppr:12:socket
>>>>>> - 24 cores / 2 sockets:
>>>>>> mpirun -n 24 --bind-to socket
>>>>>>
>>>>>> and verified the binding using --report-bindings.
>>>>>>
>>>>>> Any help or comment would be much appreciated.
>>>>>>
>>>>>> Cheers
>>>>>> Joseph
>>>>>>
>>>>>> --
>>>>>> Dipl.-Inf. Joseph Schuchart
>>>>>> High Performance Computing Center Stuttgart (HLRS)
>>>>>> Nobelstr. 19
>>>>>> D-70569 Stuttgart
>>>>>>
>>>>>> Tel.: +49(0)711-68565890
>>>>>> Fax: +49(0)711-6856832
>>>>>> E-Mail: schuch...@hlrs.de
>>>>>> <mpiwin_vs_malloc.c>_______________________________________________
>>>>>> users mailing list
>>>>>> users@lists.open-mpi.org
>>>>>> https://lists.open-mpi.org/mailman/listinfo/users
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> users@lists.open-mpi.org
>>>>> https://lists.open-mpi.org/mailman/listinfo/users
>>>> _______________________________________________
>>>> users mailing list
>>>> users@lists.open-mpi.org
>>>> https://lists.open-mpi.org/mailman/listinfo/users
>>> _______________________________________________
>>> users mailing list
>>> users@lists.open-mpi.org
>>> https://lists.open-mpi.org/mailman/listinfo/users
>> _______________________________________________
>> users mailing list
>> users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/users
>
> _______________________________________________
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
signature.asc
Description: Message signed with OpenPGP
_______________________________________________ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
