Joseph, the error message suggests that allocating memory with MPI_Win_allocate[_shared] is done by creating a file and then mmap'ing it. how much space do you have in /dev/shm ? (this is a tmpfs e.g. a RAM file system) there is likely quite some space here, so as a workaround, i suggest you use this as the shared-memory backing directory
/* i am afk and do not remember the syntax, ompi_info --all | grep backing is likely to help */ Cheers, Gilles On Thu, Aug 24, 2017 at 10:31 PM, Joseph Schuchart <schuch...@hlrs.de> wrote: > All, > > I have been experimenting with large window allocations recently and have > made some interesting observations that I would like to share. > > The system under test: > - Linux cluster equipped with IB, > - Open MPI 2.1.1, > - 128GB main memory per node > - 6GB /tmp filesystem per node > > My observations: > 1) Running with 1 process on a single node, I can allocate and write to > memory up to ~110 GB through MPI_Allocate, MPI_Win_allocate, and > MPI_Win_allocate_shared. > > 2) If running with 1 process per node on 2 nodes single large allocations > succeed but with the repeating allocate/free cycle in the attached code I > see the application being reproducibly being killed by the OOM at 25GB > allocation with MPI_Win_allocate_shared. When I try to run it under Valgrind > I get an error from MPI_Win_allocate at ~50GB that I cannot make sense of: > > ``` > MPI_Alloc_mem: 53687091200 B > [n131302:11989] *** An error occurred in MPI_Alloc_mem > [n131302:11989] *** reported by process [1567293441,1] > [n131302:11989] *** on communicator MPI_COMM_WORLD > [n131302:11989] *** MPI_ERR_NO_MEM: out of memory > [n131302:11989] *** MPI_ERRORS_ARE_FATAL (processes in this communicator > will now abort, > [n131302:11989] *** and potentially your MPI job) > ``` > > 3) If running with 2 processes on a node, I get the following error from > both MPI_Win_allocate and MPI_Win_allocate_shared: > ``` > -------------------------------------------------------------------------- > It appears as if there is not enough space for > /tmp/openmpi-sessions-31390@n131702_0/23041/1/0/shared_window_4.n131702 (the > shared-memory backing > file). It is likely that your MPI job will now either abort or experience > performance degradation. > > Local host: n131702 > Space Requested: 6710890760 B > Space Available: 6433673216 B > ``` > This seems to be related to the size limit of /tmp. MPI_Allocate works as > expected, i.e., I can allocate ~50GB per process. I understand that I can > set $TMP to a bigger filesystem (such as lustre) but then I am greeted with > a warning on each allocation and performance seems to drop. Is there a way > to fall back to the allocation strategy used in case 2)? > > 4) It is also worth noting the time it takes to allocate the memory: while > the allocations are in the sub-millisecond range for both MPI_Allocate and > MPI_Win_allocate_shared, it takes >24s to allocate 100GB using > MPI_Win_allocate and the time increasing linearly with the allocation size. > > Are these issues known? Maybe there is documentation describing > work-arounds? (esp. for 3) and 4)) > > I am attaching a small benchmark. Please make sure to adjust the > MEM_PER_NODE macro to suit your system before you run it :) I'm happy to > provide additional details if needed. > > Best > Joseph > -- > Dipl.-Inf. Joseph Schuchart > High Performance Computing Center Stuttgart (HLRS) > Nobelstr. 19 > D-70569 Stuttgart > > Tel.: +49(0)711-68565890 > Fax: +49(0)711-6856832 > E-Mail: schuch...@hlrs.de > > _______________________________________________ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users _______________________________________________ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
