Hi, All, This may seem like an odd query (or not; perhaps it has been brought up before). My work recently involves HPC usability i.e. making things easier for new users by abstracting away the scheduler. I've been working with DRMAA for interfacing with DRMs and it occurred to me: what would be the advantage to letting the scheduler itself handle farming out MPI processes as individual tasks rather than having a wrapper like mpirun to handle this task via ssh/rsh/etc.?
I thought about MPI2's ability to do dynamic process management and how scheduling environments tend to allocate static pools of resources for parallel tasks. A DRMAA-driven MPI would be able to request that the scheduler launch these tasks as resources become available enabling scheduled MPI jobs to dynamically add and remove processors during execution. Several applications that I have worked with come to mind, where pre-processing and other tasks are non-parallel whereas the various solvers are. Being able to dynamically spawn processes based on where you are in this work-flow could be very useful here. It also occurred to me that commercial application vendors tend to roll-their-own when it comes to integrating their applications with an MPI library. I've seen applications use HP-MPI, MPICH, MPICH2, Intel-MPI, (and thankfully, recently) OpenMPI and then proceed to butcher the execution mechanisms to such an extent that it makes integration with common DRM systems quite a task. With the exception of OpenMPI, none of these libraries provides turn-key compatibility with most of the major DRMs and each require some degree of manual integration and testing for use in a multi-user production environment. I would think that vendors would be falling over themselves to integrate OpenMPI with their applications for this very reason alone. Instead, some opt to develop their own scheduling environments! Don't they have bean counters that sit around and gripe about duplicated work? Then it occurred to me: with the exception of being able to easily launch an MPI job with OpenMPI, the ability to monitor it from within the application is still dependent on the vendor integrating with various DRMs! This is another area where a DRMAA RAS can come in handy. There are nice bindings for monitoring tasks and getting an idea of where you are in execution without having to resort to kludgey shell-script wrappers tailing output files. Anyway, its been a frustrating couple of weeks dealing with several commercial vendors and integrating their applications with our DRM and my mind has been trying to think of a solution that could save all of us a lot of work (though, at the same time, raise job security concerns in such turbulent times ;-/ ). What say you, MPI experts? Many thanks for your thoughts! -Brian
