I would appreciate trying to fix the multi-word argument to orte_launch_agent, unless you've already fixed it in a later series in which case I could update. My workaround with the setuid applied to a copied executable doesn't work for other applications that run on our clusters ( I mean their job scripts would also have to be modified similarly).
I think OpenMPI should be able to launch all processes with the same uid and gid even when the user has launched the job while using one of his secondary groups. Whether it should be a change request or bug (unexpected, counter-intuitive), I'll leave that to your judgment. Thank you again for the support From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain Sent: Wednesday, September 14, 2011 5:10 PM To: Open MPI Users Subject: Re: [OMPI users] EXTERNAL: Re: Can you set the gid of the processes created by mpirun? Hi Ed I finally had some time to dig around a little. I believe we could make this work if I fix the multi-word cmd line param issue. Either "newgrp" or "sg" should resolve the problem - both are user-level cmds. Up to you - let me know if you want me to pursue this. Ralph On Sep 14, 2011, at 3:31 PM, Blosch, Edwin L wrote: Thank you - I did pursue this kind of workaround, and it worked, but you'll be happy to know that nothing had to be owned by root. ASIDE ---- Just to remind: The job script is a shell script that invokes mpirun; the job script itself is run with as the correct user, but the group id may be changed to whatever the user requests of the job-scheduling system. I think it may not be uncommon to have jobs that request a specific Unix group, for many reasons, but in our case the group is an input for the scheduler's prioritization policy. Outcome: rank 0 runs user:group2 as the user requested, but the launched child processes run user:group1 where group 1 is the user's primary group. The peculiarity of this application is that each of the processes writes a file to disk, so the resulting group ownership of rank 0 files is group2, but the group ownership of all other ranks' files is group1. That was the original problem I'm trying to work around. --- END ASIDE Fortunately for me, there is another peculiarity of this application -- the executable gets copied out to /tmp (local space) on each of the hosts to be used in the job. We found this helped prevent some crashes during test phases where an executable gets overwritten while in use. Definitely a special behavior. But as a result of this peculiarity, the mpirun command ends up launching the copied executable, and I took advantage of that. I had the job script do chown user:group2 on the copied executables and then chmod 6711, and then I observed that the child processes ran as user:group2, same as the rank 0 process, so the files they created had the desired group ownership. I will explore Reuti's guidance as well. Thank you From: users-boun...@open-mpi.org<mailto:users-boun...@open-mpi.org> [mailto:users-boun...@open-mpi.org] On Behalf Of Randall Svancara Sent: Wednesday, September 14, 2011 3:07 PM To: Open MPI Users Subject: Re: [OMPI users] EXTERNAL: Re: Can you set the gid of the processes created by mpirun? You could set the setuid bit on the application and chown it to root?? It is about as secure as anything else that has been described thus far. As a system admin, I cringe at the thought of anything that would allow something to run as someone else, so there would have to be a pretty good justification for such unique use case as yours. Randall On Wed, Sep 14, 2011 at 12:56 PM, Reuti <re...@staff.uni-marburg.de<mailto:re...@staff.uni-marburg.de>> wrote: Am 14.09.2011 um 19:02 schrieb Blosch, Edwin L: > Thanks for trying. > > Do you feel that this is an impossible request without the assistance of some > process running as root, for example, as Reuti mentioned, the daemons of a > job scheduler? Or are you saying it will just not be as straightforward as > calling setgid as you had hoped? > > Also, do you think there is a way I could make use of the sg command below? > Perhaps there is a way to have the rsh/ssh launcher start the application > processes with a command like 'sg <group> <executable name>'? What about a half-tight integration (or call it: classic tight integration), i.e. no recompilation necessary? - setup your mpiexec call in the jobscript to use a plain rsh for the remote startup (no path given): -mca plm_rsh_agent rsh - the PE of SGE needs the argument -catch_rsh in start_proc_args and the supplied script in $SGE_ROOT/mpi/startmpi.sh (SGE will create a symbolic link in $TMPDIR therein [which will be called first this way] to the rsh-wrapper in $SGE_ROOT/mpi [pitfall: some applications need a -V to be added in the lines woth "qrsh", i.e. "qrsh -inherit -V ..." to send all environment variables to the slaves]) - what is your setting of qrsh_daemon/qrsh_command in `qconf -sconf`? This will then be used finally to reach the node and should be builtin or point to the SGE supplied rsh/rshd (no rshd necessary to install, no rshd is running all the time, no rshd will be started by xinet.d or alike) - like you do already: switch off the built-in SGE starter in your mpiexec call: -mca plm_rsh_disable_qrsh 1 -- Reuti PS: To avoid misunderstandings: you could also set "-mca plm_rsh_agent foobar" and in $SGE_ROOT/mpi/startmpi.sh you change it to create a symbolic link called "foobar " in $TMPDIR. It's just a name at this stage of startup. > Ed > > > NAME > sg - execute command as different group ID > > SYNOPSIS > sg [-] [group [-c ] command] > > DESCRIPTION > The sg command works similar to newgrp but accepts a command. The > command will be executed with the /bin/sh shell. With most shells you > may run sg from, you need to enclose multi-word commands in quotes. > Another difference between newgrp and sg is that some shells treat > newgrp specially, replacing themselves with a new instance of a shell > that newgrp creates. This doesn't happen with sg, so upon exit from a > sg command you are returned to your previous group ID. > > > > > -----Original Message----- > From: users-boun...@open-mpi.org<mailto:users-boun...@open-mpi.org> > [mailto:users-boun...@open-mpi.org<mailto:users-boun...@open-mpi.org>] On > Behalf Of Ralph Castain > Sent: Wednesday, September 14, 2011 11:33 AM > To: Open MPI Users > Subject: Re: [OMPI users] EXTERNAL: Re: Can you set the gid of the processes > created by mpirun? > > > On Sep 14, 2011, at 9:39 AM, Blosch, Edwin L wrote: > >> Thanks, Ralph, >> >> I get the failure messages, unfortunately: >> >> setgid FAILED >> setgid FAILED >> setgid FAILED >> >> I actually had attempted to call setgid from within the application >> previously, which looks similar to what you've done, but it failed. That was >> when I initiated the post to the mailing list. My conclusion, a guess >> really, was that Linux would not let me setgid from within my program >> because I was not root. > > I was afraid of that - the documentation seemed to indicate that would be the > case, but I figured it was worth a quick try. Sorry I can't be of help. > > >> >> >> -----Original Message----- >> From: users-boun...@open-mpi.org<mailto:users-boun...@open-mpi.org> >> [mailto:users-boun...@open-mpi.org<mailto:users-boun...@open-mpi.org>] On >> Behalf Of Ralph Castain >> Sent: Wednesday, September 14, 2011 8:15 AM >> To: Open MPI Users >> Subject: Re: [OMPI users] EXTERNAL: Re: Can you set the gid of the processes >> created by mpirun? >> >> The attached should set the gid of the remote daemons (and their children) >> to the gid of mpirun. No cmd line option or anything is required - it will >> just always do it. >> >> Would you mind giving it a try? >> >> Please let me know if/how it works. >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org<mailto:us...@open-mpi.org> >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > _______________________________________________ > users mailing list > us...@open-mpi.org<mailto:us...@open-mpi.org> > http://www.open-mpi.org/mailman/listinfo.cgi/users > _______________________________________________ > users mailing list > us...@open-mpi.org<mailto:us...@open-mpi.org> > http://www.open-mpi.org/mailman/listinfo.cgi/users _______________________________________________ users mailing list us...@open-mpi.org<mailto:us...@open-mpi.org> http://www.open-mpi.org/mailman/listinfo.cgi/users -- Randall Svancara http://knowyourlinux.com/ _______________________________________________ users mailing list us...@open-mpi.org<mailto:us...@open-mpi.org> http://www.open-mpi.org/mailman/listinfo.cgi/users
