Re: [OMPI users] Xgrid and choosing agents...
Hi Vitorio, On 11-Jul-09, at 8:40 PM, Luis Vitorio Cargnini wrote: did you saw that, maybe, just maybe using: xserve01.local slots=8 max-slots=8 xserve02.local slots=8 max-slots=8 xserve03.local slots=8 max-slots=8 xserve04.local slots=8 max-slots=8 it can set the number of process specifically for each node, the "slots" does this setting the configuration of slots per each node, try it with the old conf of Xgrid and also test with your new Xgrid conf. As per Ralph's message, the xgrid launcher ignores --hostfiles... Further, "max_slots=2" is the same as "slots=2 max_slots=2" according to the man page. Xgrid does have a somewhat convoluted, and poorly documented, method of directing jobs to specified machines. Its called Scoreboard and it allows the scheduler to query each machine with a script that gathers info about the machine and compute a "score". Nodes with the highest score get the job. However, how one would implement that using openMPI is unclear to me. Does openMPI have the capability of passing arbitrary arguments to the resource managers? Thanks, Jody Regards. Vitorio. Le 09-07-11 à 18:11, Klymak Jody a écrit : If anyone else is using xgrid, there is a mechanism to limit the processes per machine: sudo defaults write /Library/Preferences/com.apple.xgrid.agent MaximumTaskCount 8 on each of the nodes and then restarting xgrid tells the controller to only send 8 processes to that node. For now that is fine solution for my need. I'll try and figure out how to specify hosts via xgrid and get back to the list... Thanks for everyone's help, Cheers, Jody On 11-Jul-09, at 12:42 PM, Ralph Castain wrote: Looking at the code, you are correct in that the Xgrid launcher is ignoring hostfiles. I'll have to look at it to determine how to correct that situation - I didn't write that code, nor do I have a way to test any changes I might make to it. For now, though, if you add --bynode to your command line, you should get the layout you want. I'm not sure you'll get the rank layout you'll want, though...or if that is important to what you are doing. Ralph On Jul 11, 2009, at 1:18 PM, Klymak Jody wrote: Hi Vitorio, Thanks for getting back to me! My hostfile is xserve01.local max-slots=8 xserve02.local max-slots=8 xserve03.local max-slots=8 xserve04.local max-slots=8 I've now checked, and this seems to work fine just using ssh. i.e. if I turn off the Xgrid queue manager I can submit jobs manually to the appropriate nodes using --hosts. However, I'd really like to use Xgrid as my queue manager as it is already set up (though I'll happily take hints on how to set up other queue managers on an OS X cluster). So you have 4 nodes each one with 2 processors, each processor 4- core - quad-core. So you have capacity for 32 process in parallel. The new Xeon chips designate 2-processes per core, though at a reduced clock rate. This means that Xgrid believes I have 16 processors/node. For large jobs I expect that to be useful, but for my more modest jobs I really only want 8 processes/node. It appears that the default way xgrid assigns the jobs is to fill all 16 slots on one node before moving to the next. OpenMPI doesn't appear to look at the hostfile configuration when using Xgrid, so it makes it hard for me to deprecate this behaviour. Thanks, Jody I think that only using the hostfile is enough is how I use. If you to specify a specific host or a different sequence, the mpirun will obey the host sequence in your hostfile to start the process, also can you put how you configured your host files ? I'm asking this because you should have something like: # This is an example hostfile. Comments begin with # # # The following node is a single processor machine: foo.example.com # The following node is a dual-processor machine: bar.example.com slots=2 # The following node is a quad-processor machine, and we absolutely # want to disallow over-subscribing it: yow.example.com slots=4 max-slots=4 so in your case like mine you should have something like: your.hostname.domain slots=8 max-slots=8 # for each node I hope this will help you. Regards. Vitorio. Le 09-07-11 à 10:56, Klymak Jody a écrit : Hi all, Sorry in advance if these are naive questions - I'm not experienced in running a grid... I'm using openMPI on 4 duo Quad-core Xeon xserves. The 8 cores mimic 16 cores and show up in xgrid as each agent having 16 processors. However, the processing speed goes down as the used processors exceeds 8, so if possible I'd prefer to not have more than 8 processors working on each machine at a time. Unfortunately, if I submit a 16-processor job to xgrid it all goes to "xserve03". Or even worse, it does so if I submit two separate 8-processor jobs. Is there anyway to steer jobs to less-busy agents? I tried making a hostfile and then specifying the host, but I get: /usr/l
Re: [OMPI users] Xgrid and choosing agents...
On Jul 11, 2009, at 11:13 PM, Klymak Jody wrote: Hi Vitorio, On 11-Jul-09, at 8:40 PM, Luis Vitorio Cargnini wrote: did you saw that, maybe, just maybe using: xserve01.local slots=8 max-slots=8 xserve02.local slots=8 max-slots=8 xserve03.local slots=8 max-slots=8 xserve04.local slots=8 max-slots=8 it can set the number of process specifically for each node, the "slots" does this setting the configuration of slots per each node, try it with the old conf of Xgrid and also test with your new Xgrid conf. As per Ralph's message, the xgrid launcher ignores --hostfiles... Further, "max_slots=2" is the same as "slots=2 max_slots=2" according to the man page. Xgrid does have a somewhat convoluted, and poorly documented, method of directing jobs to specified machines. Its called Scoreboard and it allows the scheduler to query each machine with a script that gathers info about the machine and compute a "score". Nodes with the highest score get the job. However, how one would implement that using openMPI is unclear to me. Does openMPI have the capability of passing arbitrary arguments to the resource managers? Assuming that Scoreboard is appropriately licensed (i.e., is not licensed under GPL, but preferably something like FreeBSD), and that it has an accessible API, then we can link against it when in that environment and interact any way we desire - including asking Scoreboard for its recommended list of nodes. Thanks, Jody Regards. Vitorio. Le 09-07-11 à 18:11, Klymak Jody a écrit : If anyone else is using xgrid, there is a mechanism to limit the processes per machine: sudo defaults write /Library/Preferences/com.apple.xgrid.agent MaximumTaskCount 8 on each of the nodes and then restarting xgrid tells the controller to only send 8 processes to that node. For now that is fine solution for my need. I'll try and figure out how to specify hosts via xgrid and get back to the list... Thanks for everyone's help, Cheers, Jody On 11-Jul-09, at 12:42 PM, Ralph Castain wrote: Looking at the code, you are correct in that the Xgrid launcher is ignoring hostfiles. I'll have to look at it to determine how to correct that situation - I didn't write that code, nor do I have a way to test any changes I might make to it. For now, though, if you add --bynode to your command line, you should get the layout you want. I'm not sure you'll get the rank layout you'll want, though...or if that is important to what you are doing. Ralph On Jul 11, 2009, at 1:18 PM, Klymak Jody wrote: Hi Vitorio, Thanks for getting back to me! My hostfile is xserve01.local max-slots=8 xserve02.local max-slots=8 xserve03.local max-slots=8 xserve04.local max-slots=8 I've now checked, and this seems to work fine just using ssh. i.e. if I turn off the Xgrid queue manager I can submit jobs manually to the appropriate nodes using --hosts. However, I'd really like to use Xgrid as my queue manager as it is already set up (though I'll happily take hints on how to set up other queue managers on an OS X cluster). So you have 4 nodes each one with 2 processors, each processor 4-core - quad-core. So you have capacity for 32 process in parallel. The new Xeon chips designate 2-processes per core, though at a reduced clock rate. This means that Xgrid believes I have 16 processors/node. For large jobs I expect that to be useful, but for my more modest jobs I really only want 8 processes/node. It appears that the default way xgrid assigns the jobs is to fill all 16 slots on one node before moving to the next. OpenMPI doesn't appear to look at the hostfile configuration when using Xgrid, so it makes it hard for me to deprecate this behaviour. Thanks, Jody I think that only using the hostfile is enough is how I use. If you to specify a specific host or a different sequence, the mpirun will obey the host sequence in your hostfile to start the process, also can you put how you configured your host files ? I'm asking this because you should have something like: # This is an example hostfile. Comments begin with # # # The following node is a single processor machine: foo.example.com # The following node is a dual-processor machine: bar.example.com slots=2 # The following node is a quad-processor machine, and we absolutely # want to disallow over-subscribing it: yow.example.com slots=4 max-slots=4 so in your case like mine you should have something like: your.hostname.domain slots=8 max-slots=8 # for each node I hope this will help you. Regards. Vitorio. Le 09-07-11 à 10:56, Klymak Jody a écrit : Hi all, Sorry in advance if these are naive questions - I'm not experienced in running a grid... I'm using openMPI on 4 duo Quad-core Xeon xserves. The 8 cores mimic 16 cores and show up in xgrid as each agent having 16 processors. However, the processing speed goes down as the used processors exceeds 8, so if possibl
[OMPI users] Help: HPL Compile Problems
Hello, I use OpenMPI 1.3.2 and HPL-2.0 to measure clusters performance. However I encountered a problem when I compile the HPL. I don't know how to fill up the MPlib variable item because the default value is $(MPdir)/lib/libmpich.a. Obviously I have to choose a file from OpenMPI to replace it. # # -- # - Message Passing library (MPI) -- # -- # MPinc tells the C compiler where to find the Message Passing library # header files, MPlib is defined to be the name of the library to be # used. The variable MPdir is only used for defining MPinc and MPlib. # MPdir= /usr/local/openmpi MPinc= -I$(MPdir)/include MPlib= $(MPdir)/lib/libmpich.a Could anyone tell me how to fix that problem? Thank you very much. Regards, Amy Lee
Re: [OMPI users] Help: HPL Compile Problems
Hi, you can ignore MP... if you set the compiler and linker to mpicc. In my makefile for hpl I have # -- # - MPI directories - library -- # -- # MPinc tells the C compiler where to find the Message Passing library # header files, MPlib is defined to be the name of the library to be # used. The variable MPdir is only used for defining MPinc and MPlib. # MPdir= MPinc= MPlib= # and # -- # - Compilers / linkers - Optimization flags --- # -- # CC = $(HOME)/openmpi-1.3.2/bin/mpicc CCNOOPT = $(HPL_DEFS) CCFLAGS = $(HPL_DEFS) -fomit-frame-pointer -O3 -funroll-loops -W -Wall # LINKER = $(HOME)/openmpi-1.3.2/bin/mpicc LINKFLAGS= $(CCFLAGS) # ARCHIVER = ar ARFLAGS = r RANLIB = echo The mpicc wrapper will take care of including the correct libraries. Regards, Dorian Lee Amy wrote: Hello, I use OpenMPI 1.3.2 and HPL-2.0 to measure clusters performance. However I encountered a problem when I compile the HPL. I don't know how to fill up the MPlib variable item because the default value is $(MPdir)/lib/libmpich.a. Obviously I have to choose a file from OpenMPI to replace it. # # -- # - Message Passing library (MPI) -- # -- # MPinc tells the C compiler where to find the Message Passing library # header files, MPlib is defined to be the name of the library to be # used. The variable MPdir is only used for defining MPinc and MPlib. # MPdir= /usr/local/openmpi MPinc= -I$(MPdir)/include MPlib= $(MPdir)/lib/libmpich.a Could anyone tell me how to fix that problem? Thank you very much. Regards, Amy Lee ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] Help: HPL Compile Problems
On Sun, Jul 12, 2009 at 8:13 PM, Dorian Krause wrote: > Hi, > > you can ignore MP... if you set the compiler and linker to mpicc. In my > makefile for hpl I have > > # -- > # - MPI directories - library -- > # -- > # MPinc tells the C compiler where to find the Message Passing library > # header files, MPlib is defined to be the name of the library to be > # used. The variable MPdir is only used for defining MPinc and MPlib. > # > MPdir = > MPinc = > MPlib = > # > > and > > # -- > # - Compilers / linkers - Optimization flags --- > # -- > # > CC = $(HOME)/openmpi-1.3.2/bin/mpicc > CCNOOPT = $(HPL_DEFS) > CCFLAGS = $(HPL_DEFS) -fomit-frame-pointer -O3 -funroll-loops -W -Wall > # > LINKER = $(HOME)/openmpi-1.3.2/bin/mpicc > LINKFLAGS = $(CCFLAGS) > # > ARCHIVER = ar > ARFLAGS = r > RANLIB = echo > > The mpicc wrapper will take care of including the correct libraries. > > Regards, > Dorian > > > Lee Amy wrote: >> >> Hello, >> >> I use OpenMPI 1.3.2 and HPL-2.0 to measure clusters performance. >> However I encountered a problem when I compile the HPL. >> >> I don't know how to fill up the MPlib variable item because the >> default value is $(MPdir)/lib/libmpich.a. Obviously I have to choose a >> file from OpenMPI to replace it. >> >> # >> # -- >> # - Message Passing library (MPI) -- >> # -- >> # MPinc tells the C compiler where to find the Message Passing library >> # header files, MPlib is defined to be the name of the library to be >> # used. The variable MPdir is only used for defining MPinc and MPlib. >> # >> MPdir = /usr/local/openmpi >> MPinc = -I$(MPdir)/include >> MPlib = $(MPdir)/lib/libmpich.a >> >> Could anyone tell me how to fix that problem? >> >> Thank you very much. >> >> Regards, >> >> Amy Lee >> ___ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users > Thanks. Your solution is great. And I found that if I use libmpi.so HPL could also compile well. Regards, Amy
Re: [OMPI users] Xgrid and choosing agents...
Hi Ralph, On 12-Jul-09, at 4:07 AM, Ralph Castain wrote: Assuming that Scoreboard is appropriately licensed (i.e., is not licensed under GPL, but preferably something like FreeBSD), and that it has an accessible API, then we can link against it when in that environment and interact any way we desire - including asking Scoreboard for its recommended list of nodes. The info I have is all from http://www.macresearch.org/node/4654. To use scoreboard you pass a run-time argument to the xgrid command using "-art" and "-artid". xgrid -job submit -art scorescript.pl -artid score MyJob The script can return any score calculated from any of the machine attributes, i.e. you can send the job to machines with more than 2 Gb of memory, or that are faster than 2GHz. So its a pretty nice feature. I don't think there is any way to manually query the nodes. Unfortunately, I'm not sure if it is possible to do this via the API which does not look to have any hooks into Scorecard: http://developer.apple.com/documentation/Performance/Conceptual/XgridDeveloper/index.html#/ /apple_ref/doc/framework/XgridFoundation_reference Curiously, looking at the API, I further get the impression it has no concept of identifying the nodes. The xgrid controller assigns them, and there does not seem to be a mechanism for the user to choose. So until the API is improved to account for the Scoreboard capability I guess this can't be accessed via openMPI. Thanks a lot for thinking about this. As I said, I have a solution now that works just great, so this is all just for curiousity's sake at this point. Cheers, Jody
Re: [OMPI users] Xgrid and choosing agents...
On Jul 12, 2009, at 6:58 AM, Klymak Jody wrote: Hi Ralph, On 12-Jul-09, at 4:07 AM, Ralph Castain wrote: Assuming that Scoreboard is appropriately licensed (i.e., is not licensed under GPL, but preferably something like FreeBSD), and that it has an accessible API, then we can link against it when in that environment and interact any way we desire - including asking Scoreboard for its recommended list of nodes. The info I have is all from http://www.macresearch.org/node/4654. To use scoreboard you pass a run-time argument to the xgrid command using "-art" and "-artid". xgrid -job submit -art scorescript.pl -artid score MyJob The script can return any score calculated from any of the machine attributes, i.e. you can send the job to machines with more than 2 Gb of memory, or that are faster than 2GHz. So its a pretty nice feature. I don't think there is any way to manually query the nodes. Interesting! I'll look and see if/how we can do the query from inside OMPI. Thanks! Ralph Unfortunately, I'm not sure if it is possible to do this via the API which does not look to have any hooks into Scorecard: http://developer.apple.com/documentation/Performance/Conceptual/XgridDeveloper/index.html#/ /apple_ref/doc/framework/XgridFoundation_reference Curiously, looking at the API, I further get the impression it has no concept of identifying the nodes. The xgrid controller assigns them, and there does not seem to be a mechanism for the user to choose. So until the API is improved to account for the Scoreboard capability I guess this can't be accessed via openMPI. Thanks a lot for thinking about this. As I said, I have a solution now that works just great, so this is all just for curiousity's sake at this point. Cheers, Jody ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] Problems in OpenMPI
Can you give me a further explanation about why results are different when it ran it on mutiprocessors against single processor? Thank you! On Fri, Jul 10, 2009 at 4:20 AM, Ashley Pittman wrote: > On Thu, 2009-07-09 at 23:40 -0500, Yin Feng wrote: >> I am a beginner in MPI. >> >> I ran an example code using OpenMPI and it seems work. >> And then I tried a parallel example in PETSc tutorials folder (ex5). >> >> mpirun -np 4 ex5 >> It can do but the results are not as accurate as just running ex5. >> Is that thing normal? > > Not as accurate or just different? Different is normal and in light of > that accurate is itself a vague concept. > >> After that, send this job to supercomputer which allocates me 4 nodes >> and each node has 8 processors. When I check load on each node, I >> found: > >> Does anyone have any idea about this? > > I'd say it's obvious all 32 processes have been located on the same > node, what was the mpirun command you issued and the contents of the > machinefile you used? > > Running "orte-ps" on the machine where the mpirun command is running > will tell you the hostname where every rank is running or if you want > more information (load, cpu usage etc) you can use padb, the link for > which is in my signature. > > Ashley, > > -- > > Ashley Pittman, Bath, UK. > > Padb - A parallel job inspection tool for cluster computing > http://padb.pittman.org.uk > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users >
