i have good news. after updating to a newer kernel on ubuntu server nodes, sm is not a problem anymore for the nehalem CPUs!!! my older kernel, was Linux 2.6.32-22-server #36-Ubuntu SMP Thu Jun 3 20:38:33 UTC 2010 x86_64 GNU/Linux
and i upgraded to Linux agua 2.6.32-24-server #39-Ubuntu SMP Wed Jul 28 06:21:40 UTC 2010 x86_64 GNU/Linux that solved everything. Gus, maybe the problem you had with fedora can be solved in a similar way. we should keep this for the records. regards Cristobal On Wed, Jul 28, 2010 at 6:45 PM, Gus Correa <g...@ldeo.columbia.edu> wrote: > Cristobal Navarro wrote: > >> Gus >> my kernel for all nodes is this one: >> Linux 2.6.32-22-server #36-Ubuntu SMP Thu Jun 3 20:38:33 UTC 2010 x86_64 >> GNU/Linux >> >> > Kernel is not my league. > > However, it would be great if somebody clarified > for good these issues with Nehalem/Westmere, HT, > shared memory and what the kernel is doing, > or how to make the kernel do the right thing. > Maybe Intel could tell. > > > at least for the moment i will use this configuration, at least for >> deveplopment/testing of the parallel programs. >> lag is minimum :) >> >> whenever i get another kernel update, i will test again to check if sm >> works, would be good to know that suddenly another distribution supports >> nehalem sm. >> >> best regards and thanks again >> Cristobal >> ps: guess what are the names of the other 2 nodes lol >> > > Acatenango (I said that before), and Pacaya. > > Maybe: Santa Maria, Santiaguito, Atitlan, Toliman, San Pedro, > Cerro de Oro ... too many volcanoes, and some are multithreaded ... > You need to buy more nodes! > > Gus > > >> >> >> On Wed, Jul 28, 2010 at 5:50 PM, Gus Correa <g...@ldeo.columbia.edu<mailto: >> g...@ldeo.columbia.edu>> wrote: >> >> Hi Cristobal >> >> Please, read my answer (way down the message) below. >> >> Cristobal Navarro wrote: >> >> >> >> On Wed, Jul 28, 2010 at 3:28 PM, Gus Correa >> <g...@ldeo.columbia.edu <mailto:g...@ldeo.columbia.edu> >> <mailto:g...@ldeo.columbia.edu <mailto:g...@ldeo.columbia.edu>>> >> wrote: >> >> Hi Cristobal >> >> Cristobal Navarro wrote: >> >> >> >> On Wed, Jul 28, 2010 at 11:09 AM, Gus Correa >> <g...@ldeo.columbia.edu <mailto:g...@ldeo.columbia.edu> >> <mailto:g...@ldeo.columbia.edu <mailto:g...@ldeo.columbia.edu>> >> <mailto:g...@ldeo.columbia.edu >> <mailto:g...@ldeo.columbia.edu> <mailto:g...@ldeo.columbia.edu >> <mailto:g...@ldeo.columbia.edu>>>> >> wrote: >> >> Hi Cristobal >> >> In case you are not using full path name for >> mpiexec/mpirun, >> what does "which mpirun" say? >> >> >> --> $which mpirun >> /opt/openmpi-1.4.2 >> >> >> Often times this is a source of confusion, old >> versions may >> be first on the PATH. >> >> Gus >> >> >> openMPI version problem is now gone, i can confirm that the >> version is consistent now :), thanks. >> >> >> This is good news. >> >> >> however, i keep getting this kernel crash randomnly when i >> execute with -np higher than 5 >> these are Xeons, with Hyperthreading On, is that a problem?? >> >> >> The problem may be with Hyperthreading, maybe not. >> Which Xeons? >> >> >> --> they are not so old, not so new either >> fcluster@agua:~$ cat /proc/cpuinfo | more >> processor : 0 >> vendor_id : GenuineIntel >> cpu family : 6 >> model : 26 >> model name : Intel(R) Xeon(R) CPU E5520 @ 2.27GHz >> stepping : 5 >> cpu MHz : 1596.000 >> cache size : 8192 KB >> physical id : 0 >> siblings : 8 >> core id : 0 >> cpu cores : 4 >> apicid : 0 >> initial apicid : 0 >> fpu : yes >> fpu_exception : yes >> cpuid level : 11 >> wp : yes >> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca >> cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss h >> t tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts >> rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_ >> cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt >> lahf_lm ida tpr_shadow vnmi flexpriority ept vpid >> bogomips : 4522.21 >> clflush size : 64 >> cache_alignment : 64 >> address sizes : 40 bits physical, 48 bits virtual >> power management: >> ...same for cpu1, 2, 3, ..., 15. >> >> >> AHA! Nehalems! >> >> Here they are E5540, just a different clock speed, I suppose. >> >> >> information on how the cpu is distributed >> fcluster@agua:~$ lstopo >> System(7992MB) >> Socket#0 + L3(8192KB) >> L2(256KB) + L1(32KB) + Core#0 >> P#0 >> P#8 >> L2(256KB) + L1(32KB) + Core#1 >> P#2 >> P#10 >> L2(256KB) + L1(32KB) + Core#2 >> P#4 >> P#12 >> L2(256KB) + L1(32KB) + Core#3 >> P#6 >> P#14 >> Socket#1 + L3(8192KB) >> L2(256KB) + L1(32KB) + Core#0 >> P#1 >> P#9 >> L2(256KB) + L1(32KB) + Core#1 >> P#3 >> P#11 >> L2(256KB) + L1(32KB) + Core#2 >> P#5 >> P#13 >> L2(256KB) + L1(32KB) + Core#3 >> P#7 >> P#15 >> >> >> >> If I remember right, the old hyperthreading on old >> Xeons was >> problematic. >> >> OTOH, about 1-2 months ago I had trouble with OpenMPI on a >> relatively new Xeon Nehalem machine with (the new) >> Hyperthreading >> turned on, >> and Fedora Core 13. >> The machine would hang with the OpenMPI connectivity example. >> I reported this to the list, you may find in the archives. >> >> >> --i foudn the archives recently about an hour ago, was not sure >> if it was the same problem but i removed HT for testing with >> setting the online flag to 0 on the extra cpus showed with >> lstopo, unfortenately i also crashes, so HT may not be the problem. >> >> >> It didn't fix the problem in our Nehalem machine here either, >> although it was FC13, and I don't know what OS and kernel you're using. >> >> >> Apparently other people got everything (OpenMPI with HT on >> Nehalem) >> working in more stable distributions (CentOS, RHEL, etc). >> >> That problem was likely to be in the FC13 kernel, >> because even turning off HT I still had the machine hanging. >> Nothing worked with shared memory turned on, >> so I had to switch OpenMPI to use tcp instead, >> which is kind of ridiculous in a standalone machine. >> >> >> --> very interesting, sm can be the problem >> >> >> >> im trying to locate the kernel error on logs, but after >> rebooting a crash, the error is not in the kern.log (neither >> kern.log.1). >> all i remember is that it starts with "Kernel BUG..." >> and somepart it mentions a certain CPU X, where that cpu >> can be >> any from 0 to 15 (im testing only in main node). Someone >> knows >> where the log of kernel error could be? >> >> >> Have you tried to turn off hyperthreading? >> >> >> --> yes, tried, same crashes. >> In any case, depending on the application, it may not >> help much >> performance to have HT on. >> >> A more radical alternative is to try >> -mca btl tcp,self >> in the mpirun command line. >> That is what worked in the case I mentioned above. >> >> >> wow!, this worked really :), you pointed out the problem, it >> was shared memory. >> >> >> Great news! >> That's exactly the problem we had here. >> Glad that the same solution worked for you. >> >> Over a year ago another fellow reported the same problem on Nehalem, >> on the very early days of Nehalem. >> The thread should be in the archives. >> Somebody back then (Ralph, or Jeff, or other?) >> suggested that turning off "sm" might work. >> So, I take no credit for this. >> >> >> i have 4 nodes, so anyways there will be node comunication, do >> you think i can rely on working with -mca btl tcp,self?? i dont >> mind small lag. >> >> >> Well, this may be it, short from reinstalling the OS. >> >> Some people reported everything works with OpenMPI+HT+sm in CentOS >> and RHEL, see the thread I mentioned in the archives from 1-2 months >> ago. >> I don't administer that machine, and didn't have the time to do OS >> reinstall either. >> So I left it with -mca btl tcp,self, and the user/machine owner >> is happy that he can run his programs right, >> and with a performance that he considers good. >> >> >> i just have one more question, is this a problem of the ubuntu >> server kernel?? from the Nehalem Cpus?? from openMPI (i dont >> think) ?? >> >> >> I don't have any idea. >> It may be a problem with some kernels, not sure. >> Which kernel do you have? >> >> Ours was FC-13, maybe FC-12, I don't remember exactly. >> Currently that machine has kernel 2.6.33.6-147.fc13.x86_64 #1 SMP. >> However, it may have been a slightly older kernel when I installed >> OpenMPI there. >> It may have been 2.6.33.5-124.fc13.x86_64 or 2.6.32.14-127.fc12.x86_64. >> My colleague here updates the machines with yum, >> so it may have gotten a new kernel since then. >> >> Our workhorse machines in the clusters that I take care >> of are AMD Opteron, never had this problem there. >> Maybe the kernels have yet to catch up with Nehalem, >> now Westmere, soon another one. >> >> >> and on what depends that in the future, sm could be possible on >> the same configuration i have?? kernel update?. >> >> >> You may want to try CentOS or RHEL, but I can't guarantee the results. >> Somebody else in the list may have had the direct experience, >> and may speak out. >> >> It may be worth the effort anyway. >> After all, intra-node communication should be >> running on shared memory. >> Having to turn it off is outrageous. >> >> If you try another OS distribution, >> and if it works, please report the results back to the list: >> OS/distro, kernel, OpenMPI version, HT on or off, >> mca btl sm/tcp/self/etc choices, compilers, etc. >> This type of information is a real time saving for everybody. >> >> >> >> Thanks very much Gus, really! >> Cristobal >> >> >> >> My pleasure. >> Glad that there was a solution, even though not the best. >> Enjoy your cluster with vocano-named nodes! >> Have fun with OpenMPI and PETSc! >> >> Gus Correa >> --------------------------------------------------------------------- >> Gustavo Correa >> Lamont-Doherty Earth Observatory - Columbia University >> Palisades, NY, 10964-8000 - USA >> --------------------------------------------------------------------- >> >> >> My $0.02 >> Gus Correa >> >> >> Cristobal Navarro wrote: >> >> >> On Tue, Jul 27, 2010 at 7:29 PM, Gus Correa >> <g...@ldeo.columbia.edu >> <mailto:g...@ldeo.columbia.edu> <mailto:g...@ldeo.columbia.edu >> <mailto:g...@ldeo.columbia.edu>> >> <mailto:g...@ldeo.columbia.edu >> <mailto:g...@ldeo.columbia.edu> <mailto:g...@ldeo.columbia.edu >> <mailto:g...@ldeo.columbia.edu>>> >> <mailto:g...@ldeo.columbia.edu >> <mailto:g...@ldeo.columbia.edu> >> <mailto:g...@ldeo.columbia.edu >> <mailto:g...@ldeo.columbia.edu>> <mailto:g...@ldeo.columbia.edu >> <mailto:g...@ldeo.columbia.edu> >> <mailto:g...@ldeo.columbia.edu >> <mailto:g...@ldeo.columbia.edu>>>>> >> >> wrote: >> >> Hi Cristobal >> >> Does it run only on the head node alone? >> (Fuego? Agua? Acatenango?) >> Try to put only the head node on the hostfile >> and execute >> with mpiexec. >> >> --> i will try only with the head node, and post >> results back >> This may help sort out what is going on. >> Hopefully it will run on the head node. >> >> Also, do you have Infinband connecting the nodes? >> The error messages refer to the openib btl (i.e. >> Infiniband), >> and complains of >> >> >> no we are just using normal network 100MBit/s , >> since i >> am just >> testing yet. >> >> >> "perhaps a missing symbol, or compiled for a >> different >> version of Open MPI?". >> It sounds as a mixup of versions/builds. >> >> >> --> i agree, somewhere there must be the remains >> of the older >> version >> >> Did you configure/build OpenMPI from source, or >> did >> you install >> it with apt-get? >> It may be easier/less confusing to install from >> source. >> If you did, what configure options did you use? >> >> >> -->i installed from source, ./configure >> --prefix=/opt/openmpi-1.4.2 --with-sge --without-xgid >> --disable--static >> >> Also, as for the OpenMPI runtime environment, >> it is not enough to set it on >> the command line, because it will be effective >> only on the >> head node. >> You need to either add them to the PATH and >> LD_LIBRARY_PATH >> on your .bashrc/.cshrc files (assuming these >> files and >> your home >> directory are *also* shared with the nodes via >> NFS), >> or use the --prefix option of mpiexec to point >> to the >> OpenMPI >> main >> directory. >> >> >> yes, all nodes have their PATH and LD_LIBRARY_PATH >> set up >> properly inside the login scripts ( .bashrc in my >> case ) >> >> Needless to say, you need to check and ensure >> that the >> OpenMPI >> directory (and maybe your home directory, and >> your work >> directory) >> is (are) >> really mounted on the nodes. >> >> >> --> yes, doublechecked that they are >> >> I hope this helps, >> >> >> --> thanks really! >> >> Gus Correa >> >> Update: i just reinstalled openMPI, with the same >> parameters, >> and it >> seems that the problem has gone, i couldnt test >> entirely but >> when i >> get back to lab ill confirm. >> >> best regards! Cristobal >> >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org <mailto:us...@open-mpi.org> >> <mailto:us...@open-mpi.org <mailto:us...@open-mpi.org>> >> <mailto:us...@open-mpi.org <mailto:us...@open-mpi.org> >> <mailto:us...@open-mpi.org <mailto:us...@open-mpi.org>>> >> >> >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org <mailto:us...@open-mpi.org> >> <mailto:us...@open-mpi.org <mailto:us...@open-mpi.org>> >> <mailto:us...@open-mpi.org <mailto:us...@open-mpi.org> >> <mailto:us...@open-mpi.org <mailto:us...@open-mpi.org>>> >> >> >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org <mailto:us...@open-mpi.org> >> <mailto:us...@open-mpi.org <mailto:us...@open-mpi.org>> >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org <mailto:us...@open-mpi.org> >> <mailto:us...@open-mpi.org <mailto:us...@open-mpi.org>> >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org <mailto:us...@open-mpi.org> >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org <mailto:us...@open-mpi.org> >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users >
