Hi ALL,
I tried to run trivial test between two nodes, it seems to be running without any memory issue ,but it gives some error(path mismatch like its taking openmpi 1.3.2 instead of 1.3.3) while fetching/executing some test binaries i.e. cxx_ring, c_ring etc. I am using openmpi-1.3.3.path is /usr/mpi/gcc/openmpi-1.3.3,installed through OFED daily build. Output is attached. Please guide me on this. Regards, Vishal
Debug is 1, Verbose is 1 *** MTT: ../client/mtt --debug - *** Running on mughal Chdir ../client Chdir /root/mtt-svn/samples Copying: stdin to /tmp/DAeod33kUB.ini Expanding include_file(s) parameters in /tmp/DAeod33kUB.ini Reading ini file: stdin Validating INI inifile: /tmp/QWi6beuXYK.ini FilterINI: Final list of sections: [mtt] [mpi details: open mpi] [mpi get: my installation] [mpi install: my installation] [reporter: text file backup] [test get: trivial] [test build: trivial] [test run: trivial] Value got: Config::IniFiles=HASH(0x1aab97f0) MTT scratch Value returning: scratch: . scratch resolved: /root/mtt-svn/samples Chdir /root/mtt-svn/samples Value got: Config::IniFiles=HASH(0x1aab97f0) MTT fast_scratch Value returning: fast_scratch: /root/mtt-svn/samples fast_scratch resolved: /root/mtt-svn/samples *** Main scratch tree: /root/mtt-svn/samples *** Fast scratch tree: /root/mtt-svn/samples Value got: Config::IniFiles=HASH(0x1aab97f0) MTT force Value returning: 1 Making dir: /root/mtt-svn/samples/sources (cwd: /root/mtt-svn/samples) Making dir: /root/mtt-svn/samples/installs (cwd: /root/mtt-svn/samples) Making dir: /root/mtt-svn/samples/test_runs (cwd: /root/mtt-svn/samples) Value got: Config::IniFiles=HASH(0x1aab97f0) MTT funclet_files Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT hostfile Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT hostlist Value returning: 102.77.77.66 102.77.77.68 >> Got default hostlist: 102.77.77.66,102.77.77.68, max_np: 2 Value got: Config::IniFiles=HASH(0x1aab97f0) MTT max_np Value returning: 8 Value got: Config::IniFiles=HASH(0x1aab97f0) MTT textwrap Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT drain_timeout Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT trim_save_successful Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT trim_save_failed Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT trial Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT http_proxy Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT https_proxy Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT ftp_proxy Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT terminate_files Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT pause_files Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT min_disk_free Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT min_disk_free_wait Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT delete_fast_scratch Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT save_fast_scratch_files Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT docommand_timeout_notify_file Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT docommand_timeout_notify_email Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT docommand_timeout_notify_timeout Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT docommand_timeout_backtrace_program Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT before_all_exec Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT before_all_exec_timeout Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT before_all_exec_pass Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT before_each_exec Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT before_each_exec_timeout Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT before_each_exec_pass Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT after_all_exec Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT after_all_exec_timeout Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT after_all_exec_pass Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT after_each_exec Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT after_each_exec_timeout Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) MTT after_each_exec_pass Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mtt log_file logfile Value returning: *** Reporter initializing Value got: Config::IniFiles=HASH(0x1ab04720) reporter: text file backup module Value returning: TextFile >> Initializing reporter module: TextFile Evaluating: require MTT::Reporter::TextFile Evaluating: $ret = &MTT::Reporter::TextFile::Init(@args) Value got: Config::IniFiles=HASH(0x1ab04720) reporter: text file backup textfile_summary_header Value returning: hostname: mughal uname: Linux mughal 2.6.18-128.el5 #1 SMP Wed Jan 21 10:41:14 EST 2009 x86_64 x86_64 x86_64 GNU/Linux who am i: Value got: Config::IniFiles=HASH(0x1ab04720) reporter: text file backup textfile_summary_footer Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) reporter: text file backup textfile_detail_header Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) reporter: text file backup textfile_detail_footer Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) reporter: text file backup textfile_textwrap Value returning: 78 Value got: Config::IniFiles=HASH(0x1ab04720) reporter: text file backup textfile_filename Value returning: $phase-$section-$mpi_name-$mpi_version.txt Value got: Config::IniFiles=HASH(0x1ab04720) reporter: text file backup textfile_dirname Value returning: Making dir: /root/mtt-svn/samples (cwd: /root/mtt-svn/samples) File reporter initialized (/root/mtt-svn/samples/$phase-$section-$mpi_name-$mpi_version.txt) Value got: Config::IniFiles=HASH(0x1ab04720) reporter: text file backup email_to Value returning: *** Reporter initialized Global min free specified as: 5%, resolved as 2539267072 *** MPI Get phase starting Chdir /root/mtt-svn/samples/sources Global min free specified as: 5%, resolved as 2539267072 Global min free specified as: 5%, resolved as 2539267072 Global min free specified as: 5%, resolved as 2539267072 >> MPI Get: [mpi get: my installation] Checking for new MPI sources... Value got: Config::IniFiles=HASH(0x1ab04720) mpi get: my installation module Value returning: AlreadyInstalled Value got: Config::IniFiles=HASH(0x1ab04720) mpi get: my installation mpi_details Value returning: Open MPI Value got: Config::IniFiles=HASH(0x1ab04720) mpi get: my installation skip_section Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi get: my installation env_module Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi get: my installation env_importer Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi get: my installation setenv Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi get: my installation unsetenv Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi get: my installation prepend_path Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi get: my installation append_path Value returning: Chdir /root/mtt-svn/samples/sources Making dir: mpi_get__my_installation (cwd: /root/mtt-svn/samples/sources) Chdir /root/mtt-svn/samples/sources/mpi_get__my_installation Evaluating: require MTT::MPI::Get::AlreadyInstalled Evaluating: $ret = &MTT::MPI::Get::AlreadyInstalled::Get(@args) Value got: Config::IniFiles=HASH(0x1ab04720) mpi get: my installation alreadyinstalled_dir Value returning: >> MTT::FindProgram::FindProgram returning /usr/mpi/gcc/openmpi-1.3.3/bin/mpicc Using MPI in: /usr/mpi/gcc/openmpi-1.3.3/ Value got: Config::IniFiles=HASH(0x1ab04720) mpi get: my installation alreadyinstalled_mpi_type Value returning: *** WARNING: alreadyinstalled_mpi_type was not specified, defaulting to "OMPI". Value got: Config::IniFiles=HASH(0x1ab04720) mpi get: my installation alreadyinstalled_version Value returning: >> MTT::Values::Functions::MPI::OMPI::get_version returning 1.3.3 >> MTT::MPI::Get::AlreadyInstalled complete Got new MPI sources: version 1.3.3 Global min free specified as: 5%, resolved as 2539267072 Global min free specified as: 5%, resolved as 2539267072 Global min free specified as: 5%, resolved as 2539267072 Global min free specified as: 5%, resolved as 2539267072 Global min free specified as: 5%, resolved as 2539267072 *** MPI Get phase complete Global min free specified as: 5%, resolved as 2539267072 *** MPI Install phase starting Making dir: /root/mtt-svn/samples/mpi-install (cwd: /root/mtt-svn/samples/sources/mpi_get__my_installation) Chdir /root/mtt-svn/samples/installs Global min free specified as: 5%, resolved as 2539267072 Chdir /root/mtt-svn/samples/installs Global min free specified as: 5%, resolved as 2539267072 Chdir /root/mtt-svn/samples/installs Global min free specified as: 5%, resolved as 2539267072 Chdir /root/mtt-svn/samples/installs Global min free specified as: 5%, resolved as 2539267072 >> MPI Install [mpi install: my installation] Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation mpi_get Value returning: My Installation Chdir /root/mtt-svn/samples/installs Global min free specified as: 5%, resolved as 2539267072 Checking for [my installation] / [1.3.2] / [my installation] Installing MPI: [my installation] / [1.3.2] / [my installation]... Chdir /root/mtt-svn/samples/installs Unique directory: kA6z Making dir: kA6z (cwd: /root/mtt-svn/samples/installs) kA6z does not exist -- creating Chdir /root/mtt-svn/samples/mpi-install Unique directory: Ttfp Making dir: Ttfp (cwd: /root/mtt-svn/samples/mpi-install) Ttfp does not exist -- creating Chdir /root/mtt-svn/samples/mpi-install/Ttfp Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation skip_section Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation module Value returning: Analyze::OMPI Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation description Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) MTT description Value returning: Pushdir to /root/mtt-svn/samples/installs/kA6z Chdir .. Chdir my_installation--my_installation--1.3.2 Chdir /root/mtt-svn/samples/installs Running command: rm -rf /root/mtt-svn/samples/installs/DXzO Command complete, exit status: 0 Sym linked: kA6z to my_installation--my_installation--1.3.2 Popdir to /root/mtt-svn/samples/mpi-install/Ttfp Pushdir to /root/mtt-svn/samples/mpi-install/Ttfp Chdir .. Chdir my_installation--my_installation--1.3.2 Chdir /root/mtt-svn/samples/mpi-install Running command: rm -rf /root/mtt-svn/samples/mpi-install/rH4U Command complete, exit status: 0 Sym linked: Ttfp to my_installation--my_installation--1.3.2 Popdir to /root/mtt-svn/samples/mpi-install/Ttfp Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation env_module Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation env_importer Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation setenv Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation unsetenv Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation prepend_path Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation append_path Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation configure_arguments Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation vpath_mode Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation make_all_arguments Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation make_check Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation compiler_name Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation compiler_version Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation mpi_details Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation save_stdout_on_success Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation merge_stdout_stderr Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation stderr_save_lines Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation stdout_save_lines Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation restart_on_pattern Value returning: Chdir /root/mtt-svn/samples/mpi-install/Ttfp Running command: rm -rf src Command complete, exit status: 0 Making dir: src (cwd: /root/mtt-svn/samples/mpi-install/Ttfp) src does not exist -- creating Chdir /root/mtt-svn/samples/mpi-install/Ttfp/src Evaluating: require MTT::MPI::Get::AlreadyInstalled Evaluating: $ret = &MTT::MPI::Get::AlreadyInstalled::PrepareForInstall(@args) Chdir /root/mtt-svn/samples/mpi-install/Ttfp/src Making dir: /root/mtt-svn/samples/installs/kA6z/install (cwd: /root/mtt-svn/samples/mpi-install/Ttfp/src) /root/mtt-svn/samples/installs/kA6z/install does not exist -- creating Evaluating: require MTT::MPI::Install::Analyze::OMPI Evaluating: $ret = &MTT::MPI::Install::Analyze::OMPI::Install(@args) Have C bindings: 1 Have C++ bindings: 0 Have F77 bindings: 0 Have F90 bindings: 0 Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation mpi_bitness bitness Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation endian Value returning: Scanning fast scratch for files to save: /root/mtt-svn/samples/mpi-install/Ttfp (config.log) Saving files from fast scratch to persistent scratch... Value got: Config::IniFiles=HASH(0x1aab97f0) reporter: text file backup skip_mpi_get Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) reporter: text file backup skip_mpi_install Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) reporter: text file backup setenv Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) reporter: text file backup unsetenv Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) reporter: text file backup prepend_path Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) reporter: text file backup append_path Value returning: Evaluating: require MTT::Reporter::TextFile Evaluating: $ret = &MTT::Reporter::TextFile::Submit(@args) File reporter _get_filename returning /root/mtt-svn/samples/MPI_Install-my_installation-my_installation-1.3.2.tx t Writing to text file: /root/mtt-svn/samples/MPI_Install-my_installation-my_installation-1.3.2.ht ml >> Reported to text file /root/mtt-svn/samples/MPI_Install-my_installation-my_installation-1.3.2.htm l Writing to text file: /root/mtt-svn/samples/MPI_Install-my_installation-my_installation-1.3.2.tx t >> Reported to text file /root/mtt-svn/samples/MPI_Install-my_installation-my_installation-1.3.2.txt Writing to text file: /root/mtt-svn/samples/MPI_Install-my_installation-my_installation-1.3.2.ht ml Completed MPI Install successfully Chdir /root/mtt-svn/samples/installs Global min free specified as: 5%, resolved as 2539267072 Checking for [my installation] / [1.3.3] / [my installation] Installing MPI: [my installation] / [1.3.3] / [my installation]... Chdir /root/mtt-svn/samples/installs Unique directory: bTUy Making dir: bTUy (cwd: /root/mtt-svn/samples/installs) bTUy does not exist -- creating Chdir /root/mtt-svn/samples/mpi-install Unique directory: FaLb Making dir: FaLb (cwd: /root/mtt-svn/samples/mpi-install) FaLb does not exist -- creating Chdir /root/mtt-svn/samples/mpi-install/FaLb Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation skip_section Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation module Value returning: Analyze::OMPI Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation description Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) MTT description Value returning: Pushdir to /root/mtt-svn/samples/installs/bTUy Chdir .. Chdir my_installation--my_installation--1.3.3 Chdir /root/mtt-svn/samples/installs Running command: rm -rf /root/mtt-svn/samples/installs/_aOF Command complete, exit status: 0 Sym linked: bTUy to my_installation--my_installation--1.3.3 Popdir to /root/mtt-svn/samples/mpi-install/FaLb Pushdir to /root/mtt-svn/samples/mpi-install/FaLb Chdir .. Chdir my_installation--my_installation--1.3.3 Chdir /root/mtt-svn/samples/mpi-install Running command: rm -rf /root/mtt-svn/samples/mpi-install/1vVM Command complete, exit status: 0 Sym linked: FaLb to my_installation--my_installation--1.3.3 Popdir to /root/mtt-svn/samples/mpi-install/FaLb Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation env_module Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation env_importer Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation setenv Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation unsetenv Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation prepend_path Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation append_path Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation configure_arguments Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation vpath_mode Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation make_all_arguments Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation make_check Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation compiler_name Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation compiler_version Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation mpi_details Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation save_stdout_on_success Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation merge_stdout_stderr Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation stderr_save_lines Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation stdout_save_lines Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation restart_on_pattern Value returning: Chdir /root/mtt-svn/samples/mpi-install/FaLb Running command: rm -rf src Command complete, exit status: 0 Making dir: src (cwd: /root/mtt-svn/samples/mpi-install/FaLb) src does not exist -- creating Chdir /root/mtt-svn/samples/mpi-install/FaLb/src Evaluating: require MTT::MPI::Get::AlreadyInstalled Evaluating: $ret = &MTT::MPI::Get::AlreadyInstalled::PrepareForInstall(@args) Chdir /root/mtt-svn/samples/mpi-install/FaLb/src Making dir: /root/mtt-svn/samples/installs/bTUy/install (cwd: /root/mtt-svn/samples/mpi-install/FaLb/src) /root/mtt-svn/samples/installs/bTUy/install does not exist -- creating Evaluating: require MTT::MPI::Install::Analyze::OMPI Evaluating: $ret = &MTT::MPI::Install::Analyze::OMPI::Install(@args) Have C bindings: 1 Have C++ bindings: 1 Have F77 bindings: 1 Have F90 bindings: 1 Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation mpi_bitness bitness Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi install: my installation endian Value returning: Scanning fast scratch for files to save: /root/mtt-svn/samples/mpi-install/FaLb (config.log) Saving files from fast scratch to persistent scratch... Value got: Config::IniFiles=HASH(0x1aab97f0) reporter: text file backup skip_mpi_get Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) reporter: text file backup skip_mpi_install Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) reporter: text file backup setenv Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) reporter: text file backup unsetenv Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) reporter: text file backup prepend_path Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) reporter: text file backup append_path Value returning: Evaluating: require MTT::Reporter::TextFile Evaluating: $ret = &MTT::Reporter::TextFile::Submit(@args) File reporter _get_filename returning /root/mtt-svn/samples/MPI_Install-my_installation-my_installation-1.3.3.tx t Writing to text file: /root/mtt-svn/samples/MPI_Install-my_installation-my_installation-1.3.3.ht ml >> Reported to text file /root/mtt-svn/samples/MPI_Install-my_installation-my_installation-1.3.3.htm l Writing to text file: /root/mtt-svn/samples/MPI_Install-my_installation-my_installation-1.3.3.tx t >> Reported to text file /root/mtt-svn/samples/MPI_Install-my_installation-my_installation-1.3.3.txt Writing to text file: /root/mtt-svn/samples/MPI_Install-my_installation-my_installation-1.3.3.ht ml Completed MPI Install successfully Chdir /root/mtt-svn/samples/installs Global min free specified as: 5%, resolved as 2539267072 Chdir /root/mtt-svn/samples/installs Global min free specified as: 5%, resolved as 2539267072 Chdir /root/mtt-svn/samples/installs Global min free specified as: 5%, resolved as 2539267072 Chdir /root/mtt-svn/samples/installs Global min free specified as: 5%, resolved as 2539267072 *** MPI Install phase complete Global min free specified as: 5%, resolved as 2539267072 *** Test Get phase starting Chdir /root/mtt-svn/samples/sources Global min free specified as: 5%, resolved as 2539267072 Global min free specified as: 5%, resolved as 2539267072 Global min free specified as: 5%, resolved as 2539267072 Global min free specified as: 5%, resolved as 2539267072 Global min free specified as: 5%, resolved as 2539267072 Global min free specified as: 5%, resolved as 2539267072 >> Test Get: [test get: trivial] Checking for new test sources... Value got: Config::IniFiles=HASH(0x1ab04720) test get: trivial module Value returning: Trivial Value got: Config::IniFiles=HASH(0x1ab04720) test get: trivial skip_section Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test get: trivial setenv Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test get: trivial unsetenv Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test get: trivial prepend_path Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test get: trivial append_path Value returning: Chdir /root/mtt-svn/samples/sources Making dir: test_get__trivial (cwd: /root/mtt-svn/samples/sources) Chdir /root/mtt-svn/samples/sources/test_get__trivial Evaluating: require MTT::Test::Get::Trivial Evaluating: $ret = &MTT::Test::Get::Trivial::Get(@args) Getting Trivial Got new test sources Global min free specified as: 5%, resolved as 2539267072 Global min free specified as: 5%, resolved as 2539267072 *** Test Get phase complete Global min free specified as: 5%, resolved as 2539267072 *** Test Build phase starting Chdir /root/mtt-svn/samples/installs Global min free specified as: 5%, resolved as 2539267072 Global min free specified as: 5%, resolved as 2539267072 Global min free specified as: 5%, resolved as 2539267072 Global min free specified as: 5%, resolved as 2539267072 Global min free specified as: 5%, resolved as 2539267072 Global min free specified as: 5%, resolved as 2539267072 Global min free specified as: 5%, resolved as 2539267072 >> Test Build [test build: trivial] Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial test_get Value returning: trivial Global min free specified as: 5%, resolved as 2539267072 Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial skip_mpi_get Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial skip_mpi_install Value returning: Building for [my installation] / [1.3.2] / [my installation] / [trivial] Chdir /root/mtt-svn/samples/installs Chdir /root/mtt-svn/samples/installs/kA6z Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial skip_section Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial module Value returning: Trivial Making dir: tests (cwd: /root/mtt-svn/samples/installs/kA6z) tests does not exist -- creating Chdir /root/mtt-svn/samples/installs/kA6z/tests Making dir: trivial (cwd: /root/mtt-svn/samples/installs/kA6z/tests) trivial does not exist -- creating Chdir /root/mtt-svn/samples/installs/kA6z/tests/trivial Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial description Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) MTT description Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial test_bitness bitness Value returning: Evaluating: require MTT::Common::Copytree Evaluating: $ret = &MTT::Common::Copytree::PrepareForInstall(@args) >> copytree copying to /root/mtt-svn/samples/installs/kA6z/tests/trivial Chdir /root/mtt-svn/samples/installs/kA6z/tests/trivial Copying directory: /root/mtt-svn/samples/sources/test_get__trivial Running command: rm -rf test_get__trivial Command complete, exit status: 0 Running command: cp -r /root/mtt-svn/samples/sources/test_get__trivial . Command complete, exit status: 0 Chdir test_get__trivial >> copytree finished copying Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial save_stdout_on_success Value returning: 1 Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial merge_stdout_stderr Value returning: 1 Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial stderr_save_lines Value returning: 100 Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial stdout_save_lines Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial env_module Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial env_importer Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial setenv Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial unsetenv Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial prepend_path Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial append_path Value returning: Evaluating: require MTT::Test::Build::Trivial Evaluating: $ret = &MTT::Test::Build::Trivial::Build(@args) Building Trivial Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial trivial_tests_cflags Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial trivial_tests_fflags Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial trivial_tests_languages Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial trivial_tests_mpicc Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial trivial_tests_mpicxx Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial trivial_tests_mpif77 Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial trivial_tests_mpif90 Value returning: Test compile/link sample C MPI application Running command: mpicc hello.c -o c_hello Command complete, exit status: 0 Running command: mpicc ring.c -o c_ring Command complete, exit status: 0 MPI C++ bindings unavailable; skipping simple compile/link test MPI F77 bindings unavailable; skipping simple compile/link test MPI F90 bindings unavailable; skipping simple compile/link test Value got: Config::IniFiles=HASH(0x1aab97f0) reporter: text file backup skip_mpi_get Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) reporter: text file backup skip_mpi_install Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) reporter: text file backup setenv Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) reporter: text file backup unsetenv Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) reporter: text file backup prepend_path Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) reporter: text file backup append_path Value returning: Evaluating: require MTT::Reporter::TextFile Evaluating: $ret = &MTT::Reporter::TextFile::Submit(@args) File reporter _get_filename returning /root/mtt-svn/samples/Test_Build-trivial-my_installation-1.3.2.txt Writing to text file: /root/mtt-svn/samples/Test_Build-trivial-my_installation-1.3.2.html >> Reported to text file /root/mtt-svn/samples/Test_Build-trivial-my_installation-1.3.2.html Writing to text file: /root/mtt-svn/samples/Test_Build-trivial-my_installation-1.3.2.txt >> Reported to text file /root/mtt-svn/samples/Test_Build-trivial-my_installation-1.3.2.txt Writing to text file: /root/mtt-svn/samples/Test_Build-trivial-my_installation-1.3.2.html Completed test build successfully Global min free specified as: 5%, resolved as 2539267072 Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial skip_mpi_get Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial skip_mpi_install Value returning: Building for [my installation] / [1.3.3] / [my installation] / [trivial] Chdir /root/mtt-svn/samples/installs Chdir /root/mtt-svn/samples/installs/bTUy Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial skip_section Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial module Value returning: Trivial Making dir: tests (cwd: /root/mtt-svn/samples/installs/bTUy) tests does not exist -- creating Chdir /root/mtt-svn/samples/installs/bTUy/tests Making dir: trivial (cwd: /root/mtt-svn/samples/installs/bTUy/tests) trivial does not exist -- creating Chdir /root/mtt-svn/samples/installs/bTUy/tests/trivial Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial description Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) MTT description Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial test_bitness bitness Value returning: Evaluating: require MTT::Common::Copytree Evaluating: $ret = &MTT::Common::Copytree::PrepareForInstall(@args) >> copytree copying to /root/mtt-svn/samples/installs/bTUy/tests/trivial Chdir /root/mtt-svn/samples/installs/bTUy/tests/trivial Copying directory: /root/mtt-svn/samples/sources/test_get__trivial Running command: rm -rf test_get__trivial Command complete, exit status: 0 Running command: cp -r /root/mtt-svn/samples/sources/test_get__trivial . Command complete, exit status: 0 Chdir test_get__trivial >> copytree finished copying Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial save_stdout_on_success Value returning: 1 Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial merge_stdout_stderr Value returning: 1 Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial stderr_save_lines Value returning: 100 Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial stdout_save_lines Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial env_module Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial env_importer Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial setenv Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial unsetenv Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial prepend_path Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial append_path Value returning: Evaluating: require MTT::Test::Build::Trivial Evaluating: $ret = &MTT::Test::Build::Trivial::Build(@args) Building Trivial Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial trivial_tests_cflags Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial trivial_tests_fflags Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial trivial_tests_languages Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial trivial_tests_mpicc Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial trivial_tests_mpicxx Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial trivial_tests_mpif77 Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test build: trivial trivial_tests_mpif90 Value returning: Test compile/link sample C MPI application Running command: mpicc hello.c -o c_hello Command complete, exit status: 0 Running command: mpicc ring.c -o c_ring Command complete, exit status: 0 Test compile/link sample C++ MPI application Running command: mpicxx hello.cc -o cxx_hello Command complete, exit status: 0 Running command: mpicxx ring.cc -o cxx_ring Command complete, exit status: 0 Test compile/link sample F77 MPI application Running command: mpif77 hello.f -o f77_hello Command complete, exit status: 0 Running command: mpif77 ring.f -o f77_ring Command complete, exit status: 0 Test compile/link sample F90 MPI application Running command: mpif90 hello.f90 -o f90_hello Command complete, exit status: 0 Running command: mpif90 ring.f90 -o f90_ring Command complete, exit status: 0 Value got: Config::IniFiles=HASH(0x1aab97f0) reporter: text file backup skip_mpi_get Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) reporter: text file backup skip_mpi_install Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) reporter: text file backup setenv Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) reporter: text file backup unsetenv Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) reporter: text file backup prepend_path Value returning: Value got: Config::IniFiles=HASH(0x1aab97f0) reporter: text file backup append_path Value returning: Evaluating: require MTT::Reporter::TextFile Evaluating: $ret = &MTT::Reporter::TextFile::Submit(@args) File reporter _get_filename returning /root/mtt-svn/samples/Test_Build-trivial-my_installation-1.3.3.txt Writing to text file: /root/mtt-svn/samples/Test_Build-trivial-my_installation-1.3.3.html >> Reported to text file /root/mtt-svn/samples/Test_Build-trivial-my_installation-1.3.3.html Writing to text file: /root/mtt-svn/samples/Test_Build-trivial-my_installation-1.3.3.txt >> Reported to text file /root/mtt-svn/samples/Test_Build-trivial-my_installation-1.3.3.txt Writing to text file: /root/mtt-svn/samples/Test_Build-trivial-my_installation-1.3.3.html Completed test build successfully Global min free specified as: 5%, resolved as 2539267072 *** Test Build phase complete Global min free specified as: 5%, resolved as 2539267072 *** Test Run phase starting Global min free specified as: 5%, resolved as 2539267072 Global min free specified as: 5%, resolved as 2539267072 Global min free specified as: 5%, resolved as 2539267072 Global min free specified as: 5%, resolved as 2539267072 Global min free specified as: 5%, resolved as 2539267072 Global min free specified as: 5%, resolved as 2539267072 Global min free specified as: 5%, resolved as 2539267072 Global min free specified as: 5%, resolved as 2539267072 >> Test Run [trivial] Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial test_build Value returning: trivial Global min free specified as: 5%, resolved as 2539267072 Found a match! trivial [trivial Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial skip_mpi_get Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial skip_mpi_install Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial skip_section Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial specify_module Value returning: Simple >> Running with [my installation] / [1.3.2] / [my installation] Found mpi_details [open mpi] in MPI get [my installation] Found MPI details: [open mpi] Using MPI Details [open mpi] with MPI Install [my installation] Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial mpi_details_exec Value returning: Got final exec: mpirun &if( &have_hostfile(), &stringify("--hostfile ", &hostfile()), &if( &have_hostlist(), &stringify("--host ", &hostlist()), "" ) ) -np &test_np() --mca btl openib,sm,self --prefix &test_prefix() &test_executable() &test_argv() Chdir /root/mtt-svn/samples/installs/kA6z/tests/trivial/test_get__trivial Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial env_module Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial env_importer Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial setenv Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial unsetenv Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial prepend_path Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial append_path Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial description Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) MTT description Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi details: open mpi before_any_exec Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi details: open mpi before_any_exec_timeout Value returning: Time 10 = 10 seconds Value got: Config::IniFiles=HASH(0x1ab04720) mpi details: open mpi before_any_exec_pass Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi details: open mpi before_each_exec Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi details: open mpi before_each_exec_timeout Value returning: Time seconds Value got: Config::IniFiles=HASH(0x1ab04720) mpi details: open mpi before_each_exec_pass Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi details: open mpi after_each_exec Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi details: open mpi after_each_exec_timeout Value returning: Time 10 = 10 seconds Value got: Config::IniFiles=HASH(0x1ab04720) mpi details: open mpi after_each_exec_pass Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi details: open mpi after_all_exec Value returning: done. Value got: Config::IniFiles=HASH(0x1ab04720) mpi details: open mpi after_all_exec_timeout Value returning: Time 10 = 10 seconds Value got: Config::IniFiles=HASH(0x1ab04720) mpi details: open mpi after_all_exec_pass Value returning: Evaluating: require MTT::Test::Specify::Simple Evaluating: $ret = &MTT::Test::Specify::Simple::Specify(@args) Adding test: ./c_hello (group: only) Adding test: ./c_ring (group: only) Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial analyze_module Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mtt submit_group_results Value returning: Returining reference to an array of a single scalar Returining reference to an array of a single scalar Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial break_threshold_pass Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial break_threshold_fail Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial break_threshold_timeout Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial break_threshold_skipped Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial report_after_each_result Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial report_after_n_results Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial submit_after_each_result Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial submit_results_after_n_results Value returning: ### ### Total tests to run in this section: ### 2 test executable(s) ### 1 np value(s) ### 1 test variant(s) ### 2 total mpirun command(s) to run ### Global min free specified as: 5%, resolved as 2539267072 &have_hostfile returning 0 &have_hostlist: returning 1 &hostlist: 102.77.77.66,102.77.77.68 &hostlist_hosts &have_hostlist: returning 1 &hostlist_hosts returning 102.77.77.66,102.77.77.68 Time 8 = 8 seconds Running command: mpirun --host 102.77.77.66,102.77.77.68 -np 8 --mca btl openib,sm,self --prefix /usr/mpi/gcc/openmpi-1.3.2/ ./c_hello Timeout: 8 - 1259552291 (vs. now: 1259552283) OUT:bash: /usr/mpi/gcc/openmpi-1.3.2/bin/orted: No such file or directory OUT:bash: /usr/mpi/gcc/openmpi-1.3.2/bin/orted: No such file or directory OUT:----------------------------------------------------------------------- --- A daemon (pid 5195) died unexpectedly with status 127 while attempting to launch so we are aborting. There may be more information reported by the environment (see above). This may be because the daemon was unable to find all the needed shared libraries on the remote node. You may set your LD_LIBRARY_PATH to have the location of the shared libraries on the remote nodes and this will automatically be forwarded to the remote nodes. ------------------------------------------------------------------------ -- OUT:----------------------------------------------------------------------- --- mpirun noticed that the job aborted, but has no info as to the process that caused that situation. ------------------------------------------------------------------------ -- OUT:----------------------------------------------------------------------- --- mpirun was unable to cleanly terminate the daemons on the nodes shown below. Additional manual cleanup may be required - please refer to the "orte-clean" tool for assistance. ------------------------------------------------------------------------ -- OUT: 102.77.77.66 - daemon did not report back when launched 102.77.77.68 - daemon did not report back when launched Command complete, exit status: 127 Evaluating: require MTT::Test::Analyze Evaluating: $ret = &MTT::Test::Analyze::Analyze(@args) Evaluating: require MTT::Test::Analyze::Correctness Evaluating: $ret = &MTT::Test::Analyze::Correctness::Analyze(@args) *** WARNING: Test: c_hello, np=8, variant=1: FAILED Making dir: /root/mtt-svn/samples/test_runs/test_runs/my_installation/1.3.2/my_insta llation/trivial/trivial/c_hello (cwd: /root/mtt-svn/samples/installs/kA6z/tests/trivial/test_get__trivial) Global min free specified as: 5%, resolved as 2539267072 &have_hostfile returning 0 &have_hostlist: returning 1 &hostlist: 102.77.77.66,102.77.77.68 &hostlist_hosts &have_hostlist: returning 1 &hostlist_hosts returning 102.77.77.66,102.77.77.68 Time 8 = 8 seconds Running command: mpirun --host 102.77.77.66,102.77.77.68 -np 8 --mca btl openib,sm,self --prefix /usr/mpi/gcc/openmpi-1.3.2/ ./c_ring Timeout: 8 - 1259552292 (vs. now: 1259552284) OUT:bash: /usr/mpi/gcc/openmpi-1.3.2/bin/orted: No such file or directory OUT:bash: /usr/mpi/gcc/openmpi-1.3.2/bin/orted: No such file or directory OUT:----------------------------------------------------------------------- --- A daemon (pid 5203) died unexpectedly with status 127 while attempting to launch so we are aborting. There may be more information reported by the environment (see above). This may be because the daemon was unable to find all the needed shared libraries on the remote node. You may set your LD_LIBRARY_PATH to have the location of the shared libraries on the remote nodes and this will automatically be forwarded to the remote nodes. ------------------------------------------------------------------------ -- OUT:----------------------------------------------------------------------- --- mpirun noticed that the job aborted, but has no info as to the process that caused that situation. ------------------------------------------------------------------------ -- OUT:mpirun: clean termination accomplished Command complete, exit status: 127 Evaluating: require MTT::Test::Analyze Evaluating: $ret = &MTT::Test::Analyze::Analyze(@args) Evaluating: require MTT::Test::Analyze::Correctness Evaluating: $ret = &MTT::Test::Analyze::Correctness::Analyze(@args) *** WARNING: Test: c_ring, np=8, variant=1: FAILED Making dir: /root/mtt-svn/samples/test_runs/test_runs/my_installation/1.3.2/my_insta llation/trivial/trivial/c_ring (cwd: /root/mtt-svn/samples/installs/kA6z/tests/trivial/test_get__trivial) ### Test progress: 2 of 2 section test executables complete. Moving on. Time 10 = 10 seconds Running step: after_all_exec: done. / timeout 10 Running command: done. Timeout: 10 - 1259552296 (vs. now: 1259552286) OUT:Can't execute command: done. Error: No such file or directory Command complete, exit status: 2 Warning: step after_all_exec FAILED Output: Can't execute command: done. Error: No such file or directory Evaluating: require MTT::Reporter::TextFile Evaluating: $ret = &MTT::Reporter::TextFile::Submit(@args) File reporter _get_filename returning /root/mtt-svn/samples/Test_Run-trivial-my_installation-1.3.2.txt _get_filename returning /root/mtt-svn/samples/Test_Run-trivial-my_installation-1.3.2.txt Writing to text file: /root/mtt-svn/samples/Test_Run-trivial-my_installation-1.3.2.html >> Reported to text file /root/mtt-svn/samples/Test_Run-trivial-my_installation-1.3.2.html Writing to text file: /root/mtt-svn/samples/Test_Run-trivial-my_installation-1.3.2.txt >> Reported to text file /root/mtt-svn/samples/Test_Run-trivial-my_installation-1.3.2.txt Writing to text file: /root/mtt-svn/samples/Test_Run-trivial-my_installation-1.3.2.html Global min free specified as: 5%, resolved as 2539267072 Found a match! trivial [trivial Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial skip_mpi_get Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial skip_mpi_install Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial skip_section Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial specify_module Value returning: Simple >> Running with [my installation] / [1.3.3] / [my installation] Found mpi_details [open mpi] in MPI get [my installation] Found MPI details: [open mpi] Using MPI Details [open mpi] with MPI Install [my installation] Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial mpi_details_exec Value returning: Got final exec: mpirun &if( &have_hostfile(), &stringify("--hostfile ", &hostfile()), &if( &have_hostlist(), &stringify("--host ", &hostlist()), "" ) ) -np &test_np() --mca btl openib,sm,self --prefix &test_prefix() &test_executable() &test_argv() Chdir /root/mtt-svn/samples/installs/bTUy/tests/trivial/test_get__trivial Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial env_module Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial env_importer Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial setenv Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial unsetenv Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial prepend_path Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial append_path Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial description Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) MTT description Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi details: open mpi before_any_exec Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi details: open mpi before_any_exec_timeout Value returning: Time 10 = 10 seconds Value got: Config::IniFiles=HASH(0x1ab04720) mpi details: open mpi before_any_exec_pass Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi details: open mpi before_each_exec Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi details: open mpi before_each_exec_timeout Value returning: Time seconds Value got: Config::IniFiles=HASH(0x1ab04720) mpi details: open mpi before_each_exec_pass Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi details: open mpi after_each_exec Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi details: open mpi after_each_exec_timeout Value returning: Time 10 = 10 seconds Value got: Config::IniFiles=HASH(0x1ab04720) mpi details: open mpi after_each_exec_pass Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mpi details: open mpi after_all_exec Value returning: done. Value got: Config::IniFiles=HASH(0x1ab04720) mpi details: open mpi after_all_exec_timeout Value returning: Time 10 = 10 seconds Value got: Config::IniFiles=HASH(0x1ab04720) mpi details: open mpi after_all_exec_pass Value returning: Evaluating: require MTT::Test::Specify::Simple Evaluating: $ret = &MTT::Test::Specify::Simple::Specify(@args) Adding test: ./cxx_ring (group: only) Adding test: ./f77_hello (group: only) Adding test: ./f90_hello (group: only) Adding test: ./c_hello (group: only) Adding test: ./f90_ring (group: only) Adding test: ./f77_ring (group: only) Adding test: ./c_ring (group: only) Adding test: ./cxx_hello (group: only) Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial analyze_module Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) mtt submit_group_results Value returning: Returining reference to an array of a single scalar Returining reference to an array of a single scalar Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial break_threshold_pass Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial break_threshold_fail Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial break_threshold_timeout Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial break_threshold_skipped Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial report_after_each_result Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial report_after_n_results Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial submit_after_each_result Value returning: Value got: Config::IniFiles=HASH(0x1ab04720) test run: trivial submit_results_after_n_results Value returning: ### ### Total tests to run in this section: ### 8 test executable(s) ### 1 np value(s) ### 1 test variant(s) ### 8 total mpirun command(s) to run ### Global min free specified as: 5%, resolved as 2539267072 &have_hostfile returning 0 &have_hostlist: returning 1 &hostlist: 102.77.77.66,102.77.77.68 &hostlist_hosts &have_hostlist: returning 1 &hostlist_hosts returning 102.77.77.66,102.77.77.68 Time 8 = 8 seconds Running command: mpirun --host 102.77.77.66,102.77.77.68 -np 8 --mca btl openib,sm,self --prefix /usr/mpi/gcc/openmpi-1.3.3/ ./cxx_ring Timeout: 8 - 1259552294 (vs. now: 1259552286) OUT:----------------------------------------------------------------------- --- mpirun was unable to launch the specified application as it could not access or execute an executable: Executable: ./cxx_ring Node: 102.77.77.68 while attempting to start process rank 1. ------------------------------------------------------------------------ -- Command complete, exit status: 131 Evaluating: require MTT::Test::Analyze Evaluating: $ret = &MTT::Test::Analyze::Analyze(@args) Evaluating: require MTT::Test::Analyze::Correctness Evaluating: $ret = &MTT::Test::Analyze::Correctness::Analyze(@args) *** WARNING: Test: cxx_ring, np=8, variant=1: FAILED Making dir: /root/mtt-svn/samples/test_runs/test_runs/my_installation/1.3.3/my_insta llation/trivial/trivial/cxx_ring (cwd: /root/mtt-svn/samples/installs/bTUy/tests/trivial/test_get__trivial) Global min free specified as: 5%, resolved as 2539267072 &have_hostfile returning 0 &have_hostlist: returning 1 &hostlist: 102.77.77.66,102.77.77.68 &hostlist_hosts &have_hostlist: returning 1 &hostlist_hosts returning 102.77.77.66,102.77.77.68 Time 8 = 8 seconds Running command: mpirun --host 102.77.77.66,102.77.77.68 -np 8 --mca btl openib,sm,self --prefix /usr/mpi/gcc/openmpi-1.3.3/ ./f77_hello Timeout: 8 - 1259552295 (vs. now: 1259552287) OUT:----------------------------------------------------------------------- --- mpirun was unable to launch the specified application as it could not access or execute an executable: Executable: ./f77_hello Node: 102.77.77.66 while attempting to start process rank 0. ------------------------------------------------------------------------ -- Command complete, exit status: 131 Evaluating: require MTT::Test::Analyze Evaluating: $ret = &MTT::Test::Analyze::Analyze(@args) Evaluating: require MTT::Test::Analyze::Correctness Evaluating: $ret = &MTT::Test::Analyze::Correctness::Analyze(@args) *** WARNING: Test: f77_hello, np=8, variant=1: FAILED Making dir: /root/mtt-svn/samples/test_runs/test_runs/my_installation/1.3.3/my_insta llation/trivial/trivial/f77_hello (cwd: /root/mtt-svn/samples/installs/bTUy/tests/trivial/test_get__trivial) Global min free specified as: 5%, resolved as 2539267072 &have_hostfile returning 0 &have_hostlist: returning 1 &hostlist: 102.77.77.66,102.77.77.68 &hostlist_hosts &have_hostlist: returning 1 &hostlist_hosts returning 102.77.77.66,102.77.77.68 Time 8 = 8 seconds Running command: mpirun --host 102.77.77.66,102.77.77.68 -np 8 --mca btl openib,sm,self --prefix /usr/mpi/gcc/openmpi-1.3.3/ ./f90_hello Timeout: 8 - 1259552296 (vs. now: 1259552288) OUT:----------------------------------------------------------------------- --- mpirun was unable to launch the specified application as it could not access or execute an executable: Executable: ./f90_hello Node: 102.77.77.68 while attempting to start process rank 1. ------------------------------------------------------------------------ -- Command complete, exit status: 131 Evaluating: require MTT::Test::Analyze Evaluating: $ret = &MTT::Test::Analyze::Analyze(@args) Evaluating: require MTT::Test::Analyze::Correctness Evaluating: $ret = &MTT::Test::Analyze::Correctness::Analyze(@args) *** WARNING: Test: f90_hello, np=8, variant=1: FAILED Making dir: /root/mtt-svn/samples/test_runs/test_runs/my_installation/1.3.3/my_insta llation/trivial/trivial/f90_hello (cwd: /root/mtt-svn/samples/installs/bTUy/tests/trivial/test_get__trivial) Global min free specified as: 5%, resolved as 2539267072 &have_hostfile returning 0 &have_hostlist: returning 1 &hostlist: 102.77.77.66,102.77.77.68 &hostlist_hosts &have_hostlist: returning 1 &hostlist_hosts returning 102.77.77.66,102.77.77.68 Time 8 = 8 seconds Running command: mpirun --host 102.77.77.66,102.77.77.68 -np 8 --mca btl openib,sm,self --prefix /usr/mpi/gcc/openmpi-1.3.3/ ./c_hello Timeout: 8 - 1259552297 (vs. now: 1259552289) OUT:----------------------------------------------------------------------- --- mpirun was unable to launch the specified application as it could not access or execute an executable: Executable: ./c_hello Node: 102.77.77.66 while attempting to start process rank 0. ------------------------------------------------------------------------ -- OUT:2 total processes failed to start Command complete, exit status: 131 Evaluating: require MTT::Test::Analyze Evaluating: $ret = &MTT::Test::Analyze::Analyze(@args) Evaluating: require MTT::Test::Analyze::Correctness Evaluating: $ret = &MTT::Test::Analyze::Correctness::Analyze(@args) *** WARNING: Test: c_hello, np=8, variant=1: FAILED Making dir: /root/mtt-svn/samples/test_runs/test_runs/my_installation/1.3.3/my_insta llation/trivial/trivial/c_hello (cwd: /root/mtt-svn/samples/installs/bTUy/tests/trivial/test_get__trivial) Global min free specified as: 5%, resolved as 2539267072 &have_hostfile returning 0 &have_hostlist: returning 1 &hostlist: 102.77.77.66,102.77.77.68 &hostlist_hosts &have_hostlist: returning 1 &hostlist_hosts returning 102.77.77.66,102.77.77.68 Time 8 = 8 seconds Running command: mpirun --host 102.77.77.66,102.77.77.68 -np 8 --mca btl openib,sm,self --prefix /usr/mpi/gcc/openmpi-1.3.3/ ./f90_ring Timeout: 8 - 1259552299 (vs. now: 1259552291) OUT:----------------------------------------------------------------------- --- mpirun was unable to launch the specified application as it could not access or execute an executable: Executable: ./f90_ring Node: 102.77.77.66 while attempting to start process rank 0. ------------------------------------------------------------------------ -- Command complete, exit status: 131 Evaluating: require MTT::Test::Analyze Evaluating: $ret = &MTT::Test::Analyze::Analyze(@args) Evaluating: require MTT::Test::Analyze::Correctness Evaluating: $ret = &MTT::Test::Analyze::Correctness::Analyze(@args) *** WARNING: Test: f90_ring, np=8, variant=1: FAILED Making dir: /root/mtt-svn/samples/test_runs/test_runs/my_installation/1.3.3/my_insta llation/trivial/trivial/f90_ring (cwd: /root/mtt-svn/samples/installs/bTUy/tests/trivial/test_get__trivial) Global min free specified as: 5%, resolved as 2539267072 &have_hostfile returning 0 &have_hostlist: returning 1 &hostlist: 102.77.77.66,102.77.77.68 &hostlist_hosts &have_hostlist: returning 1 &hostlist_hosts returning 102.77.77.66,102.77.77.68 Time 8 = 8 seconds Running command: mpirun --host 102.77.77.66,102.77.77.68 -np 8 --mca btl openib,sm,self --prefix /usr/mpi/gcc/openmpi-1.3.3/ ./f77_ring Timeout: 8 - 1259552300 (vs. now: 1259552292) OUT:----------------------------------------------------------------------- --- mpirun was unable to launch the specified application as it could not access or execute an executable: Executable: ./f77_ring Node: 102.77.77.68 while attempting to start process rank 1. ------------------------------------------------------------------------ -- Command complete, exit status: 131 Evaluating: require MTT::Test::Analyze Evaluating: $ret = &MTT::Test::Analyze::Analyze(@args) Evaluating: require MTT::Test::Analyze::Correctness Evaluating: $ret = &MTT::Test::Analyze::Correctness::Analyze(@args) *** WARNING: Test: f77_ring, np=8, variant=1: FAILED Making dir: /root/mtt-svn/samples/test_runs/test_runs/my_installation/1.3.3/my_insta llation/trivial/trivial/f77_ring (cwd: /root/mtt-svn/samples/installs/bTUy/tests/trivial/test_get__trivial) Global min free specified as: 5%, resolved as 2539267072 &have_hostfile returning 0 &have_hostlist: returning 1 &hostlist: 102.77.77.66,102.77.77.68 &hostlist_hosts &have_hostlist: returning 1 &hostlist_hosts returning 102.77.77.66,102.77.77.68 Time 8 = 8 seconds Running command: mpirun --host 102.77.77.66,102.77.77.68 -np 8 --mca btl openib,sm,self --prefix /usr/mpi/gcc/openmpi-1.3.3/ ./c_ring Timeout: 8 - 1259552301 (vs. now: 1259552293) OUT:----------------------------------------------------------------------- --- mpirun was unable to launch the specified application as it could not access or execute an executable: Executable: ./c_ring Node: 102.77.77.68 while attempting to start process rank 1. ------------------------------------------------------------------------ -- Command complete, exit status: 131 Evaluating: require MTT::Test::Analyze Evaluating: $ret = &MTT::Test::Analyze::Analyze(@args) Evaluating: require MTT::Test::Analyze::Correctness Evaluating: $ret = &MTT::Test::Analyze::Correctness::Analyze(@args) *** WARNING: Test: c_ring, np=8, variant=1: FAILED Making dir: /root/mtt-svn/samples/test_runs/test_runs/my_installation/1.3.3/my_insta llation/trivial/trivial/c_ring (cwd: /root/mtt-svn/samples/installs/bTUy/tests/trivial/test_get__trivial) Global min free specified as: 5%, resolved as 2539267072 &have_hostfile returning 0 &have_hostlist: returning 1 &hostlist: 102.77.77.66,102.77.77.68 &hostlist_hosts &have_hostlist: returning 1 &hostlist_hosts returning 102.77.77.66,102.77.77.68 Time 8 = 8 seconds Running command: mpirun --host 102.77.77.66,102.77.77.68 -np 8 --mca btl openib,sm,self --prefix /usr/mpi/gcc/openmpi-1.3.3/ ./cxx_hello Timeout: 8 - 1259552302 (vs. now: 1259552294) OUT:----------------------------------------------------------------------- --- mpirun was unable to launch the specified application as it could not access or execute an executable: Executable: ./cxx_hello Node: 102.77.77.66 while attempting to start process rank 0. ------------------------------------------------------------------------ -- OUT:2 total processes failed to start Command complete, exit status: 131 Evaluating: require MTT::Test::Analyze Evaluating: $ret = &MTT::Test::Analyze::Analyze(@args) Evaluating: require MTT::Test::Analyze::Correctness Evaluating: $ret = &MTT::Test::Analyze::Correctness::Analyze(@args) *** WARNING: Test: cxx_hello, np=8, variant=1: FAILED Making dir: /root/mtt-svn/samples/test_runs/test_runs/my_installation/1.3.3/my_insta llation/trivial/trivial/cxx_hello (cwd: /root/mtt-svn/samples/installs/bTUy/tests/trivial/test_get__trivial) ### Test progress: 8 of 8 section test executables complete. Moving on. Time 10 = 10 seconds Running step: after_all_exec: done. / timeout 10 Running command: done. Timeout: 10 - 1259552305 (vs. now: 1259552295) OUT:Can't execute command: done. Error: No such file or directory Command complete, exit status: 2 Warning: step after_all_exec FAILED Output: Can't execute command: done. Error: No such file or directory Evaluating: require MTT::Reporter::TextFile Evaluating: $ret = &MTT::Reporter::TextFile::Submit(@args) File reporter _get_filename returning /root/mtt-svn/samples/Test_Run-trivial-my_installation-1.3.3.txt _get_filename returning /root/mtt-svn/samples/Test_Run-trivial-my_installation-1.3.3.txt _get_filename returning /root/mtt-svn/samples/Test_Run-trivial-my_installation-1.3.3.txt _get_filename returning /root/mtt-svn/samples/Test_Run-trivial-my_installation-1.3.3.txt _get_filename returning /root/mtt-svn/samples/Test_Run-trivial-my_installation-1.3.3.txt _get_filename returning /root/mtt-svn/samples/Test_Run-trivial-my_installation-1.3.3.txt _get_filename returning /root/mtt-svn/samples/Test_Run-trivial-my_installation-1.3.3.txt _get_filename returning /root/mtt-svn/samples/Test_Run-trivial-my_installation-1.3.3.txt Writing to text file: /root/mtt-svn/samples/Test_Run-trivial-my_installation-1.3.3.html >> Reported to text file /root/mtt-svn/samples/Test_Run-trivial-my_installation-1.3.3.html Writing to text file: /root/mtt-svn/samples/Test_Run-trivial-my_installation-1.3.3.txt >> Reported to text file /root/mtt-svn/samples/Test_Run-trivial-my_installation-1.3.3.txt Writing to text file: /root/mtt-svn/samples/Test_Run-trivial-my_installation-1.3.3.html *** Run test phase complete *** Reporter finalizing Evaluating: require MTT::Reporter::TextFile Evaluating: $ret = &MTT::Reporter::TextFile::Finalize(@args) MTT Results Summary hostname: mughal uname: Linux mughal 2.6.18-128.el5 #1 SMP Wed Jan 21 10:41:14 EST 2009 x86_64 x86_64 x86_64 GNU/Linux who am i: _get_filename returning /root/mtt-svn/samples/MPI_Install-my_installation-my_installation-1.3.2. txt _get_filename returning /root/mtt-svn/samples/MPI_Install-my_installation-my_installation-1.3.3. txt _get_filename returning /root/mtt-svn/samples/Test_Build-trivial-my_installation-1.3.2.txt _get_filename returning /root/mtt-svn/samples/Test_Build-trivial-my_installation-1.3.3.txt _get_filename returning /root/mtt-svn/samples/Test_Run-trivial-my_installation-1.3.2.txt _get_filename returning /root/mtt-svn/samples/Test_Run-trivial-my_installation-1.3.3.txt +-------------+-----------------+-------------+----------+------+------+----------+------+--------------------------------------------------------+ | Phase | Section | MPI Version | Duration | Pass | Fail | Time out | Skip | Detailed report | +-------------+-----------------+-------------+----------+------+------+----------+------+--------------------------------------------------------+ | MPI Install | my installation | 1.3.2 | 00:00 | 1 | | | | MPI_Install-my_installation-my_installation-1.3.2.html | | MPI Install | my installation | 1.3.3 | 00:00 | 1 | | | | MPI_Install-my_installation-my_installation-1.3.3.html | | Test Build | trivial | 1.3.2 | 00:00 | 1 | | | | Test_Build-trivial-my_installation-1.3.2.html | | Test Build | trivial | 1.3.3 | 00:02 | 1 | | | | Test_Build-trivial-my_installation-1.3.3.html | | Test Run | trivial | 1.3.2 | 00:03 | | 2 | | | Test_Run-trivial-my_installation-1.3.2.html | | Test Run | trivial | 1.3.3 | 00:09 | | 8 | | | Test_Run-trivial-my_installation-1.3.3.html | +-------------+-----------------+-------------+----------+------+------+----------+------+--------------------------------------------------------+ Total Tests: 14 Total Failures: 10 Total Passed: 4 Total Duration: 14 secs. (00:14) hostname: mughal uname: Linux mughal 2.6.18-128.el5 #1 SMP Wed Jan 21 10:41:14 EST 2009 x86_64 x86_64 x86_64 GNU/Linux who am i: +-------------+-----------------+-------------+----------+------+------+----------+------+--------------------------------------------------------+ | Phase | Section | MPI Version | Duration | Pass | Fail | Time out | Skip | Detailed report | +-------------+-----------------+-------------+----------+------+------+----------+------+--------------------------------------------------------+ | MPI Install | my installation | 1.3.2 | 00:00 | 1 | | | | MPI_Install-my_installation-my_installation-1.3.2.html | | MPI Install | my installation | 1.3.3 | 00:00 | 1 | | | | MPI_Install-my_installation-my_installation-1.3.3.html | | Test Build | trivial | 1.3.2 | 00:00 | 1 | | | | Test_Build-trivial-my_installation-1.3.2.html | | Test Build | trivial | 1.3.3 | 00:02 | 1 | | | | Test_Build-trivial-my_installation-1.3.3.html | | Test Run | trivial | 1.3.2 | 00:03 | | 2 | | | Test_Run-trivial-my_installation-1.3.2.html | | Test Run | trivial | 1.3.3 | 00:09 | | 8 | | | Test_Run-trivial-my_installation-1.3.3.html | +-------------+-----------------+-------------+----------+------+------+----------+------+--------------------------------------------------------+ Total Tests: 14 Total Failures: 10 Total Passed: 4 Total Duration: 14 secs. (00:14) Writing to text file: /root/mtt-svn/samples/All_phase-summary.txt >> Reported to text file /root/mtt-svn/samples/All_phase-summary.txt Writing to text file: /root/mtt-svn/samples/All_phase-summary.html >> Reported to text file /root/mtt-svn/samples/All_phase-summary.html *** Reporter finalized