On 03/04/2017 09:37, Henkel, Andreas wrote:
Dear Kenneth,
thank you very much for your explanations!
You're right, I oversaw the option of environment variables.
I think my issue is the transition from the os-dependent manual install to
easybuild, since easybuild tries to the self-contained, hence, I see more
modules which were "invisible" because they were os-libs before. Probably,
it's just me getting used to it. Then I'll stick with having no hidden
modules for now and see if our users adapt.
There's another sort of related question I have:
Do you also have many redundant modules? I mean, I install GCC 6.3.0, which
needed zlib. Later I tried to install some other software with try-toolchain
and --use-existing but now I see redundant modules, namely
zlib-ver-GCCcore-6.3.0
zlib-ver-foss2017a
Although foss2017a uses 6.3.0 in my case. My little experience with
try-toolchain is that it somehow ignores existing software/modules. Is that
correct or did I do something wrong?
Yes, it's possible you are getting the same software version installed
multiple times with different toolchains, although we try to avoid that.
By default, EasyBuild will always consider the 'full' toolchain (e.g.
foss/2017a) first when trying to resolve dependencies.
Since EasyBuild v3.0, it will also consider subtoolchains (e.g.
gompi/2017a, GCC/6.3.0-2.27 and GCCcore/6.3.0) if no matching easyconfig
file was found.
If you want to let the dependency resolution mechanism consider
subtoolchains first (i.e. use the order GCCcore/6.3.0, ..., foss/2017a),
you should configure EasyBuild with --minimal-toolchains, see
http://easybuild.readthedocs.io/en/latest/Manipulating_dependencies.html#using-minimal-toolchains-for-dependencies
.
This may become the default behavior in a future (major) release of
EasyBuild, but the downside is that this requires that easyconfigs are
available in the robot search path at all times, while the default
full-toolchain resolution doesn't (it can just check for available modules).
regards,
Kenneth
Best regards,
Andreas
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Kenneth Hoste
Sent: Monday, April 3, 2017 9:14 AM
To: [email protected]
Subject: Re: [easybuild] Common config: to hide or not to hide
On 03/04/2017 08:24, Henkel, Andreas wrote:
Dear Easybuilder,
I'm fairly new to easybuild and have just built several packages using the
--try-* options or even adapted easyconfigs. It's a very handy way for
installing software, thanks for that!
Beyond that I had a look at the different module naming schemes and ended
up with using the categorized nms.
I'm a bit surprised about the amount of modules I end up with when
installing for example GCC or Netcdf. We did manual installs before and used
System Libs a lot. Now, every lib ends up being a module and on my first
impression this pollutes the module list. I saw the option hide-deps and
filter-deps. Using those would remove some of those libs of course but as
far as I can see it would also take a way some of the "automagic" since I'd
have to specify the options for every install.
You don't have to specify --hide-deps of --filter-deps for every
installations, you can configure EasyBuild via environment variables or
configuration files as well, see
http://easybuild.readthedocs.io/en/latest/Configuration.html .
It is true that you have to gradually build up the list of modules you want
to hide though, since that's a very site-specific aspect.
Note that --filter-deps means something very different than --hide-deps.
With --filter-deps, you're telling EasyBuild not to install something, and
to assume that that particular library/tool is already provided by the OS
(which also means you're at the mercy of the OS a little bit more).
I would like to ask for some hints of experienced easybuild users. How are
you handling the dependent modules? Are they usually hidden-modules or do
the users see all the dependencies in their module environment?
Some sites (e.g. JSC) indeed to quite a bit of effort the only expose those
modules to users that they are actually interested in, and --hide-deps is a
useful way of doing that.
This is being discussed in the HUST-16 workshop paper available at
http://hpcugent.github.io/easybuild/files/eb-jsc-hust16.pdf .
At HPC-UGent, we typically don't hide modules (unless it's a temporary
installation that we want some users to evaluate first), and we're not
getting too much complaints from our users on that to be honest. That does
mean our default 'module avail' view is huge though, but it's less effort on
our part to figure out which modules would be useful to users and which ones
wouldn't be...
regards,
Kenneth
