For those of you interested in this topic, this is also being discussed
in https://github.com/easybuilders/easybuild-framework/issues/2865 .
K.
On 07/05/2019 13:21, Espen Flage-Larsen wrote:
Dear developers and users,
After looking at this a bit more there is also a more fundamental
problem about the approach we are taking; the need to create custom
build directories in an easyblock file (where I do the iteration). I
could do this in the build_step by modifying the self.builddir, but the
path_step would not see this, etc.
It then seems to me that if we were to iterate, this should happen
outside the easyblock. At least given the current configuration. As I
do not see a way to specify such an iteration from the easyconfig (even
by enabling extra parameters for the custom easyblock file), this
iteration seem to need to happen outside EB. I am now thinking of
simply adding a wrapper script that steps trough all the different
versions we need by modifying the easyconfig file and passing custom
arguments to an easyblock for each step and calling eb in each of these
steps, collecting modules as I go. Is this the way to go or is there
something I am missing?
Thanks again for any comments or suggestions.
Cheers,
Espen
On Mon, 2019-05-06 at 13:58 +0200, Espen wrote:
Dear developers and users,
We are have a core software that need to be compiled with the
following
perturbations:
- additional source files
- changes in the existing files
- additional libraries
The first two can be done together or isolated. The latter is rather
easy to do; using dependencies in the top easyconfig and writing a
custom include file.
However, we have not yet determined the best strategy to do the
additional files and changes in source files.
In total we have about 50 raw perturbations of the same software at a
single toolchain that we need to maintain and deploy. We have defined
a
build matrix in a json file that we read in into our custom easyblock
file. The number of versions that needs to maintain will probably be
even larger as we allow for different versions also of each
perturbation. However, then starts the questions. Here is the first
one:
What is the best strategy to iterate over these perturbations? Right
now we have overloaded the build_step. That works as long as we do
not
need to apply patches (say that one of these perturbations is a
simple
patch). Then it would make sense to overload patch_step, but I am not
sure how to coordinate that with the iteration we have created in
build_step. We have tried to look at the documentation to find
examples
of this, but have not yet managed to locate it. If it exists, can
anyone please point us in the right direction?
Also, since the easyconfig file does not seem to allow generic python
code we would like to load parameters from a json and only make the
easyconfig start the easyblock file. Except for the simple
dependencies, all the iterative work will be done in the easyblock.
Is
this a sensible approach or should we halt immediately?
Thanks in advance for any help. Also, keep up the good work!
Best regards,
Espen
