Hi, Watching this thread and trying to take the pulse of GWL.
Where should I look? https://git.roelj.com/guix/gwl has little documentation - it does say " GWL has a built-in getting-started guide. To use it, run: guix workflow --web-interface" - but supposing we just want to read some documentation https://www.guixwl.org/ is 503 Workflow management with GNU Guix https://archive.fosdem.org/2017/schedule/event/guixworkflowmanagement/ is interesting but not documentation Can someone please catch me up? Thx, ~malcolm_c...@stowers.org > -----Original Message----- > From: Guix-devel [mailto:guix-devel-bounces+mec=stowers....@gnu.org] > On Behalf Of Roel Janssen > Sent: Thursday, January 25, 2018 4:05 PM > To: zimoun <zimon.touto...@gmail.com> > Cc: guix-devel@gnu.org > Subject: Re: Guix Workflow Language ? > > > zimoun writes: > > > Dear Roel, > > > > Thank you for your comments. > > > > I was imaging your point 2. And the softwares come from Guix. > > The added benefit was: a controlled and reproducible environment. > > In other words, the added benefit came from the GuixWorkflow (the > > engine of workflow), and not from the Language (lisp EDSL). > > But maybe it is a wrong way. > > I get that point. Maybe it's then a better idea to write the workflow > in CWL (like you would do), and use Guix to generate Docker containers. > > Then you do get the benefit of Guix's strong reproducibility and > composability forscientific software, plus you get to keep writing the > workflow in CWL. :-) > > > > >>From my experience, the classical strategy of writing pipelines is to > > adapt an already existing workflow for one another particular > > question. We fetch bits here and there, do some ugly and dirty hacks > > to have some results; then depending on them, a cleaner pipeline is > > written (or not! :-) or other pieces are tested. > > Again from my experience, there is (at least) 3 issues: the number of > > tools to learn and know enough to be able to adapt; the bits/pieces > > already available; the environment/dependencies and how they are > > managed. > > > > In this context, since 'lispy' syntax is not mainstream (and will > > never be), it appears to me as a hard position. That's why I asked if > > a Guix-backend workflow engine for CWL specs is doable. Run CWL specs > > workflow on the top of the GWL engine. > > This is a good question, but how can you describe the origin of a > software package in CWL? In the GWL, we use the Scheme symbols, and > the > Guix programming interface directly, but that is unavailable in CWL. > > This is a real problem that I don't see we can easily solve. > > > > > > However, I got your point, I guess. > > You mean: it is a lot of work with unclear benefits over existing engines. > > So, I think it's impossible to express the deployment of a software > program in CWL. It is not as expressive as GWL in this regard. > Translating to a precise Guix package recipe and its dependencies is > very hard from what we can write in CWL. > > If I am mistaken here, please let me know. Maybe we can figure > something out. > > > > > > > Therefore, your point 1. reverses "my issue". > > Once the pipeline is well-established, write it with GWL! :-) > > Next, if it is possible to convert this GWL specs pipeline to CWL one > > [+ Docker] (with softwares coming from Guix), then we can enjoy the > > CWL-world engine capabilities. > > The benefit of that is from two sides: run the pipeline with different > > engines; and produce a clean docker image. > > > > So , instead of working on improving the GWL engine (adding features > > about efficiency, Grid, Amazon, etc.) which is a very tough task, the > > doable plan would be to add an "exporter". > > Right ? > > The plan is to implement back-ends, or 'process-engines' for GWL to work > with AWS, Kubernetes, Grid (this one is already supported). > > These back-ends are surprisingly easy to write, because the Guix > programming interface allows us to generate virtual machines, > containers, or simply store items if Guix is available locally. > > We also implemented a Bash-engine that can generate Bash scripts for > every step of the workflow. That in combination with the variety of > deployment options solves most of the challenges. > > > > > > > Another question, do you think it is doable to write "importers" ? > > > > I am not sure that the metaphor is good enough, but do you think it is > > a feasible goal from the existing GWL to go towards a kind of `Pandoc > > of workflows` ? also packing the softwares. > > > > And a start should be: > > - write a parser for (subset of) CWL yaml file and obtain the GWL > > representation of the workflow > > - write a exporter to CWL + Docker image > > > > What do you think ? > > Maybe. But in CWL we cannot describe precise software packages. So > translating these things to Guix is hard. > > > > > > > About the parser, I haven't found yet an easy-to-use Guile lib for > > parsing YAML-like files. Any pointer ? Adapt some Racket ones ? > > I don't know of one, sorry. > > > > Thank you for your insights. > > > > All the best, > > simon > > Thanks! > > Kind regards, > Roel Janssen
