Ok, will do, currently at Munich for DataSummit, on it when i get back Cheers
Toby On Wed, 5 Apr 2017, 09:27 Andras Piros, <andras.pi...@cloudera.com> wrote: Hi Toby, thanks for your interest on using Oozie! In order I can help you further, can you please: - add *Apache GitHub <https://github.com/apache/oozie>* git hashes of Oozie versions for each HDP version - add a small example of two workflows, one sets a parameter, the other reads the changed parameter - that was working in HDP versions 2.3 and 2.4, but not anymore in 2.5? Regards, Andras On Tue, Apr 4, 2017 at 4:38 PM, Toby Evans <tobysev...@gmail.com> wrote: > Hi there, > > We've got a big Oozie pipeline, which I'm having a little difficulty > upgrading from Hortonworks 2.3 > 2.4 > 2.5 > > HDP Oozie > 2.3 4.2.0.2.3.2.0-2950 > 2.4 4.2.0.2.4.2.0-258 > 2.5 4.2.0.2.5.0.0-1245 > > > So, we ingest and process data from a range of sources, eg databases, FTP, > SQL dbs etc. These are all different, but share so many common features > that we have grown various heavily templated workflows. Each of our > workflows has a set of parameters (eg date range, source type, > hasCustomStep) all that sort of thing, all stored in "profile" workflows. > We first call our "profile" workflow, which sets up the various parameters > and then calls the template workflow, which then fills in the values > supplied from the profile. > > This has run fine for a couple of years in HDP 2.3, seems OK in 2.4, but > definitely doesn't in 2.5. The problem is that once a parameter has been > set in a workflow chain, then it stays set. I understand that you don't > want to be changing a parameter inside the same workflow, and we've > refactored so that we don't have a parameter value passed to a workflow, > which then changes the parameter value then calls another sub-workflow. > However, we're getting caught out by parameters being set by a workflow and > then not being reset much further down the pipeline when used by a > different "profile", which should be isolated but clearly isn't > > Has there been a code change between the versions, and is there a config > setting that allows us to switch off cross-workflow parameter immutability? > > thanks > > Toby >
