Hi Cédrick,
2018-01-21 8:04 GMT-03:00 Cédrick Béler <cdric...@gmail.com>:
> Hi Hernan,
>
> I decided to give a try to Aare or Workflow in Pharo 6, following
> instructions in https://github.com/Netstyle/Workflow .
>
>
> Cool :)
>
> I started but had to stop until the semester is finish. Next month should be
> ok to continue on that.
>
>
> However, the
> installation script raised a Warning about missing WADynamicVariable
> class. Should I install Seaside?
>
>
>
> Happy to see people having interest in such tools :)
>
> No.
>
> I discussed with Max Leske.
>
> To solve this « pb », just create WADynamicVariable as a subclass of
> DynamicVariable already in Pharo. There are also missing part for the
> export/import in XML (should be easy to fix and Max propose to give the code
> - DOMBuilder and co.)
Thanks. I cloned the Ducasse fork which already solved the issue.
> I tried to start a booklet with Stephane but it is too complicated right
> now. Stephane started a simpler version.
>
> I put below a summary of the reflexion/discussion I had with Max.
>
>
> Cheers,
>
> Cédrick
>
>
>
>
> ========= INTRO
>
> What's missing compared to the full (proprietary) solution are user
> interfaces that were web based (Seaside \+ Magritte). Also, persistence was
> achived through Omnibase. References were removed but they are stil some
> disfunctionning. Moreover, the completion of activities were mainly manual
> and achieved through web interface (web forms). They are, for now, no real
> way to introduce different kind of activities like those being
> message-based, event-based, service-based.
>
> To evolve, one has to remove all dependancies to Omnibase and
> Magritte/Seaside.
>
> There are also important design decisions to ensure WfWorkflow being a
> powerful and extensible workflow management system for Pharo. To me there
> are 2 main tasks:
> - rethink persistence (process definition and orchestration, realization)
> => persistance is essential and was central to Aare. Beside removing
> Omnibase references (and implication of WfManagedObject), some of the
> application features were very dependant on the persistence (like logging,
> tracing process evolution). All of these features are to be thing again
> considering as much as possible a loosely coupled solution. A general
> purpose storage solution like Voyage could be used. At first, in image
> storage will be used to focus on the running aspects of the library.
> - rethink interaction
> => Designing a general interaction sub-system is probably the more
> challenging tasks. Aare was web based. We want workflow to be UI agnostic.
> Moreover, we want to associate ,
> remove Magritte (replace UI or build an independant system based on
> Annoucement for instance)
>
It could be me that I am used to execute and wait processes jobs for
hours or days in HPC: typical workflows of NGS trimming, QC, genome
indexing, sequence alignment, etc. But integration of a monitoring
system (like Ganglia) and interaction with a work load manager (like
PBS, Condor, Slurm, etc.) would be also giant tasks.
> ========= RANDOM QUESTIONS AND ANSWER (in black me - below max)
>
> 1)
>
> The static definition of workflow is done through WfWorkflow, WfOutgoingEdge
> and WfSteps (+ WfConditions). The easy part :)
>
> The runtime aspect are achieved via WfActivation, WfWorkList,
> WfWorkflowHistory et WfWorkflowManager. More difficult to understand and
> make evolve.
>
> WfWorkflowHistory represents the different versions of a workflow, so it
> belongs to the static part. WfWorkflowManager managers workflow definitions
> (static) and activations (dynamic).
> WfFrame is also a part of the dynamic side, connecting activations to a
> workflow.
> Workflows are hierarchical, i.e. a step can have a sub workflow (see
> WfWorkflowLibrary>>simpleSubflow)
>
>
>
> 2)
>
> To me activations are really central entities (and I need to be sure of its
> correct role). They are said to the realization of an activity/step.
>
> Can we consider the static definition of a workflow as « Plan » and the
> runtime as the concrete realization (like a job in a scheduler) ?
>
> Yes, I would say so.
>
>
>
> I see that:
>
>
> FUTURE: Static BP définitions with steps (plans)
>
> PRESENT: activation (realization)
>
> PAST: History (experience feedback)
>
> Yes, that's certainly one way to look at it. Although I'd say that the
> static definition is independent from time once it exists and that "future"
> would be the set of steps of the current definition that have not yet been
> activated (just a little nit picking :) ).
>
>
>
> 3)
>
> Going back to activations, here is my current understanding.
>
> Once a step is activated (all condition on Edges leading to are true - to
> evaluate outgoing conditions of a step, the step execution has to be
> complete first) , we get an activation instance (I’m not 100% sure how
> completion of an activity and outgoing conditions are managed).
>
> As far as I understand, a step can always be activated (e.g. manually).
> However, activating a step propagates the values of conditions on the edges
> to the next steps and their activations ("dead" or "alive"). Hence, an
> activation responds to #isAlive with true if at leat one incoming token was
> true ("alive") (see WfStep>>continueInFrame:fromStep:alive: and
> WfActivation>>isAlive). The conditions are evaluated once #complete is sent
> to the activation.
> Activation instances are created by two ways (slightly different for the
> start step), but generally they are created when a step receives tokens from
> the outgoing edges of an upstream activation (see
> WfStep>>continueInFrame:fromStep:alive:).
>
>
> On the exemple of BLActivationHistoryItem that you gave, I’m not sure if
> it’s an activation of a history item stored in the history (subclasss of
> WfHistory?).
>
> My idea was that it should represent a record for an activation (there might
> be multiple records for a single activation). So "HistoryItem" would be the
> general description of a record, of which "ActivationHistoryItem" is a
> special case for activations (I hope that makes sense).
>
>
> 4)
>
> Moreover I don’t see clearly the difference with a scheduler and a BP engine
> (they are at least really complementary). Do you see any ressemblance
> between Activation and ScheduledJob ?
>
> I actually would like to have a (simple) scheduler with the BP engine :)
>
> Well, it's certainly a matter of point of view. I would argue that an
> activation is something close to a "job". A "scheduled job" would be a more
> specific case of a "job", a job that is in a queue. An activation has no
> notion of scheduling but simply represents the idea of what happens when
> traversal between two nodes occurs. A scheduler would be one way of managing
> activations (WfWorklist seems pretty close to a scheduler, holding running
> and completed activations).
>
>
> To me the scheduler right now is not really existing besides having a list
> of simultaneous/parralelized steps to realize (activated or not). Activated
> means currently being done.
>
> You're right, I think. The application that Workflow was designed for
> required manual interaction, hence there was no need for other methods of
> scheduling (e.g. by time or event).
>
>
>
> This activation role is about logging and interacting with responsible
> users. But it is more than that. It’s the actual job being done.
>
> Yes.
>
>
>
> 5)
>
> Concerning run time modification of a process. Is it true that we can modify
> a currently executed workflow (steps that are not yet activated ?) ? I think
> that is possible (an cool). When activated, it shouldn’t. Would you think it
> would be interesting to allow activations to be suspended/resume (frozen),
> cancelled ?
>
> The history of workflows consists of copies. The steps of a workflow
> reference the workflow they belong to. Therefore, while it is possible to
> modify a workflow, modifications do not impact running workflows, as the
> steps of that workflow continue to reference the previous version. The model
> suggests that it is ok to create a new version of a workflow but not to
> modify existing workflows.
> The model does *allow* (in principle) to modify an existing workflow but I
> doubt that was the intention. However, I don't see why modifying an existing
> workflow has to be bad thing, as long as you can guarantee that only edges
> and steps are modified that have not yet been activated. That could open up
> some interesting possibilities.
> Cancellation of an activation could be very interesting. It would basically
> allow reversal of complex effects. Very nice idea! I'm not so sure though
> whether suspending makes much sense. Maybe when a step has the potential to
> take a long time for some operation. But then you may run into transaction
> problems. The nice thing about an activation at the moment is that it's
> atomic (in theory). But again, done right that could open up interesting
> possibilities.
>
>
>
> If a running process is modified, do we track its evolutions in
> WfWorkflowManager ?
>
> The modification of a workflow is indeed announced to WfWorkflowManager,
> yes. Workflow versions are stored in WfWorkflowHistory. For modifying a
> *running* workflow you will probably want to create a specific workflow
> manager for that, that does validation etc.
>
>
>
> The goal of history(s) are to log the realization of the process. But is it
> one of its aim to « replay » the process execution ?
>
> No. WfWorkflowHistory only stores the static definitions. However, the
> history of activations could be used to replay an execution (nice idea!).
>
> i.o.w., Is it more than log/report ?
>
> I think the activation history was designed as a log, the workflow history
> as a log and also as a way to undo changes. As you suggest, however, they
> could certainly be used for more things.
>
>
> I agree with the separation (WfWFManager for the run time evolution of the
> process definition => adaptation and WfActivationHistory for the trace of
> interactive/automatic step realizations information).
>
>
> #beginHistory implementation example was very helpful. I knew I missed
> something Just, as I see a MailManager (with role is obvious), is it
> something that has to be generalizable ? Like InteroperableManager ?
> MessageManager ? To get user interactions
>
> That would be very nice, yes. You're completely free to do what you like. It
> would be cool though if we could evolve the project as opposed to you
> creating a fork to your own liking. If you're game we can set up a workflow
> on github where you open pull requests against our repository.
>
>
>
> 6) I’d like to remove fully the reference and need of Omnibase and
> Magritte/seaside forms.
>
> Yes please!
>
>
> I think Omnibase is ok, even storing in image for instance.
>
> What’s more problematic to me is for user interactions (and even automatic
> interaction through program for instance). There are achieved in your
> example with seaside. It is ok but what if I’d like to do it more generally
> ? Would we use Annoucement for example ?
>
> I think that is completely open to implementation. In my eyes, Workflow
> should remain a separate library that does not enforce any kind of specific
> presentation or persistence. What's critical for that is of course the right
> kind of interface.
> Announcements would certainly be an interesting way to decouple Workflow
> from other parts of the system.
>
>
> I’d like to have an incomingMessageBox for each user and eventually
> transform these message in concrete action (user i/o) ? Or dispatch in
> JobQueues (each user being a distant worker thread). Again, the distinction
> between Scheduler and BPEngin is fuzzy to me.
>
>
> Does it make sense to you ?
>
> Yes, it does :) What I've asked myself (out of curiosity) is whether you
> want to have
> - a dedicated instance for executing workflows
> - a distributed workflow with autonomous synchronisation (parts of a
> workflow could run on every uses instance, others would need to be executed
> through agreement)
> - a partially distributed workflow with one or more managing nodes that
> control the core steps (e.g. you could designate parts of a workflow as
> independant)
>
>
> PS: some random comments on your proposals
>
> Parts of Workflow were never cleanly separated from the proprietary
> application, so there are indeed a couple of things that are missing (mostly
> #subclassResponsibility stuff etc.)
>
>
>
> - initializeDefaults is not present for steps but thats ok.
>
>
> - in #createIncoming: (doesn’t exist), I don’t get ….. => makePersistent
> asPersistentReference. You store persistentReference of steps. Is it like
> the step object itself ? I would store the step objects (as no persistance
> right now).
>
> An activity step is a specific subclass of a step. Such a step includes
> additional information such as documents, duration and user roles. I'm not
> sure why "makePersistent asPersistentReference" is sent there, as WfStep
> already inherits from WfManagedObject and would, therefore, already be
> managed by OmniBase. But I don't know OmniBase, maybe some things need to be
> announced specifically.
>
>
> - I don’t understand the offsetSeconds in your BLActivationHistoryItem
>
> That's really just some business rule to modify the timestamp of the history
> item.
>
>
> ============= Others responses from Max
>
> WfWorkflow - Load and Run
> All instances that inherit from WfManagedObject were managed by OmniBase,
> yes.
> DomBuilder is a small package for wrapping XML-Parser. I'll publish it so
> you can use it. The XML-Parser package has been significantly modified and
> is now named XMLParser (you can find it in the catalog). You should make the
> changes necessary to use the current incarnation of XMLParser (decide for
> yourself whether you still need DomBuilder or not).
>
>
> Those IDXXX classes belong to another package a didn't publish (but will).
> This package also includes an OmniBase specific class.
> Magritte has changed significantly over the years but the core should be
> pretty much the same. Just load the newest Magritte and try working with
> that.
>> ok but avoidable ?
>> try loading Magritte ? Long …
>
> #beginInHistory should be implemented with #subclassResponsibility on
> WfActivation.
> Your explicit implementation could then look something like this:
> beginInHistory
> self history add: (MyActivationHistoryItem new
> message: 'started';
> user: (WACurrentSession value
> ifNotNilDo: [ :session | session user ]);
> type: #started;
> yourself).
> MyMailManager noteNewActivation: self
>
> Basically, what the history contains is up to the workflow implementation. I
> think it would makes sense though to have an abstract
> WfActivationHistoryItem.
> WfActivation>>history should be implemented like this:
> history
> ^self propertyAt: #history ifAbsentPut: [ OrderedCollection new ].
>
> Although I think that it would be good to have a dedicated
> WfActivationHistory class. WfWorkflowHistory is associated with changes to
> the workflow, not activations. Both history classes should probably inherit
> from an abstract class WFAbstractHistory.
>
>
> Seaside interaction
>
> Here are a couple of action implementations in Seaside components that
> should give you an idea of how to interact with the workflow:
> MyActivationComponent>>complete
> self activation hasCompleted
> ifTrue: [ ^ self ].
> self history add: (BLActivationHistoryItem new
> message: 'completed';
> user: self session user;
> type: #completed;
> offsetSeconds: -1;
> yourself).
> self activation complete.
>
> MyActivationForm>>accept: aString
> self activation authorized: true.
> self history add: (BLActivationHistoryItem new
> comment: aString;
> message: 'accepted';
> user: self session user;
> type: #accepted;
> offsetSeconds: -2;
> yourself).
> self completeIfSatisfied.
>
> MyActivityList>>addActivity
> self call: (MyActivityEditor new
> workflow: self workflow;
> yourself).
> self refresh.
>
> MyActivityEditor>>buildActivity
> ^(self workflow newActivityStep
> initializeDefaults;
> yourself)
> asPersistentReference.
>
> MyTransitionEditor>>createIncoming
> | step |
> step := (self workflow newActivityStep
> initializeDefaults;
> yourself) makePersistent asPersistentReference.
> self addIncoming: step.
> MyCurrentWorkflowComponent value showActivity: step.
>
> MyTransitionEditor>>addIncoming: anActivity
> ^self add: anActivity to: self activity.
>
> MyTransitionEditor>>add: aSourceActivity to: aDestinationActivity
> ^aSourceActivity addOutgoingEdgeFor: aDestinationActivity.
>
> MyTransitionEditor>>editCondtions: anEdge
> | component |
> anEdge condition isTrueCondition
> ifTrue: [ anEdge condition: WfAllCondition new ].
> component := (MyConditionEditor activity: self activity)
> container: anEdge;
> addDecoration: (MyToolbarDecoration new
> title: 'Edit Transition Conditions to: ' , anEdge to name;
> toolbar: [ :html |
> html
> icon: 'icon_close.gif'
> text: 'Close'
> action: [ component answer ].
> html helpIconOn: component ];
> yourself);
> yourself.
> self call: component.
>
> MyConditionEditor>>addRule: aDescription
> | new |
> new := MyCondition newPersistent
> field: aDescription;
> asPersistentReference.
> new := self call: (self
> buildEditorFor: new
> titled: 'Add Rule for ' , aDescription asLabel).
> new isNil ifFalse: [
> self condition add: new; markDirty.
> self changedForm ].
>
>
>> WfWorflowManager class is empty defining empty stubs/protocols. Its aim
>> seems to be about “logging/noting” changes. Is is it’s only role ?
> Correct. You would use WfWorkflowManager to update the workflow in OmniBase
> (e.g. mark dirty, add new activation etc.). #generateID would generate an ID
> from OmniBase and #activationClass would answer your specific activation
> class, if you choose to override WfActivation.
>
Cool, thank you both for including your very valuable comments!
I did my first experiment and have a script to start:
| wf task1 task2 startStep frame |
wf := WfWorkflowLibrary default workflowNamed: 'Test1'.
task1 := wf newStepNamed: 'Step 1'.
task2 := wf newStepNamed: 'Step 2'.
startStep := wf start.
startStep addOutgoingEdgeFor: task1.
task1 addOutgoingEdgeFor: task2.
frame := wf executeInNewFrame.
frame workList running first complete.
frame workList runningSteps.
frame workList running first complete.
frame workList runningSteps.
wf
To see how it matches with scientific workflows I will need more experiments.
Cheers,
Hernán
