Thanks Jonah - I will digest this over coffee...
On 5 March 2015 at 16:45, Jonah Benton <jo...@jonah.com> wrote:
> Hi Colin- a simple example that uses a flow operator would be
>
> (-> context
> do-something
> do-some-io
> do-a-last-thing )
>
> with
>
> (defn do-something [context]
> (let [some-data (get-in context [:path :to :data :element])
> result (pure-edge-function some-data)]
> (assoc-in context [:path :to :revised :data] result)))
>
> and
>
> (defn do-some-io [context]
> (let [some-other-data (get-in context [:path :to :data :to :save])
> database (get-in context [:path :to :database :resource])]
> (save! database some-other-data)
> context)))
>
> Then save! can be a plain function, or
>
> (defprotocol Saver
> (save! [this data]))
>
> (defrecord DatabaseSaver [conn ... ]
> Saver
> (save! ))
>
> (defrecord AtomSaver [a ... ]
> Saver
> (save! ))
>
> And context is a map of maps that contains both application state and edge
> resources that should provide or receive data.
>
> Encoding conditional steps in a data flow presents a challenge. Several
> variants for doing this are in flow operators in Clojure and there are also
> a number of third party approaches. The below is a concrete example from an
> internal library that is not open source but doesn't differ significantly
> from approaches that are.
>
> A "flow" looks like this:
>
> (defflow renew
> { :start valid-request?
> valid-request? { true valid-token? false invalid-request }
> valid-token? { true ingest-token false not-authorized }
> ingest-token renew-session
> renew-session renewed?
> renewed? { true return-renewed-token false not-authorized
> }
> return-renewed-token :end
> not-authorized :end
> invalid-request :end })
>
> It's a regular map with at least a :start key and an :end value. Symbols in
> a flow map are plain functions that take a context map.
>
> Semantically, those functions represent either actions or decisions. A
> function whose value in the flow map is itself a map with true and false
> keys is a decision. A function whose value in the map is another function is
> an action.
>
> This particular flow performs a renewal of a web-like "session" token. It
> runs in a Pedestal interceptor, so it receives a context map (with :request,
> :response and other keys) as a parameter and at the end it populates the
> :response key of the map.
>
> The way to read this is-
>
> 0. a context map is prepared.
> 1. the data flow runner looks for the :start key. the value of :start is
> valid-request?
> 2. The value of valid-request? is a map, rather than a function, so the flow
> determines valid-request? is a decision function- an ordinary function that
> takes a context and is expected to return true or false.
> 3. the flow runner executes valid-request? If it returns true, the flow
> runner proceeds to valid-token? using the same context map. If false,
> proceed to invalid-request
> 4. valid-token? is similarly a decision function, it's executed similarly
> and is also expected to return true or false. invalid-request is an action.
> It's expected to return a new context map- in this case it populates the
> :response with a 4xx :status and some other response attributes.
> 5. the flow engine keeps going between keys and values until it comes to a
> value of :end. Then the resulting context is the output of the flow.
>
> The action function in the above called "renew-session" can do IO to renew a
> user session in the database. It's copied below.
>
> (defn renew-session [context]
> (let [database (get-in context [:components :database])
> session-key (get-in context [:work :session-key])
> renewed? (db/renew-session! database session-key)]
> (if (not (nil? renewed?))
> (assoc-in context [:work :renewed-key] session-key)
> context)))
>
> It simply pulls the database connection from the context, calls into a db
> specific function, db/renew-session!, which takes a connection and a key,
> and could be a protocol function that accepts a Record that encapsulates a
> specific storage resource, or could be a multimethod.
>
> This model of a data flow with actions and decisions follows Amazon's Simple
> Workflow Service, and the clever encoding of that flow as a map is due to
> Cognitect.
>
> A few projects exist now that encode flows in data form with an engine to
> process them. There are some trade-offs- it takes practice to read them, you
> can trade off a little in performance. In the above there's a fair amount of
> boilerplate that could be abstracted away along the lines of of Prismatic's
> Graph.
>
> You get a lot of other benefits- a consistent system shape, separation of
> concerns between core logic and edge IO, consistent error handling,
> logging/auditing, etc. Flows can be reused and can themselves compose,
> because everything takes a context map. You could also generate a visual
> decision tree of the model, or a real time heat map of decisions or of
> action performance.
>
> Anyway- sorry, this isn't intended to pitch an internal library, rather to
> share another thought process around the problems you raised that points in
> the direction of modeling and then executing the data flow, rather than
> relying solely on the call stack.
>
>
>
> On Thu, Mar 5, 2015 at 9:39 AM, Colin Yates <colin.ya...@gmail.com> wrote:
>>
>> Sounds interesting - are there are instances that I can look at?
>>
>> On 5 March 2015 at 14:15, Jonah Benton <jo...@jonah.com> wrote:
>> > Yes, exactly.
>> >
>> > That's fine, a node in the "middle" of the data flow can be responsible
>> > for
>> > reaching out to the edge to synchronously get some data to feed into a
>> > decision performed by the data flow. It would just do that through the
>> > edge
>> > functions.
>> >
>> > A request for data could also be asynchronously initiated once enough
>> > context had been determined by the data flow, and a promise added to the
>> > context map for later derefing (likely with a timeout).
>> >
>> > If the request for the data is context-free, it could be pulled in at
>> > the
>> > start of the data flow.
>> >
>> > By edge, it's is not that IO can't be done in the middle of a dataflow,
>> > it's
>> > just that functions that deal with dataflow follow one pattern (taking a
>> > context map) so they can be easily composed and rearranged, while
>> > functions
>> > that deal with edge resources take those resources along with other
>> > parameters more explicitly, so the data flow can run in an insulated
>> > fashion.
>> >
>> >
>> >
>> > On Thu, Mar 5, 2015 at 8:55 AM, Colin Yates <colin.ya...@gmail.com>
>> > wrote:
>> >>
>> >> Hi Jonah,
>> >>
>> >> This sounds very much like the model layer in DDD, which ironically is
>> >> exactly what I am building.
>> >>
>> >> However, in the middle of a data flow a function needs to reach out to
>> >> a repository to make a decision - how does that fit in with the data
>> >> flow approach?
>> >>
>> >>
>> >> On 5 March 2015 at 13:39, Jonah Benton <jo...@jonah.com> wrote:
>> >> > Hi Colin,
>> >> >
>> >> > Another option, other than HOFs or dynamic binding, is what might be
>> >> > called
>> >> > a data flow approach.
>> >> >
>> >> > At the edge of the application are the functions that explicitly take
>> >> > parameterized resources to perform edge state IO. These might be bare
>> >> > functions, or they might be protocol implementations that await the
>> >> > delivery
>> >> > of a type to perform their operation, or multimethods.
>> >> >
>> >> > At the center are functions that take a context map, which contains
>> >> > all
>> >> > relevant application state- IO resources and transient state data.
>> >> > Those
>> >> > functions are arranged in a data flow and capture the logic or state
>> >> > transitions of the application independent of any specific IO
>> >> > commitments.
>> >> > They can use schema or type annotation or pre/post conditions to
>> >> > enforce
>> >> > invariants.
>> >> >
>> >> > When data flow processing arrives at a place where some edge IO
>> >> > should
>> >> > occur, these data flow functions act as adapters to get-in the
>> >> > appropriate
>> >> > resource or Record from the context map and call the edge functions
>> >> > to
>> >> > perform IO.
>> >> >
>> >> > The result is a wider, flatter system, propagating state explicitly
>> >> > rather
>> >> > than implicitly through the call stack.
>> >> >
>> >> > Jonah
>> >> >
>> >> >
>> >> >
>> >> > On Wed, Mar 4, 2015 at 12:58 PM, Colin Yates <colin.ya...@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> I am looking for the Clojure equivalent of:
>> >> >>
>> >> >> class Whatever {
>> >> >> @Transactional
>> >> >> void doSomething(IDoSomething one, IDoSomethingElse two) {
>> >> >> one.doSomething()
>> >> >> two.doSomething()
>> >> >> }
>> >> >> }
>> >> >>
>> >> >> where both one and two are dependency injected with a proxy which
>> >> >> resolves
>> >> >> to a thread local database connection. In addition, one might itself
>> >> >> have a
>> >> >> collaborator which itself has a collaborator which needs a
>> >> >> datasource.
>> >> >>
>> >> >> So far I have two protocols:
>> >> >>
>> >> >> (defprotocol IDoSomething
>> >> >> (do-something [this ...])
>> >> >>
>> >> >> (defprotocol IDoSomethingElse
>> >> >> (do-something [this ...])
>> >> >>
>> >> >> Each protocol may have a number of implementations, one of which is
>> >> >> a
>> >> >> JDBC
>> >> >> implementation:
>> >> >>
>> >> >> (defrecord JdbcIDoSomething [db]
>> >> >> (do-something [this ...] ...))
>> >> >>
>> >> >> The problem is that the calling code only gets provided an
>> >> >> IDoSomething
>> >> >> and an IDoSomethingElse and it wants to do something like:
>> >> >>
>> >> >> (let [one (->JdbcDoSomething db) two (->JdbcDoSomethingElse db)]
>> >> >> (with-transaction [tx db]
>> >> >> (do-something one)
>> >> >> (do-something-else two)))
>> >> >>
>> >> >> The problem here is that the implementations of do-something and
>> >> >> do-something-else won't have access to the local bound 'tx', they
>> >> >> will
>> >> >> have
>> >> >> their own 'db'.
>> >> >>
>> >> >> I realise the general argument is to be explicit and pass a db as
>> >> >> the
>> >> >> first argument to the protocol but this isn't appropriate in this
>> >> >> case
>> >> >> as
>> >> >> there are validly multiple implementations. I could abstract a
>> >> >> 'unit-of-work' and pass that as the first argument to the protocols
>> >> >> but
>> >> >> that
>> >> >> seems a bit painful.
>> >> >>
>> >> >> Also, these protocols may be used quite far away from where the
>> >> >> database
>> >> >> code lives and passing a parameter all the way through the call
>> >> >> stack
>> >> >> is
>> >> >> painful.
>> >> >>
>> >> >> I am using Stuart Sierra's components if that makes any difference.
>> >> >>
>> >> >> I can't be the first person to run into this but google is
>> >> >> surprisingly
>> >> >> unhelpful which makes me think I have missed something fundamental,
>> >> >> and
>> >> >> that
>> >> >> I have something upside down.
>> >> >>
>> >> >> What do you all do?
>> >> >>
>> >> >>
>> >> >> --
>> >> >> You received this message because you are subscribed to the Google
>> >> >> Groups "Clojure" group.
>> >> >> To post to this group, send email to clojure@googlegroups.com
>> >> >> Note that posts from new members are moderated - please be patient
>> >> >> with
>> >> >> your first post.
>> >> >> To unsubscribe from this group, send email to
>> >> >> clojure+unsubscr...@googlegroups.com
>> >> >> For more options, visit this group at
>> >> >> http://groups.google.com/group/clojure?hl=en
>> >> >> ---
>> >> >> You received this message because you are subscribed to the Google
>> >> >> Groups
>> >> >> "Clojure" group.
>> >> >> To unsubscribe from this group and stop receiving emails from it,
>> >> >> send
>> >> >> an
>> >> >> email to clojure+unsubscr...@googlegroups.com.
>> >> >> For more options, visit https://groups.google.com/d/optout.
>> >> >
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups "Clojure" group.
>> >> > To post to this group, send email to clojure@googlegroups.com
>> >> > Note that posts from new members are moderated - please be patient
>> >> > with
>> >> > your
>> >> > first post.
>> >> > To unsubscribe from this group, send email to
>> >> > clojure+unsubscr...@googlegroups.com
>> >> > For more options, visit this group at
>> >> > http://groups.google.com/group/clojure?hl=en
>> >> > ---
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "Clojure" group.
>> >> > To unsubscribe from this group and stop receiving emails from it,
>> >> > send
>> >> > an
>> >> > email to clojure+unsubscr...@googlegroups.com.
>> >> > For more options, visit https://groups.google.com/d/optout.
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups "Clojure" group.
>> >> To post to this group, send email to clojure@googlegroups.com
>> >> Note that posts from new members are moderated - please be patient with
>> >> your first post.
>> >> To unsubscribe from this group, send email to
>> >> clojure+unsubscr...@googlegroups.com
>> >> For more options, visit this group at
>> >> http://groups.google.com/group/clojure?hl=en
>> >> ---
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "Clojure" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send
>> >> an
>> >> email to clojure+unsubscr...@googlegroups.com.
>> >> For more options, visit https://groups.google.com/d/optout.
>> >
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups "Clojure" group.
>> > To post to this group, send email to clojure@googlegroups.com
>> > Note that posts from new members are moderated - please be patient with
>> > your
>> > first post.
>> > To unsubscribe from this group, send email to
>> > clojure+unsubscr...@googlegroups.com
>> > For more options, visit this group at
>> > http://groups.google.com/group/clojure?hl=en
>> > ---
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Clojure" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to clojure+unsubscr...@googlegroups.com.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
