Hi Jose,
I think you should try making the core iteration purely functional,
meaning no agents, atoms, or side effecting functions like the random
generator.
I assume the number of steps you evolve the particle is encoded in
step-extract-fn?
What you probably want is something like
(loop [i 0 pos initial-position]
(if (< i num-of-steps)
(recur (+ i 1) (move pos)) ;; iterate
pos ;; if done, return final position
))
This will make it easier to benchmark the iteration step, which is an
important number to know. I'm sure you can make it much faster, if
perf is the ultimate goal its worth tuning a little.
In terms of distributing the work, I would not use atoms or agents.
They are not meant for parallelism or for work queues. With agents and
futures you need to be aware of the various thread pools involved
under the hood and make sure you are not saturating them. And combined
with laziness, it takes care to ensure work is getting done where you
are expecting it.
It would be easier to reason about what is going on by using threads
and queues directly. Enqueue a bunch of work on a queue, and directly
set up a bunch of threads that read batches of work from the queue
until its empty.
If the initial condition / other parameters are the same across
workers, you could even skip the work queue, and just set up a bunch
of threads that just do the iterations and then dump their result
somewhere.
I also definitely recommend making friends with loop/recur:
(time
(loop [i 0]
(if (< i 1000000)
(recur (+ 1 i))
true)))
=> "Elapsed time: 2.441 msecs"
(def i (atom 0))
(time
(while (< @i 1000000)
(swap! i + 1)))
=> "Elapsed time: 52.767 msecs"
loop/recur is both simpler and faster, and the best way to rapidly iterate.
On Mon, Nov 18, 2013 at 7:47 PM, Jose M. Perez Sanchez
<m...@josemperez.com> wrote:
> Hi Andy, cej38, kovas:
>
> Thanks for the replies. I plan to release the whole code soon (waiting for
> institutional authorization).
>
> I do use lazyness both within the move function to select the allowed random
> displacements and when iterating the move function to generate the
> trajectory. Lazy structures are only consumed within the thread in which
> they are created.
>
> Here is the core code where the computations happens:
>
> (defn step-particle
> "Returns a new value for particle after moving particle once to a new
> position from the current one"
> [pdf-get-fn step-condition-fn calc-value-fns particle]
> (let [pos (particle :pos)
> disp (first (filter (fn [x] (step-condition-fn (particle :pos) x))
> (repeatedly (fn [] (pdf-get-fn)))))
> new-pos (mapv + pos disp)
> new-base-particle {:pos new-pos :steps (inc (particle :steps))}
> new-trackers-results (if (seq calc-value-fns)
> (zipmap
> (keys calc-value-fns)
> ((apply juxt (vals calc-value-fns))
> particle new-base-particle))
> {})]
> (merge new-trackers-results new-base-particle)))
>
> (defn walk-particles
> "While there is work to do, create new particles, move them n-steps, then
> send them to particle container (agent)"
> [todo particles simul-info init-get-fn init-condition step-get-fn
> step-condition trackers-maps step-extract-fn]
> (let [init-value-fns (zipmap
> (keys trackers-maps)
> (map :create-fn (vals trackers-maps)))
> calc-value-fns (zipmap
> (keys trackers-maps)
> (map :compute-fn (vals trackers-maps)))
> move (partial step-particle step-get-fn step-condition
> calc-value-fns)]
>
> (while (> @todo 0)
> (swap! todo dec)
> (let [p (last (create-particle init-get-fn init-condition
> init-value-fns))
> lazy-steps (iterate move p)
> result (step-extract-fn lazy-steps)]
> (send-off particles (fn [x] (conj x result)))
> ))))
>
> Each worker is created launching a future that executes walk-particles, each
> worker has a separate Mersenne Twister random number generator embedded into
> the pdf-get-fn (using partial on a common pdf-get function and different MT
> generators). In real calculations both number of particles and steps are at
> least 1e4. In the benchmarks I'm posting particles are 1000, steps are 5000.
>
> As expected conjoining to a single global vector poses no problem, I tested
> both conjoining to a single global vector and to separate global vectors
> (one per worker) and the computing time is the same.
>
> I could test in another system with 16 cores. See the results attached for
> the 8 and 16 core systems.
>
> Best,
>
> Jose.
>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
