On Fri, Dec 17, 2010 at 5:39 PM, clj123 <ariela2...@gmail.com> wrote:
> (defn persist-rows
> [headers rows id]
> (let [mrows (transform-rows rows id)]
> (with-db *db* (try
> (apply insert-into-table
> :my-table
> [:col1 :col2 :col3]
> mrows)))
> nil ))
>
> (defn filter-data
> [rows item-id header id]
> (persist-rows
> header
> (filter #(= (:item_id %) item-id) rows)
> id))
>
> (dorun (pmap #(filter-data rows %1 header %2)
> items id ))
Rows gets traversed repeatedly, for each item/id pair. So the head
gets held onto.
To make this scale you're going to need to regenerate the rows seq for
each traversal; you need something like
(doseq [[item-id id] (map vector items ids)]
(let [rows (generate-rows-however)]
(filter-data rows item-id header id)))
That's not parallel, but I don't know that parallel buys you much when
the task is I/O bound and takes however long it takes the database
server to process that number of requests.
If parallelism really does buy you something here you could replace
(doseq ...) with (dorun (pmap identity (for ...))) and that might
parallelize the realization of the items, and thus all of the nested
operations.
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
