Hello Jose,
Thank you for the response, but pmap does not address my use case. It's
insufficient for two reasons: 1) the entire collection must fit in memory.
My use case is handling records from a Kinesis stream. and 2) pmap
parallelizes over the whole collection, whereas I want to parallelize the
collection handling while handling subsets of the data sequentially, as I
discussed in my first post.
- Tom
On Friday, June 16, 2017 at 10:13:11 AM UTC-4, Tom Connors wrote:
>
> I'm looking for a function that would likely be named something like
> "sequential-by" or "parallel-per" that takes some data-producing thing like
> a lazy seq or a core async channel, a function to split records from that
> input, and a function to handle each item. Each item with an identical
> return value from the "split" function would be handled sequentially, while
> the handling of the collection as a whole would be parallel.
>
> If we assume this signature:
> (parallel-per splitter handler inputs)
>
>
> Calling it like this:
> (parallel-per :id
> (fn [m] (prn (update m :val inc)))
> [{:id 1 :val 1}, {:id 1 :val 2}, {:id 3 :val 1}])
>
>
> Would result in the first two maps being handled sequentially, while the
> third map is handled in parallel with the first two. The order of the
> printed lines would probably be non-deterministic, except {:id 1 :val 2}
> would be printed before {:id 1 :val 3}.
>
> Note that for my use case I don't care about return values, but if
> something like this already exists it's plausible that it returns something
> equivalent to (map handler inputs).
>
> Does anything like this already exist in core or some lib? If not, any
> recommendations for how to build it?
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
