pmap is rarely actually useful, but point 1 is false, pmap doesn't require
that it's input or output fit in memory
On Fri, Jun 16, 2017 at 12:52 PM Tom Connors <t.v.conn...@gmail.com> wrote:
> Hello Jose,
> Thank you for the response, but pmap does not address my use case. It's
> insufficient for two reasons: 1) the entire collection must fit in memory.
> My use case is handling records from a Kinesis stream. and 2) pmap
> parallelizes over the whole collection, whereas I want to parallelize the
> collection handling while handling subsets of the data sequentially, as I
> discussed in my first post.
> - Tom
>
> On Friday, June 16, 2017 at 10:13:11 AM UTC-4, Tom Connors wrote:
>>
>> I'm looking for a function that would likely be named something like
>> "sequential-by" or "parallel-per" that takes some data-producing thing like
>> a lazy seq or a core async channel, a function to split records from that
>> input, and a function to handle each item. Each item with an identical
>> return value from the "split" function would be handled sequentially, while
>> the handling of the collection as a whole would be parallel.
>>
>> If we assume this signature:
>> (parallel-per splitter handler inputs)
>>
>>
>> Calling it like this:
>> (parallel-per :id
>> (fn [m] (prn (update m :val inc)))
>> [{:id 1 :val 1}, {:id 1 :val 2}, {:id 3 :val 1}])
>>
>>
>> Would result in the first two maps being handled sequentially, while the
>> third map is handled in parallel with the first two. The order of the
>> printed lines would probably be non-deterministic, except {:id 1 :val 2}
>> would be printed before {:id 1 :val 3}.
>>
>> Note that for my use case I don't care about return values, but if
>> something like this already exists it's plausible that it returns something
>> equivalent to (map handler inputs).
>>
>> Does anything like this already exist in core or some lib? If not, any
>> recommendations for how to build it?
>>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
