Thanks Justin. My mistake. Point 2 stands.
On Friday, June 16, 2017 at 3:58:38 PM UTC-4, Justin Smith wrote:
>
> pmap is rarely actually useful, but point 1 is false, pmap doesn't require
> that it's input or output fit in memory
>
> On Fri, Jun 16, 2017 at 12:52 PM Tom Connors <t.v.c...@gmail.com
> <javascript:>> wrote:
>
>> Hello Jose,
>> Thank you for the response, but pmap does not address my use case. It's
>> insufficient for two reasons: 1) the entire collection must fit in memory.
>> My use case is handling records from a Kinesis stream. and 2) pmap
>> parallelizes over the whole collection, whereas I want to parallelize the
>> collection handling while handling subsets of the data sequentially, as I
>> discussed in my first post.
>> - Tom
>>
>> On Friday, June 16, 2017 at 10:13:11 AM UTC-4, Tom Connors wrote:
>>>
>>> I'm looking for a function that would likely be named something like
>>> "sequential-by" or "parallel-per" that takes some data-producing thing like
>>> a lazy seq or a core async channel, a function to split records from that
>>> input, and a function to handle each item. Each item with an identical
>>> return value from the "split" function would be handled sequentially, while
>>> the handling of the collection as a whole would be parallel.
>>>
>>> If we assume this signature:
>>> (parallel-per splitter handler inputs)
>>>
>>>
>>> Calling it like this:
>>> (parallel-per :id
>>> (fn [m] (prn (update m :val inc)))
>>> [{:id 1 :val 1}, {:id 1 :val 2}, {:id 3 :val 1}])
>>>
>>>
>>> Would result in the first two maps being handled sequentially, while the
>>> third map is handled in parallel with the first two. The order of the
>>> printed lines would probably be non-deterministic, except {:id 1 :val 2}
>>> would be printed before {:id 1 :val 3}.
>>>
>>> Note that for my use case I don't care about return values, but if
>>> something like this already exists it's plausible that it returns something
>>> equivalent to (map handler inputs).
>>>
>>> Does anything like this already exist in core or some lib? If not, any
>>> recommendations for how to build it?
>>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clo...@googlegroups.com
>> <javascript:>
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+u...@googlegroups.com <javascript:>
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+u...@googlegroups.com <javascript:>.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
