You may find that using the Claypoole library is the easiest way to handle
threadpools: https://github.com/TheClimateCorporation/claypoole
Alan
On Thu, Feb 1, 2018 at 11:16 AM, Justin Smith <noisesm...@gmail.com> wrote:
> yes, that's the idea exactly
>
> also, you might want more fine grained control of how much parallelism
> occurs (eg. if every thread is writing to the same physical device, you can
> often get better throughput by not parallelizing at all, or keeping the
> parallelism quite limited - it's worth experimenting) - there are good ways
> to control that using ThreadPoolExecutor directly, or using
> clojure.lang.PersistentQueue/EMPTY as a control construct, or core.async
> channels, or ztellman's manifold library, or the claypoole threading library
>
> On Thu, Feb 1, 2018, 03:44 Jacek Grzebyta <grzebyta....@gmail.com> wrote:
>
>> Thanks folks. I see now! It should be a list of agents not list of
>> futures within agent..... Also any task sent to a agent is processed
>> within a thread anyway so I do not need to add future...
>>
>> On 1 February 2018 at 02:17, John Newman <john...@gmail.com> wrote:
>>
>>> Ah, he's using one agent, I see.
>>>
>>> On Jan 31, 2018 9:15 PM, "John Newman" <john...@gmail.com> wrote:
>>>
>>>> Multiple sen-doffs to one agent will serialize it's calls, but spawning
>>>> agents on each new task will spawn threads on a bounded thread pool, I
>>>> believe.
>>>>
>>>> On Jan 31, 2018 8:32 PM, "Justin Smith" <noisesm...@gmail.com> wrote:
>>>>
>>>>> Doing all the actions via one agent means that the actions are
>>>>> serialized though - you end up with no performance improvement over doing
>>>>> them all in a doseq in one future - the right way to do this tends to be
>>>>> trickier than it looks at first glance, and depends on your requirements.
>>>>> agents, the claypoole library, and reducers are all potentially useful. If
>>>>> parallelization leads to complex coordination needs, core.async can help
>>>>> too.
>>>>>
>>>>> On Wed, Jan 31, 2018 at 5:18 PM John Newman <john...@gmail.com> wrote:
>>>>>
>>>>>> Agents manage a pool of threads for you. Try doing it without the
>>>>>> future call and see if that works (unless you're trying to do something
>>>>>> else).
>>>>>>
>>>>>> John
>>>>>>
>>>>>> On Wed, Jan 31, 2018 at 7:31 PM, Jacek Grzebyta <
>>>>>> grzebyta....@gmail.com> wrote:
>>>>>>
>>>>>>> Thanks a lot. I will check it tomorrow.
>>>>>>>
>>>>>>> J
>>>>>>>
>>>>>>> On 1 Feb 2018 12:12 a.m., "Justin Smith" <noisesm...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> this is exactly the kind of problem code I was describing - there's
>>>>>>>> no backpressure on existing future tasks to hold up the launching of
>>>>>>>> more
>>>>>>>> futures - the work done by the agent calling conj is negligible. You
>>>>>>>> need
>>>>>>>> to control the size of the pool of threads used, and you need to impose
>>>>>>>> back-pressure.
>>>>>>>>
>>>>>>>> On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta <
>>>>>>>> grzebyta....@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> On 31 January 2018 at 18:08, James Reeves <ja...@booleanknot.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> On 31 January 2018 at 17:59, Jacek Grzebyta <
>>>>>>>>>> grzebyta....@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> I have application with quite intense tripe store populating
>>>>>>>>>>> ~30/40 k records per chunk (139 portions). The data are wrapped
>>>>>>>>>>> within the
>>>>>>>>>>> future:
>>>>>>>>>>>
>>>>>>>>>>> (conj agent (future (apply task args)))
>>>>>>>>>>>
>>>>>>>>>>> and that all together is send-off into (agent []).
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> What is "agent"? The first line of code indicates that it's a
>>>>>>>>>> local collection shadowing the code function, while the second code
>>>>>>>>>> snippet
>>>>>>>>>> indicates that you're using the core agent function.
>>>>>>>>>>
>>>>>>>>>> Also why are you sending off to an agent?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I have ~8sec computing task for each input dataset which generates
>>>>>>>>> those records. After that I write it into disk (in software-specific
>>>>>>>>> transaction). I just wanted to separate hard computing and io
>>>>>>>>> operations. I
>>>>>>>>> created a side-effect method which is injected together with the
>>>>>>>>> dataset
>>>>>>>>> into a future. The futures are async collected within a list wrapped
>>>>>>>>> in
>>>>>>>>> agent. After the computing the main thread is waiting until all io
>>>>>>>>> tasks
>>>>>>>>> will be finished.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> At the end of the main thread function I just use await-for and
>>>>>>>>>>> after that:
>>>>>>>>>>>
>>>>>>>>>>> (reduce + (map #(deref %) @data-loading-tasks))
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> As a control, tasks return number of written records.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> For some reason I see the happy collecting (see attached
>>>>>>>>>>> screenshot of jconsole).
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> "happy" = "heap"?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Both. As you can see on attached screenshot the heap usage grows
>>>>>>>>> easy until aver. ~2 1/4 G than keep that for a few minutes. In that
>>>>>>>>> moment
>>>>>>>>> I stopped. After that starts grow till ~4G with tendency to do jumps
>>>>>>>>> a bit
>>>>>>>>> more that 4G.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> After seeing the source code of future I suppose that the memory
>>>>>>>>>>> (data are kept as #{} set) is not released. The task returns only
>>>>>>>>>>> integer
>>>>>>>>>>> so I do not think that might cause the problem.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Can you provide more detail? You keep alluding to things that you
>>>>>>>>>> don't provide code for, such as the sets of data.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The code is attached. However the important code is
>>>>>>>>>
>>>>>>>>> L123 .
>>>>>>>>> (let [;; keeps all data loading futures.
>>>>>>>>> ;; waiting until all futures are finished
>>>>>>>>> ;; should be done outside the main loop
>>>>>>>>> data-loading-tasks (agent [])]
>>>>>>>>>
>>>>>>>>> L128
>>>>>>>>> (doseq
>>>>>>>>> (let [r1 (long operation)] L133
>>>>>>>>> (doseq
>>>>>>>>> (let [r2 (v.v. long)] L155
>>>>>>>>>
>>>>>>>>> L163 (send-off data-loading-task conj-task)
>>>>>>>>>
>>>>>>>>> )
>>>>>>>>> )
>>>>>>>>> )
>>>>>>>>> )
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I guess first I will move data-loading-tasks list into one of
>>>>>>>>> inner lets. Also I will create within an injecting function a separate
>>>>>>>>> abstract function let inside. The task will populate tmp variable
>>>>>>>>> which
>>>>>>>>> will be returned as a future result:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> L114 (conj agent (future (apply (fn [] (let [result (apply task
>>>>>>>>> args)] result)))))
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> James Reeves
>>>>>>>>>> booleanknot.com
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>>> Groups "Clojure" group.
>>>>>>>>>> To post to this group, send email to clojure@googlegroups.com
>>>>>>>>>> Note that posts from new members are moderated - please be
>>>>>>>>>> patient with your first post.
>>>>>>>>>> To unsubscribe from this group, send email to
>>>>>>>>>> clojure+unsubscr...@googlegroups.com
>>>>>>>>>> For more options, visit this group at
>>>>>>>>>> http://groups.google.com/group/clojure?hl=en
>>>>>>>>>> ---
>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>> Google Groups "Clojure" group.
>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>>> send an email to clojure+unsubscr...@googlegroups.com.
>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>> Groups "Clojure" group.
>>>>>>>>> To post to this group, send email to clojure@googlegroups.com
>>>>>>>>> Note that posts from new members are moderated - please be patient
>>>>>>>>> with your first post.
>>>>>>>>> To unsubscribe from this group, send email to
>>>>>>>>> clojure+unsubscr...@googlegroups.com
>>>>>>>>> For more options, visit this group at
>>>>>>>>> http://groups.google.com/group/clojure?hl=en
>>>>>>>>> ---
>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>> Groups "Clojure" group.
>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>> send an email to clojure+unsubscr...@googlegroups.com.
>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "Clojure" group.
>>>>>>>> To post to this group, send email to clojure@googlegroups.com
>>>>>>>> Note that posts from new members are moderated - please be patient
>>>>>>>> with your first post.
>>>>>>>> To unsubscribe from this group, send email to
>>>>>>>> clojure+unsubscr...@googlegroups.com
>>>>>>>> For more options, visit this group at
>>>>>>>> http://groups.google.com/group/clojure?hl=en
>>>>>>>> ---
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "Clojure" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to clojure+unsubscr...@googlegroups.com.
>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "Clojure" group.
>>>>>>> To post to this group, send email to clojure@googlegroups.com
>>>>>>> Note that posts from new members are moderated - please be patient
>>>>>>> with your first post.
>>>>>>> To unsubscribe from this group, send email to
>>>>>>> clojure+unsubscr...@googlegroups.com
>>>>>>> For more options, visit this group at
>>>>>>> http://groups.google.com/group/clojure?hl=en
>>>>>>> ---
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "Clojure" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to clojure+unsubscr...@googlegroups.com.
>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Clojure" group.
>>>>>> To post to this group, send email to clojure@googlegroups.com
>>>>>> Note that posts from new members are moderated - please be patient
>>>>>> with your first post.
>>>>>> To unsubscribe from this group, send email to
>>>>>> clojure+unsubscr...@googlegroups.com
>>>>>> For more options, visit this group at
>>>>>> http://groups.google.com/group/clojure?hl=en
>>>>>> ---
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Clojure" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to clojure+unsubscr...@googlegroups.com.
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Clojure" group.
>>>>> To post to this group, send email to clojure@googlegroups.com
>>>>> Note that posts from new members are moderated - please be patient
>>>>> with your first post.
>>>>> To unsubscribe from this group, send email to
>>>>> clojure+unsubscr...@googlegroups.com
>>>>> For more options, visit this group at
>>>>> http://groups.google.com/group/clojure?hl=en
>>>>> ---
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Clojure" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to clojure+unsubscr...@googlegroups.com.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clojure@googlegroups.com
>>> Note that posts from new members are moderated - please be patient with
>>> your first post.
>>> To unsubscribe from this group, send email to
>>> clojure+unsubscr...@googlegroups.com
>>> For more options, visit this group at
>>> http://groups.google.com/group/clojure?hl=en
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to clojure+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
