Multiple sen-doffs to one agent will serialize it's calls, but spawning
agents on each new task will spawn threads on a bounded thread pool, I
believe.
On Jan 31, 2018 8:32 PM, "Justin Smith" <noisesm...@gmail.com> wrote:
> Doing all the actions via one agent means that the actions are serialized
> though - you end up with no performance improvement over doing them all in
> a doseq in one future - the right way to do this tends to be trickier than
> it looks at first glance, and depends on your requirements. agents, the
> claypoole library, and reducers are all potentially useful. If
> parallelization leads to complex coordination needs, core.async can help
> too.
>
> On Wed, Jan 31, 2018 at 5:18 PM John Newman <john...@gmail.com> wrote:
>
>> Agents manage a pool of threads for you. Try doing it without the future
>> call and see if that works (unless you're trying to do something else).
>>
>> John
>>
>> On Wed, Jan 31, 2018 at 7:31 PM, Jacek Grzebyta <grzebyta....@gmail.com>
>> wrote:
>>
>>> Thanks a lot. I will check it tomorrow.
>>>
>>> J
>>>
>>> On 1 Feb 2018 12:12 a.m., "Justin Smith" <noisesm...@gmail.com> wrote:
>>>
>>>> this is exactly the kind of problem code I was describing - there's no
>>>> backpressure on existing future tasks to hold up the launching of more
>>>> futures - the work done by the agent calling conj is negligible. You need
>>>> to control the size of the pool of threads used, and you need to impose
>>>> back-pressure.
>>>>
>>>> On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta <grzebyta....@gmail.com>
>>>> wrote:
>>>>
>>>>> On 31 January 2018 at 18:08, James Reeves <ja...@booleanknot.com>
>>>>> wrote:
>>>>>
>>>>>> On 31 January 2018 at 17:59, Jacek Grzebyta <grzebyta....@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I have application with quite intense tripe store populating ~30/40
>>>>>>> k records per chunk (139 portions). The data are wrapped within the
>>>>>>> future:
>>>>>>>
>>>>>>> (conj agent (future (apply task args)))
>>>>>>>
>>>>>>> and that all together is send-off into (agent []).
>>>>>>>
>>>>>>
>>>>>> What is "agent"? The first line of code indicates that it's a local
>>>>>> collection shadowing the code function, while the second code snippet
>>>>>> indicates that you're using the core agent function.
>>>>>>
>>>>>> Also why are you sending off to an agent?
>>>>>>
>>>>>
>>>>> I have ~8sec computing task for each input dataset which generates
>>>>> those records. After that I write it into disk (in software-specific
>>>>> transaction). I just wanted to separate hard computing and io operations.
>>>>> I
>>>>> created a side-effect method which is injected together with the dataset
>>>>> into a future. The futures are async collected within a list wrapped in
>>>>> agent. After the computing the main thread is waiting until all io tasks
>>>>> will be finished.
>>>>>
>>>>>
>>>>>>
>>>>>> At the end of the main thread function I just use await-for and after
>>>>>>> that:
>>>>>>>
>>>>>>> (reduce + (map #(deref %) @data-loading-tasks))
>>>>>>>
>>>>>>
>>>>> As a control, tasks return number of written records.
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>> For some reason I see the happy collecting (see attached screenshot
>>>>>>> of jconsole).
>>>>>>>
>>>>>>
>>>>>> "happy" = "heap"?
>>>>>>
>>>>>
>>>>> Both. As you can see on attached screenshot the heap usage grows easy
>>>>> until aver. ~2 1/4 G than keep that for a few minutes. In that moment I
>>>>> stopped. After that starts grow till ~4G with tendency to do jumps a bit
>>>>> more that 4G.
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>>> After seeing the source code of future I suppose that the memory
>>>>>>> (data are kept as #{} set) is not released. The task returns only
>>>>>>> integer
>>>>>>> so I do not think that might cause the problem.
>>>>>>>
>>>>>>
>>>>>> Can you provide more detail? You keep alluding to things that you
>>>>>> don't provide code for, such as the sets of data.
>>>>>>
>>>>>
>>>>>
>>>>> The code is attached. However the important code is
>>>>>
>>>>> L123 .
>>>>> (let [;; keeps all data loading futures.
>>>>> ;; waiting until all futures are finished
>>>>> ;; should be done outside the main loop
>>>>> data-loading-tasks (agent [])]
>>>>>
>>>>> L128
>>>>> (doseq
>>>>> (let [r1 (long operation)] L133
>>>>> (doseq
>>>>> (let [r2 (v.v. long)] L155
>>>>>
>>>>> L163 (send-off data-loading-task conj-task)
>>>>>
>>>>> )
>>>>> )
>>>>> )
>>>>> )
>>>>>
>>>>>
>>>>> I guess first I will move data-loading-tasks list into one of inner
>>>>> lets. Also I will create within an injecting function a separate abstract
>>>>> function let inside. The task will populate tmp variable which will be
>>>>> returned as a future result:
>>>>>
>>>>>
>>>>> L114 (conj agent (future (apply (fn [] (let [result (apply task
>>>>> args)] result)))))
>>>>>
>>>>>
>>>>>>
>>>>>> --
>>>>>> James Reeves
>>>>>> booleanknot.com
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Clojure" group.
>>>>>> To post to this group, send email to clojure@googlegroups.com
>>>>>> Note that posts from new members are moderated - please be patient
>>>>>> with your first post.
>>>>>> To unsubscribe from this group, send email to
>>>>>> clojure+unsubscr...@googlegroups.com
>>>>>> For more options, visit this group at
>>>>>> http://groups.google.com/group/clojure?hl=en
>>>>>> ---
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Clojure" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to clojure+unsubscr...@googlegroups.com.
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Clojure" group.
>>>>> To post to this group, send email to clojure@googlegroups.com
>>>>> Note that posts from new members are moderated - please be patient
>>>>> with your first post.
>>>>> To unsubscribe from this group, send email to
>>>>> clojure+unsubscr...@googlegroups.com
>>>>> For more options, visit this group at
>>>>> http://groups.google.com/group/clojure?hl=en
>>>>> ---
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Clojure" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to clojure+unsubscr...@googlegroups.com.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Clojure" group.
>>>> To post to this group, send email to clojure@googlegroups.com
>>>> Note that posts from new members are moderated - please be patient with
>>>> your first post.
>>>> To unsubscribe from this group, send email to
>>>> clojure+unsubscr...@googlegroups.com
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/clojure?hl=en
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Clojure" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to clojure+unsubscr...@googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clojure@googlegroups.com
>>> Note that posts from new members are moderated - please be patient with
>>> your first post.
>>> To unsubscribe from this group, send email to
>>> clojure+unsubscr...@googlegroups.com
>>> For more options, visit this group at
>>> http://groups.google.com/group/clojure?hl=en
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to clojure+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
