Thanks folks. I see now! It should be a list of agents not list of futures
within agent..... Also any task sent to a agent is processed within a
thread anyway so I do not need to add future...
On 1 February 2018 at 02:17, John Newman <john...@gmail.com> wrote:
> Ah, he's using one agent, I see.
>
> On Jan 31, 2018 9:15 PM, "John Newman" <john...@gmail.com> wrote:
>
>> Multiple sen-doffs to one agent will serialize it's calls, but spawning
>> agents on each new task will spawn threads on a bounded thread pool, I
>> believe.
>>
>> On Jan 31, 2018 8:32 PM, "Justin Smith" <noisesm...@gmail.com> wrote:
>>
>>> Doing all the actions via one agent means that the actions are
>>> serialized though - you end up with no performance improvement over doing
>>> them all in a doseq in one future - the right way to do this tends to be
>>> trickier than it looks at first glance, and depends on your requirements.
>>> agents, the claypoole library, and reducers are all potentially useful. If
>>> parallelization leads to complex coordination needs, core.async can help
>>> too.
>>>
>>> On Wed, Jan 31, 2018 at 5:18 PM John Newman <john...@gmail.com> wrote:
>>>
>>>> Agents manage a pool of threads for you. Try doing it without the
>>>> future call and see if that works (unless you're trying to do something
>>>> else).
>>>>
>>>> John
>>>>
>>>> On Wed, Jan 31, 2018 at 7:31 PM, Jacek Grzebyta <grzebyta....@gmail.com
>>>> > wrote:
>>>>
>>>>> Thanks a lot. I will check it tomorrow.
>>>>>
>>>>> J
>>>>>
>>>>> On 1 Feb 2018 12:12 a.m., "Justin Smith" <noisesm...@gmail.com> wrote:
>>>>>
>>>>>> this is exactly the kind of problem code I was describing - there's
>>>>>> no backpressure on existing future tasks to hold up the launching of more
>>>>>> futures - the work done by the agent calling conj is negligible. You need
>>>>>> to control the size of the pool of threads used, and you need to impose
>>>>>> back-pressure.
>>>>>>
>>>>>> On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta <
>>>>>> grzebyta....@gmail.com> wrote:
>>>>>>
>>>>>>> On 31 January 2018 at 18:08, James Reeves <ja...@booleanknot.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> On 31 January 2018 at 17:59, Jacek Grzebyta <grzebyta....@gmail.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> I have application with quite intense tripe store populating
>>>>>>>>> ~30/40 k records per chunk (139 portions). The data are wrapped
>>>>>>>>> within the
>>>>>>>>> future:
>>>>>>>>>
>>>>>>>>> (conj agent (future (apply task args)))
>>>>>>>>>
>>>>>>>>> and that all together is send-off into (agent []).
>>>>>>>>>
>>>>>>>>
>>>>>>>> What is "agent"? The first line of code indicates that it's a local
>>>>>>>> collection shadowing the code function, while the second code snippet
>>>>>>>> indicates that you're using the core agent function.
>>>>>>>>
>>>>>>>> Also why are you sending off to an agent?
>>>>>>>>
>>>>>>>
>>>>>>> I have ~8sec computing task for each input dataset which generates
>>>>>>> those records. After that I write it into disk (in software-specific
>>>>>>> transaction). I just wanted to separate hard computing and io
>>>>>>> operations. I
>>>>>>> created a side-effect method which is injected together with the dataset
>>>>>>> into a future. The futures are async collected within a list wrapped in
>>>>>>> agent. After the computing the main thread is waiting until all io tasks
>>>>>>> will be finished.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> At the end of the main thread function I just use await-for and
>>>>>>>>> after that:
>>>>>>>>>
>>>>>>>>> (reduce + (map #(deref %) @data-loading-tasks))
>>>>>>>>>
>>>>>>>>
>>>>>>> As a control, tasks return number of written records.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>> For some reason I see the happy collecting (see attached
>>>>>>>>> screenshot of jconsole).
>>>>>>>>>
>>>>>>>>
>>>>>>>> "happy" = "heap"?
>>>>>>>>
>>>>>>>
>>>>>>> Both. As you can see on attached screenshot the heap usage grows
>>>>>>> easy until aver. ~2 1/4 G than keep that for a few minutes. In that
>>>>>>> moment
>>>>>>> I stopped. After that starts grow till ~4G with tendency to do jumps a
>>>>>>> bit
>>>>>>> more that 4G.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> After seeing the source code of future I suppose that the memory
>>>>>>>>> (data are kept as #{} set) is not released. The task returns only
>>>>>>>>> integer
>>>>>>>>> so I do not think that might cause the problem.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Can you provide more detail? You keep alluding to things that you
>>>>>>>> don't provide code for, such as the sets of data.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> The code is attached. However the important code is
>>>>>>>
>>>>>>> L123 .
>>>>>>> (let [;; keeps all data loading futures.
>>>>>>> ;; waiting until all futures are finished
>>>>>>> ;; should be done outside the main loop
>>>>>>> data-loading-tasks (agent [])]
>>>>>>>
>>>>>>> L128
>>>>>>> (doseq
>>>>>>> (let [r1 (long operation)] L133
>>>>>>> (doseq
>>>>>>> (let [r2 (v.v. long)] L155
>>>>>>>
>>>>>>> L163 (send-off data-loading-task conj-task)
>>>>>>>
>>>>>>> )
>>>>>>> )
>>>>>>> )
>>>>>>> )
>>>>>>>
>>>>>>>
>>>>>>> I guess first I will move data-loading-tasks list into one of inner
>>>>>>> lets. Also I will create within an injecting function a separate
>>>>>>> abstract
>>>>>>> function let inside. The task will populate tmp variable which will be
>>>>>>> returned as a future result:
>>>>>>>
>>>>>>>
>>>>>>> L114 (conj agent (future (apply (fn [] (let [result (apply task
>>>>>>> args)] result)))))
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> James Reeves
>>>>>>>> booleanknot.com
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "Clojure" group.
>>>>>>>> To post to this group, send email to clojure@googlegroups.com
>>>>>>>> Note that posts from new members are moderated - please be patient
>>>>>>>> with your first post.
>>>>>>>> To unsubscribe from this group, send email to
>>>>>>>> clojure+unsubscr...@googlegroups.com
>>>>>>>> For more options, visit this group at
>>>>>>>> http://groups.google.com/group/clojure?hl=en
>>>>>>>> ---
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "Clojure" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to clojure+unsubscr...@googlegroups.com.
>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "Clojure" group.
>>>>>>> To post to this group, send email to clojure@googlegroups.com
>>>>>>> Note that posts from new members are moderated - please be patient
>>>>>>> with your first post.
>>>>>>> To unsubscribe from this group, send email to
>>>>>>> clojure+unsubscr...@googlegroups.com
>>>>>>> For more options, visit this group at
>>>>>>> http://groups.google.com/group/clojure?hl=en
>>>>>>> ---
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "Clojure" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to clojure+unsubscr...@googlegroups.com.
>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Clojure" group.
>>>>>> To post to this group, send email to clojure@googlegroups.com
>>>>>> Note that posts from new members are moderated - please be patient
>>>>>> with your first post.
>>>>>> To unsubscribe from this group, send email to
>>>>>> clojure+unsubscr...@googlegroups.com
>>>>>> For more options, visit this group at
>>>>>> http://groups.google.com/group/clojure?hl=en
>>>>>> ---
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Clojure" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to clojure+unsubscr...@googlegroups.com.
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Clojure" group.
>>>>> To post to this group, send email to clojure@googlegroups.com
>>>>> Note that posts from new members are moderated - please be patient
>>>>> with your first post.
>>>>> To unsubscribe from this group, send email to
>>>>> clojure+unsubscr...@googlegroups.com
>>>>> For more options, visit this group at
>>>>> http://groups.google.com/group/clojure?hl=en
>>>>> ---
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Clojure" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to clojure+unsubscr...@googlegroups.com.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Clojure" group.
>>>> To post to this group, send email to clojure@googlegroups.com
>>>> Note that posts from new members are moderated - please be patient with
>>>> your first post.
>>>> To unsubscribe from this group, send email to
>>>> clojure+unsubscr...@googlegroups.com
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/clojure?hl=en
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Clojure" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to clojure+unsubscr...@googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clojure@googlegroups.com
>>> Note that posts from new members are moderated - please be patient with
>>> your first post.
>>> To unsubscribe from this group, send email to
>>> clojure+unsubscr...@googlegroups.com
>>> For more options, visit this group at
>>> http://groups.google.com/group/clojure?hl=en
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to clojure+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
