Agents manage a pool of threads for you. Try doing it without the future
call and see if that works (unless you're trying to do something else).
John
On Wed, Jan 31, 2018 at 7:31 PM, Jacek Grzebyta <grzebyta....@gmail.com>
wrote:
> Thanks a lot. I will check it tomorrow.
>
> J
>
> On 1 Feb 2018 12:12 a.m., "Justin Smith" <noisesm...@gmail.com> wrote:
>
>> this is exactly the kind of problem code I was describing - there's no
>> backpressure on existing future tasks to hold up the launching of more
>> futures - the work done by the agent calling conj is negligible. You need
>> to control the size of the pool of threads used, and you need to impose
>> back-pressure.
>>
>> On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta <grzebyta....@gmail.com>
>> wrote:
>>
>>> On 31 January 2018 at 18:08, James Reeves <ja...@booleanknot.com> wrote:
>>>
>>>> On 31 January 2018 at 17:59, Jacek Grzebyta <grzebyta....@gmail.com>
>>>> wrote:
>>>>
>>>>> I have application with quite intense tripe store populating ~30/40 k
>>>>> records per chunk (139 portions). The data are wrapped within the future:
>>>>>
>>>>> (conj agent (future (apply task args)))
>>>>>
>>>>> and that all together is send-off into (agent []).
>>>>>
>>>>
>>>> What is "agent"? The first line of code indicates that it's a local
>>>> collection shadowing the code function, while the second code snippet
>>>> indicates that you're using the core agent function.
>>>>
>>>> Also why are you sending off to an agent?
>>>>
>>>
>>> I have ~8sec computing task for each input dataset which generates those
>>> records. After that I write it into disk (in software-specific
>>> transaction). I just wanted to separate hard computing and io operations. I
>>> created a side-effect method which is injected together with the dataset
>>> into a future. The futures are async collected within a list wrapped in
>>> agent. After the computing the main thread is waiting until all io tasks
>>> will be finished.
>>>
>>>
>>>>
>>>> At the end of the main thread function I just use await-for and after
>>>>> that:
>>>>>
>>>>> (reduce + (map #(deref %) @data-loading-tasks))
>>>>>
>>>>
>>> As a control, tasks return number of written records.
>>>
>>>
>>>
>>>>
>>>>> For some reason I see the happy collecting (see attached screenshot of
>>>>> jconsole).
>>>>>
>>>>
>>>> "happy" = "heap"?
>>>>
>>>
>>> Both. As you can see on attached screenshot the heap usage grows easy
>>> until aver. ~2 1/4 G than keep that for a few minutes. In that moment I
>>> stopped. After that starts grow till ~4G with tendency to do jumps a bit
>>> more that 4G.
>>>
>>>
>>>>
>>>>
>>>>> After seeing the source code of future I suppose that the memory (data
>>>>> are kept as #{} set) is not released. The task returns only integer so I
>>>>> do
>>>>> not think that might cause the problem.
>>>>>
>>>>
>>>> Can you provide more detail? You keep alluding to things that you don't
>>>> provide code for, such as the sets of data.
>>>>
>>>
>>>
>>> The code is attached. However the important code is
>>>
>>> L123 .
>>> (let [;; keeps all data loading futures.
>>> ;; waiting until all futures are finished
>>> ;; should be done outside the main loop
>>> data-loading-tasks (agent [])]
>>>
>>> L128
>>> (doseq
>>> (let [r1 (long operation)] L133
>>> (doseq
>>> (let [r2 (v.v. long)] L155
>>>
>>> L163 (send-off data-loading-task conj-task)
>>>
>>> )
>>> )
>>> )
>>> )
>>>
>>>
>>> I guess first I will move data-loading-tasks list into one of inner
>>> lets. Also I will create within an injecting function a separate abstract
>>> function let inside. The task will populate tmp variable which will be
>>> returned as a future result:
>>>
>>>
>>> L114 (conj agent (future (apply (fn [] (let [result (apply task args)]
>>> result)))))
>>>
>>>
>>>>
>>>> --
>>>> James Reeves
>>>> booleanknot.com
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Clojure" group.
>>>> To post to this group, send email to clojure@googlegroups.com
>>>> Note that posts from new members are moderated - please be patient with
>>>> your first post.
>>>> To unsubscribe from this group, send email to
>>>> clojure+unsubscr...@googlegroups.com
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/clojure?hl=en
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Clojure" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to clojure+unsubscr...@googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clojure@googlegroups.com
>>> Note that posts from new members are moderated - please be patient with
>>> your first post.
>>> To unsubscribe from this group, send email to
>>> clojure+unsubscr...@googlegroups.com
>>> For more options, visit this group at
>>> http://groups.google.com/group/clojure?hl=en
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to clojure+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
