Thanks a lot. I will check it tomorrow.
J
On 1 Feb 2018 12:12 a.m., "Justin Smith" <noisesm...@gmail.com> wrote:
> this is exactly the kind of problem code I was describing - there's no
> backpressure on existing future tasks to hold up the launching of more
> futures - the work done by the agent calling conj is negligible. You need
> to control the size of the pool of threads used, and you need to impose
> back-pressure.
>
> On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta <grzebyta....@gmail.com>
> wrote:
>
>> On 31 January 2018 at 18:08, James Reeves <ja...@booleanknot.com> wrote:
>>
>>> On 31 January 2018 at 17:59, Jacek Grzebyta <grzebyta....@gmail.com>
>>> wrote:
>>>
>>>> I have application with quite intense tripe store populating ~30/40 k
>>>> records per chunk (139 portions). The data are wrapped within the future:
>>>>
>>>> (conj agent (future (apply task args)))
>>>>
>>>> and that all together is send-off into (agent []).
>>>>
>>>
>>> What is "agent"? The first line of code indicates that it's a local
>>> collection shadowing the code function, while the second code snippet
>>> indicates that you're using the core agent function.
>>>
>>> Also why are you sending off to an agent?
>>>
>>
>> I have ~8sec computing task for each input dataset which generates those
>> records. After that I write it into disk (in software-specific
>> transaction). I just wanted to separate hard computing and io operations. I
>> created a side-effect method which is injected together with the dataset
>> into a future. The futures are async collected within a list wrapped in
>> agent. After the computing the main thread is waiting until all io tasks
>> will be finished.
>>
>>
>>>
>>> At the end of the main thread function I just use await-for and after
>>>> that:
>>>>
>>>> (reduce + (map #(deref %) @data-loading-tasks))
>>>>
>>>
>> As a control, tasks return number of written records.
>>
>>
>>
>>>
>>>> For some reason I see the happy collecting (see attached screenshot of
>>>> jconsole).
>>>>
>>>
>>> "happy" = "heap"?
>>>
>>
>> Both. As you can see on attached screenshot the heap usage grows easy
>> until aver. ~2 1/4 G than keep that for a few minutes. In that moment I
>> stopped. After that starts grow till ~4G with tendency to do jumps a bit
>> more that 4G.
>>
>>
>>>
>>>
>>>> After seeing the source code of future I suppose that the memory (data
>>>> are kept as #{} set) is not released. The task returns only integer so I do
>>>> not think that might cause the problem.
>>>>
>>>
>>> Can you provide more detail? You keep alluding to things that you don't
>>> provide code for, such as the sets of data.
>>>
>>
>>
>> The code is attached. However the important code is
>>
>> L123 .
>> (let [;; keeps all data loading futures.
>> ;; waiting until all futures are finished
>> ;; should be done outside the main loop
>> data-loading-tasks (agent [])]
>>
>> L128
>> (doseq
>> (let [r1 (long operation)] L133
>> (doseq
>> (let [r2 (v.v. long)] L155
>>
>> L163 (send-off data-loading-task conj-task)
>>
>> )
>> )
>> )
>> )
>>
>>
>> I guess first I will move data-loading-tasks list into one of inner lets.
>> Also I will create within an injecting function a separate abstract
>> function let inside. The task will populate tmp variable which will be
>> returned as a future result:
>>
>>
>> L114 (conj agent (future (apply (fn [] (let [result (apply task args)]
>> result)))))
>>
>>
>>>
>>> --
>>> James Reeves
>>> booleanknot.com
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clojure@googlegroups.com
>>> Note that posts from new members are moderated - please be patient with
>>> your first post.
>>> To unsubscribe from this group, send email to
>>> clojure+unsubscr...@googlegroups.com
>>> For more options, visit this group at
>>> http://groups.google.com/group/clojure?hl=en
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to clojure+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
