As a shot in the dark, a common problem with memory usage and futures that
I have seen is the antipattern of launching a future for each piece of data
in a collection. The problem that occurs is that the code works for small
input collections and a small load of running tasks / requests, but for a
larger input and more requests it uses up the heap easily. Then, a dev
assumes that somehow the future itself is leaking or otherwise taking up
space it shouldn't, when the true problem is that the undbounded creation
of futures happens to exhaust your available memory space. If you aren't
doing such a thing feel free to disregard.
On Wed, Jan 31, 2018 at 10:09 AM James Reeves <ja...@booleanknot.com> wrote:
> On 31 January 2018 at 17:59, Jacek Grzebyta <grzebyta....@gmail.com>
> wrote:
>
>> I have application with quite intense tripe store populating ~30/40 k
>> records per chunk (139 portions). The data are wrapped within the future:
>>
>> (conj agent (future (apply task args)))
>>
>> and that all together is send-off into (agent []).
>>
>
> What is "agent"? The first line of code indicates that it's a local
> collection shadowing the code function, while the second code snippet
> indicates that you're using the core agent function.
>
> Also why are you sending off to an agent?
>
> At the end of the main thread function I just use await-for and after that:
>>
>> (reduce + (map #(deref %) @data-loading-tasks))
>>
>> For some reason I see the happy collecting (see attached screenshot of
>> jconsole).
>>
>
> "happy" = "heap"?
>
>
>> After seeing the source code of future I suppose that the memory (data
>> are kept as #{} set) is not released. The task returns only integer so I do
>> not think that might cause the problem.
>>
>
> Can you provide more detail? You keep alluding to things that you don't
> provide code for, such as the sets of data.
>
> --
> James Reeves
> booleanknot.com
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
