+1 for Claypoole, it removed the needs of using agents or futures in 95% of the cases in my code.
On Thursday, February 1, 2018 at 9:54:36 PM UTC+1, Alan Thompson wrote: > > You may find that using the Claypoole library is the easiest way to handle > threadpools: https://github.com/TheClimateCorporation/claypoole > Alan > > On Thu, Feb 1, 2018 at 11:16 AM, Justin Smith <noise...@gmail.com > <javascript:>> wrote: > >> yes, that's the idea exactly >> >> also, you might want more fine grained control of how much parallelism >> occurs (eg. if every thread is writing to the same physical device, you can >> often get better throughput by not parallelizing at all, or keeping the >> parallelism quite limited - it's worth experimenting) - there are good ways >> to control that using ThreadPoolExecutor directly, or using >> clojure.lang.PersistentQueue/EMPTY as a control construct, or core.async >> channels, or ztellman's manifold library, or the claypoole threading library >> >> On Thu, Feb 1, 2018, 03:44 Jacek Grzebyta <grzeby...@gmail.com >> <javascript:>> wrote: >> >>> Thanks folks. I see now! It should be a list of agents not list of >>> futures within agent..... Also any task sent to a agent is processed >>> within a thread anyway so I do not need to add future... >>> >>> On 1 February 2018 at 02:17, John Newman <joh...@gmail.com <javascript:> >>> > wrote: >>> >>>> Ah, he's using one agent, I see. >>>> >>>> On Jan 31, 2018 9:15 PM, "John Newman" <joh...@gmail.com <javascript:>> >>>> wrote: >>>> >>>>> Multiple sen-doffs to one agent will serialize it's calls, but >>>>> spawning agents on each new task will spawn threads on a bounded thread >>>>> pool, I believe. >>>>> >>>>> On Jan 31, 2018 8:32 PM, "Justin Smith" <noise...@gmail.com >>>>> <javascript:>> wrote: >>>>> >>>>>> Doing all the actions via one agent means that the actions are >>>>>> serialized though - you end up with no performance improvement over >>>>>> doing >>>>>> them all in a doseq in one future - the right way to do this tends to be >>>>>> trickier than it looks at first glance, and depends on your >>>>>> requirements. >>>>>> agents, the claypoole library, and reducers are all potentially useful. >>>>>> If >>>>>> parallelization leads to complex coordination needs, core.async can help >>>>>> too. >>>>>> >>>>>> On Wed, Jan 31, 2018 at 5:18 PM John Newman <joh...@gmail.com >>>>>> <javascript:>> wrote: >>>>>> >>>>>>> Agents manage a pool of threads for you. Try doing it without the >>>>>>> future call and see if that works (unless you're trying to do something >>>>>>> else). >>>>>>> >>>>>>> John >>>>>>> >>>>>>> On Wed, Jan 31, 2018 at 7:31 PM, Jacek Grzebyta <grzeby...@gmail.com >>>>>>> <javascript:>> wrote: >>>>>>> >>>>>>>> Thanks a lot. I will check it tomorrow. >>>>>>>> >>>>>>>> J >>>>>>>> >>>>>>>> On 1 Feb 2018 12:12 a.m., "Justin Smith" <noise...@gmail.com >>>>>>>> <javascript:>> wrote: >>>>>>>> >>>>>>>>> this is exactly the kind of problem code I was describing - >>>>>>>>> there's no backpressure on existing future tasks to hold up the >>>>>>>>> launching >>>>>>>>> of more futures - the work done by the agent calling conj is >>>>>>>>> negligible. >>>>>>>>> You need to control the size of the pool of threads used, and you >>>>>>>>> need to >>>>>>>>> impose back-pressure. >>>>>>>>> >>>>>>>>> On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta < >>>>>>>>> grzeby...@gmail.com <javascript:>> wrote: >>>>>>>>> >>>>>>>>>> On 31 January 2018 at 18:08, James Reeves <ja...@booleanknot.com >>>>>>>>>> <javascript:>> wrote: >>>>>>>>>> >>>>>>>>>>> On 31 January 2018 at 17:59, Jacek Grzebyta <grzeby...@gmail.com >>>>>>>>>>> <javascript:>> wrote: >>>>>>>>>>> >>>>>>>>>>>> I have application with quite intense tripe store populating >>>>>>>>>>>> ~30/40 k records per chunk (139 portions). The data are wrapped >>>>>>>>>>>> within the >>>>>>>>>>>> future: >>>>>>>>>>>> >>>>>>>>>>>> (conj agent (future (apply task args))) >>>>>>>>>>>> >>>>>>>>>>>> and that all together is send-off into (agent []). >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> What is "agent"? The first line of code indicates that it's a >>>>>>>>>>> local collection shadowing the code function, while the second code >>>>>>>>>>> snippet >>>>>>>>>>> indicates that you're using the core agent function. >>>>>>>>>>> >>>>>>>>>>> Also why are you sending off to an agent? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I have ~8sec computing task for each input dataset which >>>>>>>>>> generates those records. After that I write it into disk (in >>>>>>>>>> software-specific transaction). I just wanted to separate hard >>>>>>>>>> computing >>>>>>>>>> and io operations. I created a side-effect method which is injected >>>>>>>>>> together with the dataset into a future. The futures are async >>>>>>>>>> collected >>>>>>>>>> within a list wrapped in agent. After the computing the main thread >>>>>>>>>> is >>>>>>>>>> waiting until all io tasks will be finished. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> At the end of the main thread function I just use await-for and >>>>>>>>>>>> after that: >>>>>>>>>>>> >>>>>>>>>>>> (reduce + (map #(deref %) @data-loading-tasks)) >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> As a control, tasks return number of written records. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> For some reason I see the happy collecting (see attached >>>>>>>>>>>> screenshot of jconsole). >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> "happy" = "heap"? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Both. As you can see on attached screenshot the heap usage grows >>>>>>>>>> easy until aver. ~2 1/4 G than keep that for a few minutes. In that >>>>>>>>>> moment >>>>>>>>>> I stopped. After that starts grow till ~4G with tendency to do jumps >>>>>>>>>> a bit >>>>>>>>>> more that 4G. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> After seeing the source code of future I suppose that the >>>>>>>>>>>> memory (data are kept as #{} set) is not released. The task >>>>>>>>>>>> returns only >>>>>>>>>>>> integer so I do not think that might cause the problem. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Can you provide more detail? You keep alluding to things that >>>>>>>>>>> you don't provide code for, such as the sets of data. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> The code is attached. However the important code is >>>>>>>>>> >>>>>>>>>> L123 . >>>>>>>>>> (let [;; keeps all data loading futures. >>>>>>>>>> ;; waiting until all futures are finished >>>>>>>>>> ;; should be done outside the main loop >>>>>>>>>> data-loading-tasks (agent [])] >>>>>>>>>> >>>>>>>>>> L128 >>>>>>>>>> (doseq >>>>>>>>>> (let [r1 (long operation)] L133 >>>>>>>>>> (doseq >>>>>>>>>> (let [r2 (v.v. long)] L155 >>>>>>>>>> >>>>>>>>>> L163 (send-off data-loading-task conj-task) >>>>>>>>>> >>>>>>>>>> ) >>>>>>>>>> ) >>>>>>>>>> ) >>>>>>>>>> ) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I guess first I will move data-loading-tasks list into one of >>>>>>>>>> inner lets. Also I will create within an injecting function a >>>>>>>>>> separate >>>>>>>>>> abstract function let inside. The task will populate tmp variable >>>>>>>>>> which >>>>>>>>>> will be returned as a future result: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> L114 (conj agent (future (apply (fn [] (let [result (apply task >>>>>>>>>> args)] result))))) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> James Reeves >>>>>>>>>>> booleanknot.com >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>> Google >>>>>>>>>>> Groups "Clojure" group. >>>>>>>>>>> To post to this group, send email to clo...@googlegroups.com >>>>>>>>>>> <javascript:> >>>>>>>>>>> Note that posts from new members are moderated - please be >>>>>>>>>>> patient with your first post. >>>>>>>>>>> To unsubscribe from this group, send email to >>>>>>>>>>> clojure+u...@googlegroups.com <javascript:> >>>>>>>>>>> For more options, visit this group at >>>>>>>>>>> http://groups.google.com/group/clojure?hl=en >>>>>>>>>>> --- >>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>> Google Groups "Clojure" group. >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>> it, send an email to clojure+u...@googlegroups.com <javascript:> >>>>>>>>>>> . >>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>>> Groups "Clojure" group. >>>>>>>>>> To post to this group, send email to clo...@googlegroups.com >>>>>>>>>> <javascript:> >>>>>>>>>> Note that posts from new members are moderated - please be >>>>>>>>>> patient with your first post. >>>>>>>>>> To unsubscribe from this group, send email to >>>>>>>>>> clojure+u...@googlegroups.com <javascript:> >>>>>>>>>> For more options, visit this group at >>>>>>>>>> http://groups.google.com/group/clojure?hl=en >>>>>>>>>> --- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "Clojure" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to clojure+u...@googlegroups.com <javascript:>. >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "Clojure" group. >>>>>>>>> To post to this group, send email to clo...@googlegroups.com >>>>>>>>> <javascript:> >>>>>>>>> Note that posts from new members are moderated - please be patient >>>>>>>>> with your first post. >>>>>>>>> To unsubscribe from this group, send email to >>>>>>>>> clojure+u...@googlegroups.com <javascript:> >>>>>>>>> For more options, visit this group at >>>>>>>>> http://groups.google.com/group/clojure?hl=en >>>>>>>>> --- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "Clojure" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to clojure+u...@googlegroups.com <javascript:>. >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Clojure" group. >>>>>>>> To post to this group, send email to clo...@googlegroups.com >>>>>>>> <javascript:> >>>>>>>> Note that posts from new members are moderated - please be patient >>>>>>>> with your first post. >>>>>>>> To unsubscribe from this group, send email to >>>>>>>> clojure+u...@googlegroups.com <javascript:> >>>>>>>> For more options, visit this group at >>>>>>>> http://groups.google.com/group/clojure?hl=en >>>>>>>> --- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Clojure" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to clojure+u...@googlegroups.com <javascript:>. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Clojure" group. >>>>>>> To post to this group, send email to clo...@googlegroups.com >>>>>>> <javascript:> >>>>>>> Note that posts from new members are moderated - please be patient >>>>>>> with your first post. >>>>>>> To unsubscribe from this group, send email to >>>>>>> clojure+u...@googlegroups.com <javascript:> >>>>>>> For more options, visit this group at >>>>>>> http://groups.google.com/group/clojure?hl=en >>>>>>> --- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Clojure" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to clojure+u...@googlegroups.com <javascript:>. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Clojure" group. >>>>>> To post to this group, send email to clo...@googlegroups.com >>>>>> <javascript:> >>>>>> Note that posts from new members are moderated - please be patient >>>>>> with your first post. >>>>>> To unsubscribe from this group, send email to >>>>>> clojure+u...@googlegroups.com <javascript:> >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/group/clojure?hl=en >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Clojure" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to clojure+u...@googlegroups.com <javascript:>. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Clojure" group. >>>> To post to this group, send email to clo...@googlegroups.com >>>> <javascript:> >>>> Note that posts from new members are moderated - please be patient with >>>> your first post. >>>> To unsubscribe from this group, send email to >>>> clojure+u...@googlegroups.com <javascript:> >>>> For more options, visit this group at >>>> http://groups.google.com/group/clojure?hl=en >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "Clojure" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to clojure+u...@googlegroups.com <javascript:>. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Clojure" group. >>> To post to this group, send email to clo...@googlegroups.com >>> <javascript:> >>> Note that posts from new members are moderated - please be patient with >>> your first post. >>> To unsubscribe from this group, send email to >>> clojure+u...@googlegroups.com <javascript:> >>> For more options, visit this group at >>> http://groups.google.com/group/clojure?hl=en >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Clojure" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to clojure+u...@googlegroups.com <javascript:>. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clo...@googlegroups.com >> <javascript:> >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+u...@googlegroups.com <javascript:> >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+u...@googlegroups.com <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
