HI, So things works well. In terms of memory usage anyway. I use 50 main threads (in process-data-by-condition-set) and 20 for replications (process-growths). However I found that after good performance in the beginning it slows down rapid and quick. VisualVM shows the threads usage is poor - mainly they are waiting. Only 1 claypoole thread works + some lower level threads managed probably by Clojure, etc. CPU usage is poor as well. I guess the reason is on the lower - Java classes level with IO operations bottleneck. IMO they are working by blocking. That why I wanted to separate them. However IO speed operations go down as well. I've tried async channels but that they were flooded and some data where lost for some reason.
Regards, Jacek On 3 February 2018 at 00:14, Justin Smith <noisesm...@gmail.com> wrote: > -> is just a list transform performed after reading your code into a list > data structure containing symbols, and before compiling to byte code - it > doesn't do anything directly. > > On Fri, Feb 2, 2018 at 3:55 PM Jacek Grzebyta <grzebyta....@gmail.com> > wrote: > >> OK I found what makes the memory leak. >> >> In the project I work with I use a java Model class which is java >> Collection proxy/facade for a single transaction. Unfortunately it's not >> thread safe. In a few places I passed single instance of model into several >> threads.... Also I requested the instance with -> which makes new thread as >> well. I was surprised that -> makes trouble but after thinking that might >> be expected. Especially that internally the wrapper doesn't do simple >> mapping - it uses some iterator, etc. Anyway the machinery is fragile. It >> seems I need to rewrite code and replace all multithreading parts by >> something simpler. >> >> >> >> If you want see the stacktrace just look at: >> https://github.com/jgrzebyta/triple-loader/issues/53 >> >> On 2 February 2018 at 11:16, Jacek Grzebyta <grzebyta....@gmail.com> >> wrote: >> >>> >>> On 2 February 2018 at 08:34, Niels van Klaveren < >>> niels.vanklave...@gmail.com> wrote: >>> >>>> +1 for Claypoole, it removed the needs of using agents or futures in >>>> 95% of the cases in my code. >>>> >>>> >>> Thanks a lot. I modify the code using claypoole. I imagine with-shutdown >>> will close the pool properly after finish all tasks so there is no need to >>> watch them? >>> >>> J >>> >>> >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clojure@googlegroups.com >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
