Slow -main function termination by pmap
Hi, I have multi -mains project. Thus the execution looks like: java -cp location/file.jar some.method .. One -main method looks like: (defn -main [& args] (let [validated (validate-args args)] (if (:msg validated) (println (st/join \newline (:msg validated))) (run validated)) (log/debug "finish"))) There is nothing special in that except in the run time I have displayed the last log "finish" and the program termination is done after ~1 min later what is annoying. I found it is caused by deeply hidden pmap even if it is wrapped by doall. I thought after pmap returns results all threads are finished i.e. the main thread waits until all sub threads will be finished. If I am right what else can cause the delay? The threads pool manager? Tanks a lot, Jacek -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Slow -main function termination by pmap
any Clojure program that uses the built in thread pools (future, agent, pmap, core.async, etc.) should call (shutdown-agents) if prompt exit is needed On Tue, Dec 19, 2017 at 12:05 PM Jacek Grzebyta wrote: > Hi, > > I have multi -mains project. Thus the execution looks like: > > java -cp location/file.jar some.method .. > > > One -main method looks like: > > (defn -main > [& args] > (let [validated (validate-args args)] > (if (:msg validated) > (println (st/join \newline (:msg validated))) > (run validated)) > (log/debug "finish"))) > > > There is nothing special in that except in the run time I have displayed > the last log "finish" and the program termination is done after ~1 min > later what is annoying. I found it is caused by deeply hidden pmap even if > it is wrapped by doall. I thought after pmap returns results all threads > are finished i.e. the main thread waits until all sub threads will be > finished. If I am right what else can cause the delay? The threads pool > manager? > > Tanks a lot, > Jacek > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Slow -main function termination by pmap
Sometimes there's some weird interop with stuff like pmap and program termination. What happens if you end your program with this function: https://clojuredocs.org/clojure.core/shutdown-agents On Tue, Dec 19, 2017 at 1:05 PM, Jacek Grzebyta wrote: > Hi, > > I have multi -mains project. Thus the execution looks like: > > java -cp location/file.jar some.method .. > > > One -main method looks like: > > (defn -main > [& args] > (let [validated (validate-args args)] > (if (:msg validated) > (println (st/join \newline (:msg validated))) > (run validated)) > (log/debug "finish"))) > > > There is nothing special in that except in the run time I have displayed > the last log "finish" and the program termination is done after ~1 min > later what is annoying. I found it is caused by deeply hidden pmap even if > it is wrapped by doall. I thought after pmap returns results all threads > are finished i.e. the main thread waits until all sub threads will be > finished. If I am right what else can cause the delay? The threads pool > manager? > > Tanks a lot, > Jacek > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- “One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.” (Robert Firth) -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Slow -main function termination by pmap
That solved my problem! Thanks l lot! Jacek On 19 December 2017 at 20:08, Timothy Baldridge wrote: > Sometimes there's some weird interop with stuff like pmap and program > termination. What happens if you end your program with this function: > https://clojuredocs.org/clojure.core/shutdown-agents > > On Tue, Dec 19, 2017 at 1:05 PM, Jacek Grzebyta > wrote: > >> Hi, >> >> I have multi -mains project. Thus the execution looks like: >> >> java -cp location/file.jar some.method .. >> >> >> One -main method looks like: >> >> (defn -main >> [& args] >> (let [validated (validate-args args)] >> (if (:msg validated) >> (println (st/join \newline (:msg validated))) >> (run validated)) >> (log/debug "finish"))) >> >> >> There is nothing special in that except in the run time I have displayed >> the last log "finish" and the program termination is done after ~1 min >> later what is annoying. I found it is caused by deeply hidden pmap even if >> it is wrapped by doall. I thought after pmap returns results all threads >> are finished i.e. the main thread waits until all sub threads will be >> finished. If I am right what else can cause the delay? The threads pool >> manager? >> >> Tanks a lot, >> Jacek >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clojure@googlegroups.com >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > “One of the main causes of the fall of the Roman Empire was that–lacking > zero–they had no way to indicate successful termination of their C > programs.” > (Robert Firth) > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Slow -main function termination by pmap
last line in -main should be (System/exit 0) or (shutdown-agents). 1 min awaiting it is documented behaviour of clojure. вторник, 19 декабря 2017 г., 23:08:09 UTC+3 пользователь Justin Smith написал: > > any Clojure program that uses the built in thread pools (future, agent, > pmap, core.async, etc.) should call (shutdown-agents) if prompt exit is > needed > > On Tue, Dec 19, 2017 at 12:05 PM Jacek Grzebyta > wrote: > >> Hi, >> >> I have multi -mains project. Thus the execution looks like: >> >> java -cp location/file.jar some.method .. >> >> >> One -main method looks like: >> >> (defn -main >> [& args] >> (let [validated (validate-args args)] >> (if (:msg validated) >> (println (st/join \newline (:msg validated))) >> (run validated)) >> (log/debug "finish"))) >> >> >> There is nothing special in that except in the run time I have displayed >> the last log "finish" and the program termination is done after ~1 min >> later what is annoying. I found it is caused by deeply hidden pmap even if >> it is wrapped by doall. I thought after pmap returns results all threads >> are finished i.e. the main thread waits until all sub threads will be >> finished. If I am right what else can cause the delay? The threads pool >> manager? >> >> Tanks a lot, >> Jacek >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clo...@googlegroups.com >> >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+u...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+u...@googlegroups.com . >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Slow -main function termination by pmap
Well, this behavior is documented _somewhere_, but not in the doc string or on clojure.org that I know of (happy to be corrected if I'm wrong there). On the ClojureDoc.org web site, the docs for pmap: http://clojuredocs.org/clojure.core/pmap suggest looking at the docs for future for info about a 1-minute wait and how to avoid it, where there are lots of details: http://clojuredocs.org/clojure.core/future Andy On Tue, Dec 19, 2017 at 1:15 PM, Mike <145...@gmail.com> wrote: > last line in -main should be (System/exit 0) or (shutdown-agents). > 1 min awaiting it is documented behaviour of clojure. > > вторник, 19 декабря 2017 г., 23:08:09 UTC+3 пользователь Justin Smith > написал: >> >> any Clojure program that uses the built in thread pools (future, agent, >> pmap, core.async, etc.) should call (shutdown-agents) if prompt exit is >> needed >> >> On Tue, Dec 19, 2017 at 12:05 PM Jacek Grzebyta >> wrote: >> >>> Hi, >>> >>> I have multi -mains project. Thus the execution looks like: >>> >>> java -cp location/file.jar some.method .. >>> >>> >>> One -main method looks like: >>> >>> (defn -main >>> [& args] >>> (let [validated (validate-args args)] >>> (if (:msg validated) >>> (println (st/join \newline (:msg validated))) >>> (run validated)) >>> (log/debug "finish"))) >>> >>> >>> There is nothing special in that except in the run time I have displayed >>> the last log "finish" and the program termination is done after ~1 min >>> later what is annoying. I found it is caused by deeply hidden pmap even if >>> it is wrapped by doall. I thought after pmap returns results all threads >>> are finished i.e. the main thread waits until all sub threads will be >>> finished. If I am right what else can cause the delay? The threads pool >>> manager? >>> >>> Tanks a lot, >>> Jacek >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Clojure" group. >>> To post to this group, send email to clo...@googlegroups.com >>> Note that posts from new members are moderated - please be patient with >>> your first post. >>> To unsubscribe from this group, send email to >>> clojure+u...@googlegroups.com >>> For more options, visit this group at >>> http://groups.google.com/group/clojure?hl=en >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Clojure" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to clojure+u...@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Understanding remove-ns and require
Hi, Apologies in advance for the possibly stupid question! I was having a problem with clojure.tools.namespace.repl/refresh where a namespace that had been successfully loaded subsequently failed to be reloaded when doing a repl/refresh. Whilst investigating the problem, (this is probably totally unrelated to my issue), I realised I didn't understand how remove-ns and require actually work. To illustrate this I did the following: (require '[foo.bar :as bar]) => nil (remove-ns 'foo.bar) => #namespace[foo.bar] (require '[foo.bar :as bar]) => CompilerException java.lang.Exception: namespace 'foo.bar' not found, compiling:(*cider-repl foo*:30:7) So what does remove-ns do to cause the second require to fail? Thanks Mark -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Understanding remove-ns and require
I don't have the answer, but perhaps a clue that might lead you to one: the function remove-lib inside of the tools.namespace library does this: (defn remove-lib "Remove lib's namespace and remove lib from the set of loaded libs." [lib] (remove-ns lib) (dosync (alter @#'clojure.core/*loaded-libs* disj lib))) I suspect that perhaps since in your example REPL session you are not doing anything to change clojure.core/*loaded-libs*, that perhaps it still contains the namespace name on which you called remove-ns. *loaded-libs* is modified by functions used to implement the behavior of 'require', because require tries to avoid re-loading namespaces that have already been loaded earlier. Andy On Tue, Dec 19, 2017 at 2:14 PM, Mark Melling wrote: > Hi, > > Apologies in advance for the possibly stupid question! > > I was having a problem with clojure.tools.namespace.repl/refresh where a > namespace that had been successfully loaded subsequently failed to be > reloaded when doing a repl/refresh. > > Whilst investigating the problem, (this is probably totally unrelated to > my issue), I realised I didn't understand how remove-ns and require > actually work. > > To illustrate this I did the following: > > (require '[foo.bar :as bar]) > => nil > (remove-ns 'foo.bar) > => #namespace[foo.bar] > (require '[foo.bar :as bar]) > => CompilerException java.lang.Exception: namespace 'foo.bar' not found, > compiling:(*cider-repl foo*:30:7) > > > > > So what does remove-ns do to cause the second require to fail? > > > > Thanks > > Mark > > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
