Re: [ANN] CIDER 0.19

[Tim Visher]

Thanks to you, Bozho ( ;) ) , and all
the other contributors for your consistent hard work on CIDER! :)

It's made my life and the life of my team so much better.

On Tue, Jan 1, 2019 at 10:39 AM Bozhidar Batsov  wrote:

> Btw, I also wrote a short blog post on the new release
> https://metaredux.com/posts/2019/01/01/happy-new-cider.html
>
> On Tue, 1 Jan 2019 at 12:24, Bozhidar Batsov  wrote:
>
>> Hey everyone!
>>
>> Happy New Year! Live long, live well, have fun and prosper!
>>
>> To help you start the New Year on the right foot and with the proper mood
>> I've just cut CIDER 0.19!
>>
>> The release notes are here
>> https://github.com/clojure-emacs/cider/releases/tag/v0.19.0
>>
>> The highlights are:
>>
>> * lots of improvements to the new connection management system
>> * some fixes for the changes to the error messages in Clojure 1.10
>>
>> That's going to be the final CIDER release that will work with the legacy
>> tools.nrepl, so if you're a boot or lein user you should definitely upgrade
>> to their latest versions which target the modern nREPL.
>>
>> Cheers!
>>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Reify (run vs cloverage) (single node vs cluster)

[cloje]

Thanks Justin,

I changed to deftype and it seems to be working fine on Cloverage on my 
local machine. I'm gonna try it on cluster mode and will update if that 
worked or not. 

Happy new year! :)


On Monday, December 31, 2018 at 9:36:07 PM UTC-6, Justin Smith wrote:
>
> Just a hunch, but many cluster / distribution tools expect that a given 
> Class name will refer to the same Class on each peer. You cannot ensure 
> this with reify- the name is auto-generated. The solution might be using 
> deftype or gen-class so that the class name would be deterministic and 
> shared on each instance.
>
> Another hunch, regarding cloverage, is that an error like that just from 
> checking test coverage means you have top level side effects (eg. top level 
> forms or defs that do some kind of IO or launch a process) - Clojure 
> doesn't have a "compile only" mode, and top level side effects will be 
> launched by any program that uses Clojure to load your file. The principled 
> solution is to move side effects into function bodies or delays, and pass 
> stateful resources as arguments to other code on startup rather than 
> binding them at the top level. There are multiple lifecycle management 
> libraries designed for managing this.
>
> On Mon, Dec 31, 2018 at 1:53 PM cloje > 
> wrote:
>
>> I'm really struck at the point where I'm using reify in clojure to 
>> implement java interface "org.apache.spark.sql.api.java.UDF2" and define 
>> the method "call" in this interface.
>>
>> I've been able to use reify to implement and it works fine when I do a 
>> "lein run", I see that this udf is being applied on my dataframe. I've also 
>> written test cases to check if the udf is being applied on the dataframe 
>> and I see that udf is being applied as expected. Now, when I do a "lein 
>> cloverage" to check my code coverage, this test case fails, giving a 
>> classNotFoundException.
>>
>> Partial error stack trace: (running the app on virtual desktop, could not 
>> copy and paste) 
>> org.apache.spark.sparkException: Job aborted due to stage failure: 
>> classNotFoundException: ab.cd.ef.ef$fn$reify__2440
>>
>>
>> *On Hadoop:*
>> Getting the same kind of error when I run this application on Cluster 
>> mode used spark-submit, tried hadoop jar; all give the same error as above. 
>> *But, 
>> *when I run the same jar on a single node, it works fine without any 
>> issues. 
>>
>> Using Clojure 1.9.0, Cloverage 1.0.13, Spark 2.1.3, Java 1.8.0_141
>>
>>
>> Has anyone been facing this kind of issue? if so, was there a solution?
>>
>> -- 
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clo...@googlegroups.com 
>> 
>> Note that posts from new members are moderated - please be patient with 
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+u...@googlegroups.com 
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to clojure+u...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

How does Executors/newScheduledThreadPool know how or where to parallelize work?

[lawrence . krubner]

I guess this is more of a JVM question than a Clojure question, unless 
Clojure exerts any special magic here. I'm open to a more Clojure approach 
than what I have now. 

Someone suggested I use Executors/newScheduledThreadPool for some recurring 
work, so I set it up like this: 

(def scheduler-aggregate
  (Executors/newScheduledThreadPool 32))

at the start I call:

  (.scheduleAtFixedRate  scheduler-aggregate ^Runnable (cycle-aggregate 
to-database-queue) 1 30 TimeUnit/MINUTES)

Aside from a try/catch block (which I just removed to simplify this 
example) the inner function looks like this:

(defn- cycle-aggregate
  [to-database-queue]
  (fn []
 (let [
   transcripts (query @global-database-connection {:item-type 
:transcript :processed { operators/$exists false }})
   ]
   (doseq [x transcripts]
 (aggregate-words x)
 (set-transcript-processed  @global-database-connection x)))

The function (aggregate-words) counts up a bunch of words, doing some prep 
work for a later NLP engine, and then there is this line: 

(log "The end of aggregate-words.")))

The whole process takes about 5 minutes to run, about 300 seconds. I watch 
the database and I see the number of new records increase. About every 10 
seconds I see these words appear in the logs: 

"The end of aggregate-words."

At the end of 5 minutes, these words have appeared 30 times, one for each 
of the transcripts I'm importing. 

This seems like I've done something wrong? Since the words "The end of 
aggregate-words."
appear at roughly equal intervals, and the transcripts are all about the 
same size, it seems that all of the transcripts are being handled on one 
thread. After all, if the 30 transcripts were handled on 30 threads, I'd 
expect the 30 calls to aggregate-words would all end at roughly the same 
time, instead of sequentially. 

What else do I need to do to parallelize this work? If I call (future) 
inside of aggregate-words, would the new thread come from the pool? Is 
there a way I can call aggregate-words and make sure it runs on its own 
thread from the pool? 












-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How does Executors/newScheduledThreadPool know how or where to parallelize work?

[Justin Smith]

A ScheduledThreadPool doesn't parallelize or partition your work, it
schedules tasks and keeps a pool of Thread objects it can reuse for that
purpose. If you need a job to be broken into smaller pieces, executed on a
schedule, you'll need to implement some sort of coordination. There's some
prior art in frameworks like Quartz that assist with tracking individual
tasks across hosts or restarts of an app, and things like Spark that are
designed for coordinating subdivided tasks and combining results.

As far as I know there's no magic tool that knows how to reliably subdivide
your task.

On Wed, Jan 2, 2019 at 12:47 PM  wrote:

> I guess this is more of a JVM question than a Clojure question, unless
> Clojure exerts any special magic here. I'm open to a more Clojure approach
> than what I have now.
>
> Someone suggested I use Executors/newScheduledThreadPool for some
> recurring work, so I set it up like this:
>
> (def scheduler-aggregate
>   (Executors/newScheduledThreadPool 32))
>
> at the start I call:
>
>   (.scheduleAtFixedRate  scheduler-aggregate ^Runnable (cycle-aggregate
> to-database-queue) 1 30 TimeUnit/MINUTES)
>
> Aside from a try/catch block (which I just removed to simplify this
> example) the inner function looks like this:
>
> (defn- cycle-aggregate
>   [to-database-queue]
>   (fn []
>  (let [
>transcripts (query @global-database-connection {:item-type
> :transcript :processed { operators/$exists false }})
>]
>(doseq [x transcripts]
>  (aggregate-words x)
>  (set-transcript-processed  @global-database-connection x)))
>
> The function (aggregate-words) counts up a bunch of words, doing some prep
> work for a later NLP engine, and then there is this line:
>
> (log "The end of aggregate-words.")))
>
> The whole process takes about 5 minutes to run, about 300 seconds. I watch
> the database and I see the number of new records increase. About every 10
> seconds I see these words appear in the logs:
>
> "The end of aggregate-words."
>
> At the end of 5 minutes, these words have appeared 30 times, one for each
> of the transcripts I'm importing.
>
> This seems like I've done something wrong? Since the words "The end of
> aggregate-words."
> appear at roughly equal intervals, and the transcripts are all about the
> same size, it seems that all of the transcripts are being handled on one
> thread. After all, if the 30 transcripts were handled on 30 threads, I'd
> expect the 30 calls to aggregate-words would all end at roughly the same
> time, instead of sequentially.
>
> What else do I need to do to parallelize this work? If I call (future)
> inside of aggregate-words, would the new thread come from the pool? Is
> there a way I can call aggregate-words and make sure it runs on its own
> thread from the pool?
>
>
>
>
>
>
>
>
>
>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How does Executors/newScheduledThreadPool know how or where to parallelize work?

[Matching Socks]

You are right, this is not a Clojure question. I suppose your scheduled 
task could put a bunch of tasks on a queue and a thread-pool executor could 
work the queue in multiple threads.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How does Executors/newScheduledThreadPool know how or where to parallelize work?

[Orestis Markou]

If you call future, you will be using the future threadpool which is unbounded. 
If you know there’s always going to be a small amount of tasks in-flight, you 
can safely just use future to parallelize your work, and the scheduled executor 
to trigger trigger the work periodically. 

Have a look at https://github.com/TheClimateCorporation/claypoole as well. 

3 Ιαν 2019, 12:23 πμ, ο χρήστης «Matching Socks » έγραψε:

> You are right, this is not a Clojure question. I suppose your scheduled task 
> could put a bunch of tasks on a queue and a thread-pool executor could work 
> the queue in multiple threads.
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your 
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> --- 
> You received this message because you are subscribed to the Google Groups 
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.