Re: Poor parallelization performance across 18 cores (but not 4)

[Niels van Klaveren]

Could you also show how you are running these functions in parallel and 
time them ? The way you start the functions can have as much impact as the 
functions themselves.

Regards,
Niels

On Tuesday, November 17, 2015 at 6:38:39 AM UTC+1, David Iba wrote:
>
> I have functions f1 and f2 below, and let's say they run in T1 and T2 
> amount of time when running a single instance/thread.  The issue I'm facing 
> is that parallelizing f2 across 18 cores takes anywhere from 2-5X T2, and 
> for more complex funcs takes absurdly long.
>
>
>1. (defn f1 []
>2.   (apply + (range 2e9)))
>3.  
>4. ;; Note: each call to (f2) makes its own x* atom, so the 'swap!' 
>should never retry.
>5. (defn f2 []
>6.   (let [x* (atom {})]
>7. (loop [i 1e9]
>8.   (when-not (zero? i)
>9. (swap! x* assoc :k i)
>10. (recur (dec i))
>
>
> Of note:
> - On a 4-core machine, both f1 and f2 parallelize well (roungly T1 and T2 
> for 4 runs in parallel)
> - running 18 f1's in parallel on the 18-core machine also parallelizes 
> well.
> - Disabling hyperthreading doesn't help.
> - Based on jvisualvm monitoring, doesn't seem to be GC-related
> - also tried on dedicated 18-core ec2 instance with same issues, so not 
> shared-tenancy-related
> - if I make a jar that runs a single f2 and launch 18 in parallel, it 
> parallelizes well (so I don't think it's machine/aws-related)
>
> Could it be that the 18 f2's in parallel on a single JVM instance is 
> overworking the STM with all the swap's?  Any other theories?
>
> Thanks!
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ClassNotFoundException: clojure.lang.AFunction on OS X

[Phil Segal]

Hi

I have found the cause this morning. The ContextClassLoader on the 
currentthread was null. I managed to work around this by checking for null 
and then setting it as the SystemClassLoader if needed. 

I am using lein 2.5.3, java 1.8.0_65.

The ClassLoader issue only occurs on Mac, but there seem to be no other 
examples of this online which suggest its specific to our setup. We are 
using JavaFX, Clj, Cljs and Java. 

I will continue to explore and see whether when we remove JavaFX dependency 
in the next few months that resolves it. 

For now, I have at least found a workaround.

On Monday, 16 November 2015 23:31:49 UTC, Gary Verhaegen wrote:
>
> I have no idea what this could be. Is there any chance you could build up 
> a minimal-ish project that reproduces this ? I have been using Clojure on a 
> Mac for years without any problem.
>
> Also, at much less effort on your part, could you post the Leiningen and 
> JVM versions ? lein version should print both.
>
> On Monday, 16 November 2015, Phil Segal  > wrote:
>
>> We have been happily using clojure on windows and linux, today we tried 
>> to run on a MacBook Pro and ran into several issues.
>>
>> The current stopper is a CNFE as per the title (stack trace at the end).
>>
>> I have now replicated this on my personal Mac and seen the same 
>> behaviour. I can't find anything that suggests that there is anything 
>> special to do.
>>
>> I installed leiningen using brew, and am using clojure 1.7.0 as a maven 
>> dependency in our java code.
>>
>> The issue is occurring in a java constructor which is then calling:
>>
>> IFn require = Clojure.var("clojure.core", "require");
>>
>> require.invoke(Clojure.read("com.xxx.our-name-space"), RELOAD);
>> aFn = Clojure.var("com.xxx-our-name-space", "a-function");
>>
>>
>> Any assistance would be amazing. 
>>
>> Thanks
>> Phil
>>
>> Caused by: java.lang.NoClassDefFoundError: clojure/lang/AFunction
>> at java.lang.ClassLoader.defineClass1(Native Method)
>> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
>> at java.lang.ClassLoader.defineClass(ClassLoader.java:642)
>> at clojure.lang.DynamicClassLoader.defineClass(DynamicClassLoader.java:46)
>> at clojure.lang.Compiler$ObjExpr.getCompiledClass(Compiler.java:4833)
>> at clojure.lang.Compiler$FnExpr.parse(Compiler.java:3993)
>> at clojure.lang.Compiler.analyzeSeq(Compiler.java:6721)
>> ... 32 more
>> Caused by: java.lang.ClassNotFoundException: clojure.lang.AFunction
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>> at clojure.lang.DynamicClassLoader.findClass(DynamicClassLoader.java:69)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>> at clojure.lang.DynamicClassLoader.loadClass(DynamicClassLoader.java:77)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>> ... 39 more
>>
>> -- 
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with 
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ClassNotFoundException: clojure.lang.AFunction on OS X

[Phil Segal]

It appears this is down to JavaFX. Having skipped the javaFX part, the 
system runs fine.

On Tuesday, November 17, 2015 at 8:53:50 AM UTC, Phil Segal wrote:
>
> Hi
>
> I have found the cause this morning. The ContextClassLoader on the 
> currentthread was null. I managed to work around this by checking for null 
> and then setting it as the SystemClassLoader if needed. 
>
> I am using lein 2.5.3, java 1.8.0_65.
>
> The ClassLoader issue only occurs on Mac, but there seem to be no other 
> examples of this online which suggest its specific to our setup. We are 
> using JavaFX, Clj, Cljs and Java. 
>
> I will continue to explore and see whether when we remove JavaFX 
> dependency in the next few months that resolves it. 
>
> For now, I have at least found a workaround.
>
> On Monday, 16 November 2015 23:31:49 UTC, Gary Verhaegen wrote:
>>
>> I have no idea what this could be. Is there any chance you could build up 
>> a minimal-ish project that reproduces this ? I have been using Clojure on a 
>> Mac for years without any problem.
>>
>> Also, at much less effort on your part, could you post the Leiningen and 
>> JVM versions ? lein version should print both.
>>
>> On Monday, 16 November 2015, Phil Segal  wrote:
>>
>>> We have been happily using clojure on windows and linux, today we tried 
>>> to run on a MacBook Pro and ran into several issues.
>>>
>>> The current stopper is a CNFE as per the title (stack trace at the end).
>>>
>>> I have now replicated this on my personal Mac and seen the same 
>>> behaviour. I can't find anything that suggests that there is anything 
>>> special to do.
>>>
>>> I installed leiningen using brew, and am using clojure 1.7.0 as a maven 
>>> dependency in our java code.
>>>
>>> The issue is occurring in a java constructor which is then calling:
>>>
>>> IFn require = Clojure.var("clojure.core", "require");
>>>
>>> require.invoke(Clojure.read("com.xxx.our-name-space"), RELOAD);
>>> aFn = Clojure.var("com.xxx-our-name-space", "a-function");
>>>
>>>
>>> Any assistance would be amazing. 
>>>
>>> Thanks
>>> Phil
>>>
>>> Caused by: java.lang.NoClassDefFoundError: clojure/lang/AFunction
>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:642)
>>> at 
>>> clojure.lang.DynamicClassLoader.defineClass(DynamicClassLoader.java:46)
>>> at clojure.lang.Compiler$ObjExpr.getCompiledClass(Compiler.java:4833)
>>> at clojure.lang.Compiler$FnExpr.parse(Compiler.java:3993)
>>> at clojure.lang.Compiler.analyzeSeq(Compiler.java:6721)
>>> ... 32 more
>>> Caused by: java.lang.ClassNotFoundException: clojure.lang.AFunction
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>>> at clojure.lang.DynamicClassLoader.findClass(DynamicClassLoader.java:69)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>> at clojure.lang.DynamicClassLoader.loadClass(DynamicClassLoader.java:77)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>> ... 39 more
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clojure@googlegroups.com
>>> Note that posts from new members are moderated - please be patient with 
>>> your first post.
>>> To unsubscribe from this group, send email to
>>> clojure+unsubscr...@googlegroups.com
>>> For more options, visit this group at
>>> http://groups.google.com/group/clojure?hl=en
>>> --- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Clojure" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to clojure+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Poor parallelization performance across 18 cores (but not 4)

[David Iba]

Andy:  Interesting.  Thanks for educating me on the fact that atom swap's 
don't use the STM.  Your theory seems plausible... I will try those tests 
next time I launch the 18-core instance, but yeah, not sure how 
illuminating the results will be.

Niels: along the lines of this (so that each thread prints its time as well 
as printing the overall time):

   1.   (time
   2.(let [f f1
   3.  n-runs 18
   4.  futs (do (for [i (range n-runs)]
   5. (future (time (f)]
   6.  (doseq [fut futs]
   7.@fut)))
   

On Tuesday, November 17, 2015 at 5:33:01 PM UTC+9, Niels van Klaveren wrote:
>
> Could you also show how you are running these functions in parallel and 
> time them ? The way you start the functions can have as much impact as the 
> functions themselves.
>
> Regards,
> Niels
>
> On Tuesday, November 17, 2015 at 6:38:39 AM UTC+1, David Iba wrote:
>>
>> I have functions f1 and f2 below, and let's say they run in T1 and T2 
>> amount of time when running a single instance/thread.  The issue I'm facing 
>> is that parallelizing f2 across 18 cores takes anywhere from 2-5X T2, and 
>> for more complex funcs takes absurdly long.
>>
>>
>>1. (defn f1 []
>>2.   (apply + (range 2e9)))
>>3.  
>>4. ;; Note: each call to (f2) makes its own x* atom, so the 'swap!' 
>>should never retry.
>>5. (defn f2 []
>>6.   (let [x* (atom {})]
>>7. (loop [i 1e9]
>>8.   (when-not (zero? i)
>>9. (swap! x* assoc :k i)
>>10. (recur (dec i))
>>
>>
>> Of note:
>> - On a 4-core machine, both f1 and f2 parallelize well (roungly T1 and T2 
>> for 4 runs in parallel)
>> - running 18 f1's in parallel on the 18-core machine also parallelizes 
>> well.
>> - Disabling hyperthreading doesn't help.
>> - Based on jvisualvm monitoring, doesn't seem to be GC-related
>> - also tried on dedicated 18-core ec2 instance with same issues, so not 
>> shared-tenancy-related
>> - if I make a jar that runs a single f2 and launch 18 in parallel, it 
>> parallelizes well (so I don't think it's machine/aws-related)
>>
>> Could it be that the 18 f2's in parallel on a single JVM instance is 
>> overworking the STM with all the swap's?  Any other theories?
>>
>> Thanks!
>>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Poor parallelization performance across 18 cores (but not 4)

[David Iba]

correction: that "do" should be a "doall".  (My actual test code was a bit 
different, but each run printed some info when it started so it doesn't 
have to do with delayed evaluation of lazy seq's or anything).

On Tuesday, November 17, 2015 at 6:49:16 PM UTC+9, David Iba wrote:
>
> Andy:  Interesting.  Thanks for educating me on the fact that atom swap's 
> don't use the STM.  Your theory seems plausible... I will try those tests 
> next time I launch the 18-core instance, but yeah, not sure how 
> illuminating the results will be.
>
> Niels: along the lines of this (so that each thread prints its time as 
> well as printing the overall time):
>
>1.   (time
>2.(let [f f1
>3.  n-runs 18
>4.  futs (do (for [i (range n-runs)]
>5. (future (time (f)]
>6.  (doseq [fut futs]
>7.@fut)))
>
>
> On Tuesday, November 17, 2015 at 5:33:01 PM UTC+9, Niels van Klaveren 
> wrote:
>>
>> Could you also show how you are running these functions in parallel and 
>> time them ? The way you start the functions can have as much impact as the 
>> functions themselves.
>>
>> Regards,
>> Niels
>>
>> On Tuesday, November 17, 2015 at 6:38:39 AM UTC+1, David Iba wrote:
>>>
>>> I have functions f1 and f2 below, and let's say they run in T1 and T2 
>>> amount of time when running a single instance/thread.  The issue I'm facing 
>>> is that parallelizing f2 across 18 cores takes anywhere from 2-5X T2, and 
>>> for more complex funcs takes absurdly long.
>>>
>>>
>>>1. (defn f1 []
>>>2.   (apply + (range 2e9)))
>>>3.  
>>>4. ;; Note: each call to (f2) makes its own x* atom, so the 'swap!' 
>>>should never retry.
>>>5. (defn f2 []
>>>6.   (let [x* (atom {})]
>>>7. (loop [i 1e9]
>>>8.   (when-not (zero? i)
>>>9. (swap! x* assoc :k i)
>>>10. (recur (dec i))
>>>
>>>
>>> Of note:
>>> - On a 4-core machine, both f1 and f2 parallelize well (roungly T1 and 
>>> T2 for 4 runs in parallel)
>>> - running 18 f1's in parallel on the 18-core machine also parallelizes 
>>> well.
>>> - Disabling hyperthreading doesn't help.
>>> - Based on jvisualvm monitoring, doesn't seem to be GC-related
>>> - also tried on dedicated 18-core ec2 instance with same issues, so not 
>>> shared-tenancy-related
>>> - if I make a jar that runs a single f2 and launch 18 in parallel, it 
>>> parallelizes well (so I don't think it's machine/aws-related)
>>>
>>> Could it be that the 18 f2's in parallel on a single JVM instance is 
>>> overworking the STM with all the swap's?  Any other theories?
>>>
>>> Thanks!
>>>
>>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

ClojureExchange ticket to give away

[Torsten Uhlmann]

Hey,

I have bought a ClojureExchange [1] ticket right after last years 
conference,  but I will not be able to participate this year.

ClojureExchange takes place in London from Dec. 3 - Dec. 5.

I would primarily like to give the ticket to someone who would otherwise 
not be able to participate, I'll give it away free of charge.

Anyone interested please answer me privately through the groups website and 
I'll pass your details to the Skillmatter folks 
so they can change the ticket pass.

I wish an awesome time to all people that meet there this year!

Torsten.

[1] - https://skillsmatter.com/conferences/6861-clojure-exchange-2015

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

best practice with forked libraries

[Bobby Bobble]

Say there's a dependency that I fork and add some features I need for my 
team's project. I can use my fork locally with lein install no problem, and 
so can others *if* they clone my fork and do the same. It would be more 
convenient if I could publish my fork to Clojars and depend on that, and 
maybe down the line I can return to depending on the original if my PR is 
accepted. What's the best thing to do here ?

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ClojureExchange ticket to give away

[Torsten Uhlmann]

The ticket is gone.

Am Dienstag, 17. November 2015 11:35:36 UTC+1 schrieb Torsten Uhlmann:
>
> Hey,
>
> I have bought a ClojureExchange [1] ticket right after last years 
> conference,  but I will not be able to participate this year.
>
> ClojureExchange takes place in London from Dec. 3 - Dec. 5.
>
> I would primarily like to give the ticket to someone who would otherwise 
> not be able to participate, I'll give it away free of charge.
>
> Anyone interested please answer me privately through the groups website 
> and I'll pass your details to the Skillmatter folks 
> so they can change the ticket pass.
>
> I wish an awesome time to all people that meet there this year!
>
> Torsten.
>
> [1] - https://skillsmatter.com/conferences/6861-clojure-exchange-2015
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: best practice with forked libraries

[Ray Miller]

On 17 November 2015 at 13:21, Bobby Bobble  wrote:

> Say there's a dependency that I fork and add some features I need for my
> team's project. I can use my fork locally with lein install no problem, and
> so can others *if* they clone my fork and do the same. It would be more
> convenient if I could publish my fork to Clojars and depend on that, and
> maybe down the line I can return to depending on the original if my PR is
> accepted. What's the best thing to do here ?
>

There's a convention in Clojars of deploying a non-canonical fork by
renaming the project to org.clojars.USERNAME/PROJ_NAME and deploying that
to Clojars.

See, for example: https://clojars.org/search?q=clj-aws-s3

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Poor parallelization performance across 18 cores (but not 4)

[Andy Fingerhut]

David, you say "Based on jvisualvm monitoring, doesn't seem to be
GC-related".

What is jvisualvm showing you related to GC and/or memory allocation when
you tried the 18-core version with 18 threads in the same process?

Even memory allocation could become a point of contention, depending upon
how the memory allocation works with many threads.  e.g. Depends on whether
a thread gets a large chunk of memory on a global lock, and then locally
carves it up into the small pieces it needs for each individual Java 'new'
allocation, or gets a global lock for every 'new'.  The latter would give
terrible performance as # cores increase, but I don't know how to tell
whether that is the case, except by knowing more about how the memory
allocator is implemented in your JVM.  Maybe digging through OpenJDK source
code in the right place would tell?

Andy

On Tue, Nov 17, 2015 at 2:00 AM, David Iba  wrote:

> correction: that "do" should be a "doall".  (My actual test code was a bit
> different, but each run printed some info when it started so it doesn't
> have to do with delayed evaluation of lazy seq's or anything).
>
>
> On Tuesday, November 17, 2015 at 6:49:16 PM UTC+9, David Iba wrote:
>>
>> Andy:  Interesting.  Thanks for educating me on the fact that atom swap's
>> don't use the STM.  Your theory seems plausible... I will try those tests
>> next time I launch the 18-core instance, but yeah, not sure how
>> illuminating the results will be.
>>
>> Niels: along the lines of this (so that each thread prints its time as
>> well as printing the overall time):
>>
>>1.   (time
>>2.(let [f f1
>>3.  n-runs 18
>>4.  futs (do (for [i (range n-runs)]
>>5. (future (time (f)]
>>6.  (doseq [fut futs]
>>7.@fut)))
>>
>>
>> On Tuesday, November 17, 2015 at 5:33:01 PM UTC+9, Niels van Klaveren
>> wrote:
>>>
>>> Could you also show how you are running these functions in parallel and
>>> time them ? The way you start the functions can have as much impact as the
>>> functions themselves.
>>>
>>> Regards,
>>> Niels
>>>
>>> On Tuesday, November 17, 2015 at 6:38:39 AM UTC+1, David Iba wrote:

 I have functions f1 and f2 below, and let's say they run in T1 and T2
 amount of time when running a single instance/thread.  The issue I'm facing
 is that parallelizing f2 across 18 cores takes anywhere from 2-5X T2, and
 for more complex funcs takes absurdly long.


1. (defn f1 []
2.   (apply + (range 2e9)))
3.
4. ;; Note: each call to (f2) makes its own x* atom, so the 'swap!'
should never retry.
5. (defn f2 []
6.   (let [x* (atom {})]
7. (loop [i 1e9]
8.   (when-not (zero? i)
9. (swap! x* assoc :k i)
10. (recur (dec i))


 Of note:
 - On a 4-core machine, both f1 and f2 parallelize well (roungly T1 and
 T2 for 4 runs in parallel)
 - running 18 f1's in parallel on the 18-core machine also parallelizes
 well.
 - Disabling hyperthreading doesn't help.
 - Based on jvisualvm monitoring, doesn't seem to be GC-related
 - also tried on dedicated 18-core ec2 instance with same issues, so not
 shared-tenancy-related
 - if I make a jar that runs a single f2 and launch 18 in parallel, it
 parallelizes well (so I don't think it's machine/aws-related)

 Could it be that the 18 f2's in parallel on a single JVM instance is
 overworking the STM with all the swap's?  Any other theories?

 Thanks!

>>> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups

Re: best practice with forked libraries

[Paul deGrandis]

In addition to Ray's response (which I would also suggest), I'd encourage 
you to reach out to the project maintainer/author with your additions - 
it's always nice to see how your project is being used and the shortcomings 
that others are hacking around.

Cheers,
Paul

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: best practice with forked libraries

[Mimmo Cosenza]

A lot of time ago I wrote the following tutorial on this topic.

https://github.com/magomimmo/modern-cljs/blob/master/doc/tutorial-19.md 


HIH

mimmo

> On 17 Nov 2015, at 20:48, Paul deGrandis  wrote:
> 
> In addition to Ray's response (which I would also suggest), I'd encourage you 
> to reach out to the project maintainer/author with your additions - it's 
> always nice to see how your project is being used and the shortcomings that 
> others are hacking around.
> 
> Cheers,
> Paul
> 
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your 
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en 
> 
> --- 
> You received this message because you are subscribed to the Google Groups 
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to clojure+unsubscr...@googlegroups.com 
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Is there a "State of Clojure 2015" coming?

[Philip Markgraf]

One of the topics I had hoped to see coming out of the Conj is a 2015 
update to the State of Clojure survey. Is the survey something that is in 
the plans for 2015?

Thanks!

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Is there a "State of Clojure 2015" coming?

[Alex Miller]

Yes! It's basically ready to go but I ran out of time to post it before the 
conj. coming soon

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Is there a "State of Clojure 2015" coming?

[Philip Markgraf]

Outstanding! thank you for the effort, Alex.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.