[ANN] Tufte, simple profiling and performance monitoring for Clojure/Script

[Peter Taoussanis]

On Clojars, and GitHub at: https://github.com/ptaoussanis/tufte


Basically took the weekend to clean up Timbre's old profiling ns, 
generalize it, and move it to its own lib. Biggest difference for most 
folks will probably be the new documentation.


Enjoy, cheers! :-)

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Thoughts on clojure.spec

[Rich Hickey]

Kovas is right on the money here. Inherently when you make something 
‘programmable’ generating correct programs for it is hard.

I would say though, I frequently see people struggling to spec more complex 
logic are going directly to independent input generation. Thus, how will inputs 
be reasonably coordinated etc. The better approach is to first generate a 
consistent model, then generate inputs from that model. gen/bind is your 
friend. Typically the approach is:

(gen/bind model-generator (fn [model] data-generator))

Each model then need not be exhaustively large because many models will be 
generated, and the model lets you share things between what would otherwise be 
independent generators.

On the specific points:

> 1. No way to test function output specs.  For documentation purposes, I want 
> to have an output spec on my function.  However, as far as I know, after 
> instrumentation-triggered checking of output specs was removed a couple of 
> alphas ago, the only way remaining to check against output specs is to use 
> randomly generated tests.  So if I can't make good generators, I have no way 
> to confirm that my output spec works the way I think it does.  My 
> documentation could be totally out of touch with reality, and that displeases 
> me.

Running return-value instrument-style checking on whatever few hand-written 
tests you might have isn’t going to give you better coverage than a simple 
(even hardwired) generator that captures similar ranges. And you can directly 
exercise your :ret spec - it’s just another spec. You can also spec/assert on 
your return - the tests will disappear in production, although this is 
similarly as weak as instrument-triggered return checking, so I’m not going to 
help you do it, but you can do it yourself :)

> 2. Limited ability for testing that functions take and receive what you 
> expect.  Whereas a static type system can prove certain properties about 
> whether functions are called with valid inputs, with spec, you only get those 
> guarantees if you pump a function with enough valid random data to trigger 
> the function calling all its callees with interesting combinations of data.  
> But if I use the naive generators, the function will never even complete with 
> most of the randomly generated input, let alone call other functions in a 
> useful way.  And in many cases I don't see how to generate something of 
> better quality.

You can certainly use core.typed if you want to check the kinds of things it 
can check in the way it can check them. But saying specs aren’t types says 
little about either, and using the word ‘valid’ in both contexts without 
qualification equates things that are not the same. It’s not as if most type 
systems can check the predicates spec can (Idris et al aside). There’s a big 
difference between type-correct inputs/outputs and stakeholder-correct 
programs. spec is oriented towards the latter, but, the bar, as you have seen, 
is generation. It is, however, a local challenge - ‘this particular fn is hard 
to gen for’.  The bar for static typing is flowing types everywhere, a bar I 
consider to be more global, harder to meet, and less expressive.

> One other issue I've had, unrelated to generators, is that I'm struggling to 
> express higher-order type constraints, for example, this function takes a 
> vector v1 of anything and a vector v2 of anything, but the type of the things 
> in vector v1 better match the type of the things in vector v2.

Parameterized types definitely have advantages for the subset of things they 
can check in this regard. But again, it is a logical fallacy to equate spec == 
type then switch to static type checking ‘advantages' as if they had the same 
expressive power - in general they don’t, since specs are predicative and of 
runtime values. Most type systems can’t tell you that the things in v1 satisfy 
the same predicates as do the things in v2 for any predicate other than ‘is 
statically a T’ (Idris et al aside). That said, I have been thinking about the 
‘parameterized spec’ problem, nothing concrete to show yet.

Rich

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ANN] Clojure 1.9.0-alpha10

[Alex Miller]

Clojure 1.9.0-alpha10 is now available.

Try it via

- Download: https://repo1.maven.org/maven2/org/clojure/clojure/1.9.0-alpha10
- Leiningen: [org.clojure/clojure "1.9.0-alpha10"]

1.9.0-alpha10 includes the following changes since 1.9.0-alpha9:

- NEW clojure.core/any? - a predicate that matches anything. any? has 
built-in gen support. The :clojure.spec/any spec has been removed. 
Additionally, gen support has been added for some?.

- keys* will now gen

- gen overrides (see c.s/gen, c.s./exercise, c.s.t/check, c.s.t/instrument) 
now expect no-arg functions that return gens, rather than gens

- CLJ-1977 - fix regression from alpha9 in data conversion of Throwable 
when stack trace is empty

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[JOB] Product Engineer at CircleCI

[Brave Clojure Jobs]

CircleCI is hiring a product engineer to learn and work in Clojure and 
Clojurescript with one of the most experienced Clojure teams in the world. 
Hiring in San Francisco and Toronto.

More 
details: 
https://jobs.braveclojure.com/jobs/17592186046002/product-engineer-circleci

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Rationale for `keys` not supporting vectors?

[Sam Estep]

I asked a question about this on Stack Overflow; you may be interested in 
the discussion there: http://stackoverflow.com/q/35341969/5044950

On Saturday, July 9, 2016 at 12:50:22 AM UTC-4, Michael Gardner wrote:
>
> I've looked around, but couldn't find any discussion on the topic. Is it 
> purely an implementation thing, or a design choice? 
>
> (Yes, I realize you can just do (range (count v)).)

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Clojure 1.9.0-alpha10: Semantic mismatch: any? vs not-any?

[Alan Thompson]

In clojure we are used to shortcuts involving not:

(when-not x ...) => (when (not x) ...)
(if-not x ...)   => (if (not x) ...)
(not-every? pred coll)   => (not (every? pred coll))
(not-any? pred coll) => (not (some pred coll))

However, the new function clojure.core/any? breaks this semantic pattern:

​> ​
(doc clojure.core/any?)
-
clojure.core/any?
([x])
  Returns true given any argument.

​> ​
(doc clojure.core/not-any?)
-
clojure.core/not-any?
([pred coll])
  Returns false if (pred x) is logical true for any x in coll,
  else true.

​These two functions are not only unrelated to each other, but they don't
even accept the same number or type of arguments.  Moreover, ​there is an
even more surprising property:

> (clojure.core/any? nil)
true

I would have bet money that, at least for nil, the result would not have
been true.
​
Given the significant prior conventions in Clojure​ for functions like
some, every?, *-not, not-*, and also the general handling of nil, it seems
that the new any? function is bound to cause much confusion &
consternation, especially among people learning Clojure.

Given the degenerate definition:

> (source clojure.core/any?)
(defn any?
  [x] true)

would it not be simpler and more instinctive to rename the function
clojure.core/true:

(defn clojure.core/true
  [x] true)

We could then have code with the obvious result:

(true 1) => true
(true "hi")  => true
(true [])=> true
(true nil)   => true

I believe that such a change would help to keep Clojure in line with users
instincts and assumptions, as well as past Clojure practices.  I have often
felt that one of the most important principles in any sort of software
development is adherence to the Principle of Least Astonishment
.

Alan



On Mon, Jul 11, 2016 at 7:28 AM, Alex Miller  wrote:

>
> 1.9.0-alpha10 includes the following changes since 1.9.0-alpha9:
>
> - NEW clojure.core/any? - a predicate that matches anything. any? has
> built-in gen support. The :clojure.spec/any spec has been removed.
> Additionally, gen support has been added for some?.
>
>
>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Secondar Sorting in spark using clojure/flambo

[Blake Miller]

Hi Punit

The behavior you are referring to is a feature of the Scala compiler, which 
is why it does not happen automatically when you try to use it from Clojure.

Please see the note here:

https://github.com/t6/from-scala/blob/4e1752aaa2ef835dd67a8404273bee067510a431/test/t6/from_scala/guide.clj#L161-L166

You may find that library a useful resource, either as a dependency or 
simply as reference material.

What you want to do is find the full method signature, including the 
implicits, and invoke _that_ from clojure, passing values for all implicit 
parameters (in this case, your custom ordering function.

HTH

On Saturday, July 9, 2016 at 6:13:17 AM UTC, Punit Naik wrote:
>
> Hi Ashish
>
> The "package" is indeed the full package name.
> On 09-Jul-2016 11:02 AM, "Ashish Negi"  > wrote:
>
>> Should not be `package` in `:import` be the actual package name of  `
>> RFMCPartitioner` ?
>>
>> see examples at https://clojuredocs.org/clojure.core/import
>>
>> like :
>>
>> (ns foo.bar
>>   (:import (java.util Date
>>   Calendar)
>>(java.util.logging Logger
>>   Level)))
>>
>>
>>
>> (ns xyz
>>   (:import
>> [**  RFMCPartitioner]
>> [** RFMCKey]
>> )
>>   )
>>
>>
>> where ** is package full name.
>>
>>
>>
>> On Friday, 8 July 2016 21:31:27 UTC+5:30, Punit Naik wrote:
>>>
>>>
>>>  
>>>
>>> I have a scala program in which I have implemented a secondary sort 
>>> which works perfectly. The way I have written that program is:
>>>
>>> object rfmc {
>>>   // Custom Key and partitioner
>>>
>>>   case class RFMCKey(cId: String, R: Double, F: Double, M: Double, C: 
>>> Double)
>>>   class RFMCPartitioner(partitions: Int) extends Partitioner {
>>> require(partitions >= 0, "Number of partitions ($partitions) cannot be 
>>> negative.")
>>> override def numPartitions: Int = partitions
>>> override def getPartition(key: Any): Int = {
>>>   val k = key.asInstanceOf[RFMCKey]
>>>   k.cId.hashCode() % numPartitions
>>> }
>>>   }
>>>   object RFMCKey {
>>> implicit def orderingBycId[A <: RFMCKey] : Ordering[A] = {
>>>   Ordering.by(k => (k.R, k.F * -1, k.M * -1, k.C * -1))
>>> }
>>>   }
>>>   // The body of the code
>>>   //
>>>   //
>>>   val x = rdd.map(RFMCKey(cust,r,f,m,c), r+","+f+","+m+","+c)
>>>   val y = x.repartitionAndSortWithinPartitions(new RFMCPartitioner(1))}
>>>
>>> I wanted to implement the same thing using clojure's DSL for spark 
>>> called flambo. Since I can't write partitioner using clojure, I re-used the 
>>> code defind above, compiled it and used it as a dependency in my Clojure 
>>> code.
>>>
>>> Now I am importing the partitioner and the key in my clojure code the 
>>> following way:
>>>
>>> (ns xyz
>>>   (:import
>>> [package RFMCPartitioner]
>>> [package RFMCKey]
>>> )
>>>   )
>>>
>>> But when I try to create RFMCKey by doing (RFMCKey. cust_id r f m c), 
>>> it throws the following error:
>>>
>>> java.lang.ClassCastException: org.formcept.wisdom.RFMCKey cannot be cast to 
>>> java.lang.Comparable
>>> at 
>>> org.spark-project.guava.collect.NaturalOrdering.compare(NaturalOrdering.java:28)
>>> at 
>>> scala.math.LowPriorityOrderingImplicits$$anon$7.compare(Ordering.scala:153)
>>> at 
>>> org.apache.spark.util.collection.ExternalSorter$$anon$8.compare(ExternalSorter.scala:170)
>>> at 
>>> org.apache.spark.util.collection.ExternalSorter$$anon$8.compare(ExternalSorter.scala:164)
>>> at 
>>> org.apache.spark.util.collection.TimSort.countRunAndMakeAscending(TimSort.java:252)
>>> at org.apache.spark.util.collection.TimSort.sort(TimSort.java:110)
>>> at org.apache.spark.util.collection.Sorter.sort(Sorter.scala:37)
>>> at 
>>> org.apache.spark.util.collection.SizeTrackingPairBuffer.destructiveSortedIterator(SizeTrackingPairBuffer.scala:83)
>>> at 
>>> org.apache.spark.util.collection.ExternalSorter.partitionedIterator(ExternalSorter.scala:687)
>>> at 
>>> org.apache.spark.util.collection.ExternalSorter.iterator(ExternalSorter.scala:705)
>>> at 
>>> org.apache.spark.shuffle.hash.HashShuffleReader.read(HashShuffleReader.scala:64)
>>> at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:92)
>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:70)
>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:242)
>>> at 
>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> at 
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>  

Re: Secondar Sorting in spark using clojure/flambo

[Punit Naik]

Hi Black

Thanks for the reply but  figured it out on my own. Posting the answer 
after this.

On Monday, July 11, 2016 at 11:42:10 PM UTC+5:30, Blake Miller wrote:
>
> Hi Punit
>
> The behavior you are referring to is a feature of the Scala compiler, 
> which is why it does not happen automatically when you try to use it from 
> Clojure.
>
> Please see the note here:
>
>
> https://github.com/t6/from-scala/blob/4e1752aaa2ef835dd67a8404273bee067510a431/test/t6/from_scala/guide.clj#L161-L166
>
> You may find that library a useful resource, either as a dependency or 
> simply as reference material.
>
> What you want to do is find the full method signature, including the 
> implicits, and invoke _that_ from clojure, passing values for all implicit 
> parameters (in this case, your custom ordering function.
>
> HTH
>
> On Saturday, July 9, 2016 at 6:13:17 AM UTC, Punit Naik wrote:
>>
>> Hi Ashish
>>
>> The "package" is indeed the full package name.
>> On 09-Jul-2016 11:02 AM, "Ashish Negi"  wrote:
>>
>>> Should not be `package` in `:import` be the actual package name of  `
>>> RFMCPartitioner` ?
>>>
>>> see examples at https://clojuredocs.org/clojure.core/import
>>>
>>> like :
>>>
>>> (ns foo.bar
>>>   (:import (java.util Date
>>>   Calendar)
>>>(java.util.logging Logger
>>>   Level)))
>>>
>>>
>>>
>>> (ns xyz
>>>   (:import
>>> [**  RFMCPartitioner]
>>> [** RFMCKey]
>>> )
>>>   )
>>>
>>>
>>> where ** is package full name.
>>>
>>>
>>>
>>> On Friday, 8 July 2016 21:31:27 UTC+5:30, Punit Naik wrote:


  

 I have a scala program in which I have implemented a secondary sort 
 which works perfectly. The way I have written that program is:

 object rfmc {
   // Custom Key and partitioner

   case class RFMCKey(cId: String, R: Double, F: Double, M: Double, C: 
 Double)
   class RFMCPartitioner(partitions: Int) extends Partitioner {
 require(partitions >= 0, "Number of partitions ($partitions) cannot be 
 negative.")
 override def numPartitions: Int = partitions
 override def getPartition(key: Any): Int = {
   val k = key.asInstanceOf[RFMCKey]
   k.cId.hashCode() % numPartitions
 }
   }
   object RFMCKey {
 implicit def orderingBycId[A <: RFMCKey] : Ordering[A] = {
   Ordering.by(k => (k.R, k.F * -1, k.M * -1, k.C * -1))
 }
   }
   // The body of the code
   //
   //
   val x = rdd.map(RFMCKey(cust,r,f,m,c), r+","+f+","+m+","+c)
   val y = x.repartitionAndSortWithinPartitions(new RFMCPartitioner(1))}

 I wanted to implement the same thing using clojure's DSL for spark 
 called flambo. Since I can't write partitioner using clojure, I re-used 
 the 
 code defind above, compiled it and used it as a dependency in my Clojure 
 code.

 Now I am importing the partitioner and the key in my clojure code the 
 following way:

 (ns xyz
   (:import
 [package RFMCPartitioner]
 [package RFMCKey]
 )
   )

 But when I try to create RFMCKey by doing (RFMCKey. cust_id r f m c), 
 it throws the following error:

 java.lang.ClassCastException: org.formcept.wisdom.RFMCKey cannot be cast 
 to java.lang.Comparable
 at 
 org.spark-project.guava.collect.NaturalOrdering.compare(NaturalOrdering.java:28)
 at 
 scala.math.LowPriorityOrderingImplicits$$anon$7.compare(Ordering.scala:153)
 at 
 org.apache.spark.util.collection.ExternalSorter$$anon$8.compare(ExternalSorter.scala:170)
 at 
 org.apache.spark.util.collection.ExternalSorter$$anon$8.compare(ExternalSorter.scala:164)
 at 
 org.apache.spark.util.collection.TimSort.countRunAndMakeAscending(TimSort.java:252)
 at org.apache.spark.util.collection.TimSort.sort(TimSort.java:110)
 at org.apache.spark.util.collection.Sorter.sort(Sorter.scala:37)
 at 
 org.apache.spark.util.collection.SizeTrackingPairBuffer.destructiveSortedIterator(SizeTrackingPairBuffer.scala:83)
 at 
 org.apache.spark.util.collection.ExternalSorter.partitionedIterator(ExternalSorter.scala:687)
 at 
 org.apache.spark.util.collection.ExternalSorter.iterator(ExternalSorter.scala:705)
 at 
 org.apache.spark.shuffle.hash.HashShuffleReader.read(HashShuffleReader.scala:64)
 at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:92)
 at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
 at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:70)
 at org.apache.spark.rdd.RDD.iterator(RDD.scala:242)
 at 
 org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
 at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
 at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>

Re: Secondar Sorting in spark using clojure/flambo

[Punit Naik]

So I finally figured it out on my own. I had to basically write my custom 
ordering function as a separate scala project and then call that in clojure.

I had my scala file written in this manner:


import org.apache.spark.Partitionerimport org.apache.spark.rdd.RDD
case class RFMCKey(cId: String, R: Double, F: Long, M: Double, C: Double)class 
RFMCPartitioner(partitions: Int) extends Partitioner {
  require(partitions >= 0, "Number of partitions ($partitions) cannot be 
negative.")
  override def numPartitions: Int = partitions
  override def getPartition(key: Any): Int = {
val k = key.asInstanceOf[RFMCKey]
k.cId.hashCode() % numPartitions
  }}object RFMCKey {
  implicit def orderingBycId[A <: RFMCKey] : Ordering[A] = {
Ordering.by(k => (k.R, k.F * -1, k.M * -1, k.C * -1))
  }}
class rfmcSort {
  def sortWithRFMC(a: RDD[(String, (((Double, Long), Double), Double))], parts: 
Int): RDD[(RFMCKey, String)] = {
val x = a.map(v => v match {
case (custId, (((rVal, fVal), mVal),cVal)) => (RFMCKey(custId, 
rVal, fVal, mVal, cVal), rVal+","+fVal+","+mVal+","+cVal)
}).repartitionAndSortWithinPartitions(new RFMCPartitioner(parts))
x
  }}

I compiled it as ascala project and used it in my clojure code this way:


(:import [org.formcept.wisdom rfmcSort]
 [org.apache.spark.rdd.RDD])

sorted-rfmc-records (.toJavaRDD (.sortWithRFMC (rfmcSort.) (.rdd rfmc-records) 
num_partitions))

Please notice the way I am calling the sortWithRFMC function from the 
rfmcSort object that I created. Also one very important thing to note here 
is when you pass your JavaPairRDD to your scala function, you have to 
convert it into a normal spark RDD first by calling the .rdd method on it. 
And then you have to convert the spark RDD back to JavaPairRDD to work with 
it in clojure.


And sorry that I got your name wrong *Blake :)

On Tuesday, July 12, 2016 at 12:16:42 AM UTC+5:30, Punit Naik wrote:
>
> Hi Black
>
> Thanks for the reply but  figured it out on my own. Posting the answer 
> after this.
>
> On Monday, July 11, 2016 at 11:42:10 PM UTC+5:30, Blake Miller wrote:
>>
>> Hi Punit
>>
>> The behavior you are referring to is a feature of the Scala compiler, 
>> which is why it does not happen automatically when you try to use it from 
>> Clojure.
>>
>> Please see the note here:
>>
>>
>> https://github.com/t6/from-scala/blob/4e1752aaa2ef835dd67a8404273bee067510a431/test/t6/from_scala/guide.clj#L161-L166
>>
>> You may find that library a useful resource, either as a dependency or 
>> simply as reference material.
>>
>> What you want to do is find the full method signature, including the 
>> implicits, and invoke _that_ from clojure, passing values for all implicit 
>> parameters (in this case, your custom ordering function.
>>
>> HTH
>>
>> On Saturday, July 9, 2016 at 6:13:17 AM UTC, Punit Naik wrote:
>>>
>>> Hi Ashish
>>>
>>> The "package" is indeed the full package name.
>>> On 09-Jul-2016 11:02 AM, "Ashish Negi"  wrote:
>>>
 Should not be `package` in `:import` be the actual package name of  `
 RFMCPartitioner` ?

 see examples at https://clojuredocs.org/clojure.core/import

 like :

 (ns foo.bar
   (:import (java.util Date
   Calendar)
(java.util.logging Logger
   Level)))



 (ns xyz
   (:import
 [**  RFMCPartitioner]
 [** RFMCKey]
 )
   )


 where ** is package full name.



 On Friday, 8 July 2016 21:31:27 UTC+5:30, Punit Naik wrote:
>
>
>  
>
> I have a scala program in which I have implemented a secondary sort 
> which works perfectly. The way I have written that program is:
>
> object rfmc {
>   // Custom Key and partitioner
>
>   case class RFMCKey(cId: String, R: Double, F: Double, M: Double, C: 
> Double)
>   class RFMCPartitioner(partitions: Int) extends Partitioner {
> require(partitions >= 0, "Number of partitions ($partitions) cannot 
> be negative.")
> override def numPartitions: Int = partitions
> override def getPartition(key: Any): Int = {
>   val k = key.asInstanceOf[RFMCKey]
>   k.cId.hashCode() % numPartitions
> }
>   }
>   object RFMCKey {
> implicit def orderingBycId[A <: RFMCKey] : Ordering[A] = {
>   Ordering.by(k => (k.R, k.F * -1, k.M * -1, k.C * -1))
> }
>   }
>   // The body of the code
>   //
>   //
>   val x = rdd.map(RFMCKey(cust,r,f,m,c), r+","+f+","+m+","+c)
>   val y = x.repartitionAndSortWithinPartitions(new RFMCPartitioner(1))}
>
> I wanted to implement the same thing using clojure's DSL for spark 
> called flambo. Since I can't write partitioner using clojure, I re-used 
> the 
> code defind above, compiled it and used it as a dependency in my Clojure 
> code.
>
> Now 

Re: Rationale for `keys` not supporting vectors?

[Alex Miller]

As background re the collection 
model: http://insideclojure.org/2016/03/16/collections/

Keys and vals are both defined as functions that take a seqable of map 
entries. Given that definition, it is not easily possible to widen it to 
also take a collection with the associative trait, because a collection 
like vector will not produce a seq of entries, but rather a seq of vector 
values. The ability to seq to map entries is something only provided by 
maps. 

Yes, it would be possible to make a function that worked on either, but I 
think doing so would break the existing contract of these functions, not 
extend it.

To the design question of whether it should have been done this way or not 
in the first place, that's harder for me to say. 


On Friday, July 8, 2016 at 11:50:22 PM UTC-5, Michael Gardner wrote:
>
> I've looked around, but couldn't find any discussion on the topic. Is it 
> purely an implementation thing, or a design choice? 
>
> (Yes, I realize you can just do (range (count v)).)

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Clojure 1.9.0-alpha10: Semantic mismatch: any? vs not-any?

[Alex Miller]

On Monday, July 11, 2016 at 1:01:52 PM UTC-5, Alan Thompson wrote:
>
> In clojure we are used to shortcuts involving not:
>
> (when-not x ...) => (when (not x) ...)
> (if-not x ...)   => (if (not x) ...)
> (not-every? pred coll)   => (not (every? pred coll))
> (not-any? pred coll) => (not (some pred coll))
>
>
The latter two are collection predicates, not simple value predicates, so I 
don't think they can actually relate closely to any?.
 

> However, the new function clojure.core/any? breaks this semantic pattern:
>
> ​> ​
> (doc clojure.core/any?)
> -
> clojure.core/any?
> ([x])
>   Returns true given any argument.
>
> ​> ​
> (doc clojure.core/not-any?)
> -
> clojure.core/not-any?
> ([pred coll])
>   Returns false if (pred x) is logical true for any x in coll,
>   else true.
>
> ​These two functions are not only unrelated to each other, but they don't 
> even accept the same number or type of arguments.  
>

Well, there are only so many words. As it happens any? is best name for 
this function.
 

> Moreover, ​there is an even more surprising property:
>
> > (clojure.core/any? nil)
> true
>
> I would have bet money that, at least for nil, the result would not have 
> been true.
>

The doc string seems quite clear. The English sense of "any" also seems 
quite clear and the distinction vs "some?" is the important one. 
 

> ​
> Given the significant prior conventions in Clojure​ for functions like 
> some, every?, *-not, not-*, and also the general handling of nil, it seems 
> that the new any? function is bound to cause much confusion & 
> consternation, especially among people learning Clojure.  
>
> Given the degenerate definition:
>
> > (source clojure.core/any?)
> (defn any?
>   [x] true)
>
> would it not be simpler and more instinctive to rename the function 
> clojure.core/true:
>
> (defn clojure.core/true
>   [x] true)
>
>
I don't like the use of true. "any?" is asking a question about an input 
value. "true" is a statement about the result, which totally misses the 
point of this function. Also, it shadows an existing literal token in the 
language.

We have no plans to change the name of this function.

 

> We could then have code with the obvious result:
>
> (true 1) => true
> (true "hi")  => true
> (true [])=> true
> (true nil)   => true
>
> I believe that such a change would help to keep Clojure in line with users 
> instincts and assumptions, as well as past Clojure practices.  I have often 
> felt that one of the most important principles in any sort of software 
> development is adherence to the Principle of Least Astonishment 
> .
>
> Alan
>
>
>
> On Mon, Jul 11, 2016 at 7:28 AM, Alex Miller  wrote:
>
>>
>> 1.9.0-alpha10 includes the following changes since 1.9.0-alpha9:
>>
>> - NEW clojure.core/any? - a predicate that matches anything. any? has 
>> built-in gen support. The :clojure.spec/any spec has been removed. 
>> Additionally, gen support has been added for some?.
>>
>>
>>
>>  
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ANN] XT16 Conference in the UK

[Jon Pither]

Dear Clojurians,

Let me explain about our XT16 conference: a couple of days ago I was 
talking with a programmer from Pamplona, Spain. He was encamped in a busy 
bar with lousy WIFI because a load of bulls were about to rampage through 
the streets and most of the town was shut down; the shops boarded up.

We were talking about his project - a not for profit international group of 
developers producing online software for developing countries - when he 
made the statement: 'A project succeeds or fails depending on people being 
happy, enjoying what they do and putting everything in to make it succeed'.

Those coming to the XT16 conference agree that having fun at work does not 
subtract from the task of creating serious software, but rather it has a 
multiplying effect. Crossing the streams of advanced technology and 
developer enjoyment can result in an explosion of beauty and raw 
productivity power that can then be harnessed. This is what we want to show.

Join us at XT16 for an intimate conference in the UK countryside 
(conveniently close to a fast-line to London Kings Cross). We are offering 
a time-limited coupon code CLOJURIANPEACE for a 25% discount (we've only 
got 128 tickets and they are being allocated on a first-come first-serve 
basis).

Talks include James Lewis surveying the changing open source landscape and 
Professor James Woudhuysen will examine the future of our humanity and 
technology. Portia Tung - who inspired our theme - will dig deep into the 
science behind getting everybody out of the rut and to collectively enjoy 
our work.

We have Clojure and non-Clojure language related talks that push the 
boundaries and talks that go down to the bare-metal. If that isn't enough 
we have Sam Aaron to provide the beats and to give a talk on the 
philosophical aspects of performance coding.

We want this conference to be an *event*. We have sourced the finest coffee 
and have commissioned our very own XT16 beer. We have activities to 
challenge the faculties and to keep them burning and an after party that 
will provide the opportunity to network with similarly minded enthusiasts. 
We will also be providing Ukuleles.

Visit our page for full details https://juxt.pro/XT16.html,

Jon & Malcolm.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] Tufte, simple profiling and performance monitoring for Clojure/Script

[Gary Trakhman]

Wonderful, used it today on some core.async code.  It helped me tune
GC/buffer-sizes/parallelism and the metrics are a useful communication tool
across a team that's less familliar with clojure.

On Mon, Jul 11, 2016 at 9:45 AM Peter Taoussanis 
wrote:

> On Clojars, and GitHub at: https://github.com/ptaoussanis/tufte
>
>
> Basically took the weekend to clean up Timbre's old profiling ns,
> generalize it, and move it to its own lib. Biggest difference for most
> folks will probably be the new documentation.
>
>
> Enjoy, cheers! :-)
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Thoughts on clojure.spec

[Oliver George]

> Do you find it frustrating that there's no way to turn on instrumentation 
> of function outputs for manual testing?


Yes.  

In particular when I'm refactoring code and want to know when I'm returning 
something surprising.  

I haven't yet had much success with spec generators for CLJS / UI code. 
 Too many non-clojure types to deal with - my gen-fu is limited.

I suspect the argument for reading an instrument flag to turn on return 
value testing is just that: 

   1. to help those who aren't able to leverage generators in their problem 
   domain.
   2. informal exploration testing

I see that spec/assert would give me the same results.  That's a fantastic 
addition but is also a code intrusion.

Final though: I suspect you can have it for the cost of a small helper 
function.  The instrument-all code doesn't seem complex but there may be a 
few private vars between you and happiness.


On Sunday, 10 July 2016 20:04:39 UTC+10, puzzler wrote:
>
> I've played around now with implementing specs for a couple of my projects.
>
> What I'm finding is that writing specs to check the inputs of a function 
> is easy-ish, but building useful random generators is very hard -- in my 
> projects, this seems too hard to do with any reasonable amount of effort.
>
> This isn't due to anything inherent in clojure.spec, it's just that for 
> non-trivial functions, coming up with relevant random input is a very hard 
> problem.  For example, let's say I have a function that takes two 
> integers.  Odds are that not any two randomly chosen integers will work.  
> Some combinations of integers are non-sensical and could trigger an error, 
> other combinations may cause the function to run way too long.  As a 
> concrete example, I just tried to spec out a SAT solver (which tries to 
> solve NP-complete problems).  The input should be a vector of vectors of 
> ints, but many combinations of inputs will just run forever.  How to 
> generate "good" SAT problems?  I have no idea.
>
> So for the most part, I've ignored the generation aspect of specs because 
> it just feels like too much work.  But without the generators, 
> clojure.spec's utility is significantly diminished.
>
> 1. No way to test function output specs.  For documentation purposes, I 
> want to have an output spec on my function.  However, as far as I know, 
> after instrumentation-triggered checking of output specs was removed a 
> couple of alphas ago, the only way remaining to check against output specs 
> is to use randomly generated tests.  So if I can't make good generators, I 
> have no way to confirm that my output spec works the way I think it does.  
> My documentation could be totally out of touch with reality, and that 
> displeases me.
>
> 2. Limited ability for testing that functions take and receive what you 
> expect.  Whereas a static type system can prove certain properties about 
> whether functions are called with valid inputs, with spec, you only get 
> those guarantees if you pump a function with enough valid random data to 
> trigger the function calling all its callees with interesting combinations 
> of data.  But if I use the naive generators, the function will never even 
> complete with most of the randomly generated input, let alone call other 
> functions in a useful way.  And in many cases I don't see how to generate 
> something of better quality.
>
> So looking back at my experiments, my preliminary impression is that by 
> adding specs to my public APIs, I've gained some useful documentation, and 
> I've given users the ability to instrument functions in order to get 
> high-quality assertion-checking of the inputs.  In some cases, the error 
> messages for bad input when instrumented are also more useful than I would 
> have otherwise gotten, but in some cases they aren't.  Overall, I've 
> struggled to write generators, and without them, the value proposition 
> isn't as great.
>
> One other issue I've had, unrelated to generators, is that I'm struggling 
> to express higher-order type constraints, for example, this function takes 
> a vector v1 of anything and a vector v2 of anything, but the type of the 
> things in vector v1 better match the type of the things in vector v2.
>
> What are other people finding?  Do you find it easy/hard to write 
> generators?  (If you think it's easy, I'd love to know your tricks).  Do 
> you find it easy/hard to read specs as a form of documentation about the 
> contract of a function?  Do you find it frustrating that there's no way to 
> turn on instrumentation of function outputs for manual testing?  Do you 
> feel your generators are providing sufficient code coverage when exercising 
> callee functions?
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, se

Re: Thoughts on clojure.spec

[Alan Thompson]

Do you find it frustrating that there's no way to turn on instrumentation
of function outputs for manual testing?

Yes.


On Mon, Jul 11, 2016 at 4:16 PM, Oliver George 
wrote:

>
> Do you find it frustrating that there's no way to turn on instrumentation
>> of function outputs for manual testing?
>
>
> Yes.
>
> In particular when I'm refactoring code and want to know when I'm
> returning something surprising.
>
> I haven't yet had much success with spec generators for CLJS / UI code.
> Too many non-clojure types to deal with - my gen-fu is limited.
>
> I suspect the argument for reading an instrument flag to turn on return
> value testing is just that:
>
>1. to help those who aren't able to leverage generators in their
>problem domain.
>2. informal exploration testing
>
> I see that spec/assert would give me the same results.  That's a fantastic
> addition but is also a code intrusion.
>
> Final though: I suspect you can have it for the cost of a small helper
> function.  The instrument-all code doesn't seem complex but there may be a
> few private vars between you and happiness.
>
>
> On Sunday, 10 July 2016 20:04:39 UTC+10, puzzler wrote:
>>
>> I've played around now with implementing specs for a couple of my
>> projects.
>>
>> What I'm finding is that writing specs to check the inputs of a function
>> is easy-ish, but building useful random generators is very hard -- in my
>> projects, this seems too hard to do with any reasonable amount of effort.
>>
>> This isn't due to anything inherent in clojure.spec, it's just that for
>> non-trivial functions, coming up with relevant random input is a very hard
>> problem.  For example, let's say I have a function that takes two
>> integers.  Odds are that not any two randomly chosen integers will work.
>> Some combinations of integers are non-sensical and could trigger an error,
>> other combinations may cause the function to run way too long.  As a
>> concrete example, I just tried to spec out a SAT solver (which tries to
>> solve NP-complete problems).  The input should be a vector of vectors of
>> ints, but many combinations of inputs will just run forever.  How to
>> generate "good" SAT problems?  I have no idea.
>>
>> So for the most part, I've ignored the generation aspect of specs because
>> it just feels like too much work.  But without the generators,
>> clojure.spec's utility is significantly diminished.
>>
>> 1. No way to test function output specs.  For documentation purposes, I
>> want to have an output spec on my function.  However, as far as I know,
>> after instrumentation-triggered checking of output specs was removed a
>> couple of alphas ago, the only way remaining to check against output specs
>> is to use randomly generated tests.  So if I can't make good generators, I
>> have no way to confirm that my output spec works the way I think it does.
>> My documentation could be totally out of touch with reality, and that
>> displeases me.
>>
>> 2. Limited ability for testing that functions take and receive what you
>> expect.  Whereas a static type system can prove certain properties about
>> whether functions are called with valid inputs, with spec, you only get
>> those guarantees if you pump a function with enough valid random data to
>> trigger the function calling all its callees with interesting combinations
>> of data.  But if I use the naive generators, the function will never even
>> complete with most of the randomly generated input, let alone call other
>> functions in a useful way.  And in many cases I don't see how to generate
>> something of better quality.
>>
>> So looking back at my experiments, my preliminary impression is that by
>> adding specs to my public APIs, I've gained some useful documentation, and
>> I've given users the ability to instrument functions in order to get
>> high-quality assertion-checking of the inputs.  In some cases, the error
>> messages for bad input when instrumented are also more useful than I would
>> have otherwise gotten, but in some cases they aren't.  Overall, I've
>> struggled to write generators, and without them, the value proposition
>> isn't as great.
>>
>> One other issue I've had, unrelated to generators, is that I'm struggling
>> to express higher-order type constraints, for example, this function takes
>> a vector v1 of anything and a vector v2 of anything, but the type of the
>> things in vector v1 better match the type of the things in vector v2.
>>
>> What are other people finding?  Do you find it easy/hard to write
>> generators?  (If you think it's easy, I'd love to know your tricks).  Do
>> you find it easy/hard to read specs as a form of documentation about the
>> contract of a function?  Do you find it frustrating that there's no way to
>> turn on instrumentation of function outputs for manual testing?  Do you
>> feel your generators are providing sufficient code coverage when exercising
>> callee functions?
>>
> --
> You received this message because you are subscr

Q: Any plans to make the default value of :gen-max a dynamic var?

[Joseph Wayne Norton]

Taken from the spec documentation:

And additional args that control gen
> :gen-max - the maximum coll size to generate (default 20)


Q: Any plans to make the default value of :gen-max a dynamic var?



-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] Tufte, simple profiling and performance monitoring for Clojure/Script

[Peter Taoussanis]

Thanks Gary, happy to hear that :-)

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ANN] http-kit 2.2.0 final is out

[Peter Taoussanis]

On Clojars, and GitHub at: https://github.com/http-kit/http-kit/releases

This release was put together with the help of (23!) individual 
contributors.

Enjoy, cheers! :-)

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.