ANN - tech.resource
Clojurians - I hope your holidays are going well! We took the time to explain in some detail a small library that has been extremely useful since the days of cortex: blog post: http://techascent.com/blog/generalized-resource-management.html Library: https://github.com/techascent/tech.resource It enables either stack-based resource management similar to with-open and gc-based resource management. We hope you find it useful and that it simplifies using 'off-heap' resources. Complements from cold, snowy Boulder! Chris -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Reify (run vs cloverage) (single node vs cluster)
I'm really struck at the point where I'm using reify in clojure to implement java interface "org.apache.spark.sql.api.java.UDF2" and define the method "call" in this interface. I've been able to use reify to implement and it works fine when I do a "lein run", I see that this udf is being applied on my dataframe. I've also written test cases to check if the udf is being applied on the dataframe and I see that udf is being applied as expected. Now, when I do a "lein cloverage" to check my code coverage, this test case fails, giving a classNotFoundException. Partial error stack trace: (running the app on virtual desktop, could not copy and paste) org.apache.spark.sparkException: Job aborted due to stage failure: classNotFoundException: ab.cd.ef.ef$fn$reify__2440 *On Hadoop:* Getting the same kind of error when I run this application on Cluster mode used spark-submit, tried hadoop jar; all give the same error as above. *But, *when I run the same jar on a single node, it works fine without any issues. Using Clojure 1.9.0, Cloverage 1.0.13, Spark 2.1.3, Java 1.8.0_141 Has anyone been facing this kind of issue? if so, was there a solution? -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: clj deps.edn :git overture rejected by ssh server, citing dss
Wow, that is a super write-up on SSH and clj! I would contribute 2 tiny tips: git and jsch were reading ~/.ssh/config and coming to different conclusions about the HostKeyAlgorithms. The HostKeyAlgorithms statement said, "HostKeyAlgorithms +...". In effect, jsch may have ignored the "+", which would be a defect, but achieving the correct effect would require jsch to know OpenSSH's defaults, which may be out of the question. In a word, using "+" is asking for trouble. Of course, once "+" is there, you would not even consider changing it, lest other things break. So... ssh reached the git server OK. Evidently, ssh was satisfied with known_hosts. In the relevant known_hosts entry, I spotted the name of the algorithm ssh used. I copied that algorithm name onto the "HostKeyAlgorithms +..." declaration (it hadn't been there). Then clj reached the git server. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Reify (run vs cloverage) (single node vs cluster)
Just a hunch, but many cluster / distribution tools expect that a given Class name will refer to the same Class on each peer. You cannot ensure this with reify- the name is auto-generated. The solution might be using deftype or gen-class so that the class name would be deterministic and shared on each instance. Another hunch, regarding cloverage, is that an error like that just from checking test coverage means you have top level side effects (eg. top level forms or defs that do some kind of IO or launch a process) - Clojure doesn't have a "compile only" mode, and top level side effects will be launched by any program that uses Clojure to load your file. The principled solution is to move side effects into function bodies or delays, and pass stateful resources as arguments to other code on startup rather than binding them at the top level. There are multiple lifecycle management libraries designed for managing this. On Mon, Dec 31, 2018 at 1:53 PM cloje wrote: > I'm really struck at the point where I'm using reify in clojure to > implement java interface "org.apache.spark.sql.api.java.UDF2" and define > the method "call" in this interface. > > I've been able to use reify to implement and it works fine when I do a > "lein run", I see that this udf is being applied on my dataframe. I've also > written test cases to check if the udf is being applied on the dataframe > and I see that udf is being applied as expected. Now, when I do a "lein > cloverage" to check my code coverage, this test case fails, giving a > classNotFoundException. > > Partial error stack trace: (running the app on virtual desktop, could not > copy and paste) > org.apache.spark.sparkException: Job aborted due to stage failure: > classNotFoundException: ab.cd.ef.ef$fn$reify__2440 > > > *On Hadoop:* > Getting the same kind of error when I run this application on Cluster mode > used spark-submit, tried hadoop jar; all give the same error as above. *But, > *when I run the same jar on a single node, it works fine without any > issues. > > Using Clojure 1.9.0, Cloverage 1.0.13, Spark 2.1.3, Java 1.8.0_141 > > > Has anyone been facing this kind of issue? if so, was there a solution? > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
