Not a solution to your immediate problem, but if this is for new development (not an existing mass of clojure-hadoop code), I'd suggest looking at Parkour instead. As the main Parkour developer I'm obviously biased, but Parkour exists in part because the compilation model used by clojure-hadoop in order to meet Hadoop's expectations is very much at odds with typical Clojure development. In particular, Parkour does not require AOT compilation.
https://github.com/damballa/parkour On Thursday, January 29, 2015 at 2:39:54 AM UTC-5, Jason Wolfe wrote: > > First off, I apologize in advance for not having a reduced test case, and > express my sincere gratitude in advance for any assistance. I've been > tearing my hair out for a day or so and not making headway, and figured > someone here might recognize some keywords and have a pointer in the right > direction. (I'm admittedly pretty green when it comes to class loading, and > have largely exhausted my google fu). > > *Problem: * > > I'm submitting a hadoop job using clojure-hadoop. All is well with a > simple job, but once I require something that transitively depends on > Schema, I end up with: > > clojure.lang.Compiler$CompilerException: > java.lang.IllegalArgumentException: No implementation of method: :walker of > protocol: #'schema.core/Schema found for class: clojure.core$long, > compiling:(crane/config.clj:33:4) > > It works fine when run in-process with hadoop-mapreduce-client-jobclient, > but not with bin/hadoop -jar. This stunk of a classloader issue, and after > digging in it seems that there are multiple versions of clojure.core$long > floating around. The version on which the protocol is extended is not the > same class for the fn that the symbol 'long resolves to in client code. > > *Context: * > > clojure-hadoop is AOT-compiled, and after being loaded by hadoop it > dynamically loads the target namespace (not AOT-compiled, nor any other of > the code in question) using > https://github.com/alexott/clojure-hadoop/blob/master/src/clojure_hadoop/load.clj#L3 > > From here, schema is transitively required, and then client namespaces > attempt to use the Schema protocol to generate validators, and when the > schema 'long is used (which resolves to the fn with class > clojure.core$long), it fails to find the appropriate method. > > After repeated head-bashing, I've determined that there are (at least two) > versions of the clojure.core$long class floating around -- the one used to > extend the protocol, which stems from a DynamicClassLoader, and the one > that 'long resolves to in client code, which stems from a URLClassLoader. > The URLClassLoader is the loader of the current thread and Compiler, but > not @(clojure.lang.Compiler/LOADER). > > *Attempts:* > > I've tried wrapping the clojure-hadoop loading code with > .setContextClassLoader on some obvious candidates and binding > *use-context-classloader* around the code doing the loading, with no avail. > I've tried changing the schema code to reference the class in different > ways (class (resolve 'long)), (class 'long), etc and that hasn't made a > difference. I've checked and the clojure-hadoop jar doesn't contain any > .class files for clojure, schema, or other offending code. > > *Plea:* > > I suspect there's something obvious I'm missing. (In retrospect it seems > like the design of Schema may be suboptimal in light of this, but if > possible I'd like to figure out a workaround without changing that > substantially). Thanks in advance for your help -- any and all pointers are > welcome. > > -Jason > > > > > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
