On Friday, January 30, 2015 at 7:27:12 AM UTC-8, Marshall Bockrath-Vandegrift wrote: > > Not a solution to your immediate problem, but if this is for > new development (not an existing mass of clojure-hadoop code), I'd suggest > looking at Parkour instead. As the main Parkour developer I'm obviously > biased, but Parkour exists in part because the compilation model used by > clojure-hadoop in order to meet Hadoop's expectations is very much at odds > with typical Clojure development. In particular, Parkour does not require > AOT compilation. > > https://github.com/damballa/parkour >
Thanks for the recommendation. For now we're looking for a simple low-level interface to MR, but we're also keeping an eye on parkour and pigpen for more complex tasks down the road. Can you explain why I might prefer parkour to pigpen or vice-versa? > > > On Thursday, January 29, 2015 at 2:39:54 AM UTC-5, Jason Wolfe wrote: >> >> First off, I apologize in advance for not having a reduced test case, and >> express my sincere gratitude in advance for any assistance. I've been >> tearing my hair out for a day or so and not making headway, and figured >> someone here might recognize some keywords and have a pointer in the right >> direction. (I'm admittedly pretty green when it comes to class loading, and >> have largely exhausted my google fu). >> >> *Problem: * >> >> I'm submitting a hadoop job using clojure-hadoop. All is well with a >> simple job, but once I require something that transitively depends on >> Schema, I end up with: >> >> clojure.lang.Compiler$CompilerException: >> java.lang.IllegalArgumentException: No implementation of method: :walker of >> protocol: #'schema.core/Schema found for class: clojure.core$long, >> compiling:(crane/config.clj:33:4) >> >> It works fine when run in-process with hadoop-mapreduce-client-jobclient, >> but not with bin/hadoop -jar. This stunk of a classloader issue, and after >> digging in it seems that there are multiple versions of clojure.core$long >> floating around. The version on which the protocol is extended is not the >> same class for the fn that the symbol 'long resolves to in client code. >> >> *Context: * >> >> clojure-hadoop is AOT-compiled, and after being loaded by hadoop it >> dynamically loads the target namespace (not AOT-compiled, nor any other of >> the code in question) using >> https://github.com/alexott/clojure-hadoop/blob/master/src/clojure_hadoop/load.clj#L3 >> >> From here, schema is transitively required, and then client namespaces >> attempt to use the Schema protocol to generate validators, and when the >> schema 'long is used (which resolves to the fn with class >> clojure.core$long), it fails to find the appropriate method. >> >> After repeated head-bashing, I've determined that there are (at least >> two) versions of the clojure.core$long class floating around -- the one >> used to extend the protocol, which stems from a DynamicClassLoader, and the >> one that 'long resolves to in client code, which stems from a >> URLClassLoader. The URLClassLoader is the loader of the current thread and >> Compiler, but not @(clojure.lang.Compiler/LOADER). >> >> *Attempts:* >> >> I've tried wrapping the clojure-hadoop loading code with >> .setContextClassLoader on some obvious candidates and binding >> *use-context-classloader* around the code doing the loading, with no avail. >> I've tried changing the schema code to reference the class in different >> ways (class (resolve 'long)), (class 'long), etc and that hasn't made a >> difference. I've checked and the clojure-hadoop jar doesn't contain any >> .class files for clojure, schema, or other offending code. >> >> *Plea:* >> >> I suspect there's something obvious I'm missing. (In retrospect it seems >> like the design of Schema may be suboptimal in light of this, but if >> possible I'd like to figure out a workaround without changing that >> substantially). Thanks in advance for your help -- any and all pointers are >> welcome. >> >> -Jason >> >> >> >> >> > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
