First off, I apologize in advance for not having a reduced test case, and express my sincere gratitude in advance for any assistance. I've been tearing my hair out for a day or so and not making headway, and figured someone here might recognize some keywords and have a pointer in the right direction. (I'm admittedly pretty green when it comes to class loading, and have largely exhausted my google fu).
*Problem: * I'm submitting a hadoop job using clojure-hadoop. All is well with a simple job, but once I require something that transitively depends on Schema, I end up with: clojure.lang.Compiler$CompilerException: java.lang.IllegalArgumentException: No implementation of method: :walker of protocol: #'schema.core/Schema found for class: clojure.core$long, compiling:(crane/config.clj:33:4) It works fine when run in-process with hadoop-mapreduce-client-jobclient, but not with bin/hadoop -jar. This stunk of a classloader issue, and after digging in it seems that there are multiple versions of clojure.core$long floating around. The version on which the protocol is extended is not the same class for the fn that the symbol 'long resolves to in client code. *Context: * clojure-hadoop is AOT-compiled, and after being loaded by hadoop it dynamically loads the target namespace (not AOT-compiled, nor any other of the code in question) using https://github.com/alexott/clojure-hadoop/blob/master/src/clojure_hadoop/load.clj#L3 >From here, schema is transitively required, and then client namespaces attempt to use the Schema protocol to generate validators, and when the schema 'long is used (which resolves to the fn with class clojure.core$long), it fails to find the appropriate method. After repeated head-bashing, I've determined that there are (at least two) versions of the clojure.core$long class floating around -- the one used to extend the protocol, which stems from a DynamicClassLoader, and the one that 'long resolves to in client code, which stems from a URLClassLoader. The URLClassLoader is the loader of the current thread and Compiler, but not @(clojure.lang.Compiler/LOADER). *Attempts:* I've tried wrapping the clojure-hadoop loading code with .setContextClassLoader on some obvious candidates and binding *use-context-classloader* around the code doing the loading, with no avail. I've tried changing the schema code to reference the class in different ways (class (resolve 'long)), (class 'long), etc and that hasn't made a difference. I've checked and the clojure-hadoop jar doesn't contain any .class files for clojure, schema, or other offending code. *Plea:* I suspect there's something obvious I'm missing. (In retrospect it seems like the design of Schema may be suboptimal in light of this, but if possible I'd like to figure out a workaround without changing that substantially). Thanks in advance for your help -- any and all pointers are welcome. -Jason -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
