Thanks for the help everyone. After some more fiddling, with two changes I can hack things to work but it's not pretty.
First, this change to schema: https://github.com/Prismatic/schema/commit/e835d447ea2525471a40d65768b5e6435a469fc7 Second, if I just (require 'top-level-ns) things still break, but if I execute a careful sequence of steps like (require 'schema.core 'another-ns 'yet-another-ns 'top-level-ns) where the intermediate ones are the place things were breaking, then everything works fine. I'm guessing this is something to do with a nested sequence of classloaders created along the namespace loading chain, which this effectively flattens out so that the problem namespaces are loaded with the root classloader. But I'm admittedly out of my depth here. We'd obviously like to avoid such hackery; does this extra info give enough context for anyone to offer a better solution? Thanks! Jason On Wednesday, January 28, 2015 at 11:39:54 PM UTC-8, Jason Wolfe wrote: > > First off, I apologize in advance for not having a reduced test case, and > express my sincere gratitude in advance for any assistance. I've been > tearing my hair out for a day or so and not making headway, and figured > someone here might recognize some keywords and have a pointer in the right > direction. (I'm admittedly pretty green when it comes to class loading, and > have largely exhausted my google fu). > > *Problem: * > > I'm submitting a hadoop job using clojure-hadoop. All is well with a > simple job, but once I require something that transitively depends on > Schema, I end up with: > > clojure.lang.Compiler$CompilerException: > java.lang.IllegalArgumentException: No implementation of method: :walker of > protocol: #'schema.core/Schema found for class: clojure.core$long, > compiling:(crane/config.clj:33:4) > > It works fine when run in-process with hadoop-mapreduce-client-jobclient, > but not with bin/hadoop -jar. This stunk of a classloader issue, and after > digging in it seems that there are multiple versions of clojure.core$long > floating around. The version on which the protocol is extended is not the > same class for the fn that the symbol 'long resolves to in client code. > > *Context: * > > clojure-hadoop is AOT-compiled, and after being loaded by hadoop it > dynamically loads the target namespace (not AOT-compiled, nor any other of > the code in question) using > https://github.com/alexott/clojure-hadoop/blob/master/src/clojure_hadoop/load.clj#L3 > > From here, schema is transitively required, and then client namespaces > attempt to use the Schema protocol to generate validators, and when the > schema 'long is used (which resolves to the fn with class > clojure.core$long), it fails to find the appropriate method. > > After repeated head-bashing, I've determined that there are (at least two) > versions of the clojure.core$long class floating around -- the one used to > extend the protocol, which stems from a DynamicClassLoader, and the one > that 'long resolves to in client code, which stems from a URLClassLoader. > The URLClassLoader is the loader of the current thread and Compiler, but > not @(clojure.lang.Compiler/LOADER). > > *Attempts:* > > I've tried wrapping the clojure-hadoop loading code with > .setContextClassLoader on some obvious candidates and binding > *use-context-classloader* around the code doing the loading, with no avail. > I've tried changing the schema code to reference the class in different > ways (class (resolve 'long)), (class 'long), etc and that hasn't made a > difference. I've checked and the clojure-hadoop jar doesn't contain any > .class files for clojure, schema, or other offending code. > > *Plea:* > > I suspect there's something obvious I'm missing. (In retrospect it seems > like the design of Schema may be suboptimal in light of this, but if > possible I'd like to figure out a workaround without changing that > substantially). Thanks in advance for your help -- any and all pointers are > welcome. > > -Jason > > > > > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
