Hi Paul,
It looks like the cascading jars are not distributed to the YARN cluster.
Can you please try adding "zeppelin/interpreter/scalding/*" to the
args.string property of the scalding interpreter?
Here's the args.string we use:
-libjars /home/zeppelin-user/zeppelin/interpreter/scalding/*,/home/
zeppelin-user/deploy-bundle-201608111417/libs/*
-Dscalding.reducer.estimator.classes=com.twitter.scalding.
reducer_estimation.InputSizeReducerEstimator
-Delephantbird.use.combine.input.format=true
-Delephantbird.combine.split.size=134217728 --hdfs --repl
tmpjars contains jars that are distributed to the YARN cluster. You can see
its contents with the command below:
%scalding
mode.asInstanceOf[Hdfs].conf.get("tmpjars").split(",").foreach(println)
Thanks,
Prasad
On Thu, Dec 22, 2016 at 9:31 AM, Paul Brenner <pbren...@placeiq.com> wrote:
> I'm trying to get Scalding working on Zeppelin while using YARN. I
> followed the steps in the docs here
> <https://share.polymail.io/v1/z/b/NTg1YzBkOTc5ZTkx/2oW5SQjbADW8zb9nS3JO5g421bQMXDLTC0FeCJ_WR7eecFsW9CWa-tzokB9aSLwG5t9yQ9B6QpcS8AmXjjFFxJ31Thy9lN7HSvilaEeoI6Az7C53CrnFmUoMnta-EYrRI5uEQhbztPSzTrQle-3E00nNiVc7M6poouix37ZlX2VacVqONwmxpu6FSMs2x-_t20QRzFz8S7lneRPUBtpzIyxBRLcRL4CMf1AeMxQIVl3FkoStgA==>
> to
> build the interpreter and set up the classpath override. When I run in
> local mode, code executes properly. However when I run on my cluster via
> YARN my jobs fail with:
>
> Error: java.lang.ClassNotFoundException: cascading.CascadingException
>
> or
>
> Error: java.lang.ClassNotFoundException: cascading.tuple.TupleException
>
> What is even stranger to me is that I can go into Zeppelin and execute:
>
> import cascading.tuple.TupleException import cascading.CascadingException
>
> And both appear to have no problem finding those classes. It is only when
> I try to actually use scalding (on YARN), like loading data into a typed
> pipe and dumping that I get the ClassNotFoundException. Any ideas on how to
> debug or what to fix?
> (I already posted this on Stack Overflow with no luck:
> http://stackoverflow.com/questions/41191773/classnotfoundexception-with-
> scalding-on-zeppelin-managed-on-yarn
> <https://share.polymail.io/v1/z/b/NTg1YzBkOTc5ZTkx/2oW5SQjbADW8zb9nS3JO5g421bQMXDLTC0FeCJ_WR7eecFsW9CWa-tzokB9aSLwG5t9yQ9B6QpcS8AmXjjFFxJ31Thy9lN7HSvilaEeoI6Az7C53CrnFmUoMnta-EYrRI5uEQhbztPSzB6ElJ-PAwFLHk1sne6d3tKW61fUlXHcQZkGEK0jyp-yaS91unqy7hlcPyFvqV_lgfB3NDNd4PGFMQK4UWIUMywUEsXGgXRQGgho1Hlj-0iyZtbcZrE3vMNtVzjO8HWJpq45DjiV-N210k6sXeYh5YrKTEiEpER4B>
> )
>
