Thanks everyone for the feedback!
Indeed %dep works only for Spark interpreter, just mentioned it to show
the interpreter behavior I expected with Livy.
When setting my local maven repo and the "groupId:artifactId:version" in
the interpreter settings, I can see the dependency (i.e. a jar file)
being downloaded to the local OS under "local-repo" directory but the
dependency is not deployed with the Spark application in YARN.
Cheers,
Theo
On 2017-05-31 01:19, Ben Vogan wrote:
For what it's worth I have successfully added jar files and maven
packages to sessions using zeppelin & livy 0.3 - although not using
%dep. In the interpreter settings I set the livy.spark.jars setting
for jars that are on my HDFS cluster, and livy.spark.jars.packages for
maven packages - although only using maven central and not a local repo.
--Ben
On Tue, May 30, 2017 at 12:36 PM, Felix Cheung
<felixcheun...@hotmail.com <mailto:felixcheun...@hotmail.com>> wrote:
To add, this might be an issue with Livy.
I'm seeing something similar as well.
If you can get a repo with calling the Livy REST API directly it
will be worthwhile to follow up with the Livy community separately.
------------------------------------------------------------------------
*From:* Felix Cheung <felixcheun...@hotmail.com
<mailto:felixcheun...@hotmail.com>>
*Sent:* Tuesday, May 30, 2017 11:34:31 AM
*To:* users@zeppelin.apache.org
<mailto:users@zeppelin.apache.org>; users@zeppelin.apache.org
<mailto:users@zeppelin.apache.org>
*Subject:* Re: Livy - add external libraries from additional maven
repo
if I recall, %dep only works with the built in Spark interpreter
and not the Livy interpreter.
To manage dependency win Livy you will need to set Spark conf with
Livy.
------------------------------------------------------------------------
*From:* Theofilos Kakantousis <t...@kth.se <mailto:t...@kth.se>>
*Sent:* Tuesday, May 30, 2017 9:05:15 AM
*To:* users@zeppelin.apache.org <mailto:users@zeppelin.apache.org>
*Subject:* Livy - add external libraries from additional maven repo
Hi everyone,
I'm using Zeppelin with Livy 0.4 and trying to add external
libraries from an additional maven repo to my application
according to the documentation available here
<https://zeppelin.apache.org/docs/0.7.0/interpreter/livy.html#adding-external-libraries>.
The example works fine, but when I set the
livy.spark.jars.packages to my library the interpreter throws an
unresolved dependency error.
I have added the additional maven repository in the interpreter
settings and have also tried setting livy.spark.jars.ivy but
without luck. However, if I use the Spark interpreter with the
following code it works fine.
"%dep
z.reset();
z.addRepo("my repo").url("http://myrepo"; <http://myrepo>).snapshot
z.load("mygroup:myartifact:myversion");
Has anyone managed to do that with Livy? Thanks!
Cheers,
Theo
--
*BENJAMIN VOGAN*| Data Platform Team Lead
<http://www.shopkick.com/>
<https://www.facebook.com/shopkick><https://www.instagram.com/shopkick/><https://www.pinterest.com/shopkick/><https://twitter.com/shopkickbiz><https://www.linkedin.com/company-beta/831240/?pathWildcard=831240>
