Hi Jeff,
That is the first thing I tried, as per the documentation as well, but
it only works for the main maven repo and not my custom maven repo. The
documentation is not clear if it is supposed to work for custom maven
repos. Have you tried it?
Kind regards,
Theo
On 2017-06-01 01:45, Jeff Zhang wrote:
Ben's right, for livy interpreter, you need to specify
livy.spark.jars.packages.
Specifying interpreter dependency doesn't work for livy interpreter,
because livy run in yarn-cluster mode, while interpreter dependencies
is downloaded in the client JVM.
Felix Cheung <felixcheun...@hotmail.com
<mailto:felixcheun...@hotmail.com>>于2017年6月1日周四 上午2:52写道：
How are you setting this:
When setting my local maven repo and the
"groupId:artifactId:version" in the interpreter settings
------------------------------------------------------------------------
*From:* Theofilos Kakantousis <t...@kth.se <mailto:t...@kth.se>>
*Sent:* Wednesday, May 31, 2017 1:56:07 AM
*To:* users@zeppelin.apache.org <mailto:users@zeppelin.apache.org>
*Subject:* Re: Livy - add external libraries from additional maven
repo
Thanks everyone for the feedback!
Indeed %dep works only for Spark interpreter, just mentioned it to
show the interpreter behavior I expected with Livy.
When setting my local maven repo and the
"groupId:artifactId:version" in the interpreter settings, I can
see the dependency (i.e. a jar file) being downloaded to the local
OS under "local-repo" directory but the dependency is not deployed
with the Spark application in YARN.
Cheers,
Theo
On 2017-05-31 01:19, Ben Vogan wrote:
For what it's worth I have successfully added jar files and maven
packages to sessions using zeppelin & livy 0.3 - although not
using %dep. In the interpreter settings I set the
livy.spark.jars setting for jars that are on my HDFS cluster, and
livy.spark.jars.packages for maven packages - although only using
maven central and not a local repo.
--Ben
On Tue, May 30, 2017 at 12:36 PM, Felix Cheung
<felixcheun...@hotmail.com <mailto:felixcheun...@hotmail.com>> wrote:
To add, this might be an issue with Livy.
I'm seeing something similar as well.
If you can get a repo with calling the Livy REST API directly
it will be worthwhile to follow up with the Livy community
separately.
------------------------------------------------------------------------
*From:* Felix Cheung <felixcheun...@hotmail.com
<mailto:felixcheun...@hotmail.com>>
*Sent:* Tuesday, May 30, 2017 11:34:31 AM
*To:* users@zeppelin.apache.org
<mailto:users@zeppelin.apache.org>; users@zeppelin.apache.org
<mailto:users@zeppelin.apache.org>
*Subject:* Re: Livy - add external libraries from additional
maven repo
if I recall, %dep only works with the built in Spark
interpreter and not the Livy interpreter.
To manage dependency win Livy you will need to set Spark conf
with Livy.
------------------------------------------------------------------------
*From:* Theofilos Kakantousis <t...@kth.se <mailto:t...@kth.se>>
*Sent:* Tuesday, May 30, 2017 9:05:15 AM
*To:* users@zeppelin.apache.org
<mailto:users@zeppelin.apache.org>
*Subject:* Livy - add external libraries from additional
maven repo
Hi everyone,
I'm using Zeppelin with Livy 0.4 and trying to add external
libraries from an additional maven repo to my application
according to the documentation available here
<https://zeppelin.apache.org/docs/0.7.0/interpreter/livy.html#adding-external-libraries>.
The example works fine, but when I set the
livy.spark.jars.packages to my library the interpreter throws
an unresolved dependency error.
I have added the additional maven repository in the
interpreter settings and have also tried setting
livy.spark.jars.ivy but without luck. However, if I use the
Spark interpreter with the following code it works fine.
"%dep
z.reset();
z.addRepo("my repo").url("http://myrepo";
<http://myrepo>).snapshot
z.load("mygroup:myartifact:myversion");
Has anyone managed to do that with Livy? Thanks!
Cheers,
Theo
--
*BENJAMIN VOGAN*| Data Platform Team Lead
<http://www.shopkick.com/>
<https://www.facebook.com/shopkick><https://www.instagram.com/shopkick/><https://www.pinterest.com/shopkick/><https://twitter.com/shopkickbiz><https://www.linkedin.com/company-beta/831240/?pathWildcard=831240>
