Re: Closure issue with spark 1.4.1

2015-08-24 Thread David Salinas
Hi Moon, Today I cannot reproduce the bug with elementary example either but it is still impacting all my notebooks. The weird thing is that when calling a transformation with map, it takes Zeppelin Context in the closure which gives these java.lang.NoClassDefFoundError: Lorg/apache/zeppelin/spark

Re: Closure issue with spark 1.4.1

2015-08-24 Thread David Salinas
Sorry I forgot to mention my environment: mesos 0.17, spark 1.4.1, scala 2.10.4, java 1.8 On Mon, Aug 24, 2015 at 10:32 AM, David Salinas wrote: > Hi Moon, > > Today I cannot reproduce the bug with elementary example either but it is > still impacting all my notebooks. The weird thing is that wh

Getting data from scala to JS

2015-08-24 Thread hasan türken
Hi, I am trying to visualize a hierarchy and need to get children dynamically in javascript function just like the following: $.getJSON(addthese.json, function(addTheseJSON) { var newnodes = tree.nodes(addTheseJSON.children).reverse(); d.children = newnodes[0]; update(d);}); (http://

Fwd: Use Zeppelin in CDH cluster, using Yarn

2015-08-24 Thread Gianluigi Barbarano
Hi, some of you could help me? thanks a lot! -- Forwarded message -- From: Gianluigi Barbarano Date: 2015-08-21 12:35 GMT+02:00 Subject: Use Zeppelin in CDH cluster, using Yarn To: users@zeppelin.incubator.apache.org Hi all, I want to use Zeppelin in my CDH cluster, to run s

Re: %spark.dep only loading deps for driver?

2015-08-24 Thread Randy Gelhausen
Any ideas? Is z.load supposed to make dependencies available to all Spark JVMs (driver AND executors)? Thanks, -Randy On Sun, Aug 23, 2015 at 2:41 PM, Randy Gelhausen wrote: > It seems Spark executors are not being provided with the requisite > dependencies. With spark-shell I can pass --jars

Re: Closure issue with spark 1.4.1

2015-08-24 Thread David Salinas
Hi Moon, I found another way to reproduce the problem: //cell 1 does not work val file = "hdfs://someclusterfile.json" val s = z.input("Foo").toString val textFile = sc.textFile(file) textFile.filter(_.contains(s)).count //org.apache.spark.SparkException: Job aborted due to stage failure: Task 41

Re: Closure issue with spark 1.4.1

2015-08-24 Thread David Salinas
I have looked at the SparkInterpreter.java code and this is indeed the issue. Whenever one uses an instruction with z.input("...") something then no spark transformation can work as z will be shipped to the slaves where Zeppelin is not installed as showed by the example I sent. A workaround could b

Re: Closure issue with spark 1.4.1

2015-08-24 Thread Jerry Lam
Hi Zeppelin developers, This issue sounds very serious. Is this specific to David's use case here? Best Regards, Jerry On Mon, Aug 24, 2015 at 1:28 PM, David Salinas wrote: > I have looked at the SparkInterpreter.java code and this is indeed the > issue. Whenever one uses an instruction with

Re: %spark.dep only loading deps for driver?

2015-08-24 Thread moon soo Lee
Hi Randy, z.load() supposed to make dependencies available to all driver and executors. However, it might not work correctly in yarn-client mode. Are you using yarn-client mode? Best, moon On Mon, Aug 24, 2015 at 9:12 AM Randy Gelhausen wrote: > Any ideas? > > Is z.load supposed to make depen

Re: Use Zeppelin in CDH cluster, using Yarn

2015-08-24 Thread moon soo Lee
Hi, I'm not sure it'll help, but can you try add (change path to your actual location) *export SPARK_YARN_USER_ENV="PYTHONPATH=* $SPARK_HOME/ *python:/usr/spark-1.4.1-bin/*$SPARK_HOME/*lib/py4j-0.8.2.1-src.zip"* Thanks, moon On Mon, Aug 24, 2015 at 7:40 AM Gianluigi Barbarano < gianluigi.barbar

Re: %spark.dep only loading deps for driver?

2015-08-24 Thread Randy Gelhausen
I am. Any suggested workarounds? I know I can add to ZEPPELIN_JAVA_OPTS, but seems less than ideal, given different users will have different dependencies. Doesn't seem wise to give end-users access to customize ZEPPELIN_JAVA_OPTS and restart zeppelin server. On Mon, Aug 24, 2015 at 6:14 PM, moon

Re: Getting data from scala to JS

2015-08-24 Thread moon soo Lee
Hi Hasan, This might help http://zeppelin.incubator.apache.org/docs/displaysystem/angular.html You can get an event (ie. mouse click, function call, etc) from javascript side using angularWatch(). And you can push data to Javascript side using angularBind(). With angularWatch() and angularBind(),

Re: Confused string field formatting

2015-08-24 Thread moon soo Lee
Hi, That is currently zeppelin's limitation. One trick is transform the your values from "value" to "%html value" That'll try to format cell contents as a html and removes ',' You can try two different codes below %spark print(s"""%table value\n111""") %spark print(s"""%table value\n%html 1

Re: %spark.dep only loading deps for driver?

2015-08-24 Thread moon soo Lee
Let me look into it On Mon, Aug 24, 2015 at 3:41 PM Randy Gelhausen wrote: > I am. Any suggested workarounds? I know I can add to ZEPPELIN_JAVA_OPTS, > but seems less than ideal, given different users will have different > dependencies. > > Doesn't seem wise to give end-users access to customize