Re: Integrating with Airflow

2017-05-19 Thread Ben Vogan
Thanks for sharing this Ruslan - I will take a look. I agree that paragraphs can form tasks within a DAG. My point was that ideally a DAG could encompass multiple notes. I.e. the completion of one note triggers another and so on to complete an entire chain of dependent tasks. For example team A

Re: Integrating with Airflow

2017-05-19 Thread Ruslan Dautkhanov
Thanks for sharing this Ben. I agree Zeppelin is a better fit with tighter integration with Spark and built-in visualizations. We have pretty much standardized on pySpark, so here's one of the scripts we use internally to extract %pyspark, %sql and %md paragraphs into a standalone script (that ca

Re: Create cassandra interpreter for Zeppelin on AWS EMR spark cluster

2017-05-19 Thread shyla deshpande
I now have the cassandra interpreter setup. But I am getting the following error java.lang.NoClassDefFoundError: Could not initialize class org.apache.zeppelin.cassandra.DisplaySystem$NoResultDisplay$ at org.apache.zeppelin.cassandra.EnhancedSession.(EnhancedSession.scala:40) Please help. On Fri

Re: Integrating with Airflow

2017-05-19 Thread Ben Vogan
I do not expect the relationship between DAGs to be described in Zeppelin - that would be done in Airflow. It just seems that Zeppelin is such a great tool for a data scientists workflow that it would be nice if once they are done with the work the note could be productionized directly. I could e

Re: Create cassandra interpreter for Zeppelin on AWS EMR spark cluster

2017-05-19 Thread DuyHai Doan
Just download a binary of Zeppelin with all interpreters on your AWS EMR cluster and install it there On Sat, May 20, 2017 at 12:21 AM, shyla deshpande wrote: > I don't see the cassandra interpreter, so I need to create the interpreter. > > Locally, I installed the Binary package with all interp

Re: Create cassandra interpreter for Zeppelin on AWS EMR spark cluster

2017-05-19 Thread shyla deshpande
I don't see the cassandra interpreter, so I need to create the interpreter. Locally, I installed the Binary package with all interpreters and works fine. On AWS EMR, all the binaries may not be there, and thats where I need help. Thanks On Fri, May 19, 2017 at 3:04 PM, DuyHai Doan wrote: > So

Re: Create cassandra interpreter for Zeppelin on AWS EMR spark cluster

2017-05-19 Thread DuyHai Doan
So just use the Cassandra interpreter, I don't see where is the problem, there is even documentation here: http://zeppelin.apache.org/docs/0.7.1/interpreter/cassandra.html On Fri, May 19, 2017 at 11:51 PM, shyla deshpande wrote: > Yes, we do have cassandra cluster on AWS-EC2. > > On Fri, May 19

Re: Create cassandra interpreter for Zeppelin on AWS EMR spark cluster

2017-05-19 Thread shyla deshpande
Yes, we do have cassandra cluster on AWS-EC2. On Fri, May 19, 2017 at 2:39 PM, DuyHai Doan wrote: > If you're running on AWS EMR Spark Cluster, why do you need a Cassandra > interpreter Is Cassandra installed on your AWS cluster ? > > On Fri, May 19, 2017 at 11:23 PM, shyla deshpande < > de

Re: Create cassandra interpreter for Zeppelin on AWS EMR spark cluster

2017-05-19 Thread DuyHai Doan
If you're running on AWS EMR Spark Cluster, why do you need a Cassandra interpreter Is Cassandra installed on your AWS cluster ? On Fri, May 19, 2017 at 11:23 PM, shyla deshpande wrote: > Hello all, > > I need help in creating cassandra interpreter for Zeppelin on AWS EMR > spark cluster. P

Create cassandra interpreter for Zeppelin on AWS EMR spark cluster

2017-05-19 Thread shyla deshpande
Hello all, I need help in creating cassandra interpreter for Zeppelin on AWS EMR spark cluster. Please give me the info on what to install and configure. Thanks

Re: Integrating with Airflow

2017-05-19 Thread Ruslan Dautkhanov
We also use both Zeppelin and Airflow. I'm interested in hearing what others are doing here too. Although honestly there might be some challenges - Airflow expects a DAG structure, while a notebook has pretty linear structure; - Airflow is Python-based; Zeppelin is all Java (REST API might be of

Integrating with Airflow

2017-05-19 Thread Ben Vogan
Hi all, We are really enjoying the workflow of interacting with our data via Zeppelin, but are not sold on using the built in cron scheduling capability. We would like to be able to create more complex DAGs that are better suited for something like Airflow. I was curious as to whether anyone has

Re: Hive interpreter Error as soon as Hive query uses MapRed

2017-05-19 Thread Ben Vogan
I am running CDH 5.7 and Spark 1.6 as well and hive is working for me with the following configuration: Properties namevalue common.max_count 1000 default.driver org.apache.hive.jdbc.HiveDriver default.password default.url jdbc:hive2://hdfs004:1 default.user hive zeppelin.interpreter.localRepo

Re: angularBinding a map to a dictionary

2017-05-19 Thread moon soo Lee
Hi, You can try create java Map instead of scala Map. That'll give expected output. Thanks, moon On Fri, May 19, 2017 at 4:51 AM Andrea Sosio wrote: > Hello list, > > is there a way to bind a map to a dictionary in such a way that the map > (key, value) structure is actually preserved? > > I'm

Re: Hive interpreter Error as soon as Hive query uses MapRed

2017-05-19 Thread Meier, Alexander
Yes, the script (i.e. The select statement) runs fine in hive cli, hue and also in spark sql ( spark sql also in zeppelin). Just not when using the hive interpreter in zeppelin. Sent from my iPhone Am 19.05.2017 um 19:35 schrieb Jongyoul Lee mailto:jongy...@gmail.com>>: Can you check your sc

Re: Setting Note permission

2017-05-19 Thread Jongyoul Lee
Simply, you'd better enables personalized mode in the top of note. Then, one user's behaviors doesn't affect another one. Try it and leave comments. Thanks, Jongyoul Lee On Tue, May 16, 2017 at 1:36 PM, shyla deshpande wrote: > I want to know if this is possible. Works great for a single user

Re: Hive interpreter Error as soon as Hive query uses MapRed

2017-05-19 Thread Jongyoul Lee
Can you check your script works in native hive environment? On Fri, May 19, 2017 at 10:20 AM, Meier, Alexander < alexander.me...@t-systems-dmc.com> wrote: > Hi list > > I’m trying to get a Hive interpreter correctly running on a CDH 5.7 > Cluster with Spark 1.6. Simple queries are running fine, b

Re: Does Zeppelin 0.7.1 work with Spark Structured Streaming 2.1.1?

2017-05-19 Thread kant kodali
Look for something that is more like in this video where the graphs automatically update themselves. Is that possible in Zeppelin? https://www.youtube.com/watch?v=IJmFTXvUZgY You can watch it from 9:20 On Fri, May 19, 2017 at 9:21 AM, kant kodali wrote: > Hi All, > > I have the following code

Does Zeppelin 0.7.1 work with Spark Structured Streaming 2.1.1?

2017-05-19 Thread kant kodali
Hi All, I have the following code StreamingQuery query = df2.writeStream().outputMode("complete").queryName( "foo").option("truncate", "false").format("console").start(); query.awaitTermination(); and it works fine however when I change it to the below code. I do get the output but only once a

Hive interpreter Error as soon as Hive query uses MapRed

2017-05-19 Thread Meier, Alexander
Hi list I’m trying to get a Hive interpreter correctly running on a CDH 5.7 Cluster with Spark 1.6. Simple queries are running fine, but as soon as a query needs a MapRed tasks in order to complete, the query fails with: java.sql.SQLException: Error while processing statement: FAILED: Execution

angularBinding a map to a dictionary

2017-05-19 Thread Andrea Sosio
Hello list, is there a way to bind a map to a dictionary in such a way that the map (key, value) structure is actually preserved? I'm doing something like %spark val mymap = Map("mykey"->"myvalue") z.angularBind("mydictionary", mymap) %angular my dictionary is: {{mydictionary}} I