Re: Python script calls R script in Zeppelin on Hadoop

2018-08-31 Thread Lian Jiang
;>>>>> Lian Jiang 于2018年8月29日周三 上午1:35写道： >>>>>> >>>>>>> Thanks Lucas. We tried and got the same error. Below is the code: >>>>>>> >>>>>>> %livy2.pyspark >>>>>>> import subprocess

Re: Python script calls R script in Zeppelin on Hadoop

2018-08-29 Thread Jeff Zhang
gt;> import subprocess >>>>>> sc.addFile("hdfs:///user/zeppelin/test.r") >>>>>> stdoutdata = subprocess.getoutput("Rscript test.r") >>>>>> print(stdoutdata) >>>>>> >>>>>> Fatal error: cannot open fi

Re: Python script calls R script in Zeppelin on Hadoop

2018-08-29 Thread Lian Jiang
gt;> Fatal error: cannot open file 'test.r': No such file or directory >>>>> >>>>> >>>>> sc.addFile adds test.r to spark context. However, subprocess does not >>>>> use spark context. >>>>> >>>>>

Re: Python script calls R script in Zeppelin on Hadoop

2018-08-29 Thread Jeff Zhang
t;> >>>> >>>> sc.addFile adds test.r to spark context. However, subprocess does not >>>> use spark context. >>>> >>>> Hdfs path does not work either: subprocess.getoutput("Rscript >>>> hdfs:///user/zeppelin/test.r")

Re: Python script calls R script in Zeppelin on Hadoop

2018-08-29 Thread Lian Jiang
t;> Any idea how to make python call R script? Appreciate! >>> >>> >>> >>> >>> On Tue, Aug 28, 2018 at 1:13 AM Partridge, Lucas (GE Aviation) < >>> lucas.partri...@ge.com> wrote: >>> >>>> Have you tried SparkContext.ad

Re: Python script calls R script in Zeppelin on Hadoop

2018-08-28 Thread Lian Jiang
at 1:13 AM Partridge, Lucas (GE Aviation) < >> lucas.partri...@ge.com> wrote: >> >>> Have you tried SparkContext.addFile() (not addPyFile()) to add your R >>> script? >>> >>> >>> https://spark.apache.org/docs/2.2.0/api/python/pyspark.html#pyspark.

Re: Python script calls R script in Zeppelin on Hadoop

2018-08-28 Thread Jeff Zhang
ipt? >> >> >> https://spark.apache.org/docs/2.2.0/api/python/pyspark.html#pyspark.SparkContext.addFile >> >> >> >> *From:* Lian Jiang >> *Sent:* 27 August 2018 22:42 >> *To:* users@zeppelin.apache.org >> *Subject:* EXT: Python script calls R

Re: Python script calls R script in Zeppelin on Hadoop

2018-08-28 Thread Lian Jiang
n Jiang > *Sent:* 27 August 2018 22:42 > *To:* users@zeppelin.apache.org > *Subject:* EXT: Python script calls R script in Zeppelin on Hadoop > > > > Hi, > > > > We are using HDP3.0 (using zeppelin 0.8.0) and are migrating Jupyter > notebooks to Zeppelin. One issu

RE: Python script calls R script in Zeppelin on Hadoop

2018-08-28 Thread Partridge, Lucas (GE Aviation)
Have you tried SparkContext.addFile() (not addPyFile()) to add your R script? https://spark.apache.org/docs/2.2.0/api/python/pyspark.html#pyspark.SparkContext.addFile From: Lian Jiang Sent: 27 August 2018 22:42 To: users@zeppelin.apache.org Subject: EXT: Python script calls R script in Zeppelin

Python script calls R script in Zeppelin on Hadoop

2018-08-27 Thread Lian Jiang
Hi, We are using HDP3.0 (using zeppelin 0.8.0) and are migrating Jupyter notebooks to Zeppelin. One issue we came across is that a python script calling R script does not work in Zeppelin. %livy2.pyspark import os sc.addPyFile("hdfs:///user/zeppelin/my.py") import my my.test() my.test() calls R