Have you tried SparkContext.addFile() (not addPyFile()) to add your R script?
https://spark.apache.org/docs/2.2.0/api/python/pyspark.html#pyspark.SparkContext.addFile
From: Lian Jiang <jiangok2...@gmail.com>
Sent: 27 August 2018 22:42
To: users@zeppelin.apache.org
Subject: EXT: Python script calls R script in Zeppelin on Hadoop
Hi,
We are using HDP3.0 (using zeppelin 0.8.0) and are migrating Jupyter notebooks
to Zeppelin. One issue we came across is that a python script calling R script
does not work in Zeppelin.
%livy2.pyspark
import os
sc.addPyFile("hdfs:///user/zeppelin/my.py")
import my
my.test()
my.test() calls R script like: ['Rscript', 'myR.r']
Fatal error: cannot open file 'myR.r': No such file or directory
When running this notebook in jupyter, both my.py and myR.r exist in the same
folder. I understand the story changes on hadoop because the scripts run in
containers.
My question:
Is this scenario supported in zeppelin? How to add a R script into a python
spark context so that the Python script can find the R script? Appreciate!
