Hi,
I run into the problem that the 'global' sqlContext variable is not
available in the pyspark interpreter.
If I have for example the folowing code:
%pyspark
df = sqlContext.createDataFrame(...)
I get the error:
(<type 'exceptions.NameError'>, NameError("name 'sqlContext'is not
defined",)
When I add the sqlContext explicitly:
from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)
the df will be created, but if I register it as a (temp) table, it is not
available in the sql interpreter! (or in the SQLcontext in Scala)
If I do the same thing in Scala it works fine, for example if I run the
example notebook with the 'bank' table.
Some info about my enivornment:
I'm running spark in yarn-client mode, the spark.home and
zeppelin.pyspark.python properties of the interpreter are set, to resp
spark 1.3 and python 2.7.
Thanks in advance for your help,
Dafne
