Hi,
I'm puzzled - I have zeppelin 0.10 with spark 3.1.2.
The driver has Anaconda with python 3.6.5 installed.
Running pyspark paragraph, I'm getting some weird behavior. Paragraph runs
successfully on first attempt, but then failing on successive attempts
(until interpreter restarted).
Error is:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-13-4fe403202586> in <module>() ----> 1 sc.setJobGroup(
'zeppelin|admin|2GAWSQT96|paragraph_1629208863735_110330802', 'Started by:
admin') AttributeError: 'SparkSession' object has no attribute 'setJobGroup'
Fail to setJobGroup
For instance, this paragraph:
%spark.pyspark
import pandas as pd
from pyspark.sql.types import StringType
import numpy as np
def np_sqrt(v):
return np.__path__
spark.udf.register("np_sqrt", np_sqrt, StringType())
df = spark.range(10).createOrReplaceTempView("d")
spark.sql("select np_sqrt(id) as arr from d").show(truncate=False)
BTW, I can run different pyspark paragraphs, each will succeed at the first
attempt. Once re-running a paragraph, every pyspark paragraph will fail.
Any idea what may cause it?
Thanks,
Lior
