Hello,
I am following the tutorial code on sql programming
guide<https://spark.apache.org/docs/1.2.1/sql-programming-guide.html#inferring-the-schema-using-reflection>
to try out Python on spark 1.2.1.
SaveAsTable function works on Scala bur fails on python with "Unresolved plan
found".
Broken Python code:
from pyspark.sql import SQLContext, Row
sqlContext = SQLContext(sc)
lines = sc.textFile("data.txt")
parts = lines.map(lambda l: l.split(","))
people = parts.map(lambda p: Row(id=p[0], name=p[1]))
schemaPeople = sqlContext.inferSchema(people)
schemaPeople.saveAsTable("peopletable")
saveAsTable fails with Unresolved plan found.
org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Unresolved plan
found, tree:
'CreateTableAsSelect None, pytable, false, None
This scala code works fine:
from pyspark.sql import SQLContext, Row
sqlContext = SQLContext(sc)
lines = sc.textFile("data.txt")
parts = lines.map(lambda l: l.split(","))
people = parts.map(lambda p: Row(id=p[0], name=p[1]))
schemaPeople = sqlContext.inferSchema(people)
schemaPeople.saveAsTable("peopletable")
Is this a known issue? Or am I not using Python correctly?
Thanks,
Judy