Hi Hitesh,
Schema of the table is inferred automatically if you are reading from JSON
file, wherein when you are reading from a text file you will have to
provide a schema for the table you want to create (JSON has schema within
it).
You can create a data frames and register them as tables.
1. In
Hi team,
I want to read the text file from s3. I am doing it using DataFrame. Like
below:-
DataFrame d=sql.read().text("s3://my_first_text_file.txt");
d.registerTempTable("table1");
DataFrame d1=sql.sql("Select * from table1");
d1.printSchema();
Hyeok
>>
>> Did you set HADOOP_CONF_DIR? It seems that spark cannot find AWS key
>> properties
>>
>> If it doesn't work after set, How about export AWS_ACCESS_KEY_ID,
>> AWS_SECRET_ACCESS_KEY before running py-spark shell?
>>
>> BR
>>
>&g
;
> BR
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Directly-reading-data-from-S3-to-EC2-with-PySpark-tp24638p24698.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---
01560.n3.nabble.com/Directly-reading-data-from-S3-to-EC2-with-PySpark-tp24638p24698.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For addit
