Re: reading data from s3

2016-12-09 Thread Sudev A C
Hi Hitesh, Schema of the table is inferred automatically if you are reading from JSON file, wherein when you are reading from a text file you will have to provide a schema for the table you want to create (JSON has schema within it). You can create a data frames and register them as tables. 1. In

reading data from s3

2016-12-08 Thread Hitesh Goyal
Hi team, I want to read the text file from s3. I am doing it using DataFrame. Like below:- DataFrame d=sql.read().text("s3://my_first_text_file.txt"); d.registerTempTable("table1"); DataFrame d1=sql.sql("Select * from table1"); d1.printSchema();

Re: Directly reading data from S3 to EC2 with PySpark

2015-09-15 Thread ayan guha
Hyeok >> >> Did you set HADOOP_CONF_DIR? It seems that spark cannot find AWS key >> properties >> >> If it doesn't work after set, How about export AWS_ACCESS_KEY_ID, >> AWS_SECRET_ACCESS_KEY before running py-spark shell? >> >> BR >> >&g

Re: Directly reading data from S3 to EC2 with PySpark

2015-09-15 Thread Gourav Sengupta
; > BR > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Directly-reading-data-from-S3-to-EC2-with-PySpark-tp24638p24698.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > ---

Re: Directly reading data from S3 to EC2 with PySpark

2015-09-15 Thread Cazen
01560.n3.nabble.com/Directly-reading-data-from-S3-to-EC2-with-PySpark-tp24638p24698.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For addit