Hi,
I am newbie to spark sql and i would like to know about how to read all the
columns from a file in spark sql. I have referred the programming guide
here:
http://people.apache.org/~tdas/spark-1.0-docs/sql-programming-guide.html
The example says:
val people =
sc.textFile("examples/src/main/resources/people.txt").map(_.split(",")).map(p
=> Person(p(0), p(1).trim.toInt))
But, instead of explicitly specifying p(0),p(1) I would like to read all
the columns from a file. It would be difficult if my source dataset has
more no of columns.
Is there any shortcut for that?
And instead of a single file, i would like to read multiple files which
shares a similar structure from a directory.
Could you please share your thoughts on this?
It would be great , if you share any documentation which has details on
these?
Thanks
