Spark-csv data source: infer data types

2015-04-18 Thread barmaley
.html), for built-in sources such as JSON, the schema with data types can be inferred automatically. Can the types of columns in CSV file be inferred automatically? Thanks -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-csv-data-source-infer-data-types-tp

Spark-csv data source: infer data types

2015-04-18 Thread Oleg Shirokikh
I'm experimenting with Spark-CSV package (https://github.com/databricks/spark-csv) for reading csv files into Spark DataFrames. Everything works but all columns are assumed to be of StringType. As shown in Spark SQL documentation (https://spark.apache.org/docs/latest/sql-programming-guide.html