GitHub user falaki opened a pull request:
https://github.com/apache/spark/pull/1351
[WIP] SPARK-2360: CSV import to SchemaRDDs
Implements RFC 4180 for parsing comma separated value files into Schema
RDDs, which includes:
* Optional header line
* Handling quoted fields
* Handling new lines inside quoted fields
Adds two new methods to SQLContext
* csvFile: Takes a path and returns a SchemaRDD
* csvRDD: Takes an RDD[String] and returns a SchemaRDD
__TODO:__
* Every field is assumed to be of type String. We can either infer types
(using a sample of data) or let user specify types of columns.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/falaki/spark csv
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1351.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1351
----
commit 177eb062f6bdbf0f59d8a29ce1ff620ec8fe5ec9
Author: Hossein <[email protected]>
Date: 2014-07-08T02:12:54Z
Basic version of csv parsing
commit 510df2ef8bec5a963892af49cfcc88716abf9c93
Author: Hossein <[email protected]>
Date: 2014-07-10T02:19:58Z
RFC 4180 compatible tokenizer
commit 30a5ae52983c70816330a9c467c0d7a125564529
Author: Hossein <[email protected]>
Date: 2014-07-10T02:20:21Z
Added API documentation
commit ac95fcba25ab5175c58d6833c5d35bf7ac652370
Author: Hossein <[email protected]>
Date: 2014-07-10T02:20:58Z
Added unit tests
commit 95a7a1ac49af6e87cc2b85f1f186d1a88ad47cc3
Author: Hossein <[email protected]>
Date: 2014-07-10T02:24:15Z
Organized imports
commit 65f7e95da29cb603af3c0919bebeb0d4e8ef1dcb
Author: Hossein <[email protected]>
Date: 2014-07-10T02:32:35Z
Style cleanup
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
