Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1351#discussion_r14856106
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/csv/CsvTokenizer.scala ---
@@ -0,0 +1,120 @@
+/*
--- End diff --
Hey Cheng. Thanks for the suggestion and other comments. I looked into this
and another CSV tokenizer. Unfortunately, these tokenizers do not support
reading from an Iterator. The support/code needed to make them work with an
Iterator is no less than doing it from scratch, especially because a CSV row
can potentially span multiple lines of the input iterator.
I applied your other comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
