Interesting finding: Escaping works if data is quoted but not otherwise.
From: "Jain, Nishit" <[email protected]<mailto:[email protected]>>
Date: Thursday, October 27, 2016 at 10:54 AM
To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Subject: CSV escaping not working
I am using spark-core version 2.0.1 with Scala 2.11. I have simple code to read
a csv file which has \ escapes.
val myDA = spark.read
.option("quote",null)
.schema(mySchema)
.csv(filePath)
As per documentation \ is default escape for csv reader. But it does not work.
Spark is reading \ as part of my data. For Ex: City column in csv file is north
rocks\,au . I am expecting city column should read in code as northrocks,au.
But instead spark reads it as northrocks\ and moves au to next column.
I have tried following but did not work:
* Explicitly defined escape .option("escape",”\\")
* Changed escape to | or : in file and in code
* I have tried using spark-csv library
Any one facing same issue? Am I missing something?
Thanks
