Hi Swetha, I always look into the source code a lot but it never occured to me to look into the test suite, thank a ton for the tip. Does definitely give quite a few ideas - thanks a ton.
Thanks and Regards, Gourav On Fri, Jul 12, 2019 at 6:51 AM Swetha Ramaiah <swetha.rama...@gmail.com> wrote: > Hi Anil > > That was an example. You can replace quote with what double quotes. But > these options should give you an idea on how you want treat nulls, empty > values and quotes. > > When I faced this issues, I forked Spark repo and looked at the test > suite. This definitely helped me solve my issue. > > https://github.com/apache/spark/blob/v2.4.3/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala > > Regards > Swetha > > On Jul 11, 2019, at 4:09 PM, Anil Kulkarni <anil...@gmail.com> wrote: > > Hi Swetha, > > Thank you. > But we need the data to be quoted with ". > and when a field is null, we dont need the quotes around it. > > Example: > "A",,"B","C" > Thanks > Anil > > On Thu, Jul 11, 2019, 1:51 PM Swetha Ramaiah <swetha.rama...@gmail.com> > wrote: > >> If you are using Spark 2.4.0, I think you can try something like this: >> >> .option("quote", "\u0000") >> .option("emptyValue", “”) >> >> .option("nullValue", null) >> >> Regards >> Swetha >> >> >> >> On Jul 11, 2019, at 1:45 PM, Anil Kulkarni <anil...@gmail.com> wrote: >> >> Hi Spark users, >> >> My question is : >> I am writing a Dataframe to csv. >> Option i am using as >> .option("quoteAll","true"). >> >> This is quoting even null values and making them appear as an empty >> string. >> >> How do i make sure that quotes are enabled only for non null values? >> >> -- >> Cheers, >> Anil Kulkarni >> about.me/anilkulkarni >> [image: Anil Kulkarni on about.me] >> http://anilkulkarni.com/ <http://about.me/anilkulkarni> >> >> >> >
