Re: changed behavior for csv datasource and quoting in spark 2.0.0-SNAPSHOT

2016-05-26 Thread Reynold Xin
Yup - but the reason we did the null handling that way was for Python, which also affects Scala. On Thu, May 26, 2016 at 4:17 PM, Koert Kuipers wrote: > ok, thanks for creating ticket. > > just to be clear: my example was in scala > > On Thu, May 26, 2016 at 7:07 PM, Reynold Xin wrote: > >> Th

Re: changed behavior for csv datasource and quoting in spark 2.0.0-SNAPSHOT

2016-05-26 Thread Koert Kuipers
ok, thanks for creating ticket. just to be clear: my example was in scala On Thu, May 26, 2016 at 7:07 PM, Reynold Xin wrote: > This is unfortunately due to the way we set handle default values in > Python. I agree it doesn't follow the principle of least astonishment. > > Maybe the best thing

Re: changed behavior for csv datasource and quoting in spark 2.0.0-SNAPSHOT

2016-05-26 Thread Reynold Xin
This is unfortunately due to the way we set handle default values in Python. I agree it doesn't follow the principle of least astonishment. Maybe the best thing to do here is to put the actual default values in the Python API for csv (and json, parquet, etc), rather than using None in Python. This