That does not work.
This is Spark 3.0 by the way.
I have been looking at the Spark unit tests and there does not seem to be any
that load a CSV text file and verify that an empty string maps to an empty
string which I think is supposed to be the default behaviour because the
“nullValue” option
Try setting nullValue to anything besides the empty string. Because its
default is the empty string, empty strings become null by default.
On Fri, Jul 31, 2020 at 3:20 AM Stephen Coy
wrote:
> That does not work.
>
> This is Spark 3.0 by the way.
>
> I have been looking at the Spark unit tests an
Would *df.na.fill("") *do the trick?
On Fri, Jul 31, 2020 at 8:43 AM Sean Owen wrote:
> Try setting nullValue to anything besides the empty string. Because its
> default is the empty string, empty strings become null by default.
>
> On Fri, Jul 31, 2020 at 3:20 AM Stephen Coy
> wrote:
>
>> That
Python doesn't allow abbreviating () with no param, whereas Scala does. Use
`write()`, not `write`.
On Wed, Jul 29, 2020 at 9:09 AM muru wrote:
> In a pyspark SS job, trying to use sql instead of sql functions in
> foreachBatch sink
> throws AttributeError: 'JavaMember' object has no attribute '
