Would *df.na.fill("") *do the trick?
On Fri, Jul 31, 2020 at 8:43 AM Sean Owen <sro...@gmail.com> wrote:
> Try setting nullValue to anything besides the empty string. Because its
> default is the empty string, empty strings become null by default.
>
> On Fri, Jul 31, 2020 at 3:20 AM Stephen Coy <s...@infomedia.com.au.invalid>
> wrote:
>
>> That does not work.
>>
>> This is Spark 3.0 by the way.
>>
>> I have been looking at the Spark unit tests and there does not seem to be
>> any that load a CSV text file and verify that an empty string maps to an
>> empty string which I think is supposed to be the default behaviour because
>> the “nullValue” option defaults to “".
>>
>> Thanks anyway
>>
>> Steve C
>>
>> On 30 Jul 2020, at 10:01 pm, German Schiavon Matteo <
>> gschiavonsp...@gmail.com> wrote:
>>
>> Hey,
>>
>> I understand that your empty values in your CSV are "" , if so, try this
>> option:
>>
>> *.option("emptyValue", "\"\"")*
>>
>> Hope it helps
>>
>> On Thu, 30 Jul 2020 at 08:49, Stephen Coy <s...@infomedia.com.au.invalid>
>> wrote:
>>
>>> Hi there,
>>>
>>> I’m trying to import a tab delimited file with:
>>>
>>> Dataset<Row> catalogData = sparkSession
>>> .read()
>>> .option("sep", "\t")
>>> .option("header", "true")
>>> .csv(args[0])
>>> .cache();
>>>
>>>
>>> This works great, except for the fact that any column that is empty is
>>> given the value null, when I need these values to be literal empty strings.
>>>
>>> Is there any option combination that will achieve this?
>>>
>>> Thanks,
>>>
>>> Steve C
>>>
>>>
>>> <https://www.infomedia.com.au/driving-force/?utm_campaign=200630%20Email%20Signature&utm_source=Internal&utm_medium=Email&utm_content=Driving%20Force>
>>> This email contains confidential information of and is the copyright of
>>> Infomedia. It must not be forwarded, amended or disclosed without consent
>>> of the sender. If you received this message by mistake, please advise the
>>> sender and delete all copies. Security of transmission on the internet
>>> cannot be guaranteed, could be infected, intercepted, or corrupted and you
>>> should ensure you have suitable antivirus protection in place. By sending
>>> us your or any third party personal details, you consent to (or confirm you
>>> have obtained consent from such third parties) to Infomedia’s privacy
>>> policy. http://www.infomedia.com.au/privacy-policy/
>>>
>>
>>
