Dejan Krakovic created SPARK-59477:
--------------------------------------
Summary: Add CSV write option treatNullAsEmptyString
Key: SPARK-59477
URL: https://issues.apache.org/jira/browse/SPARK-59477
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 4.2.0
Reporter: Dejan Krakovic
Adds a per-write CSV option {{treatNullAsEmptyString}} that overrides the
session-level {{spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv}}
config for a single write.
When set to false, nulls are written as a bare, unquoted empty token while
empty strings stay quoted ({{{}""{}}}), so downstream consumers can
differentiate null from an empty string. When set to true, nulls are written as
the {{{}emptyValue{}}}. When unset, the session config decides the behavior, so
existing writes are unaffected.
This lets a workload opt into the differentiation without changing the
session/workspace default.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
