[
https://issues.apache.org/jira/browse/SPARK-60324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60324:
-----------------------------------
Labels: pull-request-available (was: )
> Partitioned write fails with FileAlreadyExistsException when a FLOAT or
> DOUBLE partition column holds 0.0 and -0.0
> ------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60324
> URL: https://issues.apache.org/jira/browse/SPARK-60324
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.3, 4.2.0, 4.0.3
> Reporter: sepuri sai krishna
> Priority: Major
> Labels: pull-request-available
>
> A partitioned write fails when a FLOAT or DOUBLE partition column holds both
> 0.0 and -0.0 in the same task, and nothing is written. With the default
> configuration:
> {code:sql}
> CREATE TABLE t (id INT, d DOUBLE) USING parquet PARTITIONED BY (d);
> INSERT INTO t SELECT /*+ COALESCE(1) */ * FROM VALUES
> (1, 0.0D), (2, -0.0D), (3, 0.0D), (4, 1.5D) AS v(id, d);
> -- [TASK_WRITE_FAILED] Task failed while writing rows to file:/.../t.
> -- Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: File already
> exists:
> --
> file:/.../t/_temporary/0/_temporary/attempt_.../d=0.0/part-00000-<uuid>.c000.snappy.parquet
> SELECT count(*) FROM t; -- 0
> {code}
> Same on 3.5.3, 4.0.3 and 4.2.0 (official binaries) and on master, for
> Parquet, ORC, CSV and JSON, {{INSERT INTO}}, {{INSERT OVERWRITE}}, CTAS and
> {{df.write.partitionBy}}, FLOAT and DOUBLE, with other partition columns
> before it, and Hive tables. NaNs with different bits, e.g. {{NaN}} and
> {{-NaN}}, fail the same way, on {{d=NaN}}.
> Cause: the write sorts the rows of a task by the dynamic partition columns,
> where -0.0 = 0.0 and all NaNs are equal, so the sort can leave them
> interleaved: 0.0, -0.0, 0.0. {{DynamicPartitionDataSingleWriter}} starts a
> new partition directory whenever the bytes of the partition values change, so
> it writes d=0.0, then d=-0.0, then d=0.0 again, and the second file of d=0.0
> has the same name as the first ({{c000}}), which exists. With concurrent
> writers ({{spark.sql.maxConcurrentOutputFileWriters}}) the writers are keyed
> by those bytes, so NaNs with different bits open the same {{d=NaN}} file, and
> when the writers fall back to sorting, the zeros fail as above.
> When the zeros are in different tasks the write succeeds, but creates both
> d=0.0 and d=-0.0, so where a row ends up depends on how the rows are split
> into tasks. The file stream sink names every file with a random UUID, so it
> does not fail, but it writes both d=0.0 and d=-0.0 even within one task.
> Spark already normalizes -0.0 to 0.0 and NaNs to one NaN for grouping, join
> and window partition keys (SPARK-26021, NormalizeFloatingNumbers), and for
> array set operations (SPARK-54918). Dynamic partition values are grouping
> keys as well. SPARK-26012 fixed the same failure for null and empty strings,
> which both go to the default partition, by converting empty strings to null
> before the write.
> Workaround for the zeros: write {{d + 0.0D}} instead of {{d}}, which turns
> -0.0 into 0.0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
