[
https://issues.apache.org/jira/browse/SPARK-58946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subhramit Basu Bhowmick updated SPARK-58946:
--------------------------------------------
Summary: CSV extension option validation accepts invalid extensions (was:
CSV extension option validation uses `&&` instead of `||`, accepting invalid
extensions)
> CSV extension option validation accepts invalid extensions
> ----------------------------------------------------------
>
> Key: SPARK-58946
> URL: https://issues.apache.org/jira/browse/SPARK-58946
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Subhramit Basu Bhowmick
> Priority: Minor
> Labels: pull-request-available
>
> [{{CSVOptions.scala:127}}|https://github.com/apache/spark/blob/fa6f71301587395d3576acb09b7bba2d8d3afc74/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala#L127]
> validates the {{extension}} write option with:
> {code:scala}
> if (ext.size != 3 && !ext.forall(_.isLetter))
> {code}
> The error message states the extension is limited to exactly three letters.
> The {{&&}} rejects only values that fail *both* clauses, so invalid
> extensions are accepted:
> {noformat}
> "ab1" three chars, not all letters accepted
> "777" three chars, all numbers accepted
> "a/b" three chars, path separator accepted
> "12" rejected (fails both clauses)
> {noformat}
> The value flows into the output filename at {{CSVWrite.scala:61}} and
> {{CSVFileFormat.scala:91}}, so {{"a/b"}} places a path separator inside a
> filename component.
> h3. Repro
> {code:scala}
> val path = "/tmp/spark58946-demo"
> spark.range(1).write.mode("overwrite").option("extension", "a/b").csv(path)
> {code}
> is currently successfully accepted.
> h3. Dependency
> Depends on SPARK-58945. Until that lands, this error path raises
> {{INTERNAL_ERROR}} rather than {{INVALID_PARAMETER_VALUE.EXTENSION}}, because
> {{invalidFileExtensionError}} passes a {{fileExtension}} key that the
> template does not declare.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
