[
https://issues.apache.org/jira/browse/SPARK-60201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60201:
-----------------------------------
Labels: pull-request-available (was: )
> Spark 4 XML silently changes ISO date interpretation when a custom dateFormat
> is configured
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-60201
> URL: https://issues.apache.org/jira/browse/SPARK-60201
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Matthew Bayer
> Priority: Major
> Labels: pull-request-available
>
> h3. Description
> Migrating from \{{spark-xml}} to Spark 4’s built-in XML reader can silently
> change date values when an input matches both ISO format and a configured
> \{{dateFormat}}.
> The previous connector tries ISO parsing first, falling back to the custom
> format. Spark 4 uses the configured format directly.
> For example, \{{2024-03-04}} with \{{dateFormat=yyyy-dd-MM}} changes from
> *March 4* to *April 3*, without an error.
> April 3 is a valid interpretation of the configured format. This report
> concerns the backward-incompatible change in parsing precedence, rather than
> an incorrectly implemented format.
> h3. Reproduction
> Compare \{{spark-xml}} 0.17.0 with Spark 4’s built-in XML reader, using the
> same input and options.
> {\{input.xml}}:
> {code:xml}
> <ROW><event_date>2024-03-04</event_date></ROW>
> {code}
> {code:python}
> for policy in ["LEGACY", "CORRECTED", "EXCEPTION"]:
> spark.conf.set("spark.sql.legacy.timeParserPolicy", policy)
> result = (
> spark.read.format("xml")
> .option("rowTag", "ROW")
> .option("dateFormat", "yyyy-dd-MM")
> .option("mode", "FAILFAST")
> .schema("event_date DATE")
> .load("/path/to/input.xml")
> .first()
> )
> print(policy, result.event_date)
> {code}
> h3. Actual behavior
> || Reader || LEGACY || CORRECTED || EXCEPTION ||
> | \{{spark-xml}} 0.17.0 | \{{2024-03-04}} | \{{2024-03-04}} | \{{2024-03-04}}
> |
> | Spark 4 built-in XML | \{{2024-04-03}} | \{{2024-04-03}} | \{{2024-04-03}} |
> {\{FAILFAST}} does not protect against this change because parsing succeeds.
> Retaining \{{spark.sql.legacy.timeParserPolicy=LEGACY}} also does not
> preserve the previous result.
> h3. Expected behavior
> Provide a compatibility path that preserves the previous interpretation, or
> fails explicitly instead of silently returning a different date.
> A narrowly scoped solution would restore the old XML date-parsing precedence
> under \{{LEGACY}}: try ISO first, then the explicitly configured custom
> format. Reading and schema inference should use the same parsing rules,
> without changing shared date formatters or other data sources.
> h3. Impact
> Existing pipelines containing ISO dates alongside a custom date-format option
> can produce different dates after upgrading. Ambiguous day/month values can
> affect date filters, joins, partitioning, and aggregations without failing
> the build.
> For ISO-only inputs, removing \{{dateFormat}} or setting it to
> \{{yyyy-MM-dd}} is a workaround. That does not preserve the old behavior for
> inputs mixing ISO and custom formats.
> h3. Reference
> The previous ISO-first behavior is implemented in [spark-xml 0.17.0’s
> parseXmlDate|https://github.com/databricks/spark-xml/blob/v0.17.0/src/main/scala/com/databricks/spark/xml/util/TypeCast.scala].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
