[
https://issues.apache.org/jira/browse/SPARK-60199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60199:
-----------------------------------
Labels: correctness pull-request-available (was: correctness)
> XML reader silently changes microsecond timestamps under LEGACY
> timeParserPolicy, even in FAILFAST mode
> -------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60199
> URL: https://issues.apache.org/jira/browse/SPARK-60199
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Matthew Bayer
> Priority: Major
> Labels: correctness, pull-request-available
>
> When \{{spark.sql.legacy.timeParserPolicy=LEGACY}} is explicitly configured,
> Spark’s built-in XML reader misinterprets fractional seconds in otherwise
> valid ISO-8601 timestamps.
> For example:
> {noformat}
> Input: 2024-02-29T12:34:56.123456Z
> Expected: 2024-02-29T12:34:56.123456Z
> Actual: 2024-02-29T12:36:59.456Z
> {noformat}
> This affects both \{{PERMISSIVE}} and \{{FAILFAST}}: parsing succeeds, so
> \{{FAILFAST}} does not protect against the incorrect value.
> This is a migration concern for pipelines moving from Spark 3 with
> \{{spark-xml}} to Spark 4’s built-in XML reader while retaining \{{LEGACY}}
> for compatibility with other timestamp operations. The old connector parsed
> these ISO timestamps independently of Spark’s legacy parser and preserved the
> fractional seconds.
> h3. Reproduction
> Run in \{{spark-shell}}; no external connector or input files are required:
> {code:language=scala}
> import spark.implicits._
> println(spark.version)
> spark.conf.set("spark.sql.session.timeZone", "UTC")
> spark.conf.set("spark.sql.legacy.timeParserPolicy", "LEGACY")
> val xml = Seq(
> "<ROW><stamp>2024-02-29T12:34:56.123456Z</stamp></ROW>"
> ).toDS()
> Seq("PERMISSIVE", "FAILFAST").foreach { mode =>
> println(s"mode=$mode")
> spark.read
> .schema("stamp TIMESTAMP")
> .option("rowTag", "ROW")
> .option("mode", mode)
> .xml(xml)
> .selectExpr(
> "CAST(stamp AS STRING) AS parsed",
> "unix_micros(stamp) AS epoch_micros"
> )
> .show(false)
> }
> {code}
> Both modes produce:
> {noformat}
> parsed epoch_micros
> 2024-02-29 12:36:59.456 1709210219456000
> {noformat}
> Expected:
> {noformat}
> parsed epoch_micros
> 2024-02-29 12:34:56.123456 1709210096123456
> {noformat}
> Changing the policy to \{{CORRECTED}} produces the expected value for this
> example. However, globally changing that policy also changes non-XML parsing
> behavior.
> h3. Apparent cause
> Under \{{LEGACY}},
> [XmlOptions|https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlOptions.scala#L161-L167]
> supplies the default pattern:
> {noformat}
> yyyy-MM-dd'T'HH:mm:ss.SSSXXX
> {noformat}
> [StaxXmlParser|https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala#L60-L65]
> selects \{{FAST_DATE_FORMAT}}.
> [LegacyFastTimestampFormatter|https://github.com/apache/spark/blob/v4.2.0/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala#L414-L471]
> accepts the six-digit fraction but scales it using the pattern’s three
> \{{S}} characters:
> {noformat}
> 123456 × 1,000,000 / 10³ = 123,456,000 microseconds
> = 123.456 seconds
> {noformat}
> That value is added to the whole-second timestamp instead of being
> interpreted as 0.123456 seconds.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
