Matthew Bayer created SPARK-60199:
-------------------------------------
Summary: XML reader silently changes microsecond timestamps under
LEGACY timeParserPolicy, even in FAILFAST mode
Key: SPARK-60199
URL: https://issues.apache.org/jira/browse/SPARK-60199
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.2.0
Reporter: Matthew Bayer
When {{spark.sql.legacy.timeParserPolicy=LEGACY}} is explicitly configured,
Spark’s built-in XML reader misinterprets fractional seconds in otherwise valid
ISO-8601 timestamps.
For example:
{{Input: 2024-02-29T12:34:56.123456Z
Expected: 2024-02-29T12:34:56.123456Z
Actual: 2024-02-29T12:36:59.456Z}}
This affects both {{PERMISSIVE}} and {{{}FAILFAST{}}}: parsing succeeds, so
{{FAILFAST}} does not protect against the incorrect value.
This is a migration concern for pipelines moving from Spark 3 with
{{spark-xml}} to Spark 4’s built-in XML reader while retaining {{LEGACY}} for
compatibility with other timestamp operations. The old connector parsed these
ISO timestamps independently of Spark’s legacy parser and preserved the
fractional seconds.
h3. Reproduction
Run in {{{}spark-shell{}}}; no external connector or input files are required:
{{import spark.implicits._
println(spark.version)
spark.conf.set("spark.sql.session.timeZone", "UTC")
spark.conf.set("spark.sql.legacy.timeParserPolicy", "LEGACY")
val xml = Seq(
"<ROW><stamp>2024-02-29T12:34:56.123456Z</stamp></ROW>"
).toDS()
Seq("PERMISSIVE", "FAILFAST").foreach \{ mode =>
println(s"mode=$mode")
spark.read
.schema("stamp TIMESTAMP")
.option("rowTag", "ROW")
.option("mode", mode)
.xml(xml)
.selectExpr(
"CAST(stamp AS STRING) AS parsed",
"unix_micros(stamp) AS epoch_micros"
)
.show(false)
}}}
Both modes produce:
{{parsed epoch_micros
2024-02-29 12:36:59.456 1709210219456000}}
Expected:
{{parsed epoch_micros
2024-02-29 12:34:56.123456 1709210096123456}}
Changing the policy to {{CORRECTED}} produces the expected value for this
example. However, globally changing that policy also changes non-XML parsing
behavior.
h3. Apparent cause
Under {{{}LEGACY{}}},
[XmlOptions|https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlOptions.scala#L161-L167]
supplies the default pattern:
{{yyyy-MM-dd'T'HH:mm:ss.SSSXXX}}
[StaxXmlParser|https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala#L60-L65]
selects {{{}FAST_DATE_FORMAT{}}}.
[LegacyFastTimestampFormatter|https://github.com/apache/spark/blob/v4.2.0/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala#L414-L471]
accepts the six-digit fraction but scales it using the pattern’s three {{S}}
characters:
{{123456 × 1,000,000 / 10³ = 123,456,000 microseconds
= 123.456 seconds}}
That value is added to the whole-second timestamp instead of being interpreted
as {{0.123456}} seconds.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
