[
https://issues.apache.org/jira/browse/SPARK-60199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthew Bayer updated SPARK-60199:
----------------------------------
Description:
When \{{spark.sql.legacy.timeParserPolicy=LEGACY}} is explicitly configured,
Spark’s built-in XML reader misinterprets fractional seconds in otherwise valid
ISO-8601 timestamps.
For example:
{noformat}
Input: 2024-02-29T12:34:56.123456Z
Expected: 2024-02-29T12:34:56.123456Z
Actual: 2024-02-29T12:36:59.456Z
{noformat}
This affects both \{{PERMISSIVE}} and \{{FAILFAST}}: parsing succeeds, so
\{{FAILFAST}} does not protect against the incorrect value.
This is a migration concern for pipelines moving from Spark 3 with
\{{spark-xml}} to Spark 4’s built-in XML reader while retaining \{{LEGACY}} for
compatibility with other timestamp operations. The old connector parsed these
ISO timestamps independently of Spark’s legacy parser and preserved the
fractional seconds.
h3. Reproduction
Run in \{{spark-shell}}; no external connector or input files are required:
{code:language=scala}
import spark.implicits._
println(spark.version)
spark.conf.set("spark.sql.session.timeZone", "UTC")
spark.conf.set("spark.sql.legacy.timeParserPolicy", "LEGACY")
val xml = Seq(
"<ROW><stamp>2024-02-29T12:34:56.123456Z</stamp></ROW>"
).toDS()
Seq("PERMISSIVE", "FAILFAST").foreach { mode =>
println(s"mode=$mode")
spark.read
.schema("stamp TIMESTAMP")
.option("rowTag", "ROW")
.option("mode", mode)
.xml(xml)
.selectExpr(
"CAST(stamp AS STRING) AS parsed",
"unix_micros(stamp) AS epoch_micros"
)
.show(false)
}
{code}
Both modes produce:
{noformat}
parsed epoch_micros
2024-02-29 12:36:59.456 1709210219456000
{noformat}
Expected:
{noformat}
parsed epoch_micros
2024-02-29 12:34:56.123456 1709210096123456
{noformat}
Changing the policy to \{{CORRECTED}} produces the expected value for this
example. However, globally changing that policy also changes non-XML parsing
behavior.
h3. Apparent cause
Under \{{LEGACY}},
[XmlOptions|https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlOptions.scala#L161-L167]
supplies the default pattern:
{noformat}
yyyy-MM-dd'T'HH:mm:ss.SSSXXX
{noformat}
[StaxXmlParser|https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala#L60-L65]
selects \{{FAST_DATE_FORMAT}}.
[LegacyFastTimestampFormatter|https://github.com/apache/spark/blob/v4.2.0/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala#L414-L471]
accepts the six-digit fraction but scales it using the pattern’s three \{{S}}
characters:
{noformat}
123456 × 1,000,000 / 10³ = 123,456,000 microseconds
= 123.456 seconds
{noformat}
That value is added to the whole-second timestamp instead of being interpreted
as 0.123456 seconds.
was:
When {{spark.sql.legacy.timeParserPolicy=LEGACY}} is explicitly configured,
Spark’s built-in XML reader misinterprets fractional seconds in otherwise valid
ISO-8601 timestamps.
For example:
Input: 2024-02-29T12:34:56.123456Z
Expected: 2024-02-29T12:34:56.123456Z
Actual: 2024-02-29T12:36:59.456Z
This affects both {{PERMISSIVE}} and {{{}FAILFAST{}}}: parsing succeeds, so
{{FAILFAST}} does not protect against the incorrect value.
This is a migration concern for pipelines moving from Spark 3 with
{{spark-xml}} to Spark 4’s built-in XML reader while retaining {{LEGACY}} for
compatibility with other timestamp operations. The old connector parsed these
ISO timestamps independently of Spark’s legacy parser and preserved the
fractional seconds.
h3. Reproduction
Run in {{{}spark-shell{}}}; no external connector or input files are required:
```python
import spark.implicits._
println(spark.version)
spark.conf.set("spark.sql.session.timeZone", "UTC")
spark.conf.set("spark.sql.legacy.timeParserPolicy", "LEGACY")
val xml = Seq(
"<ROW><stamp>2024-02-29T12:34:56.123456Z</stamp></ROW>"
).toDS()
Seq("PERMISSIVE", "FAILFAST").foreach { mode =>
println(s"mode=$mode")
spark.read
.schema("stamp TIMESTAMP")
.option("rowTag", "ROW")
.option("mode", mode)
.xml(xml)
.selectExpr(
"CAST(stamp AS STRING) AS parsed",
"unix_micros(stamp) AS epoch_micros"
)
.show(false)
```
Both modes produce:
{{parsed epoch_micros
2024-02-29 12:36:59.456 1709210219456000}}
Expected:
{{parsed epoch_micros
2024-02-29 12:34:56.123456 1709210096123456}}
Changing the policy to {{CORRECTED}} produces the expected value for this
example. However, globally changing that policy also changes non-XML parsing
behavior.
h3. Apparent cause
Under {{{}LEGACY{}}},
[XmlOptions|https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlOptions.scala#L161-L167]
supplies the default pattern:
{{yyyy-MM-dd'T'HH:mm:ss.SSSXXX}}
[StaxXmlParser|https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala#L60-L65]
selects {{{}FAST_DATE_FORMAT{}}}.
[LegacyFastTimestampFormatter|https://github.com/apache/spark/blob/v4.2.0/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala#L414-L471]
accepts the six-digit fraction but scales it using the pattern’s three {{S}}
characters:
{{123456 × 1,000,000 / 10³ = 123,456,000 microseconds
= 123.456 seconds}}
That value is added to the whole-second timestamp instead of being interpreted
as {{0.123456}} seconds.
> XML reader silently changes microsecond timestamps under LEGACY
> timeParserPolicy, even in FAILFAST mode
> -------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60199
> URL: https://issues.apache.org/jira/browse/SPARK-60199
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Matthew Bayer
> Priority: Major
> Labels: correctness
>
> When \{{spark.sql.legacy.timeParserPolicy=LEGACY}} is explicitly configured,
> Spark’s built-in XML reader misinterprets fractional seconds in otherwise
> valid ISO-8601 timestamps.
> For example:
> {noformat}
> Input: 2024-02-29T12:34:56.123456Z
> Expected: 2024-02-29T12:34:56.123456Z
> Actual: 2024-02-29T12:36:59.456Z
> {noformat}
> This affects both \{{PERMISSIVE}} and \{{FAILFAST}}: parsing succeeds, so
> \{{FAILFAST}} does not protect against the incorrect value.
> This is a migration concern for pipelines moving from Spark 3 with
> \{{spark-xml}} to Spark 4’s built-in XML reader while retaining \{{LEGACY}}
> for compatibility with other timestamp operations. The old connector parsed
> these ISO timestamps independently of Spark’s legacy parser and preserved the
> fractional seconds.
> h3. Reproduction
> Run in \{{spark-shell}}; no external connector or input files are required:
> {code:language=scala}
> import spark.implicits._
> println(spark.version)
> spark.conf.set("spark.sql.session.timeZone", "UTC")
> spark.conf.set("spark.sql.legacy.timeParserPolicy", "LEGACY")
> val xml = Seq(
> "<ROW><stamp>2024-02-29T12:34:56.123456Z</stamp></ROW>"
> ).toDS()
> Seq("PERMISSIVE", "FAILFAST").foreach { mode =>
> println(s"mode=$mode")
> spark.read
> .schema("stamp TIMESTAMP")
> .option("rowTag", "ROW")
> .option("mode", mode)
> .xml(xml)
> .selectExpr(
> "CAST(stamp AS STRING) AS parsed",
> "unix_micros(stamp) AS epoch_micros"
> )
> .show(false)
> }
> {code}
> Both modes produce:
> {noformat}
> parsed epoch_micros
> 2024-02-29 12:36:59.456 1709210219456000
> {noformat}
> Expected:
> {noformat}
> parsed epoch_micros
> 2024-02-29 12:34:56.123456 1709210096123456
> {noformat}
> Changing the policy to \{{CORRECTED}} produces the expected value for this
> example. However, globally changing that policy also changes non-XML parsing
> behavior.
> h3. Apparent cause
> Under \{{LEGACY}},
> [XmlOptions|https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlOptions.scala#L161-L167]
> supplies the default pattern:
> {noformat}
> yyyy-MM-dd'T'HH:mm:ss.SSSXXX
> {noformat}
> [StaxXmlParser|https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala#L60-L65]
> selects \{{FAST_DATE_FORMAT}}.
> [LegacyFastTimestampFormatter|https://github.com/apache/spark/blob/v4.2.0/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala#L414-L471]
> accepts the six-digit fraction but scales it using the pattern’s three
> \{{S}} characters:
> {noformat}
> 123456 × 1,000,000 / 10³ = 123,456,000 microseconds
> = 123.456 seconds
> {noformat}
> That value is added to the whole-second timestamp instead of being
> interpreted as 0.123456 seconds.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
