[
https://issues.apache.org/jira/browse/SPARK-60199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthew Bayer updated SPARK-60199:
----------------------------------
Description:
When {{spark.sql.legacy.timeParserPolicy=LEGACY}} is explicitly configured,
Spark’s built-in XML reader misinterprets fractional seconds in otherwise valid
ISO-8601 timestamps.
For example:
Input: 2024-02-29T12:34:56.123456Z
Expected: 2024-02-29T12:34:56.123456Z
Actual: 2024-02-29T12:36:59.456Z
This affects both {{PERMISSIVE}} and {{{}FAILFAST{}}}: parsing succeeds, so
{{FAILFAST}} does not protect against the incorrect value.
This is a migration concern for pipelines moving from Spark 3 with
{{spark-xml}} to Spark 4’s built-in XML reader while retaining {{LEGACY}} for
compatibility with other timestamp operations. The old connector parsed these
ISO timestamps independently of Spark’s legacy parser and preserved the
fractional seconds.
h3. Reproduction
Run in {{{}spark-shell{}}}; no external connector or input files are required:
```python
import spark.implicits._
println(spark.version)
spark.conf.set("spark.sql.session.timeZone", "UTC")
spark.conf.set("spark.sql.legacy.timeParserPolicy", "LEGACY")
val xml = Seq(
"<ROW><stamp>2024-02-29T12:34:56.123456Z</stamp></ROW>"
).toDS()
Seq("PERMISSIVE", "FAILFAST").foreach { mode =>
println(s"mode=$mode")
spark.read
.schema("stamp TIMESTAMP")
.option("rowTag", "ROW")
.option("mode", mode)
.xml(xml)
.selectExpr(
"CAST(stamp AS STRING) AS parsed",
"unix_micros(stamp) AS epoch_micros"
)
.show(false)
```
Both modes produce:
{{parsed epoch_micros
2024-02-29 12:36:59.456 1709210219456000}}
Expected:
{{parsed epoch_micros
2024-02-29 12:34:56.123456 1709210096123456}}
Changing the policy to {{CORRECTED}} produces the expected value for this
example. However, globally changing that policy also changes non-XML parsing
behavior.
h3. Apparent cause
Under {{{}LEGACY{}}},
[XmlOptions|https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlOptions.scala#L161-L167]
supplies the default pattern:
{{yyyy-MM-dd'T'HH:mm:ss.SSSXXX}}
[StaxXmlParser|https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala#L60-L65]
selects {{{}FAST_DATE_FORMAT{}}}.
[LegacyFastTimestampFormatter|https://github.com/apache/spark/blob/v4.2.0/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala#L414-L471]
accepts the six-digit fraction but scales it using the pattern’s three {{S}}
characters:
{{123456 × 1,000,000 / 10³ = 123,456,000 microseconds
= 123.456 seconds}}
That value is added to the whole-second timestamp instead of being interpreted
as {{0.123456}} seconds.
was:
When {{spark.sql.legacy.timeParserPolicy=LEGACY}} is explicitly configured,
Spark’s built-in XML reader misinterprets fractional seconds in otherwise valid
ISO-8601 timestamps.
For example:
{{Input: 2024-02-29T12:34:56.123456Z
Expected: 2024-02-29T12:34:56.123456Z
Actual: 2024-02-29T12:36:59.456Z}}
This affects both {{PERMISSIVE}} and {{{}FAILFAST{}}}: parsing succeeds, so
{{FAILFAST}} does not protect against the incorrect value.
This is a migration concern for pipelines moving from Spark 3 with
{{spark-xml}} to Spark 4’s built-in XML reader while retaining {{LEGACY}} for
compatibility with other timestamp operations. The old connector parsed these
ISO timestamps independently of Spark’s legacy parser and preserved the
fractional seconds.
h3. Reproduction
Run in {{{}spark-shell{}}}; no external connector or input files are required:
{{import spark.implicits._
println(spark.version)
spark.conf.set("spark.sql.session.timeZone", "UTC")
spark.conf.set("spark.sql.legacy.timeParserPolicy", "LEGACY")
val xml = Seq(
"<ROW><stamp>2024-02-29T12:34:56.123456Z</stamp></ROW>"
).toDS()
Seq("PERMISSIVE", "FAILFAST").foreach \{ mode =>
println(s"mode=$mode")
spark.read
.schema("stamp TIMESTAMP")
.option("rowTag", "ROW")
.option("mode", mode)
.xml(xml)
.selectExpr(
"CAST(stamp AS STRING) AS parsed",
"unix_micros(stamp) AS epoch_micros"
)
.show(false)
}}}
Both modes produce:
{{parsed epoch_micros
2024-02-29 12:36:59.456 1709210219456000}}
Expected:
{{parsed epoch_micros
2024-02-29 12:34:56.123456 1709210096123456}}
Changing the policy to {{CORRECTED}} produces the expected value for this
example. However, globally changing that policy also changes non-XML parsing
behavior.
h3. Apparent cause
Under {{{}LEGACY{}}},
[XmlOptions|https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlOptions.scala#L161-L167]
supplies the default pattern:
{{yyyy-MM-dd'T'HH:mm:ss.SSSXXX}}
[StaxXmlParser|https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala#L60-L65]
selects {{{}FAST_DATE_FORMAT{}}}.
[LegacyFastTimestampFormatter|https://github.com/apache/spark/blob/v4.2.0/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala#L414-L471]
accepts the six-digit fraction but scales it using the pattern’s three {{S}}
characters:
{{123456 × 1,000,000 / 10³ = 123,456,000 microseconds
= 123.456 seconds}}
That value is added to the whole-second timestamp instead of being interpreted
as {{0.123456}} seconds.
> XML reader silently changes microsecond timestamps under LEGACY
> timeParserPolicy, even in FAILFAST mode
> -------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60199
> URL: https://issues.apache.org/jira/browse/SPARK-60199
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Matthew Bayer
> Priority: Major
> Labels: correctness
>
> When {{spark.sql.legacy.timeParserPolicy=LEGACY}} is explicitly configured,
> Spark’s built-in XML reader misinterprets fractional seconds in otherwise
> valid ISO-8601 timestamps.
> For example:
>
> Input: 2024-02-29T12:34:56.123456Z
> Expected: 2024-02-29T12:34:56.123456Z
> Actual: 2024-02-29T12:36:59.456Z
> This affects both {{PERMISSIVE}} and {{{}FAILFAST{}}}: parsing succeeds, so
> {{FAILFAST}} does not protect against the incorrect value.
> This is a migration concern for pipelines moving from Spark 3 with
> {{spark-xml}} to Spark 4’s built-in XML reader while retaining {{LEGACY}} for
> compatibility with other timestamp operations. The old connector parsed these
> ISO timestamps independently of Spark’s legacy parser and preserved the
> fractional seconds.
>
> h3. Reproduction
> Run in {{{}spark-shell{}}}; no external connector or input files are required:
>
> ```python
> import spark.implicits._
> println(spark.version)
> spark.conf.set("spark.sql.session.timeZone", "UTC")
> spark.conf.set("spark.sql.legacy.timeParserPolicy", "LEGACY")
> val xml = Seq(
> "<ROW><stamp>2024-02-29T12:34:56.123456Z</stamp></ROW>"
> ).toDS()
> Seq("PERMISSIVE", "FAILFAST").foreach { mode =>
> println(s"mode=$mode")
> spark.read
> .schema("stamp TIMESTAMP")
> .option("rowTag", "ROW")
> .option("mode", mode)
> .xml(xml)
> .selectExpr(
> "CAST(stamp AS STRING) AS parsed",
> "unix_micros(stamp) AS epoch_micros"
> )
> .show(false)
> ```
> Both modes produce:
>
> {{parsed epoch_micros
> 2024-02-29 12:36:59.456 1709210219456000}}
> Expected:
>
> {{parsed epoch_micros
> 2024-02-29 12:34:56.123456 1709210096123456}}
> Changing the policy to {{CORRECTED}} produces the expected value for this
> example. However, globally changing that policy also changes non-XML parsing
> behavior.
>
> h3. Apparent cause
> Under {{{}LEGACY{}}},
> [XmlOptions|https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlOptions.scala#L161-L167]
> supplies the default pattern:
>
> {{yyyy-MM-dd'T'HH:mm:ss.SSSXXX}}
> [StaxXmlParser|https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala#L60-L65]
> selects {{{}FAST_DATE_FORMAT{}}}.
> [LegacyFastTimestampFormatter|https://github.com/apache/spark/blob/v4.2.0/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala#L414-L471]
> accepts the six-digit fraction but scales it using the pattern’s three {{S}}
> characters:
>
> {{123456 × 1,000,000 / 10³ = 123,456,000 microseconds
> = 123.456 seconds}}
> That value is added to the whole-second timestamp instead of being
> interpreted as {{0.123456}} seconds.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
