Haoyan Geng created SPARK-59699:
-----------------------------------
Summary: Spark Connect DataFrameReader options should be
case-insensitive with last-call-wins semantics
Key: SPARK-59699
URL: https://issues.apache.org/jira/browse/SPARK-59699
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 4.3.0
Reporter: Haoyan Geng
Spark Connect's Python DataFrameReader stores reader options in a regular
case-sensitive dictionary. As a result, option names that differ only by case
are serialized as separate entries in the protobuf map.
The DataFrameReader contract states that option keys are case-insensitive and
that a later option with the same key should override the earlier value.
{code:python}
reader = (
spark.read
.option("versionAsOf", 0)
.option("versionasof", 1)
.option("versionAsOf", 2)
)
{code}
The expected serialized options contain one logical entry:
{code:python}
{"versionAsOf": "2"}
{code}
Currently, multiple case variants can reach the protobuf map. The server later
constructs a case-insensitive map from that unordered map, so the selected
value can depend on map iteration order rather than user call order. For
options such as versionAsOf, this can select an unintended table snapshot.
Classic DataFrameReader already provides deterministic, case-insensitive,
last-call-wins behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
