Jiwon Park created SPARK-58824:
----------------------------------
Summary: Decide how Spark Connect error condition names may evolve
without breaking released clients
Key: SPARK-58824
URL: https://issues.apache.org/jira/browse/SPARK-58824
Project: Spark
Issue Type: Improvement
Components: Spark Core, Connect
Affects Versions: 5.0.0
Reporter: Jiwon Park
Follow-up to SPARK-58619.
SPARK-58619 set out to move {{INVALID_HANDLE.SESSION_CHANGED}},
{{SESSION_CLOSED}} and {{SESSION_NOT_FOUND}} into a condition of their own in
error class {{08}}. The rename was rejected during review: released clients
match the condition names to drive session invalidation and reattach
re-execution, and a deployed client cannot be repaired from the server side
(analysis: https://github.com/apache/spark/pull/57831#issuecomment-5277458439).
A sub-condition level {{sqlState}} override landed instead, documented in the
error README as a one-off exception and pinned by an allowlist invariant in
{{SparkThrowableSuite}}.
The result is that a condition name, once released, cannot be corrected, while
nothing states which names carry that status. A major release is not an escape
hatch, since Spark Connect supports independent client and server upgrades.
Options raised in the review thread, proposed in
https://github.com/apache/spark/pull/57831#issuecomment-5282766346 and assessed
in https://github.com/apache/spark/pull/57831#issuecomment-5284768317:
* Server-side translation layer: rewrite the {{errorClass}} metadata and the
{{[CONDITION]}} message prefix per client version, from a rename registry.
Feasible without a protocol change, since clients already send
{{spark/<version>}} in {{client_type}} and error serialization funnels through
{{ErrorUtils}}. Open questions: what to do for clients whose version is
unknown, given that third-party clients send no version token and defaulting to
the old name means new names never reach them; promoting {{client_type}}, which
{{base.proto}} documents as logging-only and never interpreted by the server;
and the registry's scope and lifetime. The layer would also have to ship no
later than the first rename that relies on it.
* Explicit negotiation: a dedicated protocol field rather than parsing
{{client_type}}. A cleaner contract at the cost of a protocol change, and
clients predating the field would still need the fallback above.
* Keep the names frozen and turn the README wording into a standing policy.
This ticket is for choosing among them and recording the outcome. Implementing
a layer, renaming the conditions, and removing the SPARK-58619 exception would
each be separate follow-ups.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
