Chao Sun created SPARK-57990:
--------------------------------
Summary: Extend shared get_json_object parsing to array-index
paths and object-or-array fallbacks
Key: SPARK-57990
URL: https://issues.apache.org/jira/browse/SPARK-57990
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.3.0
Reporter: Chao Sun
Assignee: Chao Sun
Fix For: 4.3.0, 5.0.0
SPARK-47670 introduced opt-in shared parsing for repeated {{get_json_object}}
calls over the same JSON input. SPARK-57626 extended it from top-level fields
to literal nested named object paths. Array subscripts are still intentionally
excluded, and the optimizer does not traverse conditional expressions, so a
common object-or-array fallback pattern still reparses the same input for each
projected field.
{code:sql}
SELECT
coalesce(
get_json_object(json, '$.model'),
get_json_object(json, '$[0].model')) AS model,
coalesce(
get_json_object(json, '$.request_id'),
get_json_object(json, '$[0].request_id')) AS request_id
FROM events
{code}
For an array-shaped row, each {{coalesce}} evaluates both extractions, and the
parsing cost continues to scale with the number of projected fields.
With {{spark.sql.optimizer.getJsonObjectSharedParsing.enabled=true}},
nonnegative literal array-index paths such as {{$[0].model}},
{{$.items[0].id}}, and {{$.matrix[0][1]}} should be eligible for the existing
shared parser. Safe object-or-array {{coalesce}} fallbacks over the same JSON
attribute should also share compatible object-root and array-root paths without
making later legacy fallback branches eager.
The follow-up should preserve the existing {{get_json_object}} behavior for
duplicate keys and parents, nulls, missing indexes, malformed or trailing JSON,
non-container intermediate values, raw strings, rendering failures, and
ancestor/descendant prefix conflicts. In particular, a terminal array-index
JSON null must retain the existing string {{"null"}} result, while an
object-key JSON null remains SQL null.
Dynamic paths, wildcards, negative, invalid, or overflowing indexes,
excessively deep paths, guarded conditionals, and unsafe {{coalesce}} shapes
should continue using independent legacy evaluation. This should reuse the
existing internal, default-disabled configuration and should not add a new
user-facing expression or configuration.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
