David Mollitor created SPARK-59617:
--------------------------------------
Summary: Reuse a per-evaluator output buffer in
JsonExpressionEvalUtils (json_tuple, JSON_TABLE)
Key: SPARK-59617
URL: https://issues.apache.org/jira/browse/SPARK-59617
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.0
Reporter: David Mollitor
h3. Problem
{{JsonExpressionEvalUtils}} has several JSON-extraction evaluators. Two already
reuse a single instance buffer across rows: {{GetJsonObjectEvaluator}} and
{{MultiGetJsonObjectEvaluator}} each hold
{{@transient private lazy val outputBuffer = new ByteArrayOutputStream()}} and
call
{{outputBuffer.reset()}} before each use. The other two allocate a fresh
{{ByteArrayOutputStream}} on every field/element:
* {{JsonTupleEvaluator.parseRow}} -- a new buffer per extracted field
(unbounded per row for a wide {{json_tuple(...)}}).
* {{JsonTableEvaluator.serializeCurrentValue}} -- a new buffer per call; the
hot caller expands a JSON array one element at a time.
A no-arg {{ByteArrayOutputStream}} starts at the JDK default of 32 bytes and
grows by doubling, so allocating (and then GC-ing) a fresh one per
field/element is steady allocation churn on the json_tuple / JSON_TABLE
evaluation path.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
