[
https://issues.apache.org/jira/browse/SPARK-59543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-59543.
----------------------------------
Fix Version/s: 4.4.0
Resolution: Fixed
Issue resolved by pull request 58832
[https://github.com/apache/spark/pull/58832]
> Avoid a per-record SQLConf.get in the JSON charset-decode path
> --------------------------------------------------------------
>
> Key: SPARK-59543
> URL: https://issues.apache.org/jira/browse/SPARK-59543
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.1.0
> Reporter: David Mollitor
> Assignee: David Mollitor
> Priority: Minor
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> h2. Summary
> When reading JSON with an explicit {{encoding}} option, Spark re-materializes
> SQLConf *once per input record*. {{CreateJacksonParser.getStreamDecoder}}
> builds a {{CharsetDecoder}} per record via
> {{{}CharsetProvider.newDecoder(enc, caller = "Jackson Parser"){}}}, and
> {{{}newDecoder{}}}'s two default arguments each call {{{}SQLConf.get{}}}:
> {code:scala}
> def newDecoder(charset: String,
> legacyCharsets: Boolean = SQLConf.get.legacyJavaCharsets,
> legacyErrorAction: Boolean = SQLConf.get.legacyCodingErrorAction,
> caller: String = "decode"): CharsetDecoder
> {code}
> On an executor task thread {{SQLConf.get}} constructs a fresh
> {{ReadOnlySQLConf}} +
> {{ConfigReader}} + {{HashMap}} each call, so this is pure per-record churn to
> read two flags that are constant for the whole task.
> h2. Details
> The single-line/text read path is affected:
> {{CreateJacksonParser.text/internalRow/bytes(enc, ...)}}
> call {{{}getStreamDecoder{}}}. (The multi-line path uses {{new
> InputStreamReader(is, enc)}} and does not hit {{{}newDecoder{}}}, so it is
> unaffected.)
> The fix resolves the two flags once per reader in {{JSONOptions}} and passes
> them explicitly to {{{}newDecoder{}}}, so no {{SQLConf.get}} runs per record.
> This mirrors what CSV already does: {{CSVOptions}} resolves
> {{legacyJavaCharsets}} at construction via
> {{{}CharsetProvider.forName(_, SQLConf.get.legacyJavaCharsets, ...){}}}.
> * {{{}JSONOptions{}}}: add {{val legacyJavaCharsets}} and {{{}val
> legacyCodingErrorAction{}}}, resolved from
> {{SQLConf.get}} at construction (matching the existing SQLConf-derived fields
> such as {{{}writeNullIfWithDefaultValue{}}}).
> * {{{}CreateJacksonParser{}}}: thread the two flags through
> {{getStreamDecoder}} and the {{{}text{}}}, {{{}internalRow{}}}, and {{bytes}}
> encoding variants into {{{}CharsetProvider.newDecoder{}}}.
> * {{{}TextInputJsonDataSource{}}}: pass the resolved flags from the options
> at the three closure sites.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
