[
https://issues.apache.org/jira/browse/SPARK-59430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59430:
-----------------------------------
Labels: pull-request-available (was: )
> Cache UTF8String.hashCode
> -------------------------
>
> Key: SPARK-59430
> URL: https://issues.apache.org/jira/browse/SPARK-59430
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 4.1.0
> Reporter: David Mollitor
> Priority: Minor
> Labels: pull-request-available
>
> h2. Summary
> {{UTF8String.hashCode()}} recomputes a Murmur3 hash over *all* of the
> string's bytes on every call:
> {code:java}
> public int hashCode() {
> return Murmur3_x86_32.hashUnsafeBytes(base, offset, numBytes, 42);
> }
> {code}
> For a {{UTF8String}} that is hashed more than once -- e.g. a key in a Scala
> {{Map}}/{{Set}}, or the repeated {{contains}}/{{apply}}/{{update}} probes
> against an aggregation buffer -- that repeats an O(length) scan each time.
> {{java.lang.String}} avoids this by caching its hash in a field; this does
> the same for {{UTF8String}}.
> h2. Change
> Cache the hash in a non-volatile {{int}}, using the modern
> {{java.lang.String}} technique so a
> *legitimately* zero hash is remembered rather than recomputed on every call
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
