David Mollitor created SPARK-59430:
--------------------------------------
Summary: Cache UTF8String.hashCode
Key: SPARK-59430
URL: https://issues.apache.org/jira/browse/SPARK-59430
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 4.1.0
Reporter: David Mollitor
h2. Summary
{{UTF8String.hashCode()}} recomputes a Murmur3 hash over *all* of the string's
bytes on every call:
{code:java}
public int hashCode() {
return Murmur3_x86_32.hashUnsafeBytes(base, offset, numBytes, 42);
}
{code}
For a {{UTF8String}} that is hashed more than once -- e.g. a key in a Scala
{{Map}}/{{Set}}, or the repeated {{contains}}/{{apply}}/{{update}} probes
against an aggregation buffer -- that repeats an O(length) scan each time.
{{java.lang.String}} avoids this by caching its hash in a field; this does the
same for {{UTF8String}}.
h2. Change
Cache the hash in a non-volatile {{int}}, using the modern {{java.lang.String}}
technique so a
*legitimately* zero hash is remembered rather than recomputed on every call
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
