[
https://issues.apache.org/jira/browse/SPARK-58677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nathan Holland updated SPARK-58677:
-----------------------------------
Description:
This ticket expands on SPARK-45900, to explicitly mention the desire for a
BINARY return type for xxhash3_128 as well as additional functionality that a
modern hashing algorithm on Spark could implement. (such as deterministic
support for complex types)
The original intent of that ticket was to capture a new hashing model through
XXH3 that allowed for high performance hashing using XXH3 with the API of the
earlier XXHash64 hash family, rather than some of the cryptographic hash
functions (SHA/MD5 etc)
One difference in this model I am proposing - SPARK-45900 has been resolved by
returning a string, which is more consistent with the older hashing models.
When writing SPARK-45900 I mentioned that this was a downside of the existing
SHA models, to clarifify - returning a STRING type has heavier overhead and
prevents future optimisations that occur by allowing HotSpot (and C
interpreters like Photon) to take advantage of SIMD instructions. Furthermore
I've been experimenting with expanding the BINARY type to have fixed byte types
and have seen some promising performance increases that may be useful at a
later date.
As XXH3 was built specifically around SIMD instructions to provide speed ups
over XXHash64 and Spark 4.4 is not out yet, I believe now would be the time to
discuss what this API should return - and either choose with a seperate API
that returns the BINARY 128 bytes or make the change to the solution of
SPARK-45900 to return a BINARY type by default and let users use the "hex"
function if they want the result as a Binary String.
I have heard from customers that one of xxhash64's benefits is the simple API
to support multiple columns which conceptually allow the hashing of columns by
passing them in natively without having to implement brittle solutions (often
something like concat_ws) to hash data. I believe an expansion to the
functionality to SPARK-45900 to support columns would allow XXH3 to be seen
successor to the XXHash64 function with this user friendly API - while keeping
all the semantics that a unitary function would support - including byte
compatability with the original C reference code.
My current codebase can be found here for a proposed implementation of the
above two key changes (better support for hashing data types and multiple
columns, and the return of a binary type which better captures the hash results
rather than a string)
*
[https://github.com/NathanNZ/spark/commit/d4317b9853a9b1f775c1ab11534dc8ea5a048040]
* I've also got a work in progress solution with pre-built containers to test
the functionality.
** [https://github.com/NathanNZ/spark-nathan-containers]
Links:
- https://issues.apache.org/jira/browse/SPARK-45900 (Original SPARK-45900
ticket)
was:
This ticket expands on completing the areas where I did not go into enough
detail about the return types of SPARK-45900 as well as the functionality that
a modern hashing algorithm on Spark could implement.
The original intent was to capture a new hashing model through XXH3 that
allowed for high performance hashing using XXH3 with the API of the earlier
XXHash64 model, while developing SPARK-45900 I found some limitations to the
XXHash64 model which I have fixed up, but have not landed in SPARK-45900.
One difference in this model I am proposing is that SPARK-45900 was resolved by
returning a string, this is more consistent with the older hashing models but
has heavier overhead and prevents future optimisations that could occur by
introducing a FixedBinaryType (not in scope) that allows HotSpot (C
interpreters like Photon) to take advantage of SIMD instructions.
As XXH3 was built specifically around these speed ups over XXHash64 and Spark
4.4 is not out yet, I believe now would be the time to discuss what this API
should return - and either choose with a seperate API that returns the BINARY
128 bytes or make the change to the solution of SPARK-45900 to return a BINARY
type by default and let users use the "hex" function if they want the result as
a Binary String.
In addition one of the benefits of xxhash64 I have heard from customers over
the other API's is it's simple API to support multiple columns to conceptually
allow the hashing of columns without having to implement brittle solutions
(often something like concat_ws) to do so. I believe an expansion to the
functionality of the proposed Spark 4.4 solution of XXH3 should be seen as a
successor to the XXHash64 function and this user friendly API.
My current codebase can be found here for a proposed implementation of the
above two key changes (better support for hashing data types and multiple
columns, and the return of a binary type which better captures the hash results
rather than a string)
*
[https://github.com/NathanNZ/spark/commit/d4317b9853a9b1f775c1ab11534dc8ea5a048040]
* I've also got a work in progress repository to demonstrate the improvements
of such a solution over the older hashing methods discussed in this repository,
with pre-baked containers to test the functionality.
** https://github.com/NathanNZ/spark-nathan-containers
Links:
- https://issues.apache.org/jira/browse/SPARK-45900 (Original SPARK-45900
ticket)
> Expand hash functionalities of XXH3
> -----------------------------------
>
> Key: SPARK-58677
> URL: https://issues.apache.org/jira/browse/SPARK-58677
> Project: Spark
> Issue Type: Improvement
> Components: PySpark, SQL
> Affects Versions: 4.4.0
> Reporter: Nathan Holland
> Priority: Major
>
> This ticket expands on SPARK-45900, to explicitly mention the desire for a
> BINARY return type for xxhash3_128 as well as additional functionality that a
> modern hashing algorithm on Spark could implement. (such as deterministic
> support for complex types)
> The original intent of that ticket was to capture a new hashing model through
> XXH3 that allowed for high performance hashing using XXH3 with the API of the
> earlier XXHash64 hash family, rather than some of the cryptographic hash
> functions (SHA/MD5 etc)
> One difference in this model I am proposing - SPARK-45900 has been resolved
> by returning a string, which is more consistent with the older hashing
> models. When writing SPARK-45900 I mentioned that this was a downside of the
> existing SHA models, to clarifify - returning a STRING type has heavier
> overhead and prevents future optimisations that occur by allowing HotSpot
> (and C interpreters like Photon) to take advantage of SIMD instructions.
> Furthermore I've been experimenting with expanding the BINARY type to have
> fixed byte types and have seen some promising performance increases that may
> be useful at a later date.
> As XXH3 was built specifically around SIMD instructions to provide speed ups
> over XXHash64 and Spark 4.4 is not out yet, I believe now would be the time
> to discuss what this API should return - and either choose with a seperate
> API that returns the BINARY 128 bytes or make the change to the solution of
> SPARK-45900 to return a BINARY type by default and let users use the "hex"
> function if they want the result as a Binary String.
> I have heard from customers that one of xxhash64's benefits is the simple API
> to support multiple columns which conceptually allow the hashing of columns
> by passing them in natively without having to implement brittle solutions
> (often something like concat_ws) to hash data. I believe an expansion to the
> functionality to SPARK-45900 to support columns would allow XXH3 to be seen
> successor to the XXHash64 function with this user friendly API - while
> keeping all the semantics that a unitary function would support - including
> byte compatability with the original C reference code.
> My current codebase can be found here for a proposed implementation of the
> above two key changes (better support for hashing data types and multiple
> columns, and the return of a binary type which better captures the hash
> results rather than a string)
> *
> [https://github.com/NathanNZ/spark/commit/d4317b9853a9b1f775c1ab11534dc8ea5a048040]
> * I've also got a work in progress solution with pre-built containers to
> test the functionality.
> ** [https://github.com/NathanNZ/spark-nathan-containers]
> Links:
> - https://issues.apache.org/jira/browse/SPARK-45900 (Original SPARK-45900
> ticket)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
