[
https://issues.apache.org/jira/browse/HIVE-29762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dayakar M updated HIVE-29762:
-----------------------------
Description:
Synchronization logic added as a part of HIVE-24428 is having hash collision
issue.
To prevent concurrent issues as a part of _add_partitions_ request, added
_com.google.common.util.concurrent.Striped.lock(stripes)_ mechanism which
internally creates lock objects by hashing the given key, here it also uses the
stripes value while calculating the hash. Sometimes for unrelated keys it is
generating the same hash which resulting same lock object.
The intent of this locking mechanism is to prevent concurrent requests for the
add_partitions request for the same table but due to hash collision for
unrelated keys(db.table names) it is using same lock object which is preventing
other unrelated add_partition requests.
In this case if stripe size increased to some bigger value like 1024 then it is
generating different hash which results different lock keys but there could be
still hash collision can happen and increasing stripe size will increase memory
overhead.
So this has to be checked and need a proper fix.
was:
Synchronization logic added as a part of HIVE-24428 is having hash collision
issue.
To prevent concurrent issues as a part of _add_partitions_ request, added
_com.google.common.util.concurrent.Striped.lock(stripes)_ mechanism which
internally creates lock objects by hashing the given key. Here it also uses the
stripes value while calculating the hash. Due to which for some unrelated keys
it is generating the same hash which resulting same key used for locking.
The intent of this locking mechanism is to prevent concurrent requests for the
add_partitions request for the same table but due to hash collision for
unrelated keys(db.table names) it is using same lock object which is preventing
other unrelated add_partition requests.
In this case if stripe size increased to some bigger value like 1024 then it is
generating different hash which results different keys but there could be still
hash collision can happen which need to share same lock for unrelated keys.
So this has to be checked and need a proper fix.
> Synchronization logic added as a part of HIVE-24428 is having hash collision
> issue.
> -----------------------------------------------------------------------------------
>
> Key: HIVE-29762
> URL: https://issues.apache.org/jira/browse/HIVE-29762
> Project: Hive
> Issue Type: Bug
> Reporter: Dayakar M
> Assignee: Dayakar M
> Priority: Major
>
> Synchronization logic added as a part of HIVE-24428 is having hash collision
> issue.
> To prevent concurrent issues as a part of _add_partitions_ request, added
> _com.google.common.util.concurrent.Striped.lock(stripes)_ mechanism which
> internally creates lock objects by hashing the given key, here it also uses
> the stripes value while calculating the hash. Sometimes for unrelated keys it
> is generating the same hash which resulting same lock object.
> The intent of this locking mechanism is to prevent concurrent requests for
> the add_partitions request for the same table but due to hash collision for
> unrelated keys(db.table names) it is using same lock object which is
> preventing other unrelated add_partition requests.
> In this case if stripe size increased to some bigger value like 1024 then it
> is generating different hash which results different lock keys but there
> could be still hash collision can happen and increasing stripe size will
> increase memory overhead.
> So this has to be checked and need a proper fix.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
