[
https://issues.apache.org/jira/browse/HIVE-26632?focusedWorklogId=817200&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-817200
]
ASF GitHub Bot logged work on HIVE-26632:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 14/Oct/22 22:18
Start Date: 14/Oct/22 22:18
Worklog Time Spent: 10m
Work Description: cnauroth opened a new pull request, #3673:
URL: https://github.com/apache/hive/pull/3673
…vent erroneous database updates.
### What changes were proposed in this pull request?
Update DelegationTokenSecretManager current key ID to prevent erroneous
database updates.
### Why are the changes needed?
When multiple HiveMetaStore processes share a database for `MASTER_KEYS`,
this bug can cause incorrect update attempts and increased database load.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Run multiple HiveMetaStore processes sharing the same database. To make it
easier to expose the problem, downtune configuration properties related to
master key rollover to artificially low values:
* `hive.cluster.delegation.token.gc-interval`
* `hive.cluster.delegation.key.update-interval`
* `hive.cluster.delegation.token.max-lifetime`
WIthout the patch, stack traces like this are visible in the logs:
```
2022-08-03T00:09:48,744 ERROR [Thread[Thread-9,5,main]]
thrift.TokenStoreDelegationTokenSecretManager: ExpiredTokenRemover thread
received unexpected exception.
org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException:
NoSuchObjectException(message:No key found with keyId: 1)
org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException:
NoSuchObjectException(message:No key found with keyId: 1)
at
org.apache.hadoop.hive.thrift.DBTokenStore.invokeOnTokenStore(DBTokenStore.java:170)
~[hive-exec-2.3.7.jar:2.3.7]
at
org.apache.hadoop.hive.thrift.DBTokenStore.updateMasterKey(DBTokenStore.java:51)
~[hive-exec-2.3.7.jar:2.3.7]
at
org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.rollMasterKeyExt(TokenStoreDelegationTokenSecretManager.java:269)
~[hive-exec-2.3.7.jar:2.3.7]
at
org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager$ExpiredTokenRemover.run(TokenStoreDelegationTokenSecretManager.java:301)
[hive-exec-2.3.7.jar:2.3.7]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
Caused by: org.apache.hadoop.hive.metastore.api.NoSuchObjectException: No
key found with keyId: 1
at
org.apache.hadoop.hive.metastore.ObjectStore.updateMasterKey(ObjectStore.java:7727)
~[hive-exec-2.3.7.jar:2.3.7]
```
After applying the patch, the problem goes away.
The exact timing makes it difficult to reproduce accurately in a unit test.
Issue Time Tracking
-------------------
Worklog Id: (was: 817200)
Remaining Estimate: 0h
Time Spent: 10m
> Update DelegationTokenSecretManager current key ID to prevent erroneous
> database updates.
> -----------------------------------------------------------------------------------------
>
> Key: HIVE-26632
> URL: https://issues.apache.org/jira/browse/HIVE-26632
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> While rolling a new master key, {{TokenStoreDelegationTokenSecretManager}}
> does not update a base class member variable that tracks the current key ID.
> This can cause situations later where it attempts to update a key using an
> incorrect ID. This update attempt fails, even though the process had
> successfully generated a new master key. Since it appears to be a failure
> though, the thread immediately attempts to roll a new master key again,
> resulting in excess database load.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
