Hi folks, Thanx for the discussion so far, I concur with Sai: combining Option 1 (renew date column so cleanup can decide renew vs remove without per-token getToken calls) with Option 2 (batched reads) looks like the best path forward for HIVE-27260.
If doing both in a single PR feels too heavy, we can land it in stages—for example schema + bulk fetch with renew date first, then batching/streaming and any cross-instance rate limiting as follow-ups. -Ayush On Wed, 21 Oct 2026 at 23:42, Sai Hemanth Gantasala <[email protected]> wrote: > > Hi Zsolt, > > Thanks for bringing this up! The high volume of database calls during token > cleanup across multiple HMS instances is a significant pain point for large > clusters. > > Here is my feedback on the proposed options: > > - Option 1 & 2: Combining a renewal date column (Option 1) with batching > (Option 2) is the most robust approach, efficiently handling this on the DB > side without pounding the metastore with 100k individual calls. > - Option 3: Fetching both token IDs and full tokens in a single bulk query > raises security concerns about loading sensitive token material into memory > at once and doubles transferred data. > - Option 4: Passing large collections of IDs in WHERE token_ident IN (...) > clauses will quickly hit SQL statement length limits or degrade query planner > performance given the varchar(767) size. > > A few additional ideas to consider alongside Options 1 and 2: > > - Cursor-based pagination / Streaming: Avoids loading massive result sets > into memory and keeps HMS memory usage flat. > - Asynchronous / Rate-limited worker: Running cleanup via a background > worker thread with rate limiting and random jitter across HMS instances would > prevent thundering herd problems from concurrent metastores. > > My opinon: Combining Options 1 and 2 with batching seems like a solid > direction forward. > > Thanks, > Sai > > > On Mon, Oct 5, 2026 at 6:16 AM Zsolt Miskolczi <[email protected]> > wrote: >> >> Hi community, >> >> I'm planning to pick up HIVE-27260 and please let me bring up some possible >> ways to solve the problem. >> >> So, the issue that we faced at multiple customers is when the expired token >> cleanup and renewal process >> (https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/security/TokenStoreDelegationTokenSecretManager.java#L263) >> runs getToken for each token id previously collected in >> tokenStore.getAllDelegationTokenIdentifiers(). >> >> In a really bad case scenario, assuming the customer has ~200k tokens and >> only half of them pass to that point, it is ~100k calls to the metastore >> database. And it is worse if they have multiple HMS instances and the clean >> up fires in an overlapping time window. >> >> I see multiple possible ways to handle the issue, let me bring some of them >> up. As I'm mostly familiar with the DbTokenStore I'm especially curious if >> the possible solutions can potentially bring up doubts at >> ZooKeeperTokenStore. >> >> So, the possible solutions came into my mind: >> >> 1.: Modify the data structure: Add renew date as a new column. So that, >> getAllDelegationTokenIdentifier can be modified to contain the renew date >> and we can make a decision about removing or renewing with a single database >> call. Please note, with this, there will still be large reads (a full table >> scan, no filtering or batching). But at least there will be only one call, >> not 100k. >> >> 2.: Combine possible solution 1 with batching. Multiple small calls, like 5k >> items in a batch. Honestly, I would do it in case solution 1 still has >> performance problems due to the large amount of data transferred in a single >> call. >> >> 3.: Instead of using getAllDelegationTokenIdentifiers() introduce a new >> method that queries both the ids and the tokens together (basically the >> whole table). The drawback is the traffic will be doubled (both of the >> columns are varchar(767)). And also, I'm not completely sure if it can lead >> to security issues (querying both the token ids and the tokens theirself >> together). >> >> 4.: Instead of using a single getToken() call for each record, do it in >> batches, like 1000 tokens in a batch. The problem with this solution is the >> size of the token id (varchar(767)). Using this solution can lead to really >> huge select texts (where token_ident in (....)). >> >> I personally think solution 1 can have the best performance but as it also >> means schema change, it is the solution that will be hard to modify later. >> Solution 4 can look like an easy solution but due to the large size of the >> id, I believe we should choose a really small batch size and after all, >> there would be no significant performance gain. >> >> Now I want to start an open discussion about the possible solutions. In case >> there will be no second thoughts against that, I'm willing to start >> developing solution 1. >> >> >> So, my question is: what is your opinion? >> >> Thank you, >> Zsolt
