Re: FYI: MetaStore running out of threads

2022-08-31 Thread Rajesh Balamohan
>> In hive, FileUtils.checkFileAccessWithImpersonation can be fixed to use create UGI once to reduce the impact (suspecting this will have 50% impact). Looked closely at the method impl for "FileUtils.checkFileAccessWithImpersonation". It doesn't make 2 connections; 50% impact may not be relevant

Re: FYI: MetaStore running out of threads

2022-08-31 Thread Rajesh Balamohan
W.r.t to connection reuse issues, LLAP had a similar issue (not in HMS) https://issues.apache.org/jira/browse/HIVE-16020. It was making a connection in every task and UGI had to be persisted in the QueryInfo level to reduce the impact. In hive, FileUtils.checkFileAccessWithImpersonation can be fix

FYI: MetaStore running out of threads

2022-08-31 Thread Owen O'Malley
We're using HMS with Storage-Based Authorization and have been having trouble with the HMS running out of threads. Looking at the jstack & code, it appears to that the problem is that RPC's ConnectionId is using UGI's equal/hash, which uses the Subject's Object equals/hash. Proxy user UGI's always