[
https://issues.apache.org/jira/browse/HIVE-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13411908#comment-13411908
]
Daryn Sharp commented on HIVE-3098:
-----------------------------------
bq. This is a metastore. So, in this context we only perform metadata
operations, like create/rm dir/file. We never open/read/write files in this
use-case. Similarly, Daryn's point about cyclic dependency between dfsclient
and renewal thread is moot here, since there are no write calls that we do.
I don't think you can create a file in dfs w/o creating a stream which starts
the lease renewer. I believe creation of an empty file involves an open/close.
bq. the FSCache is not fully beneficial though its still very useful for client
applications. Only server applications have this problem
The cache is actually useful for both clients and servers if used as designed.
In fact, {{closeAllForUGI}} is specifically for servers. Servers like the JT
use it to avoid leaking filesystems like a sieve. The general idea is to
service a request something like this:
{code}
UserGroupInformation remoteUser = <get from somewhere or create it>;
remoteUser.doAs(...);
{code}
followed by, or included at the end of the {{doAs}}:
{code}FileSystem.closeAllForUGI(remoteUser);{code}
This pattern allows arbitrary code to freely obtain cached fs instances w/o
requiring an explicit close. When the request is finished, you know it's safe
to close all the request's cached fs instances. There's no need to track
activity, etc.
I'm not going to tell anyone what to do. I just think trying to roll a custom
ugi/fs shared cache will add a lot of complexity and subtle edge-cases.
> Memory leak from large number of FileSystem instances in FileSystem.CACHE.
> (Must cache UGIs.)
> ---------------------------------------------------------------------------------------------
>
> Key: HIVE-3098
> URL: https://issues.apache.org/jira/browse/HIVE-3098
> Project: Hive
> Issue Type: Bug
> Components: Shims
> Affects Versions: 0.9.0
> Environment: Running with Hadoop 20.205.0.3+ / 1.0.x with security
> turned on.
> Reporter: Mithun Radhakrishnan
> Assignee: Mithun Radhakrishnan
> Attachments: HIVE-3098.patch
>
>
> The problem manifested from stress-testing HCatalog 0.4.1 (as part of testing
> the Oracle backend).
> The HCatalog server ran out of memory (-Xmx2048m) when pounded by 60-threads,
> in under 24 hours. The heap-dump indicates that hadoop::FileSystem.CACHE had
> 1000000 instances of FileSystem, whose combined retained-mem consumed the
> entire heap.
> It boiled down to hadoop::UserGroupInformation::equals() being implemented
> such that the "Subject" member is compared for equality ("=="), and not
> equivalence (".equals()"). This causes equivalent UGI instances to compare as
> unequal, and causes a new FileSystem instance to be created and cached.
> The UGI.equals() is so implemented, incidentally, as a fix for yet another
> problem (HADOOP-6670); so it is unlikely that that implementation can be
> modified.
> The solution for this is to check for UGI equivalence in HCatalog (i.e. in
> the Hive metastore), using an cache for UGI instances in the shims.
> I have a patch to fix this. I'll upload it shortly. I just ran an overnight
> test to confirm that the memory-leak has been arrested.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
