[
https://issues.apache.org/jira/browse/HIVE-29818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-29818:
----------------------------------
Labels: pull-request-available (was: )
> [REST Catalog] Prevent FileSystem memory leak when executing operations under
> proxy UGI
> ---------------------------------------------------------------------------------------
>
> Key: HIVE-29818
> URL: https://issues.apache.org/jira/browse/HIVE-29818
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Affects Versions: 4.2.0
> Reporter: Henri Biestro
> Assignee: Henri Biestro
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> *Problem Statement*
> When the REST Catalog service executes operations on behalf of multiple
> end-users using proxy authentication
> (`UserGroupInformation.createProxyUser`), Hadoop's internal
> `FileSystem.CACHE` retains references to both the `FileSystem` instances and
> their originating `UserGroupInformation` (UGI) objects.
> Because `createProxyUser` creates un-managed UGI instances, short-lived or
> high-cardinality proxy requests lead to continuous memory accumulation in
> `FileSystem.CACHE`. Over time, this causes significant JVM heap pressure and
> eventual `OutOfMemoryError` (OOM) failures in long-running REST catalog
> deployments.
> *Current Behavior*
> 1. The REST catalog creates a proxy `UserGroupInformation` context per
> user/request.
> 2. File system operations executed inside `ugi.doAs(...)` register new
> `FileSystem` handles inside Hadoop's static `FileSystem.CACHE`.
> 3. When the request or user session completes, the UGI reference remains held
> by `FileSystem.CACHE`, preventing garbage collection of the UGI, associated
> `Subject`/`Principal` objects, and socket/IPC connections.
> *Expected Behavior*
> The REST Catalog service should cleanly manage the lifecycle of proxy
> `UserGroupInformation` contexts and their underlying file system resources.
> Upon session termination or cache eviction of a proxy UGI, all associated
> file system handles should be explicitly closed using
> `FileSystem.closeAllForUGI(ugi)`.
> *Proposed Solution / Implementation Details*
> * *UGI Cache Management:* Introduce a bounded, time-evicting
> `LoadingCache<String, UserGroupInformation>` (or leverage existing session
> management) for proxy users.
> * *Eviction Listener Cleanup:* Register a removal/eviction listener on the
> UGI cache that explicitly calls `FileSystem.closeAllForUGI(evictedUgi)` upon
> session expiration or cache displacement.
> * *Concurrency Guard:* Ensure eviction timeouts are configured with
> appropriate buffer margins to prevent `FileSystem closed` exceptions during
> active mid-flight `doAs` execution threads.
> * {*}(Alternative / Configurable option){*}: Provide an optional
> configuration flag to disable Hadoop FileSystem caching
> (`fs.hdfs.impl.disable.cache=true`) for environments with highly ephemeral
> proxy user activity.
> *Impact*
> * Eliminates memory leaks in long-running REST catalog server instances
> handling multi-tenant workloads.
> * Ensures proper cleanup of underlying RPC sockets and IPC client resources
> tied to proxy credentials.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
