[
https://issues.apache.org/jira/browse/SPARK-58438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18105828#comment-18105828
]
Vicky Lu commented on SPARK-58438:
----------------------------------
Hi, would like to work on this!
> HadoopDelegationTokenManager leaks FileSystem cache entries on every token
> renewal
> ----------------------------------------------------------------------------------
>
> Key: SPARK-58438
> URL: https://issues.apache.org/jira/browse/SPARK-58438
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 4.2.0
> Reporter: Parth Chandra
> Priority: Major
>
> {{{}HadoopDelegationTokenManager{}}}'s periodic token-renewal path creates a
> fresh
> {{UserGroupInformation}} (UGI) on every renewal cycle but never releases the
> {{FileSystem}} instances cached for that UGI, so entries accumulate in the
> static
> {{FileSystem.CACHE}} for the lifetime of the driver JVM.
> h4. Details
> On each renewal cycle {{updateTokensTask()}} calls {{{}doLogin(){}}}, which
> returns a
> *new* UGI in the two credential modes:
> * keytab: {{UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal,
> keytab)}}
> * ccache: {{UserGroupInformation.getUGIFromTicketCache(ccache, user)}}
> Providers invoked inside {{freshUGI.doAs
> { ... }
> }} call {{{}FileSystem.get(...){}}}, which
> caches {{FileSystem}} instances in {{FileSystem.CACHE}} keyed (in part) by
> the UGI.
> Cache entries are only evicted by {{FileSystem.close}} /
> {{{}FileSystem.closeAllForUGI{}}}.
> The renewal path never calls either, so each cycle leaves behind
> {{FileSystem}}
> instances for a throwaway UGI that is never used again.
> The one-shot path {{obtainDelegationTokens(creds)}} already cleans up
> correctly:
> {code:scala}
> if (!currentUser.equals(freshUGI)) {
> FileSystem.closeAllForUGI(freshUGI)
> }
> {code}
> but the equivalent cleanup is missing from {{obtainTokensAndScheduleRenewal}}
> /
> {{{}updateTokensTask{}}}.
> h4. Impact
> Long-running applications that renew delegation tokens (keytab, or
> ccache-based
> renewal) accumulate one UGI's worth of cached {{FileSystem}} handles per
> renewal
> interval. With default settings the renewal interval is on the order of
> hours, so the
> growth is slow, but it is unbounded and can lead to gradually increasing
> memory and
> open file/connection usage (and eventually resource exhaustion) over the
> lifetime of a
> driver that runs for days or weeks.
> The proxy-user branch of {{doLogin()}} returns the shared
> {{getCurrentUser()}} and must
> NOT be closed; the guard {{!currentUser.equals(freshUGI)}} correctly excludes
> it.
> h4. Suggested fix
> Mirror the one-shot path: after the renewal {{doAs}} completes, close the
> fresh UGI's
> cached filesystems, guarded by {{!currentUser.equals(freshUGI)}} so the shared
> current/proxy user is never closed.
> Notes
> Discovered during review of SPARK-38954 (PR #57285); it is a pre-existing
> issue in the
> Kerberos renewal path and is being tracked separately per reviewer request so
> that
> SPARK-38954 keeps the Kerberos paths unchanged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
