Thanks Peter! You are correct CoarseGrainedExecutorBackend does wrap SparkHadoopUtil.runAsSparkUser. However, as you pointed out, the end result does not change. You are also correct that HADOOP-19906 is an orthogonal issue and must be addressed.
Also, Steve's suggestion is a real simplification. We can remove the DirectTokenProvider trait and just change HadoopDelegationTokenManager. We sort of lose the explicit signal that the provider does not use Kerberos, though that is now signalled by the provider indirectly, so we do not lose much. Parth On Wed, Jul 8, 2026 at 8:59 AM Peter Toth <[email protected]> wrote: > Thanks all for the great discussion. > > First, a small correction on the JDK 25 / Subject-propagation point. I > went through the code and it doesn't seem quite right that "on executor > task threads there is no active doAs() scope" because > CoarseGrainedExecutorBackend wraps everything in > SparkHadoopUtil.runAsSparkUser, which does createSparkUser().doAs(...). > If I'm not mistaken what happens is that JEP 486 stops that Subject from > propagating to the RPC/task threads, so getCurrentUser() falls back to the > static login user, and the pushed credentials are read back from there, > which is why the outcome Parth described still holds, just via the > login-user fallback rather than the absence of a doAs. (Parth, please > correct me if I'm misreading the code here.) > > IMO, HADOOP-19906 looks orthogonal to this SPIP rather than a requirement > of it. It's the same pre-existing JDK 25 dependency that the current > Kerberos and Kafka delegation-token paths already need. So I'd suggest we > track it separately and not treat it as a blocker here. > > That leaves the main open question: the overlap with the OIDC Credential > Propagation SPIP. > Parth listed three cases that the OIDC approach can't cover and that fit > naturally into the existing delegation-token framework: > 1. Kafka delegation tokens over SCRAM — they target brokers (not URIs) and > need no UserContext, so they can't be expressed as > CredentialProvider.resolve(UserContext, URI). > 2. The existing S3A/ABFS Hadoop delegation-token bindings — these already > work under Kerberos and just need the activation gates removed; the OIDC > manager doesn't run them. > 3. Proprietary, non-JWT IdP tokens — the OIDC path requires an OIDC JWT. > > Cheng Pan (and anyone else who has the redundancy concern), does this > resolve it for you? My read is that the two are complementary rather than > redundant: > - this SPIP unlocks the existing, provider-agnostic DT mechanism for any > non-Kerberos provider; > - while the OIDC SPIP adds per-user/session identity propagation for OIDC. > > I'd like to make sure we agree on that framing, and that you're > comfortable with the direction of extending the existing DT management, > before we go further and discuss the implementation options that Steve > mentioned earlier. > > Best, > Peter > > On Mon, Jun 29, 2026 at 11:58 PM Parth Chandra <[email protected]> wrote: > >> Hi Cheng Pan >> >> >> then we decided to keep them independent >> >> > I see that, but if we decide to accept and implement both SPIPs, then >> we are going to provide two approaches for users that enable cloud >> credentials refresh, this is functionality redundant, and as you know, this >> part usually involve private data and 3rd party services dependencies, when >> user report issues, they are likely limited to share the related part of >> logs and environment information to provide a minimal reproducible cases, >> this makes diagnosis extremely difficult. Offering two distinct cloud >> credential refresh mechanisms undoubtedly increases system complexity. >> > I would lean towards to the OIDC Credential Propagation approach unless >> it does not cover the functionality (user perspective) provided by this >> SPIP. >> >> There are a few cases covered by this not covered by the OIDC credential >> propagation approach. For instance, - >> 1. Kafka delegation tokens over SCRAM — the existing >> KafkaDelegationTokenProvider is currently blocked by the Kerberos gates. It >> targets brokers, not URIs, and doesn't need a UserContext. However, the >> OIDC approach depends on a UserContext (an OIDC JWT with >> principal/issuer/rawToken). It cannot be reimplemented as a >> CredentialProvider.resolve(UserContext, URI). >> 2. Existing S3A/ABFS delegation token bindings (Steve's point in this >> thread) — these are already implemented as HadoopDelegationTokenProvider >> and work today in Kerberos environments. They just need the activation >> gates removed to work without Kerberos. The OIDC SPIP's parallel manager >> does not unblock them. >> 3. There can also be proprietary IdP systems which have non JWT tokens >> (which is what prompted this SPIP in the first place) >> >> >> The DirectProviderPath proposed in this SPIP does not go >> through Subject.doAs() or UserGroupInformation.doAs() and will be >> unaffected. The existing Kerberos path will have to be updated. >> > Sorry, I overlook this reply. I think this is also affected. The JDK >> change breaks the Subject propagation between threads, that means you can >> not get the same Subject (UGI) instance from the task thread as >> the updateTokensTask thread, so you can not access any kind of the >> credential you offered from the task thread. >> >> On executor task threads, there is no active doAs()/callAs() scope. >> getCurrentUser() sees null from Subject.current() (or Subject.getSubject() >> on older JDKs) and falls back to getLoginUser() — which is a static field >> and is not based on Subject propagation. The credentials added via >> addCredentials() on the RPC handler thread are added to this same static >> login user instance so task threads reading from getLoginUser() should see >> them >> However, broadly speaking, we do need HADOOP-19906 >> >> I've updated the SPIP document to include these points >> >> Parth >> >
