andygrove opened a new issue, #7202:
URL: https://github.com/apache/datafusion-comet/issues/7202
### Describe the bug
Since #6059, the native Parquet scan resolves ABFS authentication through
Hadoop on the driver. When the driver declines an auth type it does not model,
the scan still plans natively, and every task then fails with the native
"unsupported" auth error. The scan does not fall back to Spark's own reader.
Hadoop declines these, so the task fails:
- `fs.azure.account.auth.type=Custom`
- `UserboundSASWithOAuth`
- a custom `fs.azure.sas.token.provider.type`
- OAuth with `RefreshTokenBasedTokenProvider` or `UserPasswordTokenProvider`
- the hadoop-azure 3.5.0 Workload Identity client-assertion provider
1.1.0 ran these scans natively. It ignored `fs.azure.account.auth.type` and
the provider class, and used the environment instead: AKS Workload Identity
`AZURE_*` variables, or the IMDS managed identity. So 1.1.0 worked whenever the
ambient identity could read the container, and Spark works through
hadoop-azure. With 1.2.0, those jobs fail.
The flow is:
1. `AbfsAuthResolver.resolve` returns `Declined`.
2. `NativeConfig.extractScanObjectStoreOptions` still returns `Right` for a
single-container scan.
3. `CometNativeScan` stays native.
4. `builder_for` in `native/core/src/parquet/objectstore/azure.rs` raises
`unsupported` for the `declined` marker. It does the same in
`resolved_settings` for RefreshToken and UserPassword.
There is a second case. When the driver cannot reflect into hadoop-azure,
every task fails as well, through the `Failed` outcome and the `error` marker.
That happens with a hadoop-azure whose `AbfsConfiguration` constructor or
`getTokenProviderClass` signature differs from 3.3.4 to 3.5.0 (for example a
vendor fork), or with a conf class loader that cannot see hadoop-azure. The
handles are resolved before the `Unconfigured` check, so it also hits users who
configure Azure only through environment variables and set no `fs.azure.*`
keys. On 1.1.0 those users read natively from the environment.
`datasources.md` documents the failure ("the scan fails ... keep the path on
Spark's own scan"). Not authenticating as an identity Spark would not use is
the right goal. But a decline that is known at planning time could fall back to
Spark instead of failing the job.
### Steps to reproduce
Found by reading the code; not run end to end.
1. Run on AKS with Workload Identity environment variables that can read
container `c`.
2. Set `spark.hadoop.fs.azure.account.auth.type=Custom` and
`spark.hadoop.fs.azure.account.oauth.provider.type=<a custom
AccessTokenProvider>`.
3. Run
`spark.read.parquet("abfss://[email protected]/t").count()`.
### Expected behavior
The scan falls back to Spark whenever `AbfsAuthResolver` declines or fails
on the driver, as it already does for multi-account scans and for ABFS mixed
with other schemes. Spark returns the count through the custom provider, as it
did on 1.1.0, where Comet returned it natively through the ambient identity.
### Additional context
This is a regression since 1.1.0, found in the branch-1.2 audit. #6605
covers other differences between Hadoop and native ABFS handling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
