djouallah opened a new issue, #68877: URL: https://github.com/apache/doris/issues/68877
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description Doris 4.1.4.1 reads OneLake (Microsoft Fabric) Iceberg tables through the Iceberg REST catalog. It cannot read a raw file at the same kind of path, `abfss://<workspace>@onelake.dfs.fabric.microsoft.com/<lakehouse>/Files/...`. The reader is already there. For Iceberg scans, [`LocationPath`](https://github.com/apache/doris/blob/4.1.4.1/fe/fe-core/src/main/java/org/apache/doris/common/util/LocationPath.java#L392-L394) sends OneLake `abfs[s]://` paths to `FILE_HDFS`, and the BE reads them with hadoop-azure. This works for us with workload identity, set as raw `fs.azure.account.*` keys on the catalog (`WorkloadIdentityTokenProvider`). TPC-H 22/22 runs on OneLake Iceberg data this way. The file table functions do not reach that reader: - `hdfs()` already uses `FILE_HDFS`, but [`HdfsProperties`](https://github.com/apache/doris/blob/4.1.4.1/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/HdfsProperties.java#L91) accepts only `hdfs`, `viewfs` and `jfs`. Error: `Can not build hdfs(): Unsupported schema: abfss`. On master the list is `hdfs`, `viewfs`. - `s3()` / `file()` with Azure properties use the native Azure client, which takes only an account key. OneLake has no account key. OAuth2 is refused there: [`OAuth2 auth type is only supported for iceberg rest catalog`](https://github.com/apache/doris/blob/4.1.4.1/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/AzureProperties.java#L147-L148). Request: let `hdfs()` (or `file()`) read OneLake `abfss://` paths through the same hadoop-azure route as the Iceberg catalog, with the same raw `fs.azure.*` keys (workload identity, SAS) or the OAuth2 properties. ### Use case The data lands in the lakehouse as CSV files under `Files/`. The ETL reads them and writes an Iceberg table in one statement: ```sql INSERT INTO onelake.db.t SELECT ... FROM hdfs('uri' = 'abfss://<ws>@onelake.dfs.fabric.microsoft.com/<lh>/Files/csv/*.CSV', 'format' = 'csv', ...); ``` Today Doris can write the Iceberg table but cannot read the CSV files from the same lakehouse. We have no client secret (OIDC workload identity only). ### Related issues #68103 moves Azure data access to native credentials, but keeps OneLake on the Hadoop path, so it does not cover this. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
