dwsmith1983 commented on code in PR #6059:
URL: https://github.com/apache/datafusion-comet/pull/6059#discussion_r4206380864
##########
spark/src/main/scala/org/apache/comet/objectstore/NativeConfig.scala:
##########
@@ -214,6 +216,13 @@ object NativeConfig {
}
val s3CompliantSchemes = parseSchemeSet(s3CompliantRaw)
+ // ABFS: Hadoop resolves the mechanism and values for this URI's account
on the driver, and
+ // only those travel to native (never another account's keys, never the
whole namespace).
+ if (scheme == "abfs" || scheme == "abfss") {
+ return options.toMap ++ AbfsAuthResolver.toOptions(
Review Comment:
> Could resolution preserve account/container identity through task
planning, or make such scans fall back instead of reusing the first credential?
Fixed in 8f77b4165 by falling back. A native scan whose `abfs`/`abfss` files
span more than one container or account now declines, as does one that mixes
ABFS with another scheme. Your `pathA, pathB` case reads through Spark.
Matching credentials are not enough to stay native: the native scan builds one
store per file partition from that partition's first file and keeps only each
file's path, and Spark packs files from different roots into the same
partition. A file from the second container would be fetched from the first.
`AbfsScanFallbackSuite` runs the two-account and two-container cases end to
end, for both the Parquet and the CSV native scans.
Carrying the store identity per file through to the executors would let
these scans stay native, but that is a change to the native planner rather than
to the Azure resolution, so I would rather do it separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
