andygrove commented on code in PR #2874:
URL: https://github.com/apache/datafusion-comet/pull/2874#discussion_r2608635616
##########
common/src/main/java/org/apache/comet/parquet/CometFileKeyUnwrapper.java:
##########
@@ -101,13 +101,33 @@ public class CometFileKeyUnwrapper {
// Cache the hadoopConf just to assert the assumption above.
private Configuration conf = null;
+ /**
+ * Normalizes S3 URI schemes to a canonical form. S3 can be accessed via
multiple schemes (s3://,
+ * s3a://, s3n://) that refer to the same logical filesystem. This method
ensures consistent cache
+ * lookups regardless of which scheme is used.
+ *
+ * @param filePath The file path that may contain an S3 URI
+ * @return The file path with normalized S3 scheme (s3a://)
+ */
+ private String normalizeS3Scheme(final String filePath) {
+ // Normalize s3:// and s3n:// to s3a:// for consistent cache lookups
+ // This handles the case where ObjectStoreUrl uses s3:// but Spark uses
s3a://
+ if (filePath.startsWith("s3://")) {
+ return "s3a://" + filePath.substring(5);
+ } else if (filePath.startsWith("s3n://")) {
+ return "s3a://" + filePath.substring(6);
+ }
Review Comment:
nit: rather than hard-code the lengths, I wonder if this would be clearer:
```suggestion
String s3Prefix = "s3://";
String s3nPrefix = "s3n://";
if (filePath.startsWith(s3Prefix)) {
return "s3a://" + filePath.substring(s3Prefix.length());
} else if (filePath.startsWith(s3nPrefix)) {
return "s3a://" + filePath.substring(s3nPrefix.length());
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
