xintongsong commented on a change in pull request #18489:
URL: https://github.com/apache/flink/pull/18489#discussion_r791523758
##########
File path:
flink-filesystems/flink-gs-fs-hadoop/src/main/java/org/apache/flink/fs/gs/GSFileSystem.java
##########
@@ -39,17 +45,67 @@
GSFileSystem(GoogleHadoopFileSystem googleHadoopFileSystem,
GSFileSystemOptions options) {
super(Preconditions.checkNotNull(googleHadoopFileSystem));
- LOGGER.info("Creating GSFileSystem with options {}", options);
-
this.options = Preconditions.checkNotNull(options);
+ LOGGER.info("Creating GSFileSystem with options {}", options);
}
@Override
- public RecoverableWriter createRecoverableWriter() {
- LOGGER.info("Creating recoverable writer with options {}", options);
+ public RecoverableWriter createRecoverableWriter() throws IOException {
+
+ // follow the same rules as for the Hadoop connector, i.e.
+ // 1) only use service credentials at all if Hadoop
+ // "google.cloud.auth.service.account.enable" is true (default: true)
+ // 2) use GOOGLE_APPLICATION_CREDENTIALS as location of credentials,
if supplied
+ // 3) use Hadoop "google.cloud.auth.service.account.json.keyfile" as
location of
+ // credentials, if supplied
+ // 4) use no credentials
Review comment:
Looking into this a bit more, I think it's probably fine as is.
The module currently consists of two parts leveraging different underlying
libaraies: the FileSystem that uses `gcs-connector`, and the RecoverableWriter
that uses `google-cloud-storage`. Hadoop configurations
(`core-site/default.xml`) can be applied directly on `gcs-connector` but not
`google-cloud-storage`.
In that sense, it makes sense to me that
RecoverableWriter/`google-cloud-storage` only support selective Hadoop
configurations. As a first step, the supported configurations includes only
`account.enable` and `account.json.keyfile`. We can add more if new demands
emerge later.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
