[
https://issues.apache.org/jira/browse/FLINK-11895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-11895:
-----------------------------------
Component/s: Connectors / FileSystem
> Allow FileSystem Configs to be altered at Runtime
> -------------------------------------------------
>
> Key: FLINK-11895
> URL: https://issues.apache.org/jira/browse/FLINK-11895
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem
> Reporter: Luka Jurukovski
> Priority: Minor
>
> This stems from a need to be able to pass in S3 auth keys at runtime in order
> to allow users to specify the keys they want to use. Based on the
> documentation it seems that currently S3 keys need to be part of the Flink
> cluster configuration, in a hadoop file (which the cluster needs to pointed
> to) or JVM args.
> This only seems to apply to the streaming API. Also Feel free to correct the
> following if I am wrong, as there may be pieces I have no run across, or
> parts of the code I have misunderstood.
> Currently it seems that FileSystems are inferred based on the extension type
> and a set of cached Filesystems that are generated in the background. These
> seem to use the config as defined at the time they are stood up.
> Unfortunately there is no way to tap into this control mechanism or override
> this behavior as many places in the code pulls from this cache. This is
> particularly painful in the sink instance as there are places where this is
> used that are not accessible outside the package it is implemented.
> Through a pretty hacky mechanism I have proved out that this is a self
> imposed limitation, as I was able to change the code to pass in a Filesystem
> from the top level and have it read and write to S3 given keys I set at
> runtime.
> The current methodology is convenient, however there should be finer grain
> controls for instances where the cluster is in a multitenant environment.
> As a final note it seems like both the FileSystem and FileSystemFactory
> classes are not Serializable. I can see why this would be the case in former,
> but I am not clear as to why a factory class would not be Serializable (like
> in the case of BucketFactory). If this can be made serializable this should
> make this a much cleaner process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
