Hi Community, I am trying to create an amazon S3 filesystem distributor using flink and for this I am using hadoop S3a connector with Flink filesystem sink. My flink application would run in a non-AWS environment, on native cluster; so I need to put my access keys in flink configuration.
For connecting to S3 storage, i am configuring flink-conf.yaml with the access credentials like s3.access.key: <access key> s3.secret.key: <secret key> ... and some other parameters required for assuming AWS IAM role with s3a AssumedRoleCredentialProvider Is there a way to encrypt these parameters rather than putting them directly or is there any other way to supply them programmatically. I tried to set them programmatically using the Configuration object and supplying them with StreamExecutionEnvironment.getExecutionEnvironment(Configuration), in my job (rather than from flink-conf.yaml) but then the S3 connection failed. I think flink creates the connection pool at startup even before the job is started. Thanks and Regards Anuj Jain
