Hi Robert, thanks for your reply.
GlobalConfiguration.loadConfiguration was useful to check if a
flink-conf.yml file was on resources, for the integration tests that I'm
doing. On the cluster I will use the default configurations.
On Fri, Feb 21, 2020 at 10:58 AM Robert Metzger <rmetz...@apache.org> wrote:
> There are multiple ways of passing configuration parameters to your user
> defined code in Flink
>
> a) use getRuntimeContext().getUserCodeClassLoader().getResource() to
> load a config file from your user code jar or the classpath.
> b) use getRuntimeContext().getExecutionConfig().getGlobalJobParameters() to
> access a configuration object serialized from the main method.
> you can pass a custom object to the job parameters, or use Flink's
> "Configuration" object in your main method:
>
> final StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
>
> Configuration config = new Configuration();
> config.setString("foo", "bar");
> env.getConfig().setGlobalJobParameters(config);
>
> c) Load the flink-conf.yaml:
>
> Configuration conf = GlobalConfiguration.loadConfiguration();
>
> I'm not 100% sure if this approach works, as it is not intended to be used
> in user code (I believe).
>
>
> Let me know if this helps!
>
> Best,
> Robert
>
> On Thu, Feb 20, 2020 at 1:50 PM Chesnay Schepler <ches...@apache.org>
> wrote:
>
>> First things first, we do not intend for users to use anything in the S3
>> filesystem modules except the filesystems itself,
>> meaning that you're somewhat treading on unsupported ground here.
>>
>> Nevertheless, the S3 modules contain a large variety of AWS-provided
>> CerentialsProvider implementations,
>> that can derive credentials from environment variables, system
>> properties, files on the classpath and many more.
>>
>> Ultimately though, you're kind of asking us how to use AWS APIs, for
>> which I would direct you to the AWS documentation.
>>
>> On 20/02/2020 13:16, David Magalhães wrote:
>>
>> I'm using
>> org.apache.flink.fs.s3base.shaded.com.amazonaws.client.builder.AwsClientBuilder
>> to create a S3 client to copy objects and delete object inside
>> a TwoPhaseCommitSinkFunction.
>>
>> Shouldn't be another way to set up configurations without put them
>> hardcoded ? Something like core-site.xml or flink-conf.yaml ?
>>
>> Right now I need to have them hardcoded like this.
>>
>> AmazonS3ClientBuilder.standard
>> .withPathStyleAccessEnabled(true)
>> .withEndpointConfiguration(
>> new EndpointConfiguration("http://minio:9000";, "us-east-1")
>> )
>> .withCredentials(
>> new AWSStaticCredentialsProvider(new BasicAWSCredentials("minio",
>> "minio123"))
>> )
>> .build
>>
>> Thanks
>>
>>
>>
