Hi Patricia, Different S3 buckets can be accessed with different S3A client configurations. This allows for different endpoints, data read and write strategies, as well as login details. A bucket s3a://nightly/ used for nightly data can then be given a session key: fs.s3a.bucket.nightly.access.key="..." fs.s3a.bucket.nightly.secret.key="..." fs.s3a.bucket.nightly.session.key="..." ...
The task is to forward these configs to the underlying Hadoop instance. BR, G On Wed, Dec 18, 2024 at 2:56 PM patricia lee <plee3...@gmail.com> wrote: > > I have to configure my project to use TWO S3s. > > 1- S3 for Statebackend > 2 - S3 for FileSink > > How can I configure 2 different s3 endpoint and different credentials for > each of the s3. > > When I checked the documentation, there is only a single property > attribute for s3 access keys and bucket endpoint. > > Is it possible to achieve this? > > Source: > https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/deployment/filesystems/s3/ >
