Hi Ricardo, one option is to use s3p for checkpointing (Presto) and s3a for custom applications and attach different configurations.
In general, I'd recommend to use a cluster per application to exactly avoid such issues. I'd use K8s and put the respective IAM roles on each application pod (e.g. with kiam). On Thu, Jun 18, 2020 at 1:46 AM Ricardo Cardante < ricardocarda...@tutanota.com> wrote: > Hi! > > > We are working in a use case where we have a shared Flink cluster to > deploy multiple jobs from different teams. With this strategy, we are > facing a challenge regarding the interaction with S3. Given that we already > configured S3 for the state backend (through flink-conf.yaml) every time we > use API functions that communicate with the file system (e.g., DataStream > readFile) the applicational configurations appear to be overridden by those > of the cluster while attempting to communicate with external S3 buckets. > What we've thought so far: > > > 1. Provide a core-site.xml resource file targeting the external S3 buckets > we want to interact with. We've tested, and the credentials ultimately seem > to be ignored in behalf of the IAM roles that are pre-loaded with the > instances; > > 2. Load the cluster instances with multiple IAM roles. The problem with > this is that we would allow each job to interact with out-of-scope buckets; > > 3. Spin multiple clusters with different configurations - we would like to > avoid this since we started from the premise of sharing a single cluster > per context; > > > What would be a clean/recommended solution to interact with multiple S3 > buckets with different security policies from a shared Flink cluster? > > Thanks in advance. > -- Arvid Heise | Senior Java Developer <https://www.ververica.com/> Follow us @VervericaData -- Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference Stream Processing | Event Driven | Real Time -- Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng
