Hi, We are trying to deploy our Flink application (already running on AWS EMR successfully) in Kubernetes in native mode. The base image used to bake the Flink container image is flink:1.13.1-java8. For S3 access required to save checkpoints - we have placed the jar "flink-s3-fs-presto-1.13.1.jar" under $FLINK_HOME/plugins/s3-fs-presto/ directory. We have provided the service account (-Dkubernetes.service-account) in the command-line having full AWS-S3 and all kubernetes api access.
Scenario-1 [we provided -Dkubernetes.service-account parameter] flink run-application -c com.************* -Dkubernetes.cluster-id=k8s-test-app -Dkubernetes.service-account=*************-rbac -Dkubernetes.container.image=*************:1.1 -Djobmanager.memory.process.size=1024m -Dtaskmanager.memory.process.size=2048m --target kubernetes-application local:///opt/flink/usrlib/*************.jar /opt/flink/usrlib/*************.properties /opt/flink/usrlib/*************.properties /opt/flink/usrlib/*************.properties /opt/flink/usrlib/*************.properties s3://k8s-*************/app_checkpoints/ Scenario-1 Result: Checkpoints doesn't work and throws the following error: Caused by: com.facebook.presto.hive.s3.PrestoS3FileSystem$UnrecoverableS3OperationException: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 5HVT9TVT23VYKGD5; S3 Extended Request ID: S3LpXthsyJTHO/aLGsEb6yFfthI81TXSiWr48bxroxYeuLsHw9c/31UmjKBbbBHinyPGPfVMu7k=; Proxy: null), S3 Extended Request ID: S3LpXthsyJTHO/aLGsEb6yFfthI81TXSiWr48bxroxYeuLsHw9c/31UmjKBbbBHinyPGPfVMu7k= (Path: s3://k8s-*************/app_checkpoints/7e5f3729ad8669b085158823e69edadd/chk-3/_metadata) at com.facebook.presto.hive.s3.PrestoS3FileSystem.lambda$getS3ObjectMetadata$2(PrestoS3FileSystem.java:573) ~[?:?] Scenario-2 [we provided -Dkubernetes.service-account, -Ds3.access-key and -Ds3.secret-key parameters] flink run-application -c com.************* -Dkubernetes.cluster-id=k8s-test-app -Dkubernetes.service-account=*************-rbac -Dkubernetes.container.image=*************:1.1 -Djobmanager.memory.process.size=1024m -Dtaskmanager.memory.process.size=2048m -Ds3.access-key=************* -Ds3.secret-key=************* --target kubernetes-application local:///opt/flink/usrlib/*************.jar /opt/flink/usrlib/*************.properties /opt/flink/usrlib/*************.properties /opt/flink/usrlib/*************.properties /opt/flink/usrlib/*************.properties s3://k8s-*************/app_checkpoints/ Scenario-2 Result: works fine and all checkpoints are triggered, completed and copied to S3. Please advice, how can we make checkpoints work with just service-account without providing -Ds3.access-key and -Ds3.secret-key parameters. Is there any way to provide IAM role or let presto pick from service account? Thanks, Saurabh
