Re: Potential bug when assuming roles from AWS EKS when using S3 as RocksDb checkpoint backend?

Ingo, I dig into the Flink code a little bit. It looks like the key for specifying the roleArn and roleSessionName are fs.s3a.aws.credentials.provider: com.amazonaws.auth.WebIdentityTokenCredentialsProvider fs.s3a.aws.credentials.provider.role.arn: arn:aws:iam::...:role/... fs.s3a.aws.credentials.

Re: Potential bug when assuming roles from AWS EKS when using S3 as RocksDb checkpoint backend?

Ingo, It looks like I'm now seeing "Caused by: java.lang.NullPointerException: You must specify a value for roleArn and roleSessionName". I assume I would also need to specify that through the configuration file. Could you suggest the key for this configuration? Thanks. Thomas On Sat, Sep 25, 20

Re: Write Streaming data to S3 in Parquet files

Hi, Harshvardhan I think CaiZhi is right. I only have a small addition. Because I see that you want to convert Table to DataStream, you can look at FileSink (ParquetWriterFactory)[1]. [1] https://ci.apache.org/projects/flink/flink-docs-master/docs/connectors/datastream/file_sink/#bulk-encoded-for

Re: Job Manager went down on cancelling job with savepoint

Hi, Puneet Could you share whether you are using Flink's session mode or application mode? >From the log, you are using `StandaloneDispatcher`, but you will use it in both session and application mode. If you use application mode, this might be in line with expectations. Best, Guowei On Fri, Se

Re: Write Streaming data to S3 in Parquet files

Hi! Try the PARTITIONED BY clause. See https://ci.apache.org/projects/flink/flink-docs-master/docs/connectors/table/formats/parquet/ Harshvardhan Shinde 于2021年9月24日周五 下午5:52写道： > Hi, > I wanted to know if we can write streaming data to S3 in parquet format > with partitioning. > Here's what I w

Re: Potential bug when assuming roles from AWS EKS when using S3 as RocksDb checkpoint backend?

Thanks Ingo. Adding the following setting worked. fs.s3a.aws.credentials.provider: com.amazonaws.auth.WebIdentityTokenCredentialsProvider Thomas On Sat, Sep 25, 2021 at 1:12 PM Ingo Bürk wrote: > Hi Thomas, > > I think you might be looking for this: > https://github.com/apache/flink/pull/16717

Re: Potential bug when assuming roles from AWS EKS when using S3 as RocksDb checkpoint backend?

We need to overwrite using    WebIdentityTokenFileCredentialsProviderhttps://github.com/aws/aws-sdk-java-v2/issues/1470#issuecomment-543601232.  otherwise java takes presidency to secret key and access keys than SA On Saturday, September 25, 2021, 04:37:22 PM EDT, Xiangyu Su wrote: Hi

Re: Potential bug when assuming roles from AWS EKS when using S3 as RocksDb checkpoint backend?

Hi Thomas, did you try to login to EKS node and run some aws command like : aws s3 ls ? It sounds like EKS issue, but not 100% sure. Best On Sat, 25 Sept 2021 at 22:12, Ingo Bürk wrote: > Hi Thomas, > > I think you might be looking for this: > https://github.com/apache/flink/pull/16717 > > > Be

Re: Potential bug when assuming roles from AWS EKS when using S3 as RocksDb checkpoint backend?

Hi Thomas, I think you might be looking for this: https://github.com/apache/flink/pull/16717 Best Ingo On Sat, Sep 25, 2021, 20:46 Thomas Wang wrote: > Hi, > > I'm using the official docker image: apache/flink:1.12.1-scala_2.11-java11 > > I'm trying to run a Flink job on an EKS cluster. The j

Potential bug when assuming roles from AWS EKS when using S3 as RocksDb checkpoint backend?

Hi, I'm using the official docker image: apache/flink:1.12.1-scala_2.11-java11 I'm trying to run a Flink job on an EKS cluster. The job is running under a k8s service account that is tied to an IAM role. If I'm not using s3 as RocksDB checkpoint backend, everything works just fine. However, when

Re: how to run job and make jobmanager HA

1)You can use the application cluster mode you can find how to configure in the official flink documentation https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html#deploy-application-cluster 2)for HA you can use kubernetes HA: http