Hi all, We store Flink checkpoints in Amazon S3. Flink periodically sends out GET, PUT, LIST, DELETE requests to S3, to store-clear checkpoints. From the logs, we see that GET, PUT and LIST requests are successful but it throws an AWS access denied error for DELETE request.
Here’s a snippet of the logs for DELETE request – 2018-10-15 04:13:22,819 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency - ServiceName=[Amazon S3], AWSErrorCode=[AccessDenied], StatusCode=[403], ServiceEndpoint=[https://xxx-xxx-prod.s3.amazonaws.com], Exception=[org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: xxxxxxxxxxxxx), S3 Extended Request ID: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx], RequestType=[DeleteObjectRequest], AWSRequestID=[XXXXXXXXXXXXXXXXXX], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[4.984], HttpClientSendRequestTime=[0.029], HttpRequestTime=[4.84], RequestSigningTime=[0.038], CredentialsRequestTime=[0.0, 0.0], HttpClientReceiveResponseTime=[4.78] Is there some configuration that we’re forgetting that is preventing Flink from sending DELETE requests to S3? I’d be happy to provide more information if needed. Thanks, Harshith
