Thanks Amit, I’m now in the process of checking our IAM roles to see if the user has been given DeleteObject permission to S3. I’m guessing that’s the most likely cause for this error.
- Harshith From: Amit Jain <aj201...@gmail.com> Date: Monday, 15 October 2018 at 4:46 PM To: Harshith Kumar Bolar <hk...@arity.com> Cc: "user@flink.apache.org" <user@flink.apache.org> Subject: [External] Re: Why am I getting AWS access denied error for request type [DeleteObjectRequest] in S3? Hi Harshith, Did you enable delete permission on S3 for running machines? Are you using IAM roles or access key id and secret access key combo? -- Thanks, Amit On Mon, Oct 15, 2018 at 3:15 PM Kumar Bolar, Harshith <hk...@arity.com<mailto:hk...@arity.com>> wrote: Hi all, We store Flink checkpoints in Amazon S3. Flink periodically sends out GET, PUT, LIST, DELETE requests to S3, to store-clear checkpoints. From the logs, we see that GET, PUT and LIST requests are successful but it throws an AWS access denied error for DELETE request. Here’s a snippet of the logs for DELETE request – 2018-10-15 04:13:22,819 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency - ServiceName=[Amazon S3], AWSErrorCode=[AccessDenied], StatusCode=[403], ServiceEndpoint=[https://xxx-xxx-prod.s3.amazonaws.com<https://urldefense.proofpoint.com/v2/url?u=https-3A__xxx-2Dxxx-2Dprod.s3.amazonaws.com&d=DwMFaQ&c=gtIjdLs6LnStUpy9cTOW9w&r=61bFb6zUNKZxlAQDRo_jKA&m=-YN-g71CJxTL2Wc8jPfRJMEE1-hKsivZMlFPHqVDUaE&s=XrrhcZp8eXHOAskhK-3sjYrx_XbFCd5uAIDGtDPQ4ug&e=>], Exception=[org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: xxxxxxxxxxxxx), S3 Extended Request ID: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx], RequestType=[DeleteObjectRequest], AWSRequestID=[XXXXXXXXXXXXXXXXXX], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[4.984], HttpClientSendRequestTime=[0.029], HttpRequestTime=[4.84], RequestSigningTime=[0.038], CredentialsRequestTime=[0.0, 0.0], HttpClientReceiveResponseTime=[4.78] Is there some configuration that we’re forgetting that is preventing Flink from sending DELETE requests to S3? I’d be happy to provide more information if needed. Thanks, Harshith
