I’m testing out my flink 1.13.0 cluster with the latest Minio
RELEASE.2021-04-06T23-11-00Z but flink fails to connect to the bucket.
(s3://flink/checkpoints) The 400 Bad Request error does not give any
context for the precise problem. I'm wondering if this is a compatibility
issue with AWS Signature V4 and the flink client . Here is a sample of the
logging:
2021-05-14 15:29:41,886 DEBUG
org.apache.hadoop.security.UserGroupInformation [] -
hadoop login
2021-05-14 15:29:41,886 DEBUG
org.apache.hadoop.security.UserGroupInformation [] -
hadoop login commit
2021-05-14 15:29:41,887 DEBUG
org.apache.hadoop.security.UserGroupInformation [] -
using local user:UnixPrincipal: flink
2021-05-14 15:29:41,887 DEBUG
org.apache.hadoop.security.UserGroupInformation [] -
Using user: "UnixPrincipal: flink" with name flink
2021-05-14 15:29:41,887 DEBUG
org.apache.hadoop.security.UserGroupInformation [] - User
entry: "flink"
2021-05-14 15:29:41,887 DEBUG
org.apache.hadoop.security.UserGroupInformation [] - UGI
loginUser:flink (auth:SIMPLE)
2021-05-14 15:29:41,890 DEBUG org.apache.hadoop.fs.s3a.S3AUtils
[] - Credential provider class is
org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
2021-05-14 15:29:41,892 DEBUG org.apache.hadoop.fs.s3a.S3AUtils
[] - For URI s3://flink//checkpoints, using
credentials AWSCredentialProviderList: SimpleAWSCredentialsProvider
2021-05-14 15:29:41,994 DEBUG org.apache.hadoop.fs.s3a.S3AUtils
[] - Value of fs.s3a.connection.maximum is 15
2021-05-14 15:29:41,994 DEBUG org.apache.hadoop.fs.s3a.S3AUtils
[] - Value of fs.s3a.attempts.maximum is 20
2021-05-14 15:29:41,994 DEBUG org.apache.hadoop.fs.s3a.S3AUtils
[] - Value of fs.s3a.connection.establish.timeout
is 5000
2021-05-14 15:29:41,994 DEBUG org.apache.hadoop.fs.s3a.S3AUtils
[] - Value of fs.s3a.connection.timeout is 200000
2021-05-14 15:29:41,994 DEBUG org.apache.hadoop.fs.s3a.S3AUtils
[] - Value of fs.s3a.socket.send.buffer is 8192
2021-05-14 15:29:41,995 DEBUG org.apache.hadoop.fs.s3a.S3AUtils
[] - Value of fs.s3a.socket.recv.buffer is 8192
2021-05-14 15:29:41,996 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem
[] - Using User-Agent: Hadoop 3.1.0
2021-05-14 15:29:42,013 DEBUG com.amazonaws.AmazonWebServiceClient
[] - Internal logging successfully configured to
commons logger: true
2021-05-14 15:29:42,016 DEBUG com.amazonaws.metrics.AwsSdkMetrics
[] - Admin mbean registered under
com.amazonaws.management:type=AwsSdkMetrics
2021-05-14 15:29:42,167 DEBUG
com.amazonaws.monitoring.CsmConfigurationProviderChain [] -
Unable to load configuration from
com.amazonaws.monitoring.EnvironmentVariableCsmConfigurationProvider@48442209:
Unable to load Client Side Monitoring configurations from environment
variables!
2021-05-14 15:29:42,167 DEBUG
com.amazonaws.monitoring.CsmConfigurationProviderChain [] -
Unable to load configuration from
com.amazonaws.monitoring.SystemPropertyCsmConfigurationProvider@209e60f7:
Unable to load Client Side Monitoring configurations from system
properties variables!
2021-05-14 15:29:42,168 DEBUG
com.amazonaws.monitoring.CsmConfigurationProviderChain [] -
Unable to load configuration from
com.amazonaws.monitoring.ProfileCsmConfigurationProvider@419cb23f:
Unable to load config file
2021-05-14 15:29:42,210 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem
[] - Enabling path style access!
2021-05-14 15:29:42,213 DEBUG org.apache.hadoop.fs.s3a.S3AUtils
[] - Value of fs.s3a.paging.maximum is 5000
2021-05-14 15:29:42,213 DEBUG org.apache.hadoop.fs.s3a.S3AUtils
[] - Value of fs.s3a.block.size is 33554432
2021-05-14 15:29:42,214 DEBUG org.apache.hadoop.fs.s3a.S3AUtils
[] - Value of fs.s3a.readahead.range is 65536
2021-05-14 15:29:42,214 DEBUG org.apache.hadoop.fs.s3a.S3AUtils
[] - Value of fs.s3a.max.total.tasks is 5
2021-05-14 15:29:42,214 DEBUG org.apache.hadoop.fs.s3a.S3AUtils
[] - Value of fs.s3a.threads.keepalivetime is 60
2021-05-14 15:29:42,226 DEBUG
org.apache.hadoop.fs.s3a.AWSCredentialProviderList [] -
Using credentials from SimpleAWSCredentialsProvider
2021-05-14 15:29:42,228 DEBUG com.amazonaws.request
[] - Sending Request: HEAD http://10.42.0.133:9000
/flink/ Headers: (amz-sdk-invocation-id:
a4bff47d-d82c-bd3f-8bd8-93289f81cb91, Content-Type:
application/octet-stream, User-Agent: Hadoop 3.1.0,
aws-sdk-java/1.11.788 Linux/5.8.0-50-generic
OpenJDK_64-Bit_Server_VM/25.292-b10 java/1.8.0_292 scala/2.12.7
vendor/Oracle_Corporation, )
2021-05-14 15:29:42,240 DEBUG com.amazonaws.auth.AWS4Signer
[] - AWS4 Canonical Request: '"HEAD
/flink/
amz-sdk-invocation-id:a4bff47d-d82c-bd3f-8bd8-93289f81cb91
amz-sdk-request:attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:10.42.0.133:9000
user-agent:Hadoop 3.1.0, aws-sdk-java/1.11.788 Linux/5.8.0-50-generic
OpenJDK_64-Bit_Server_VM/25.292-b10 java/1.8.0_292 scala/2.12.7
vendor/Oracle_Corporation
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20210514T152942Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
2021-05-14 15:29:42,241 DEBUG com.amazonaws.auth.AWS4Signer
[] - AWS4 String to Sign: '"AWS4-HMAC-SHA256
20210514T152942Z
20210514/us-east-1/s3/aws4_request
d26fcfe35c3fe7fa67c5adfb227f8f08498d0808b255a13378fb6b1b018be40e"
2021-05-14 15:29:42,241 DEBUG com.amazonaws.auth.AWS4Signer
[] - Generating a new signing key as the signing
key not available in the cache for the date 1620950400000
2021-05-14 15:29:42,264 DEBUG com.amazonaws.request
[] - Received error response:
com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request
(Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request;
Request ID: null; S3 Extended Request ID: null; Proxy: null), S3
Extended Request ID: null
--
Robert Cullen
240-475-4490
