Hey Robert, I’m not sure why you need to set env var. but That’s a flink configuration, See https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/filesystems/s3/#configure-path-style-access
Thanks, Youngwoo 2021년 5월 17일 (월) 오후 10:47, Robert Cullen <cinquate...@gmail.com>님이 작성: > Arvid, Is there a way to set environment variables in the flink-conf.yaml? > Setting them on the CLI isn't working. > > On Sat, May 15, 2021 at 3:56 PM Arvid Heise <ar...@apache.org> wrote: > >> Hi Robert, >> >> we have an end-to-end-test with minio. You have to use >> s3.path.style.access; I think the actual key depends on the used s3 >> connector, so just try out both. >> For reference, the actual test can be found here [2]. >> >> [1] >> https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_s3_minio.sh#L114-L115 >> [2] >> https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/test_batch_wordcount.sh#L42 >> >> On Fri, May 14, 2021 at 6:48 PM Robert Cullen <cinquate...@gmail.com> >> wrote: >> >>> I’m testing out my flink 1.13.0 cluster with the latest Minio >>> RELEASE.2021-04-06T23-11-00Z but flink fails to connect to the bucket. >>> (s3://flink/checkpoints) The 400 Bad Request error does not give any >>> context for the precise problem. I'm wondering if this is a compatibility >>> issue with AWS Signature V4 and the flink client . Here is a sample of the >>> logging: >>> >>> 2021-05-14 15:29:41,886 DEBUG >>> org.apache.hadoop.security.UserGroupInformation [] - hadoop >>> login >>> 2021-05-14 15:29:41,886 DEBUG >>> org.apache.hadoop.security.UserGroupInformation [] - hadoop >>> login commit >>> 2021-05-14 15:29:41,887 DEBUG >>> org.apache.hadoop.security.UserGroupInformation [] - using >>> local user:UnixPrincipal: flink >>> 2021-05-14 15:29:41,887 DEBUG >>> org.apache.hadoop.security.UserGroupInformation [] - Using >>> user: "UnixPrincipal: flink" with name flink >>> 2021-05-14 15:29:41,887 DEBUG >>> org.apache.hadoop.security.UserGroupInformation [] - User >>> entry: "flink" >>> 2021-05-14 15:29:41,887 DEBUG >>> org.apache.hadoop.security.UserGroupInformation [] - UGI >>> loginUser:flink (auth:SIMPLE) >>> 2021-05-14 15:29:41,890 DEBUG org.apache.hadoop.fs.s3a.S3AUtils >>> [] - Credential provider class is >>> org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider >>> 2021-05-14 15:29:41,892 DEBUG org.apache.hadoop.fs.s3a.S3AUtils >>> [] - For URI s3://flink//checkpoints, using credentials >>> AWSCredentialProviderList: SimpleAWSCredentialsProvider >>> 2021-05-14 15:29:41,994 DEBUG org.apache.hadoop.fs.s3a.S3AUtils >>> [] - Value of fs.s3a.connection.maximum is 15 >>> 2021-05-14 15:29:41,994 DEBUG org.apache.hadoop.fs.s3a.S3AUtils >>> [] - Value of fs.s3a.attempts.maximum is 20 >>> 2021-05-14 15:29:41,994 DEBUG org.apache.hadoop.fs.s3a.S3AUtils >>> [] - Value of fs.s3a.connection.establish.timeout is 5000 >>> 2021-05-14 15:29:41,994 DEBUG org.apache.hadoop.fs.s3a.S3AUtils >>> [] - Value of fs.s3a.connection.timeout is 200000 >>> 2021-05-14 15:29:41,994 DEBUG org.apache.hadoop.fs.s3a.S3AUtils >>> [] - Value of fs.s3a.socket.send.buffer is 8192 >>> 2021-05-14 15:29:41,995 DEBUG org.apache.hadoop.fs.s3a.S3AUtils >>> [] - Value of fs.s3a.socket.recv.buffer is 8192 >>> 2021-05-14 15:29:41,996 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem >>> [] - Using User-Agent: Hadoop 3.1.0 >>> 2021-05-14 15:29:42,013 DEBUG com.amazonaws.AmazonWebServiceClient >>> [] - Internal logging successfully configured to commons >>> logger: true >>> 2021-05-14 15:29:42,016 DEBUG com.amazonaws.metrics.AwsSdkMetrics >>> [] - Admin mbean registered under >>> com.amazonaws.management:type=AwsSdkMetrics >>> 2021-05-14 15:29:42,167 DEBUG >>> com.amazonaws.monitoring.CsmConfigurationProviderChain [] - Unable to >>> load configuration from >>> com.amazonaws.monitoring.EnvironmentVariableCsmConfigurationProvider@48442209: >>> Unable to load Client Side Monitoring configurations from environment >>> variables! >>> 2021-05-14 15:29:42,167 DEBUG >>> com.amazonaws.monitoring.CsmConfigurationProviderChain [] - Unable to >>> load configuration from >>> com.amazonaws.monitoring.SystemPropertyCsmConfigurationProvider@209e60f7: >>> Unable to load Client Side Monitoring configurations from system properties >>> variables! >>> 2021-05-14 15:29:42,168 DEBUG >>> com.amazonaws.monitoring.CsmConfigurationProviderChain [] - Unable to >>> load configuration from >>> com.amazonaws.monitoring.ProfileCsmConfigurationProvider@419cb23f: Unable >>> to load config file >>> 2021-05-14 15:29:42,210 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem >>> [] - Enabling path style access! >>> 2021-05-14 15:29:42,213 DEBUG org.apache.hadoop.fs.s3a.S3AUtils >>> [] - Value of fs.s3a.paging.maximum is 5000 >>> 2021-05-14 15:29:42,213 DEBUG org.apache.hadoop.fs.s3a.S3AUtils >>> [] - Value of fs.s3a.block.size is 33554432 >>> 2021-05-14 15:29:42,214 DEBUG org.apache.hadoop.fs.s3a.S3AUtils >>> [] - Value of fs.s3a.readahead.range is 65536 >>> 2021-05-14 15:29:42,214 DEBUG org.apache.hadoop.fs.s3a.S3AUtils >>> [] - Value of fs.s3a.max.total.tasks is 5 >>> 2021-05-14 15:29:42,214 DEBUG org.apache.hadoop.fs.s3a.S3AUtils >>> [] - Value of fs.s3a.threads.keepalivetime is 60 >>> 2021-05-14 15:29:42,226 DEBUG >>> org.apache.hadoop.fs.s3a.AWSCredentialProviderList [] - Using >>> credentials from SimpleAWSCredentialsProvider >>> 2021-05-14 15:29:42,228 DEBUG com.amazonaws.request >>> [] - Sending Request: HEAD http://10.42.0.133:9000 /flink/ >>> Headers: (amz-sdk-invocation-id: a4bff47d-d82c-bd3f-8bd8-93289f81cb91, >>> Content-Type: application/octet-stream, User-Agent: Hadoop 3.1.0, >>> aws-sdk-java/1.11.788 Linux/5.8.0-50-generic >>> OpenJDK_64-Bit_Server_VM/25.292-b10 java/1.8.0_292 scala/2.12.7 >>> vendor/Oracle_Corporation, ) >>> 2021-05-14 15:29:42,240 DEBUG com.amazonaws.auth.AWS4Signer >>> [] - AWS4 Canonical Request: '"HEAD >>> /flink/ >>> >>> amz-sdk-invocation-id:a4bff47d-d82c-bd3f-8bd8-93289f81cb91 >>> amz-sdk-request:attempt=1;max=21 >>> amz-sdk-retry:0/0/500 >>> content-type:application/octet-stream >>> host:10.42.0.133:9000 >>> user-agent:Hadoop 3.1.0, aws-sdk-java/1.11.788 Linux/5.8.0-50-generic >>> OpenJDK_64-Bit_Server_VM/25.292-b10 java/1.8.0_292 scala/2.12.7 >>> vendor/Oracle_Corporation >>> x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 >>> x-amz-date:20210514T152942Z >>> >>> amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date >>> e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" >>> 2021-05-14 15:29:42,241 DEBUG com.amazonaws.auth.AWS4Signer >>> [] - AWS4 String to Sign: '"AWS4-HMAC-SHA256 >>> 20210514T152942Z >>> 20210514/us-east-1/s3/aws4_request >>> d26fcfe35c3fe7fa67c5adfb227f8f08498d0808b255a13378fb6b1b018be40e" >>> 2021-05-14 15:29:42,241 DEBUG com.amazonaws.auth.AWS4Signer >>> [] - Generating a new signing key as the signing key not >>> available in the cache for the date 1620950400000 >>> 2021-05-14 15:29:42,264 DEBUG com.amazonaws.request >>> [] - Received error response: >>> com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: >>> Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: null; >>> S3 Extended Request ID: null; Proxy: null), S3 Extended Request ID: null >>> >>> -- >>> Robert Cullen >>> 240-475-4490 >>> >> > > -- > Robert Cullen > 240-475-4490 >
