Thanks, Robert. // ah
From: Robert Metzger <rmetz...@apache.org> Sent: Wednesday, September 22, 2021 1:49 PM To: Hailu, Andreas [Engineering] <andreas.ha...@ny.email.gs.com> Cc: user@flink.apache.org Subject: Re: Many S3V4AuthErrorRetryStrategy warn logs while reading/writing from S3 Hey Andreas, This could be related too https://github.com/apache/hadoop/pull/110/files#diff-0a2e55a2f79ea4079eb7b77b0dc3ee562b383076fa0ac168894d50c80a95131dR950<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_hadoop_pull_110_files-23diff-2D0a2e55a2f79ea4079eb7b77b0dc3ee562b383076fa0ac168894d50c80a95131dR950&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=hRr4SA7BtUvKoMBP6VDhfisy2OJ1ZAzai-pcCC6TFXM&m=XrWyobBln-QQ652Sml6hW8XEUlQrYjx_rDoyvyu456U&s=EKNCzrkaFqM8LySKDKyE0xKQqHI_pE00Fxg2feI15Gg&e=> I guess in Flink this would be s3.endpoint: your-endpoint-hostname Where your-endpoint-hostname is a region-specific endpoint, which you can probably look up from the S3 docs. On Wed, Sep 22, 2021 at 7:07 PM Hailu, Andreas <andreas.ha...@gs.com<mailto:andreas.ha...@gs.com>> wrote: Hi, When reading/writing to and from S3 using the flink-fs-s3-hadoop plugin on 1.11.2, we observe a lot of these WARN log statements in the logs: WARN S3V4AuthErrorRetryStrategy - Attempting to re-send the request to s3.amazonaws.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__s3.amazonaws.com&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=hRr4SA7BtUvKoMBP6VDhfisy2OJ1ZAzai-pcCC6TFXM&m=XrWyobBln-QQ652Sml6hW8XEUlQrYjx_rDoyvyu456U&s=nIZDYIPynFqOi400vqQM3FygFfZ-mwWYLqpD5v4w7K0&e=> with AWS V4 authentication. To avoid this warning in the future, please use region-specific endpoint to access buckets located in regions that require V4 signing. The applications complete successfully which is great, but I’m not sure what the root of the error is and I’m hesitant to silence it through our logging configurations. I saw something that looks similar here[1]. Is there a way for us to similarly have Flink’s AWS S3 client to use V4 strategy to begin with? [1] https://stackoverflow.com/questions/39513518/aws-emr-writing-to-kms-encrypted-s3-parquet-files<https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_39513518_aws-2Demr-2Dwriting-2Dto-2Dkms-2Dencrypted-2Ds3-2Dparquet-2Dfiles&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=hRr4SA7BtUvKoMBP6VDhfisy2OJ1ZAzai-pcCC6TFXM&m=XrWyobBln-QQ652Sml6hW8XEUlQrYjx_rDoyvyu456U&s=wxsQRAaNJ8CqtAAJ6a4Klr26_e486CWtF8GWqvnQb4k&e=> ____________ Andreas Hailu Data Lake Engineering | Goldman Sachs & Co. ________________________________ Your Personal Data: We may collect and process information about you that may be subject to data protection laws. For more information about how we use and disclose your personal data, how we protect your information, our legal basis to use your information, your rights and who you can contact, please refer to: www.gs.com/privacy-notices<http://www.gs.com/privacy-notices> ________________________________ Your Personal Data: We may collect and process information about you that may be subject to data protection laws. For more information about how we use and disclose your personal data, how we protect your information, our legal basis to use your information, your rights and who you can contact, please refer to: www.gs.com/privacy-notices<http://www.gs.com/privacy-notices>
