I haven't configured this myself but I would guess that you need to set the parameters defined here under S3A Authentication methods [1]. If the environment variables don't work, then I would try to set the authentication properties.
[1] https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#S3A Cheers, Till On Wed, Jan 16, 2019 at 11:09 AM Vinay Patil <vinay18.pa...@gmail.com> wrote: > Hi Till, > > Can you please let us know the configurations that we need to set for > Profile based credential provider in flink-conf.yaml > > Exporting AWS_PROFILE property on EMR did not work. > > Regards, > Vinay Patil > > > On Wed, Jan 16, 2019 at 3:05 PM Till Rohrmann <trohrm...@apache.org> > wrote: > >> The old BucketingSink was using Hadoop's S3 filesystem directly whereas >> the new StreamingFileSink uses Flink's own FileSystem which need to be >> configured via the flink-conf.yaml. >> >> Cheers, >> Till >> >> On Wed, Jan 16, 2019 at 10:31 AM Vinay Patil <vinay18.pa...@gmail.com> >> wrote: >> >>> Hi Till, >>> >>> We are not providing `fs.s3a.access.key: access_key`, >>> `fs.s3a.secret.key: secret_key` in flink-conf.yaml as we are using Profile >>> based credentials provider. The older BucketingSink code is able to get the >>> credentials and write to S3. We are facing this issue only with >>> StreamingFileSink. We tried adding fs.s3a.impl to core-site.xml when the >>> default configurations were not working. >>> >>> Regards, >>> Vinay Patil >>> >>> >>> On Wed, Jan 16, 2019 at 2:55 PM Till Rohrmann <trohrm...@apache.org> >>> wrote: >>> >>>> Hi Vinay, >>>> >>>> Flink's file systems are self contained and won't respect the >>>> core-site.xml if I'm not mistaken. Instead you have to set the credentials >>>> in the flink configuration flink-conf.yaml via `fs.s3a.access.key: >>>> access_key`, `fs.s3a.secret.key: secret_key` and so on [1]. Have you tried >>>> this out? >>>> >>>> This has been fixed with Flink 1.6.2 and 1.7.0 [2]. >>>> >>>> [1] >>>> https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems.html#built-in-file-systems >>>> [2] https://issues.apache.org/jira/browse/FLINK-10383 >>>> >>>> Cheers, >>>> Till >>>> >>>> On Wed, Jan 16, 2019 at 10:10 AM Kostas Kloudas < >>>> k.klou...@da-platform.com> wrote: >>>> >>>>> Hi Taher, >>>>> >>>>> So you are using the same configuration files and everything and the >>>>> only thing you change is the "s3://" to "s3a://" and the sink cannot find >>>>> the credentials? >>>>> Could you please provide the logs of the Task Managers? >>>>> >>>>> Cheers, >>>>> Kostas >>>>> >>>>> On Wed, Jan 16, 2019 at 9:13 AM Dawid Wysakowicz < >>>>> dwysakow...@apache.org> wrote: >>>>> >>>>>> Forgot to cc ;) >>>>>> On 16/01/2019 08:51, Vinay Patil wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> Can someone please help on this issue. We have even tried to set >>>>>> fs.s3a.impl in core-site.xml, still its not working. >>>>>> >>>>>> Regards, >>>>>> Vinay Patil >>>>>> >>>>>> >>>>>> On Fri, Jan 11, 2019 at 5:03 PM Taher Koitawala [via Apache Flink >>>>>> User Mailing List archive.] <ml+s2336050n25464...@n4.nabble.com> >>>>>> wrote: >>>>>> >>>>>>> Hi All, >>>>>>> We have implemented S3 sink in the following way: >>>>>>> >>>>>>> StreamingFileSink sink= StreamingFileSink.forBulkFormat(new >>>>>>> Path("s3a://mybucket/myfolder/output/"), >>>>>>> ParquetAvroWriters.forGenericRecord(schema)) >>>>>>> .withBucketCheckInterval(50l).withBucketAssigner(new >>>>>>> CustomBucketAssigner()).build(); >>>>>>> >>>>>>> The problem we are facing is that StreamingFileSink is initializing >>>>>>> S3AFileSystem class to write to s3 and is not able to find the s3 >>>>>>> credentials to write data, However other flink application on the >>>>>>> same cluster use "s3://" paths are able to write data to the same s3 >>>>>>> bucket >>>>>>> and folders, we are only facing this issue with StreamingFileSink. >>>>>>> >>>>>>> Regards, >>>>>>> Taher Koitawala >>>>>>> GS Lab Pune >>>>>>> +91 8407979163 >>>>>>> >>>>>>> >>>>>>> ------------------------------ >>>>>>> If you reply to this email, your message will be added to the >>>>>>> discussion below: >>>>>>> >>>>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/StreamingFileSink-cannot-get-AWS-S3-credentials-tp25464.html >>>>>>> To start a new topic under Apache Flink User Mailing List archive., >>>>>>> email ml+s2336050n1...@n4.nabble.com >>>>>>> To unsubscribe from Apache Flink User Mailing List archive., click >>>>>>> here >>>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx> >>>>>>> . >>>>>>> NAML >>>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>>>>>> >>>>>>
