cz <
>> dwysakow...@apache.org>; Taher Koitawala [via Apache Flink User Mailing
>> List archive.] ; user <
>> user@flink.apache.org>
>> *Subject:* EXT :Re: StreamingFileSink cannot get AWS S3 credentials
>>
>>
>>
>> I haven't configured t
Flink User Mailing
> List archive.] ; user <
> user@flink.apache.org>
> *Subject:* EXT :Re: StreamingFileSink cannot get AWS S3 credentials
>
>
>
> I haven't configured this myself but I would guess that you need to set
> the parameters defined here under S3A Authenticati
Kostas Kloudas ; Dawid Wysakowicz
; Taher Koitawala [via Apache Flink User Mailing List
archive.] ; user
Subject: EXT :Re: StreamingFileSink cannot get AWS S3 credentials
I haven't configured this myself but I would guess that you need to set the
parameters defined here under S3A Aut
I haven't configured this myself but I would guess that you need to set the
parameters defined here under S3A Authentication methods [1]. If the
environment variables don't work, then I would try to set the
authentication properties.
[1]
https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoo
Hi Till,
Can you please let us know the configurations that we need to set for
Profile based credential provider in flink-conf.yaml
Exporting AWS_PROFILE property on EMR did not work.
Regards,
Vinay Patil
On Wed, Jan 16, 2019 at 3:05 PM Till Rohrmann wrote:
> The old BucketingSink was using
The old BucketingSink was using Hadoop's S3 filesystem directly whereas the
new StreamingFileSink uses Flink's own FileSystem which need to be
configured via the flink-conf.yaml.
Cheers,
Till
On Wed, Jan 16, 2019 at 10:31 AM Vinay Patil
wrote:
> Hi Till,
>
> We are not providing `fs.s3a.access.
Hi Till,
We are not providing `fs.s3a.access.key: access_key`, `fs.s3a.secret.key:
secret_key` in flink-conf.yaml as we are using Profile based credentials
provider. The older BucketingSink code is able to get the credentials and
write to S3. We are facing this issue only with StreamingFileSink. W
Actually Till is right.
Sorry, my fault, I did not read your second email where Vinay mentions the
core-site.xml.
Cheers,
Kostas
On Wed, Jan 16, 2019 at 10:25 AM Till Rohrmann wrote:
> Hi Vinay,
>
> Flink's file systems are self contained and won't respect the
> core-site.xml if I'm not mistak
Hi Vinay,
Flink's file systems are self contained and won't respect the core-site.xml
if I'm not mistaken. Instead you have to set the credentials in the flink
configuration flink-conf.yaml via `fs.s3a.access.key: access_key`,
`fs.s3a.secret.key: secret_key` and so on [1]. Have you tried this out?
Hi Taher,
So you are using the same configuration files and everything and the only
thing you change is the "s3://" to "s3a://" and the sink cannot find the
credentials?
Could you please provide the logs of the Task Managers?
Cheers,
Kostas
On Wed, Jan 16, 2019 at 9:13 AM Dawid Wysakowicz
wrote
Forgot to cc ;)
On 16/01/2019 08:51, Vinay Patil wrote:
> Hi,
>
> Can someone please help on this issue. We have even tried to set
> fs.s3a.impl in core-site.xml, still its not working.
>
> Regards,
> Vinay Patil
>
>
> On Fri, Jan 11, 2019 at 5:03 PM Taher Koitawala [via Apache Flink User
> Mailin
Hi,
I cc Kostas who should be able to help you.
Best,
Dawid
On 16/01/2019 08:51, Vinay Patil wrote:
> Hi,
>
> Can someone please help on this issue. We have even tried to set
> fs.s3a.impl in core-site.xml, still its not working.
>
> Regards,
> Vinay Patil
>
>
> On Fri, Jan 11, 2019 at 5:03 PM
Hi,
Can someone please help on this issue. We have even tried to set
fs.s3a.impl in core-site.xml, still its not working.
Regards,
Vinay Patil
On Fri, Jan 11, 2019 at 5:03 PM Taher Koitawala [via Apache Flink User
Mailing List archive.] wrote:
> Hi All,
> We have implemented S3 sink
13 matches
Mail list logo
