Hi Fabian,

Thank you for the response. So I am currently using .writeAsText() to print out 
9 different datastreams in one Flink job as I am printing my original 
datastream with various filters applied to it. I usually see around 6-7 of my 
datastreams successfully list the JSON file in my S3 bucket upon cancelling my 
Flink job.

Even in my situation, would this still be an issue with S3’s file listing 
command?

Thanks,
Michael

From: Fabian Hueske <fhue...@gmail.com>
Date: Friday, October 25, 2019 at 6:04 AM
To: Michael Nguyen <michael.nguye...@t-mobile.com>
Cc: "user@flink.apache.org" <user@flink.apache.org>
Subject: Re: Issue with writeAsText() to S3 bucket

[External]

Hi Michael,

One reason might be that S3's file listing command is only eventually 
consistent.
It might take some time until the file appears and is listed.

Best, Fabian

Am Mi., 23. Okt. 2019 um 22:41 Uhr schrieb Nguyen, Michael 
<michael.nguye...@t-mobile.com<mailto:michael.nguye...@t-mobile.com>>:
Hello all,

I am running into issues at the moment trying to print my DataStreams to an S3 
bucket using writeAsText(“s3://bucket/result.json”) in my Flink job. I used 
print() on the same DataStream and I see the output I am looking for in 
standard output. I first confirm that my datastream has data by looking at the 
standard output, then I cancel my Flink job. After cancelling the job, 
result.json only gets created in my S3 bucket some of the time. It does not 
always gets created, but I confirmed that I see my data in standard output.

I understand writeAsText() should be used for debugging purposes only according 
to Flink’s documentation, but I’m just curious as to why I can’t get 
writeAsText() to always work every time I cancel my job.

Thank you for your help,
Michael

Reply via email to