Hi Fabian, Thank you for the response. So I am currently using .writeAsText() to print out 9 different datastreams in one Flink job as I am printing my original datastream with various filters applied to it. I usually see around 6-7 of my datastreams successfully list the JSON file in my S3 bucket upon cancelling my Flink job.
Even in my situation, would this still be an issue with S3’s file listing command? Thanks, Michael From: Fabian Hueske <fhue...@gmail.com> Date: Friday, October 25, 2019 at 6:04 AM To: Michael Nguyen <michael.nguye...@t-mobile.com> Cc: "user@flink.apache.org" <user@flink.apache.org> Subject: Re: Issue with writeAsText() to S3 bucket [External] Hi Michael, One reason might be that S3's file listing command is only eventually consistent. It might take some time until the file appears and is listed. Best, Fabian Am Mi., 23. Okt. 2019 um 22:41 Uhr schrieb Nguyen, Michael <michael.nguye...@t-mobile.com<mailto:michael.nguye...@t-mobile.com>>: Hello all, I am running into issues at the moment trying to print my DataStreams to an S3 bucket using writeAsText(“s3://bucket/result.json”) in my Flink job. I used print() on the same DataStream and I see the output I am looking for in standard output. I first confirm that my datastream has data by looking at the standard output, then I cancel my Flink job. After cancelling the job, result.json only gets created in my S3 bucket some of the time. It does not always gets created, but I confirmed that I see my data in standard output. I understand writeAsText() should be used for debugging purposes only according to Flink’s documentation, but I’m just curious as to why I can’t get writeAsText() to always work every time I cancel my job. Thank you for your help, Michael
