We used EMR version 5.20 which has Flink 1.6.2 and all other libraries were according to this version. So flink-s3-fs-hadoop was 1.6.2 as well.
On Sun, Aug 18, 2019, 9:55 PM Ayush Verma <ayushver...@gmail.com> wrote: > Hello, could you tell us the version of flink-s3-fs-hadoop library that > you are using ? > > On Sun 18 Aug 2019 at 16:24, taher koitawala <taher...@gmail.com> wrote: > >> Hi Swapnil, >> We faced this problem once, I think changing checkpoint dir to >> hdfs and keeping sink dir to s3 with EMRFS s3 consistency enabled solves >> this problem. If you are not using emr then I don't know how else it can be >> solved. But in a nutshell because EMRFS s3 consistency uses Dynamo DB in >> the back end to check for all files being written to s3. It kind of makes >> s3 consistent and Streaming file sink works just fine. >> >> >> >> On Sat, Aug 17, 2019, 3:32 AM Swapnil Kumar <swku...@zendesk.com> wrote: >> >>> Hello, We are using Flink to process input events and aggregate and >>> write o/p of our streaming job to S3 using StreamingFileSink but whenever >>> we try to restore the job from a savepoint, the restoration fails with >>> missing part files error. As per my understanding, s3 deletes those >>> part(intermittent) files and can no longer be found on s3. Is there a >>> workaround for this, so that we can use s3 as a sink? >>> >>> -- >>> Thanks, >>> Swapnil Kumar >>> >>
