Re: Weekly Community Update 2019/33, Personal Chinese Version

2019-08-18 Thread Paul Lam
Hi Tison, Big +1 for the Chinese Weekly Community Update. The content is well-organized, and I believe it would be very helpful for Chinese users to get an overview of what’s going on in the community. Best, Paul Lam > 在 2019年8月19日,12:27,Zili Chen 写道: > > Hi community, > > Inspired by weekl

Weekly Community Update 2019/33, Personal Chinese Version

2019-08-18 Thread Zili Chen
Hi community, Inspired by weekly community updates thread, regards the growth of Chinese community and kind of different concerns for community members I'd like to start a personally maintained Chinese version of Weekly Community Update. Right now I posted these weeks' updates on this page[1] whe

Re: Update Checkpoint and/or Savepoint Timeout of Running Job without restart?

2019-08-18 Thread Congxian Qiu
Hi Currently, we can't change a running job's checkpoint timeout, but there is an issue[1] which wants to set a separate timeout for savepoint. [1] https://issues.apache.org/jira/browse/FLINK-9465 Best, Congxian Aaron Levin 于2019年8月17日周六 上午12:37写道: > Hello, > > Question: Is it possible to upd

Re: Using S3 as a sink (StreamingFileSink)

2019-08-18 Thread Ayush Verma
Hi, I would suggest you upgrade flink to 1.7.x and flink-s3-fs-hadoop to 1.7.2. You might be facing this issue: - https://issues.apache.org/jira/browse/FLINK-11496 - https://issues.apache.org/jira/browse/FLINK-11302 Kind regards Ayush Verma On Sun, Aug 18, 2019 at 6:02 PM taher koitawala

Re: Using S3 as a sink (StreamingFileSink)

2019-08-18 Thread taher koitawala
We used EMR version 5.20 which has Flink 1.6.2 and all other libraries were according to this version. So flink-s3-fs-hadoop was 1.6.2 as well. On Sun, Aug 18, 2019, 9:55 PM Ayush Verma wrote: > Hello, could you tell us the version of flink-s3-fs-hadoop library that > you are using ? > > On Sun

Re: Using S3 as a sink (StreamingFileSink)

2019-08-18 Thread Ayush Verma
Hello, could you tell us the version of flink-s3-fs-hadoop library that you are using ? On Sun 18 Aug 2019 at 16:24, taher koitawala wrote: > Hi Swapnil, >We faced this problem once, I think changing checkpoint dir to hdfs > and keeping sink dir to s3 with EMRFS s3 consistency enabled so

Re: Using S3 as a sink (StreamingFileSink)

2019-08-18 Thread taher koitawala
Hi Swapnil, We faced this problem once, I think changing checkpoint dir to hdfs and keeping sink dir to s3 with EMRFS s3 consistency enabled solves this problem. If you are not using emr then I don't know how else it can be solved. But in a nutshell because EMRFS s3 consistency uses Dynamo D

Re: Using S3 as a sink (StreamingFileSink)

2019-08-18 Thread Stephan Ewen
My first guess would also be the same as Rafi's: The lifetime of the MPU part files is so too low for that use case. Maybe this can help: - If you want to stop a job with a savepoint and plan to restore later from it (possible much later, so that the MPU Part lifetime might be exceeded), then