Re: Retrieving Flink job ID/YARN Id programmatically

2019-11-21 Thread Lei Nie
checkpoint has form ".../checkpoints/flink_app_id/chk-123" On Thu, Nov 21, 2019 at 11:28 AM Lei Nie wrote: > > This does not get the correct id: > StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID = > eea5abc21dd8743a4090f4a3a660f9e8 >

Re: Retrieving Flink job ID/YARN Id programmatically

2019-11-21 Thread Lei Nie
This does not get the correct id: StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID = eea5abc21dd8743a4090f4a3a660f9e8 Actual job ID (from webUI): 1357d21be640b6a3b8a86a063f4bba8a On Thu, Nov 7, 2019 at 6:56 PM vino yang wrote: > > Hi Lei Nie, > >

StreamingFileSink duplicate data

2019-11-20 Thread Lei Nie
Hello, I would like clarification on the StreamingFileSink, thank you. >From my testing, it seems that resuming job from checkpoint does *not* also restore the rolling part counter. E.g, job may have stopped with last file: *part-6-71* But when resuming from most recent checkpoint: *part-6-89* (

Retrieving Flink job ID/YARN Id programmatically

2019-11-07 Thread Lei Nie
Hello, I am currently executing streaming jobs via StreamExecutionEnvironment. Is it possible to retrieve the Flink job ID/YARN ID within the context of a job? I'd like to be able to automatically register the job such that monitoring jobs can run (REST api requires for example job id). Thanks