Re: Does the job name affect checkpointing?

2016-09-30 Thread Ufuk Celebi
Hey Dan! On Thu, Sep 29, 2016 at 10:58 PM, dan bress wrote: > If I deploy a job to Flink called "A"(set my > streamExecutionEnvironment.execute("A"), that checkpoints state. Then I > cancel "A" and deploy the same job but call it "B", will it pick up A's > state? Or is checkpointing key'd by th

[jira] [Created] (FLINK-4719) KryoSerializer random exception

2016-09-30 Thread Flavio Pompermaier (JIRA)
Flavio Pompermaier created FLINK-4719: - Summary: KryoSerializer random exception Key: FLINK-4719 URL: https://issues.apache.org/jira/browse/FLINK-4719 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-4720) Implement an archived version of the execution graph

2016-09-30 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-4720: --- Summary: Implement an archived version of the execution graph Key: FLINK-4720 URL: https://issues.apache.org/jira/browse/FLINK-4720 Project: Flink Issu

Re: currentLowWatermark metric not reported for all tasks?

2016-09-30 Thread Chesnay Schepler
Hello Dan, the technical reason is that this metric is only collected in the *InputProcessor classes, which aren't used for source tasks. I do recall that there were discussions about source watermarks, but frankly i don't remember why we didn't add them. In order to add them one would only

Re: currentLowWatermark metric not reported for all tasks?

2016-09-30 Thread Stephan Ewen
I think what you describe, Chesnay, is exactly what we should do... On Fri, Sep 30, 2016 at 1:15 PM, Chesnay Schepler wrote: > Hello Dan, > > the technical reason is that this metric is only collected in the > *InputProcessor classes, which aren't used for source tasks. > > I do recall that ther

Re: Use specific Task Manager for heavy computations

2016-09-30 Thread Mariano Gonzalez
I think what you're suggesting is to load a large file into Kafka which will replicate it and make it available to all nodes. However, that is not what I want. What I want is to run a specific transformation step on a specific TaskManager. -- View this message in context: http://apache-flink-m

Re: currentLowWatermark metric not reported for all tasks?

2016-09-30 Thread dan bress
Awesome! It would definitely help me troubleshoot lagging watermarks if i can see what watermark all my sources have seen. Thanks for looking into this! Dan On Fri, Sep 30, 2016 at 5:48 AM Stephan Ewen wrote: > I think what you describe, Chesnay, is exactly what we should do... > > On Fri, Se

Re: Does the job name affect checkpointing?

2016-09-30 Thread dan bress
Ufuk, Thanks for the answer. In 1.1.X, If I deploy job "Job-V1", run for a while, trigger a save point, cancel the job, submit job "Job-V2" and resume save point. Will Job-V2 understand the savepoint from Job-V1? What is the scope of a savepoint? Who does it belong to? A Job? A TaskMana

[jira] [Created] (FLINK-4721) Getting exception when using a key of type Option[String] in distinct operation

2016-09-30 Thread Ashish Walia (JIRA)
Ashish Walia created FLINK-4721: --- Summary: Getting exception when using a key of type Option[String] in distinct operation Key: FLINK-4721 URL: https://issues.apache.org/jira/browse/FLINK-4721 Project: