Re: JobManager not reachable

2015-10-15 Thread Matthias J. Sax
Please find the logs here http://www2.informatik.hu-berlin.de/~saxmatti/flink-mjsax-jobmanager-0-dbis21.log http://www2.informatik.hu-berlin.de/~saxmatti/flink-mjsax-taskmanager-0-dbis34.log -Matthias On 10/15/2015 12:16 PM, Stephan Ewen wrote: > Blocking actor calls should not be an issue (eve

[jira] [Created] (FLINK-2858) Cannot build Flink Scala 2.11 with IntelliJ

2015-10-15 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-2858: Summary: Cannot build Flink Scala 2.11 with IntelliJ Key: FLINK-2858 URL: https://issues.apache.org/jira/browse/FLINK-2858 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-2857) Gelly API improvements

2015-10-15 Thread Vasia Kalavri (JIRA)
Vasia Kalavri created FLINK-2857: Summary: Gelly API improvements Key: FLINK-2857 URL: https://issues.apache.org/jira/browse/FLINK-2857 Project: Flink Issue Type: Improvement Compon

[jira] [Created] (FLINK-2856) Introduce flink.version property into quickstart archetype

2015-10-15 Thread Robert Metzger (JIRA)
Robert Metzger created FLINK-2856: - Summary: Introduce flink.version property into quickstart archetype Key: FLINK-2856 URL: https://issues.apache.org/jira/browse/FLINK-2856 Project: Flink Is

[jira] [Created] (FLINK-2855) Add a documentation section for Gelly library methods

2015-10-15 Thread Vasia Kalavri (JIRA)
Vasia Kalavri created FLINK-2855: Summary: Add a documentation section for Gelly library methods Key: FLINK-2855 URL: https://issues.apache.org/jira/browse/FLINK-2855 Project: Flink Issue Typ

A proposal about skew data handling in Flink

2015-10-15 Thread Li, Chengxiang
Dear all, In many real world use case, data are nature to be skewed. For example, in social network, famous people get much more "follow" than others, a hot tweet would be transferred millions of times. and the purchased records of normal product can never compared to hot products. While at the

Re: JobManager not reachable

2015-10-15 Thread Stephan Ewen
Blocking actor calls should not be an issue (even if they are there), because the heartbeats go between the actor systems, rather than the actors... On Thu, Oct 15, 2015 at 12:14 PM, Till Rohrmann wrote: > And please set akka.log.lifecycle.events: true to let Akka log also its > lifecycle events

Re: JobManager not reachable

2015-10-15 Thread Till Rohrmann
And please set akka.log.lifecycle.events: true to let Akka log also its lifecycle events. ​ On Thu, Oct 15, 2015 at 12:12 PM, Robert Metzger wrote: > Can you start flink with logging level DEBUG ? > Then we can see from the TaskManager logs when the TM became inactive. > Maybe an Akka message is

Re: JobManager not reachable

2015-10-15 Thread Robert Metzger
Can you start flink with logging level DEBUG ? Then we can see from the TaskManager logs when the TM became inactive. Maybe an Akka message is causing the actor to block? You can also monitor the GC from the TaskManager view in the web interface (for example by looking at the total time spend for

Re: JobManager not reachable

2015-10-15 Thread Stephan Ewen
Does not quite sound like GC is an issue. Hmmm, what else can make the failure detector kick in unexpectedly? On Thu, Oct 15, 2015 at 12:05 PM, Till Rohrmann wrote: > To verify wether GC is a problem you can enable logging of memory usage of > the JVM via taskmanager.debug.memory.startLogThread

Re: JobManager not reachable

2015-10-15 Thread Till Rohrmann
To verify wether GC is a problem you can enable logging of memory usage of the JVM via taskmanager.debug.memory.startLogThread: true. The interval of the logging is configured via taskmanager.debug.memory.logIntervalMs. ​ On Thu, Oct 15, 2015 at 12:00 PM, Matthias J. Sax wrote: > The problem is

Re: JobManager not reachable

2015-10-15 Thread Matthias J. Sax
The problem is reproducible (it happens on each run). I doubt that GC is an issue here (at least from an UDF point of view), because I read the file once and keep a String object for each line. This objects are kept to the very end; the UDF does not release them until it returns from "run()" metho

Re: Extending and improving our "How to contribute" page

2015-10-15 Thread Till Rohrmann
Thanks for leading the effort Fabian! On Fri, Oct 9, 2015 at 10:07 AM, Maximilian Michels wrote: > Very nice work, Fabian. I think we'll have to send around a reminder > from time to time and, perhaps, evaluate the new guidelines after some > period of time. It's great to have these documents no

Re: JobManager not reachable

2015-10-15 Thread Stephan Ewen
>From what the logs show, the TaskManager does not send pings any more for a long time and is then considered failed and the tasks running on that TaskManager are considered failed as well. So far, nothing unusual... Question is, why is it considered failed? Is this a reproducible problem? Or a on

[jira] [Created] (FLINK-2854) KafkaITCase.testOneSourceMultiplePartitions failed on Travis

2015-10-15 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-2854: Summary: KafkaITCase.testOneSourceMultiplePartitions failed on Travis Key: FLINK-2854 URL: https://issues.apache.org/jira/browse/FLINK-2854 Project: Flink I

Re: Question on Storm Split Streams

2015-10-15 Thread Stephan Ewen
Sorry, I did not fully understand this. Is this splitting a feature in the topology builder that is not supported at runtime? On Wed, Oct 14, 2015 at 10:10 PM, Matthias J. Sax wrote: > SplitStreamKeySelector was build for TupleX output type only > (FlinkTopologyBuilder never used primitive or PO