question please

2015-05-21 Thread Eng Fawzya
hi, i want to know what is the difference between FLink and Hadoop? -- Fawzya Ramadan Sayed, Teaching Assistant, Computer Science Department, Faculty of Computers and Information, Fayoum University

[jira] [Created] (FLINK-2077) Rework Path class and add extend support for Windows paths

2015-05-21 Thread Fabian Hueske (JIRA)
Fabian Hueske created FLINK-2077: Summary: Rework Path class and add extend support for Windows paths Key: FLINK-2077 URL: https://issues.apache.org/jira/browse/FLINK-2077 Project: Flink Issu

Re: [DISCUSS] Dedicated streaming mode

2015-05-21 Thread Henry Saputra
Hi Stephan, Gyula, Paris, How does streaming currently different in term of memory management? Currently we only have one MemoryManager which is used by both modes I believe. - Henry On Thu, May 21, 2015 at 12:34 PM, Stephan Ewen wrote: > I discussed a bit via Skype with Gyula and Paris. > > >

Re: [DISCUSS] Dedicated streaming mode

2015-05-21 Thread Gyula Fóra
Huge +1 from my side :) Sorry for the late response. On Thu, May 21, 2015 at 9:54 PM, Aljoscha Krettek wrote: > This sounds very reasonable. > On May 21, 2015 9:34 PM, "Stephan Ewen" wrote: > > > I discussed a bit via Skype with Gyula and Paris. > > > > > > We thought about the following way t

Re: [DISCUSS] Dedicated streaming mode

2015-05-21 Thread Aljoscha Krettek
This sounds very reasonable. On May 21, 2015 9:34 PM, "Stephan Ewen" wrote: > I discussed a bit via Skype with Gyula and Paris. > > > We thought about the following way to do it: > > - We add a dedicated streaming mode for now. The streaming mode supersedes > the batch mode, so it can run both t

Re: [DISCUSS] Dedicated streaming mode

2015-05-21 Thread Stephan Ewen
I discussed a bit via Skype with Gyula and Paris. We thought about the following way to do it: - We add a dedicated streaming mode for now. The streaming mode supersedes the batch mode, so it can run both type of programs. - The streaming mode sets the memory manager to "lazy allocation".

[jira] [Created] (FLINK-2076) Bug in re-openable hash join

2015-05-21 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-2076: --- Summary: Bug in re-openable hash join Key: FLINK-2076 URL: https://issues.apache.org/jira/browse/FLINK-2076 Project: Flink Issue Type: Bug Components

Debugging non-heap memory usage in Java (and Flink)

2015-05-21 Thread Stephan Ewen
Hi all! We are currently investigating the memory usage of the Flink JVMs. While we have the consumed memory in the JVM heap well under control, the memory that the JVM itself allocates outside the heap seems to be quite a bit. The non-heap memory goes up to a gigabyte for a 3GB heap space in som

[jira] [Created] (FLINK-2075) Shade akka and protobuf dependencies away

2015-05-21 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-2075: Summary: Shade akka and protobuf dependencies away Key: FLINK-2075 URL: https://issues.apache.org/jira/browse/FLINK-2075 Project: Flink Issue Type: Improveme

difference between reducefunction and GroupReduceFunction

2015-05-21 Thread santosh_rajaguru
i am new to flink and map reduce. My query is Apart from incrementally combing 2 elements, what are the merits of using reduceFunction over GroupReduceFunction. which usecases suits what functions the most!!! -- View this message in context: http://apache-flink-mailing-list-archive.1008284

Re: [DISCUSS] Dedicated streaming mode

2015-05-21 Thread Aljoscha Krettek
Would it not be possible to start the snapshot service once the user starts the first streaming job? About 2) with checkpointing coming up, would it not make sense to shift to managed memory rather sooner than later. Then this point would become moot. On Thu, May 21, 2015 at 3:47 PM, Matthias J. S

Re: [DISCUSS] Dedicated streaming mode

2015-05-21 Thread Matthias J. Sax
What would be the consequences on "mixed" programs? (If there is any plan to support those?) Would it be necessary to have a third mode? Or would those programs simple run in streaming mode? -Matthias On 05/21/2015 03:12 PM, Stephan Ewen wrote: > Hi all! > > We discussed a while back about intr

Re: Package multiple jobs in a single jar

2015-05-21 Thread Matthias J. Sax
Hi, I had a look into the current Workflow of Flink with regard to the progressing steps of a jar file. If I got it right it works as follows (not sure if this is documented somewhere): 1) check, if "-c" flag is used to set program entry point if yes, goto 4 2) try to extract "program-class"

[jira] [Created] (FLINK-2074) Sliding Window Keeps Emitting Elements After Source Stops Producing

2015-05-21 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-2074: --- Summary: Sliding Window Keeps Emitting Elements After Source Stops Producing Key: FLINK-2074 URL: https://issues.apache.org/jira/browse/FLINK-2074 Project: Flin

[DISCUSS] Dedicated streaming mode

2015-05-21 Thread Stephan Ewen
Hi all! We discussed a while back about introducing a dedicated streaming mode for Flink. I would like to take a go at this and implement the changes, but discuss them before. Here is a brief summary why we wanted to introduce the dedicated streaming mode: Even though both batch and streaming ar

[jira] [Created] (FLINK-2073) Add contribution guide for FlinkML

2015-05-21 Thread Theodore Vasiloudis (JIRA)
Theodore Vasiloudis created FLINK-2073: -- Summary: Add contribution guide for FlinkML Key: FLINK-2073 URL: https://issues.apache.org/jira/browse/FLINK-2073 Project: Flink Issue Type: New

[jira] [Created] (FLINK-2072) Add a quickstart guide for FlinkML

2015-05-21 Thread Theodore Vasiloudis (JIRA)
Theodore Vasiloudis created FLINK-2072: -- Summary: Add a quickstart guide for FlinkML Key: FLINK-2072 URL: https://issues.apache.org/jira/browse/FLINK-2072 Project: Flink Issue Type: New

Re: [GitHub] incubator-zeppelin pull request: ZEPPELIN-44 Interpreter for Apach...

2015-05-21 Thread Stephan Ewen
True, may still have another issue. On Thu, May 21, 2015 at 1:45 PM, Till Rohrmann wrote: > Hmm that is interesting. I always thought that in the case of multiple > SLF4J bindings, slf4j will simply pick one of them (first, last, random). > > On Thu, May 21, 2015 at 12:34 PM, Stephan Ewen wrote

[jira] [Created] (FLINK-2071) Projection function can fail non serializable

2015-05-21 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-2071: --- Summary: Projection function can fail non serializable Key: FLINK-2071 URL: https://issues.apache.org/jira/browse/FLINK-2071 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-2070) Confusing methods print() that print on client vs on TaskManager

2015-05-21 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-2070: --- Summary: Confusing methods print() that print on client vs on TaskManager Key: FLINK-2070 URL: https://issues.apache.org/jira/browse/FLINK-2070 Project: Flink

[jira] [Created] (FLINK-2069) writeAsCSV function in DataStream Scala API creates no file

2015-05-21 Thread Faye Beligianni (JIRA)
Faye Beligianni created FLINK-2069: -- Summary: writeAsCSV function in DataStream Scala API creates no file Key: FLINK-2069 URL: https://issues.apache.org/jira/browse/FLINK-2069 Project: Flink

Re: [GitHub] incubator-zeppelin pull request: ZEPPELIN-44 Interpreter for Apach...

2015-05-21 Thread Till Rohrmann
Hmm that is interesting. I always thought that in the case of multiple SLF4J bindings, slf4j will simply pick one of them (first, last, random). On Thu, May 21, 2015 at 12:34 PM, Stephan Ewen wrote: > The actor system fails to start. Here are the suspicious log lines: > > SLF4J: Class path c

[jira] [Created] (FLINK-2068) WebUI support multiple jobs per jar file

2015-05-21 Thread Matthias J. Sax (JIRA)
Matthias J. Sax created FLINK-2068: -- Summary: WebUI support multiple jobs per jar file Key: FLINK-2068 URL: https://issues.apache.org/jira/browse/FLINK-2068 Project: Flink Issue Type: Improv

[jira] [Created] (FLINK-2067) Chained streaming operators should not throw chained exceptions

2015-05-21 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-2067: --- Summary: Chained streaming operators should not throw chained exceptions Key: FLINK-2067 URL: https://issues.apache.org/jira/browse/FLINK-2067 Project: Flink

Re: [GitHub] incubator-zeppelin pull request: ZEPPELIN-44 Interpreter for Apach...

2015-05-21 Thread Stephan Ewen
The actor system fails to start. Here are the suspicious log lines: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/travis/.m2/ repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12- 1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4

Re: [GitHub] incubator-zeppelin pull request: ZEPPELIN-44 Interpreter for Apach...

2015-05-21 Thread Till Rohrmann
I'll try to reproduce this problem locally on my machine. On Thu, May 21, 2015 at 11:25 AM, Fabian Hueske wrote: > Hi Flink folks, > > the Flink interpreter PR for Apache Zeppelin is blocked by a failing test > case (see below). > Does anybody have an idea what is going on and can maybe help to

Fwd: [GitHub] incubator-zeppelin pull request: ZEPPELIN-44 Interpreter for Apach...

2015-05-21 Thread Fabian Hueske
Hi Flink folks, the Flink interpreter PR for Apache Zeppelin is blocked by a failing test case (see below). Does anybody have an idea what is going on and can maybe help to resolve the problem? Thanks, Fabian -- Forwarded message -- From: Leemoonsoo Date: 2015-05-21 6:35 GMT+02: