Re: [Discussion] Query regarding Join and Windows

2016-06-30 Thread Vinay Patil
Hi, Just watched the video on Robust Stream Processing . So when we say Window is a stateful operator , does it mean that even if the task manager doing the window operation fails, will it pick up from the state left earlier when it comes up ? (Have not read more on state for now) Also in one o

Re: [Discuss] Why different job's tasks can run in the single process.

2016-06-30 Thread Jark Wu
As standalone mode has the disadvantage that the TaskManager JVM can’t isolate different jobs. Does we have any plan to improve this ? - Jark Wu > 在 2016年6月30日,下午4:19,Kevin Jacobs 写道: > > In my opinion the streaming process can be perfectly simulated on a single > node. You can setup a mess

Re: [DISCUSS] Releasing Flink 1.1.0

2016-06-30 Thread Greg Hogan
It would be great if hash-based combine (FLINK-3477) could make it in to be tested for this release. We've seen impressive improvements in performance (though, admittedly, some sort-based enhancements are yet to be worked on). This PR looks to be ripe. Also, as we tidy up a few things with Gelly a

[jira] [Created] (FLINK-4138) error with Kinesis connector

2016-06-30 Thread Zhenhao Li (JIRA)
Zhenhao Li created FLINK-4138: - Summary: error with Kinesis connector Key: FLINK-4138 URL: https://issues.apache.org/jira/browse/FLINK-4138 Project: Flink Issue Type: Bug Reporter: Zh

[jira] [Created] (FLINK-4137) JobManager web frontend does not shut down on OOM exception on JM

2016-06-30 Thread Robert Metzger (JIRA)
Robert Metzger created FLINK-4137: - Summary: JobManager web frontend does not shut down on OOM exception on JM Key: FLINK-4137 URL: https://issues.apache.org/jira/browse/FLINK-4137 Project: Flink

Re: [Discuss] Why different job's tasks can run in the single process.

2016-06-30 Thread Longda Feng
What I means is standalone mode is not suitable for production enironment. It is just for testing or prototype. regardsLongda --From:Kevin Jacobs Send Time:2016年6月30日(星期四) 16:20To:dev ; 封仲淹(纪君祥) Subject:Re: [Discuss] Why differen

Re: Failure in flink-storm when building Flink

2016-06-30 Thread F. Beligianni
Hi, Build succeeds now. The latest commit in master that stables the building of flink-storm is the following: "commit 6c6b17b4d47d281b0e5dcf4413fd1ad53ce49eee" by Aljoscha Krettek. Faye. 2016-06-30 16:58 GMT+03:00 Vladislav Pernin : > Could you tell us about the commit which fix it ? > I'm try

[jira] [Created] (FLINK-4136) Add a JobHistory Server service

2016-06-30 Thread JIRA
Márton Balassi created FLINK-4136: - Summary: Add a JobHistory Server service Key: FLINK-4136 URL: https://issues.apache.org/jira/browse/FLINK-4136 Project: Flink Issue Type: New Feature

Re: [Discussion] Query regarding Join and Windows

2016-06-30 Thread Vinay Patil
Hi Aljoscha, Just wanted to check if it works with it. Anyways to solve the problem what we have thought of is to push heartbeat message to Kafka after certain interval, so that we get continuous stream always and that edge case will never occur, right ? One more question I have regarding the fai

Re: Failure in flink-storm when building Flink

2016-06-30 Thread Vladislav Pernin
Could you tell us about the commit which fix it ? I'm trying to have a stable build against an almost up to date master before submitting a PR. Thanks. Le 30 juin 2016 15:41, "Aljoscha Krettek" a écrit : Yes, that's the one I fixed. :-) On Thu, 30 Jun 2016 at 15:40 F. Beligianni wrote: > Hi,

Re: Failure in flink-storm when building Flink

2016-06-30 Thread Aljoscha Krettek
Yes, that's the one I fixed. :-) On Thu, 30 Jun 2016 at 15:40 F. Beligianni wrote: > Hi, I suppose you mean this part of the log: > > Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.875 sec > > <<< FAILURE! - in org.apache.flink.storm.wrappers.WrapperSetupHelperTest > > > test

Re: Failure in flink-storm when building Flink

2016-06-30 Thread F. Beligianni
Hi, I suppose you mean this part of the log: Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.875 sec > <<< FAILURE! - in org.apache.flink.storm.wrappers.WrapperSetupHelperTest > testCreateTopologyContext(org.apache.flink.storm.wrappers.WrapperSetupHelperTest) > Time elapsed: 2.

Re: Failure in flink-storm when building Flink

2016-06-30 Thread Aljoscha Krettek
Hi, you didn't post the relevant part of the log where we see the actual failing test but just seconds ago I pushed a fix to master that fixes a problem in one of the Storm tests. I think this should fix the failure you are seeing. Cheers, Aljoscha On Thu, 30 Jun 2016 at 15:26 F. Beligianni wrot

Failure in flink-storm when building Flink

2016-06-30 Thread F. Beligianni
Hello, I did some changes in flink-gelly and flink-gelly-scala components and when I try to build the Flink project I get the following error in flink-storm component (where I made no changes). The same happens to Travis only for JDK: oraclejdk8. For JDK: oraclejdk7,openjdk7, the builds succeed in

[jira] [Created] (FLINK-4135) Replace ChecksumHashCode as GraphAnalytic

2016-06-30 Thread Greg Hogan (JIRA)
Greg Hogan created FLINK-4135: - Summary: Replace ChecksumHashCode as GraphAnalytic Key: FLINK-4135 URL: https://issues.apache.org/jira/browse/FLINK-4135 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-4134) EventTimeSessionWindows trigger for empty windows when dropping late events

2016-06-30 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-4134: - Summary: EventTimeSessionWindows trigger for empty windows when dropping late events Key: FLINK-4134 URL: https://issues.apache.org/jira/browse/FLINK-4134 Project:

Re: Web dashboard binaries and licensing

2016-06-30 Thread Till Rohrmann
Yes, that is also how I understood the Apache License requirements. Having a mere dependency on some external library which is not shipped as part of the source release does not require to include it in the LICENSE/NOTICE file. I think that you only have to include a reference in the LICENSE/NOTICE

Re: Web dashboard binaries and licensing

2016-06-30 Thread Aljoscha Krettek
AFAIK the Apache License requires that we have an entry in the LICENSE/NOTICE file for all external code that we ship with our source. If we have vendor.js in our source we have to include everything that is in there. If we don't have any actual external code in our source but only specified depend

Re: Web dashboard binaries and licensing

2016-06-30 Thread Maximilian Michels
I'm afraid I don't think integration in the Maven build process makes a difference in terms of licensing. It is already clearly specified what dependencies the web frontend uses (see README, bower.json, package.json). It won't get any easier with something integrated in the build process. We still

Re: Web dashboard binaries and licensing

2016-06-30 Thread Aljoscha Krettek
I think it's not a question of easy-of-use but one of licensing. I don't think anyone really knows what code ends up in vendor.js, so it is very hard to figure out what we have to put into our LICENSE file. On Thu, 30 Jun 2016 at 12:14 Maximilian Michels wrote: > Hi Till, > > Thanks for checking

Re: Web dashboard binaries and licensing

2016-06-30 Thread Maximilian Michels
Hi Till, Thanks for checking the licenses for our web frontend. I think the reason why we added a big binary Javascript blob into our repository was that it was easiest for most developers to deal with. We don't have much Javascript expertise in the Flink community. Incorporating the web frontend

Re: [Discussion] Query regarding Join and Windows

2016-06-30 Thread Aljoscha Krettek
Hi, I think the problem is that the DeltaFunction needs to have this signature: DeltaFunction, Tuple2>> because the Trigger will see elements from both input streams which are represented as a TaggedUnion that can contain an element from either side. May I ask why you want to use the DeltaTrigge

Re: [Discuss] Why different job's tasks can run in the single process.

2016-06-30 Thread Kevin Jacobs
In my opinion the streaming process can be perfectly simulated on a single node. You can setup a message distribution system like Kafka on a single node, you can run Spark on a single node and the only thing you need to change when running it on a cluster is that you need to change the environm

Re: [Discuss] Why different job's tasks can run in the single process.

2016-06-30 Thread Aljoscha Krettek
Hi, no, it is not just for prototyping. Since it is actually the oldest mode of execution it is also the most stable. You should have no problem using it. Cheers, Aljoscha On Thu, 30 Jun 2016 at 09:54 Longda Feng wrote: > > > This means Standalone mode is just for prototype. > But I think we ne

Re: [Discuss] Why different job's tasks can run in the single process.

2016-06-30 Thread Longda Feng
This means Standalone mode is just for prototype. But I think we need a lightweight solution for streaming process, standalone is the best solution. Some times, we need setup a flink cluster on a small cluster. setup a yarn cluster isn't convenient.  (1) in small company, the number of machine