Re: [DISCUSS][FLIP-4] Enhance Window Evictor in Flink

Hi Radu, - Yes we can remove elements from the iterator. - Right now the EvictingWindowOperator just skips the elements from the Iterable before passing to the window function(Yes this has to be changed in the new API) - Regarding how the last question on how elements are being removed from the wi

[jira] [Created] (FLINK-4259) Unclosed FSDataOutputStream in FileCache#copy()

Ted Yu created FLINK-4259: - Summary: Unclosed FSDataOutputStream in FileCache#copy() Key: FLINK-4259 URL: https://issues.apache.org/jira/browse/FLINK-4259 Project: Flink Issue Type: Bug R

[jira] [Created] (FLINK-4258) Potential null pointer dereference in SavepointCoordinator#onFullyAcknowledgedCheckpoint

Ted Yu created FLINK-4258: - Summary: Potential null pointer dereference in SavepointCoordinator#onFullyAcknowledgedCheckpoint Key: FLINK-4258 URL: https://issues.apache.org/jira/browse/FLINK-4258 Project: Fli

[DISCUSS] FLIP-6 - Flink Deployment and Process Model - Standalone, Yarn, Mesos, Kubernetes, etc.

Hi all! Here comes a pretty big FLIP: "Improvements to the Flink Deployment and Process Model", to better support Yarn, Mesos, Kubernetes, and whatever else Google, Elon Musk, and all the other folks will think up next. https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077 It

Question about Apache Flink Use Case

Hi, Good Afternoon! I work as an engineer at Symantec. My team works on Multi-tenant Event Processing System. Just a high level background, our customers write data to kafka brokers though agents like logstash and we process the events and save the log data in Elastic Search and S3. Use Case:

Question about Apache Flink Use Case

Hi, Good Afternoon! I work as an engineer at Symantec. My team works on Multi-tenant Event Processing System. Just a high level background, our customers write data to kafka brokers though agents like logstash and we process the events and save the log data in Elastic Search and S3. Use Case:

[jira] [Created] (FLINK-4257) Handle delegating algorithm change of class

Greg Hogan created FLINK-4257: - Summary: Handle delegating algorithm change of class Key: FLINK-4257 URL: https://issues.apache.org/jira/browse/FLINK-4257 Project: Flink Issue Type: Bug

RE: [DISCUSS][FLIP-4] Enhance Window Evictor in Flink

Hi, Overall I believe that the interfaces and the proposal is good. I have the following question though: can you delete via the iterator (Iterable> elements) the elements? I tried to look over the code where the eviction happens (I did not do these since version 0.10...looks very different no

[jira] [Created] (FLINK-4256) Fine-grained recovery

Stephan Ewen created FLINK-4256: --- Summary: Fine-grained recovery Key: FLINK-4256 URL: https://issues.apache.org/jira/browse/FLINK-4256 Project: Flink Issue Type: Improvement Component

Re: [DISCUSS] FLIP-5 Only send data to each taskmanager once for broadcasts

Hi Felix! Interesting suggestion. Here are some thoughts on the design. The two core changes needed to send data once to the TaskManagers are: (1) Every sender needs to produce its stuff once (rather than for every target task), there should not be redundancy there. (2) Every TaskManager sho

[jira] [Created] (FLINK-4255) Unstable test WebRuntimeMonitorITCase.testNoEscape

Kostas Kloudas created FLINK-4255: - Summary: Unstable test WebRuntimeMonitorITCase.testNoEscape Key: FLINK-4255 URL: https://issues.apache.org/jira/browse/FLINK-4255 Project: Flink Issue Type

[DISCUSS] FLIP-5 Only send data to each taskmanager once for broadcasts

Hi everybody, I want to improve the performance of broadcasts in Flink. Therefore Till told me to start a FLIP on this topic to discuss how to go forward to solve the current issues for broadcasts. The problem in a nutshell: Instead of sending data to each taskmanager only once, at the moment the

Re: [DISCUSS] Allowed Lateness in Flink

Forgot to say that the signature for the onFire() that I think fits should be: void onFire(Window window, TriggerContext ctx) throws Exception; > On Jul 22, 2016, at 12:47 PM, Kostas Kloudas > wrote: > > Hi, > > I started working on the new triggers proposed here and so far I can see > two s

Re: [DISCUSS] Allowed Lateness in Flink

Hi, I started working on the new triggers proposed here and so far I can see two shortcomings in the current state of the triggers that do not play well with the new proposals, and more specifically the composite triggers All and Any. So here it goes: 1) In the document posted above, there a

Re: [DISCUSS][FLIP-4] Enhance Window Evictor in Flink

Hi, I have created a FLIP page for this enhancement https://cwiki.apache.org/confluence/display/FLINK/FLIP-4+%3A+Enhance+Window+Evictor Thanks, Vishnu On Thu, Jul 21, 2016 at 6:53 AM, Vishnu Viswanath < vishnu.viswanat...@gmail.com> wrote: > Thanks Aljoscha. > > On Thu, Jul 21, 2016 at 4:46 AM,

Re: Broadcast data sent increases with # slots per TM

Hi Felix, thanks for all the work you've put into the design document and your experiments. The Flink community has recently agreed to publish these design documents as part of the FLIP (Flink improvement proposal) process in the Flink wiki. It would be great if you could post your design document

[jira] [Created] (FLINK-4254) Add file system based high-availability mode

Ufuk Celebi created FLINK-4254: -- Summary: Add file system based high-availability mode Key: FLINK-4254 URL: https://issues.apache.org/jira/browse/FLINK-4254 Project: Flink Issue Type: New Featur

[jira] [Created] (FLINK-4253) Rename "recovery.mode" config key to "high-availability"

Ufuk Celebi created FLINK-4253: -- Summary: Rename "recovery.mode" config key to "high-availability" Key: FLINK-4253 URL: https://issues.apache.org/jira/browse/FLINK-4253 Project: Flink Issue Type