Re: finite subset of an infinite data stream

2015-11-20 Thread Aljoscha Krettek
Hi, I’m very sorry, yes you would need my custom branch: https://github.com/aljoscha/flink/commits/state-enhance Cheers, Aljoscha > On 20 Nov 2015, at 10:13, rss rss wrote: > > Hello Aljoscha, > > very thanks. I tried to build your example but have an obstacle with > org.apache.flink.runtim

Re: finite subset of an infinite data stream

2015-11-20 Thread rss rss
Hello Aljoscha, very thanks. I tried to build your example but have an obstacle with org.apache.flink.runtime.state.AbstractStateBackend class. Where to get it? I guess it stored in your local branch only. Would you please to send me patches for public branch or share the branch with me? Best r

Re: finite subset of an infinite data stream

2015-11-18 Thread Aljoscha Krettek
Hi, I wrote a little example that could be what you are looking for: https://github.com/dataArtisans/query-window-example It basically implements a window operator with a modifiable window size that also allows querying the current accumulated window contents using a second input stream. There

Re: finite subset of an infinite data stream

2015-11-11 Thread Robert Metzger
I think what you call "union" is a "connected stream" in Flink. Have a look at this example: https://gist.github.com/fhueske/4ea5422edb5820915fa4 It shows how to dynamically update a list of filters by external requests. Maybe that's what you are looking for? On Wed, Nov 11, 2015 at 12:15 PM, St

Re: finite subset of an infinite data stream

2015-11-11 Thread Stephan Ewen
Hi! I don not really understand what exactly you want to do, especially the "union an infinite real time data stream with filtered persistent data where the condition of filtering is provided by external requests". If you want to work on substreams in general, there are two options: 1) Create th

Re: finite subset of an infinite data stream

2015-11-09 Thread rss rss
Hello, thanks for the answer but windows produce periodical results. I used your example but the data source is changed to TCP stream: DataStream text = env.socketTextStream("localhost", 2015, '\n'); DataStream> wordCounts = text .flatMap(new Line

Re: finite subset of an infinite data stream

2015-11-09 Thread Stephan Ewen
Hi! If you want to work on subsets of streams, the answer is usually to use windows, "stream.keyBy(...).timeWindow(Time.of(1, MINUTE))". The transformations that you want to make, do they fit into a window function? There are thoughts to introduce something like global time windows across the en

finite subset of an infinite data stream

2015-11-08 Thread rss rss
Hello, I need to extract a finite subset like a data buffer from an infinite data stream. The best way for me is to obtain a finite stream with data accumulated for a 1minute before (as example). But I not found any existing technique to do it. As a possible ways how to do something near to a