Re: Apache Flink Operator State as Query Cache

2015-11-17 Thread Aljoscha Krettek
Hi, yes, it will be in 1.0. I’m working on adding it to master right now. Cheers, Aljoscha > On 17 Nov 2015, at 02:46, Welly Tambunan wrote: > > Hi Stephan, > > So that will be in Flink 1.0 right ? > > Cheers > > On Mon, Nov 16, 2015 at 9:06 PM, Stephan Ewen wrote: > Hi Anwar! > > 0.10.0 w

Re: Apache Flink Operator State as Query Cache

2015-11-16 Thread Welly Tambunan
Hi Stephan, So that will be in Flink 1.0 right ? Cheers On Mon, Nov 16, 2015 at 9:06 PM, Stephan Ewen wrote: > Hi Anwar! > > 0.10.0 was feature frozen at that time already and under testing. > Key/value state on connected streams will have to go into the next > release... > > Stephan > > > On

Re: Apache Flink Operator State as Query Cache

2015-11-16 Thread Stephan Ewen
Hi Anwar! 0.10.0 was feature frozen at that time already and under testing. Key/value state on connected streams will have to go into the next release... Stephan On Mon, Nov 16, 2015 at 3:00 PM, Anwar Rizal wrote: > Stephan, > > Having a look at the brand new 0.10 release, I noticed that Oper

Re: Apache Flink Operator State as Query Cache

2015-11-16 Thread Anwar Rizal
Stephan, Having a look at the brand new 0.10 release, I noticed that OperatorState is not implemented for ConnectedStream, which is quite the opposite of what you said below. Or maybe I misunderstood your sentence here ? Thanks, Anwar. On Wed, Nov 11, 2015 at 10:49 AM, Stephan Ewen wrote: >

Re: Apache Flink Operator State as Query Cache

2015-11-15 Thread Welly Tambunan
Hi Kostas, Yes. Exactly. Thanks a lot for this one. That's really what we need ! Cheers On Sun, Nov 15, 2015 at 8:53 PM, Kostas Tzoumas wrote: > Hi Wally, > > This version adds support for specifying and switching between time > semantics - processing time, ingestion time, or event time. > >

Re: Apache Flink Operator State as Query Cache

2015-11-15 Thread Kostas Tzoumas
Hi Wally, This version adds support for specifying and switching between time semantics - processing time, ingestion time, or event time. When working with event time, you can specify watermarks to track the progress of event time. So, even if events arrive out of order, windows will be specified

Re: Apache Flink Operator State as Query Cache

2015-11-13 Thread Welly Tambunan
Hi Robert, Is this version has already handle the stream perfection or out of order event ? Any resource on how this work and the API reference ? Cheers On Fri, Nov 13, 2015 at 4:00 PM, Welly Tambunan wrote: > Awesome ! > > This is really the best weekend gift ever. :) > > Cheers > > On Fri,

Re: Apache Flink Operator State as Query Cache

2015-11-13 Thread Welly Tambunan
Awesome ! This is really the best weekend gift ever. :) Cheers On Fri, Nov 13, 2015 at 3:54 PM, Robert Metzger wrote: > Hi Welly, > Flink 0.10.0 is out, its just not announced yet. > Its available on maven central and the global mirrors are currently > syncing it. This mirror for example has t

Re: Apache Flink Operator State as Query Cache

2015-11-13 Thread Robert Metzger
Hi Welly, Flink 0.10.0 is out, its just not announced yet. Its available on maven central and the global mirrors are currently syncing it. This mirror for example has the update already: http://apache.mirror.digionline.de/flink/flink-0.10.0/ On Fri, Nov 13, 2015 at 9:50 AM, Welly Tambunan wrote:

Re: Apache Flink Operator State as Query Cache

2015-11-13 Thread Welly Tambunan
Hi Aljoscha, Thanks for this one. Looking forward for 0.10 release version. Cheers On Thu, Nov 12, 2015 at 5:34 PM, Aljoscha Krettek wrote: > Hi, > I don’t know yet when the operator state will be transitioned to managed > memory but it could happen for 1.0 (which will come after 0.10). The go

Re: Apache Flink Operator State as Query Cache

2015-11-12 Thread Aljoscha Krettek
Hi, I don’t know yet when the operator state will be transitioned to managed memory but it could happen for 1.0 (which will come after 0.10). The good thing is that the interfaces won’t change, so state can be used as it is now. For 0.10, the release vote is winding down right now, so you can ex

Re: Apache Flink Operator State as Query Cache

2015-11-11 Thread Welly Tambunan
Hi Stephan, >Storing the model in OperatorState is a good idea, if you can. On the roadmap is to migrate the operator state to managed memory as well, so that should take care of the GC issues. Is this using off the heap memory ? Which version we expect this one to be available ? Another question

Re: Apache Flink Operator State as Query Cache

2015-11-11 Thread Stephan Ewen
Hi! In general, if you can keep state in Flink, you get better throughput/latency/consistency and have one less system to worry about (external k/v store). State outside means that the Flink processes can be slimmer and need fewer resources and as such recover a bit faster. There are use cases for

Re: Apache Flink Operator State as Query Cache

2015-11-08 Thread Welly Tambunan
Thanks for the answer. Currently the approach that i'm using right now is creating a base/marker interface to stream different type of message to the same operator. Not sure about the performance hit about this compare to the CoFlatMap function. Basically this one is providing query cache, so i'm

Re: Apache Flink Operator State as Query Cache

2015-11-06 Thread Anwar Rizal
Let me understand your case better here. You have a stream of model and stream of data. To process the data, you will need a way to access your model from the subsequent stream operations (map, filter, flatmap, ..). I'm not sure in which case Operator State is a good choice, but I think you can als

Apache Flink Operator State as Query Cache

2015-11-06 Thread Welly Tambunan
Hi All, We have a high density data that required a downsample. However this downsample model is very flexible based on the client device and user interaction. So it will be wasteful to precompute and store to db. So we want to use Apache Flink to do downsampling and cache the result for subseque