Re: Streaming - memory management

2016-09-15 Thread Fabian Hueske
tate, you have to manually clear it. >>>>>>>>>>> Otherwise, it will stay in the state backend (heap or RocksDB) >>>>>>>>>>> until the >>>>>>>>>>> job goes down (planned or due to an OOM error). >>>>>>&

Re: Streaming - memory management

2016-09-01 Thread vinay patil
will constantly add state for each new key but won't be >>>>>>>>>> able to >>>>>>>>>> clean up the state for "expired" keys. >>>>>>>>>> >>>>>>>>>> You could impl

Re: Streaming - memory management

2016-09-01 Thread Fabian Hueske
>>>>> (state will be >>>>>>>>> discarded if not requested or updated for x minutes). >>>>>>>>> >>>>>>>>> Regarding the second question: Does state remain local after >>>>>>>>> che

Re: Streaming - memory management

2016-09-01 Thread vinay patil
in the operator. So the state is not gone after a >>>>>>>> checkpoint is >>>>>>>> completed. >>>>>>>> >>>>>>>> Hope this helps, >>>>>>>> Fabian >>>>>>>> >>>>>>>> 201

Re: Streaming - memory management

2016-09-01 Thread Fabian Hueske
this discussion regarding state. >>>>>>> > >>>>>>> > So do you mean that if we maintain user-defined state (for >>>>>>> non-window >>>>>>> > operators), then if we do not clear it explicitly will the data >>>>

Re: Streaming - memory management

2016-09-01 Thread vinay patil
t after >>>>>> > the checkpoint happens the rocksDB data is pushed to the desired >>>>>> location >>>>>> > (hdfs or s3 or other fs), so for user-defined state does the data >>>>>> still >>>>>> > remain in RocksDB after checkpoi

Re: Streaming - memory management

2016-09-01 Thread Fabian Hueske
;>> > in documentation so we are going for Cassandra now (to store records >>>>> and >>>>> > query them for a special case) >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > Regards, >>&g

Re: Streaming - memory management

2016-09-01 Thread vinay patil
; > > In streaming, memory is mainly needed for state (key/value state). >>>> The >>>> > > exact representation depends on the chosen StateBackend. >>>> > > >>>> > > State is explicitly released: For windows, state is cleaned up

Re: Streaming - memory management

2016-09-01 Thread Fabian Hueske
presentation depends on the chosen StateBackend. >>> > > >>> > > State is explicitly released: For windows, state is cleaned up >>> > > automatically (firing / expiry), for user-defined state, keys have >>> to be >>> > > explici

Re: Streaming - memory management

2016-09-01 Thread vinay patil
is currently RocksDB, which >> > > internally uses native (off-heap) memory to keep the data. >> > > >> > > Does that help? >> > > >> > > Stephan >> > > >> > > >> > > On Tue, Aug 30, 2016 at 11:52 PM, Roshan Naik > > >> > > wrote: >> > > >> > > > As per the docs, in Batch mode, dynamic memory allocation is >> avoided by >> > > > storing messages being processed in ByteBuffers via Unsafe methods. >> > > > >> > > > Couldn't find any docs describing mem mgmt in Streamingn mode. >> So... >> > > > >> > > > - Am wondering if this is also the case with Streaming ? >> > > > >> > > > - If so, how does Flink detect that an object is no longer being >> used >> > and >> > > > can be reclaimed for reuse once again ? >> > > > >> > > > -roshan >> > > > >> > > >> > >> > > -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Streaming-memory-management-tp8829.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Streaming - memory management

2016-08-31 Thread vinay patil
gt; > > > > > > As per the docs, in Batch mode, dynamic memory allocation is avoided > by > > > > storing messages being processed in ByteBuffers via Unsafe methods. > > > > > > > > Couldn't find any docs describing mem mgmt in Streamingn mode. So... > > > > > > > > - Am wondering if this is also the case with Streaming ? > > > > > > > > - If so, how does Flink detect that an object is no longer being used > > and > > > > can be reclaimed for reuse once again ? > > > > > > > > -roshan > > > > > > > > > > -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Streaming-memory-management-tp8820.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Streaming - memory management

2016-08-31 Thread vinay patil
> > > - If so, how does Flink detect that an object is no longer being used and > > can be reclaimed for reuse once again ? > > > > -roshan > > > -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Streaming-memory-management-tp8816.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.